2024-09-21T08:07:58.128 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-21T08:07:58.150 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-21T08:07:58.241 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913148 branch: squid description: orch/cephadm/thrash/{0-distro/centos_9.stream_runc 1-start 2-thrash 3-tasks/snaps-few-objects fixed-2 msgr/async-v1only root} email: ceph-qa@ceph.com first_in_suite: false job_id: '7913148' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi name: teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 4 size: 10 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: global: ms bind msgr2: false ms type: async mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon min osdmap epochs: 50 mon osdmap full prune interval: 2 mon osdmap full prune min: 15 mon osdmap full prune txsize: 2 paxos service trim min: 10 osd: debug ms: 1 debug osd: 20 osd debug reject backfill probability: 0.3 osd delete sleep: 1 osd max backfills: 3 osd scrub max interval: 120 osd scrub min interval: 60 osd shutdown pgref assert: true osd snap trim sleep: 2 flavor: default log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - CEPHADM_STRAY_DAEMON - CEPHADM_FAILED_DAEMON - but it is still running - objects unfound and apparently lost - \(POOL_APP_NOT_ENABLED\) log-only-match: - CEPHADM_ mon_bind_msgr2: false sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} cephadm: cephadm_mode: root install: ceph: flavor: default sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 workunit: branch: squid sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mon.c - mgr.y - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - ceph.rgw.foo.a - node-exporter.a - alertmanager.a - - mon.b - mgr.x - osd.4 - osd.5 - osd.6 - osd.7 - client.1 - prometheus.a - grafana.a - node-exporter.b - ceph.iscsi.iscsi.a seed: 9344 sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 sleep_before_teardown: 0 subset: 56/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: 4e45e6b490e6cbe082cdf155fea780438ec53fc7 targets: smithi137.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAQ+1zN4pWgP3LAr/S0CPum3ceI2q9CO+vl094uKEIfk6z+rlKlgQpHQuj4Xa3jnVbLR1G9MPvIzZdqptk2UVPo= smithi165.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJvo0j8Im56GA0sQWFYhMmefS7xvBJzosj9jRqcB+lwA1+ZVdcjK3KCVXfVBHnedYfNR5Ep0Zmwug6pbis+0VLM= tasks: - pexec: all: - sudo dnf remove nvme-cli -y - sudo dnf install runc nvmetcli nvme-cli -y - sudo sed -i 's/^#runtime = "crun"/runtime = "runc"/g' /usr/share/containers/containers.conf - sudo sed -i 's/runtime = "crun"/#runtime = "crun"/g' /usr/share/containers/containers.conf - install: null - cephadm: conf: mgr: debug mgr: 20 debug ms: 1 - thrashosds: chance_pgnum_grow: 1 chance_pgnum_shrink: 1 chance_pgpnum_fix: 1 timeout: 1200 - rados: clients: - client.0 objects: 50 op_weights: copy_from: 50 delete: 50 read: 100 rollback: 50 snap_create: 50 snap_remove: 50 write: 100 ops: 4000 teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-20_21:08:02 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2902 2024-09-21T08:07:58.242 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa; will attempt to use it 2024-09-21T08:07:58.242 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_4e45e6b490e6cbe082cdf155fea780438ec53fc7/qa/tasks 2024-09-21T08:07:58.243 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-21T08:07:58.244 INFO:teuthology.task.internal:Checking packages... 2024-09-21T08:07:58.264 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '9e2ac70bb7aea67d34f0d50002344abbe7803736' 2024-09-21T08:07:58.264 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-21T08:07:58.265 INFO:teuthology.packaging:ref: None 2024-09-21T08:07:58.265 INFO:teuthology.packaging:tag: None 2024-09-21T08:07:58.265 INFO:teuthology.packaging:branch: squid 2024-09-21T08:07:58.265 INFO:teuthology.packaging:sha1: 9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-21T08:07:58.265 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-09-21T08:07:58.507 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-373.g4e45e6b4 2024-09-21T08:07:58.509 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-21T08:07:58.518 INFO:teuthology.task.internal:no buildpackages task found 2024-09-21T08:07:58.518 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-21T08:07:58.532 INFO:teuthology.task.internal:Saving configuration 2024-09-21T08:07:58.544 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-21T08:07:58.556 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-21T08:07:58.582 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi137.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 08:02:49.165085', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBAQ+1zN4pWgP3LAr/S0CPum3ceI2q9CO+vl094uKEIfk6z+rlKlgQpHQuj4Xa3jnVbLR1G9MPvIzZdqptk2UVPo='} 2024-09-21T08:07:58.606 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi165.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-20_21:08:02-orch-squid-distro-default-smithi/7913148', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-21 08:02:49.163436', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJvo0j8Im56GA0sQWFYhMmefS7xvBJzosj9jRqcB+lwA1+ZVdcjK3KCVXfVBHnedYfNR5Ep0Zmwug6pbis+0VLM='} 2024-09-21T08:07:58.606 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-21T08:07:58.617 INFO:teuthology.task.internal:roles: ubuntu@smithi137.front.sepia.ceph.com - ['mon.a', 'mon.c', 'mgr.y', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0', 'ceph.rgw.foo.a', 'node-exporter.a', 'alertmanager.a'] 2024-09-21T08:07:58.617 INFO:teuthology.task.internal:roles: ubuntu@smithi165.front.sepia.ceph.com - ['mon.b', 'mgr.x', 'osd.4', 'osd.5', 'osd.6', 'osd.7', 'client.1', 'prometheus.a', 'grafana.a', 'node-exporter.b', 'ceph.iscsi.iscsi.a'] 2024-09-21T08:07:58.617 INFO:teuthology.run_tasks:Running task console_log... 2024-09-21T08:07:58.685 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0d902f0160>, signals=[15]) 2024-09-21T08:07:58.685 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-21T08:07:58.697 INFO:teuthology.task.internal:Opening connections... 2024-09-21T08:07:58.697 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi137.front.sepia.ceph.com 2024-09-21T08:07:58.698 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T08:07:58.775 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi165.front.sepia.ceph.com 2024-09-21T08:07:58.776 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T08:07:58.849 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-21T08:07:58.858 DEBUG:teuthology.orchestra.run.smithi137:> uname -m 2024-09-21T08:07:58.876 INFO:teuthology.orchestra.run.smithi137.stdout:x86_64 2024-09-21T08:07:58.876 DEBUG:teuthology.orchestra.run.smithi137:> cat /etc/os-release 2024-09-21T08:07:58.933 INFO:teuthology.orchestra.run.smithi137.stdout:NAME="CentOS Stream" 2024-09-21T08:07:58.933 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION="9" 2024-09-21T08:07:58.933 INFO:teuthology.orchestra.run.smithi137.stdout:ID="centos" 2024-09-21T08:07:58.933 INFO:teuthology.orchestra.run.smithi137.stdout:ID_LIKE="rhel fedora" 2024-09-21T08:07:58.933 INFO:teuthology.orchestra.run.smithi137.stdout:VERSION_ID="9" 2024-09-21T08:07:58.933 INFO:teuthology.orchestra.run.smithi137.stdout:PLATFORM_ID="platform:el9" 2024-09-21T08:07:58.933 INFO:teuthology.orchestra.run.smithi137.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T08:07:58.933 INFO:teuthology.orchestra.run.smithi137.stdout:ANSI_COLOR="0;31" 2024-09-21T08:07:58.933 INFO:teuthology.orchestra.run.smithi137.stdout:LOGO="fedora-logo-icon" 2024-09-21T08:07:58.933 INFO:teuthology.orchestra.run.smithi137.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T08:07:58.934 INFO:teuthology.orchestra.run.smithi137.stdout:HOME_URL="https://centos.org/" 2024-09-21T08:07:58.934 INFO:teuthology.orchestra.run.smithi137.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T08:07:58.934 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T08:07:58.934 INFO:teuthology.orchestra.run.smithi137.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T08:07:58.934 INFO:teuthology.lock.ops:Updating smithi137.front.sepia.ceph.com on lock server 2024-09-21T08:07:58.960 DEBUG:teuthology.orchestra.run.smithi165:> uname -m 2024-09-21T08:07:58.978 INFO:teuthology.orchestra.run.smithi165.stdout:x86_64 2024-09-21T08:07:58.978 DEBUG:teuthology.orchestra.run.smithi165:> cat /etc/os-release 2024-09-21T08:07:59.036 INFO:teuthology.orchestra.run.smithi165.stdout:NAME="CentOS Stream" 2024-09-21T08:07:59.036 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION="9" 2024-09-21T08:07:59.036 INFO:teuthology.orchestra.run.smithi165.stdout:ID="centos" 2024-09-21T08:07:59.036 INFO:teuthology.orchestra.run.smithi165.stdout:ID_LIKE="rhel fedora" 2024-09-21T08:07:59.036 INFO:teuthology.orchestra.run.smithi165.stdout:VERSION_ID="9" 2024-09-21T08:07:59.036 INFO:teuthology.orchestra.run.smithi165.stdout:PLATFORM_ID="platform:el9" 2024-09-21T08:07:59.036 INFO:teuthology.orchestra.run.smithi165.stdout:PRETTY_NAME="CentOS Stream 9" 2024-09-21T08:07:59.037 INFO:teuthology.orchestra.run.smithi165.stdout:ANSI_COLOR="0;31" 2024-09-21T08:07:59.037 INFO:teuthology.orchestra.run.smithi165.stdout:LOGO="fedora-logo-icon" 2024-09-21T08:07:59.037 INFO:teuthology.orchestra.run.smithi165.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-09-21T08:07:59.037 INFO:teuthology.orchestra.run.smithi165.stdout:HOME_URL="https://centos.org/" 2024-09-21T08:07:59.037 INFO:teuthology.orchestra.run.smithi165.stdout:BUG_REPORT_URL="https://issues.redhat.com/" 2024-09-21T08:07:59.037 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-09-21T08:07:59.037 INFO:teuthology.orchestra.run.smithi165.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-09-21T08:07:59.037 INFO:teuthology.lock.ops:Updating smithi165.front.sepia.ceph.com on lock server 2024-09-21T08:07:59.060 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-21T08:07:59.070 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-21T08:07:59.083 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-21T08:07:59.083 DEBUG:teuthology.orchestra.run.smithi137:> test '!' -e /home/ubuntu/cephtest 2024-09-21T08:07:59.085 DEBUG:teuthology.orchestra.run.smithi165:> test '!' -e /home/ubuntu/cephtest 2024-09-21T08:07:59.102 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-21T08:07:59.113 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-21T08:07:59.113 DEBUG:teuthology.orchestra.run.smithi137:> test -z $(ls -A /var/lib/ceph) 2024-09-21T08:07:59.142 DEBUG:teuthology.orchestra.run.smithi165:> test -z $(ls -A /var/lib/ceph) 2024-09-21T08:07:59.245 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-21T08:07:59.302 INFO:teuthology.run_tasks:Running task kernel... 2024-09-21T08:07:59.320 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-21T08:07:59.321 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-21T08:07:59.321 DEBUG:teuthology.orchestra.run.smithi137:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T08:07:59.321 DEBUG:teuthology.orchestra.run.smithi165:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-21T08:07:59.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:07:59.339 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-21T08:07:59.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:07:59.340 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-21T08:07:59.397 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-503.el9.x86_64 2024-09-21T08:07:59.397 INFO:teuthology.task.kernel:Running kernel on smithi137: 5.14.0-503.el9.x86_64 2024-09-21T08:07:59.397 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum install -y kernel 2024-09-21T08:07:59.398 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-503.el9.x86_64 2024-09-21T08:07:59.398 INFO:teuthology.task.kernel:Running kernel on smithi165: 5.14.0-503.el9.x86_64 2024-09-21T08:07:59.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum install -y kernel 2024-09-21T08:08:01.871 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.3 MB 00:00 2024-09-21T08:08:03.155 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.3 MB 00:01 2024-09-21T08:08:03.656 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 29 MB/s | 20 MB 00:00 2024-09-21T08:08:05.614 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 15 MB/s | 20 MB 00:01 2024-09-21T08:08:07.948 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 14 MB/s | 6.5 MB 00:00 2024-09-21T08:08:09.482 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 59 kB/s | 19 kB 00:00 2024-09-21T08:08:10.252 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 35 MB/s | 23 MB 00:00 2024-09-21T08:08:11.055 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 3.9 MB/s | 6.5 MB 00:01 2024-09-21T08:08:12.806 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 19 kB 00:00 2024-09-21T08:08:13.877 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 24 MB/s | 23 MB 00:00 2024-09-21T08:08:15.633 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 48 kB/s | 1.7 kB 00:00 2024-09-21T08:08:17.573 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T08:08:17.573 INFO:teuthology.orchestra.run.smithi137.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T08:08:17.652 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:08:17.657 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:08:17.657 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-09-21T08:08:17.658 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:08:17.659 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 79 M 2024-09-21T08:08:17.659 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 126 M 2024-09-21T08:08:17.659 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:08:17.909 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 176 kB/s | 36 kB 00:00 2024-09-21T08:08:19.274 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 40 kB/s | 1.7 kB 00:00 2024-09-21T08:08:21.105 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-09-21T08:08:21.105 INFO:teuthology.orchestra.run.smithi165.stdout:Package kernel-5.14.0-503.el9.x86_64 is already installed. 2024-09-21T08:08:21.196 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout: kernel x86_64 5.14.0-508.el9 baseos 36 k 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core x86_64 5.14.0-508.el9 baseos 16 M 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules x86_64 5.14.0-508.el9 baseos 35 M 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core x86_64 5.14.0-508.el9 baseos 29 M 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:08:21.208 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:08:21.209 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:08:21.209 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-09-21T08:08:21.209 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:08:21.209 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 79 M 2024-09-21T08:08:21.209 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 126 M 2024-09-21T08:08:21.209 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:08:21.350 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): kernel-5.14.0-508.el9.x86_64.rpm 300 kB/s | 36 kB 00:00 2024-09-21T08:08:21.634 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 4.0 MB/s | 16 MB 00:03 2024-09-21T08:08:22.201 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): kernel-core-5.14.0-508.el9.x86_64.rpm 16 MB/s | 16 MB 00:00 2024-09-21T08:08:22.759 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): kernel-modules-core-5.14.0-508.el9.x86_6 20 MB/s | 29 MB 00:01 2024-09-21T08:08:23.142 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 18 MB/s | 35 MB 00:01 2024-09-21T08:08:23.143 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:08:23.143 INFO:teuthology.orchestra.run.smithi165.stdout:Total 41 MB/s | 79 MB 00:01 2024-09-21T08:08:23.819 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:08:24.174 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:08:24.174 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:08:25.155 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:08:25.155 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:08:26.792 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): kernel-modules-5.14.0-508.el9.x86_64.rpm 3.8 MB/s | 35 MB 00:09 2024-09-21T08:08:27.276 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): kernel-modules-core-5.14.0-508.el9.x86_6 3.1 MB/s | 29 MB 00:09 2024-09-21T08:08:27.276 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:08:27.276 INFO:teuthology.orchestra.run.smithi137.stdout:Total 8.2 MB/s | 79 MB 00:09 2024-09-21T08:08:27.581 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:08:27.964 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:08:28.231 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T08:08:28.310 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T08:08:28.324 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:08:28.324 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:08:29.299 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:08:29.300 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:08:31.058 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T08:08:31.202 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T08:08:31.725 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:08:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-core-5.14.0-508.el9.x86_64 1/4 2024-09-21T08:08:32.542 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T08:08:35.311 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T08:08:35.445 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T08:08:36.052 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T08:08:36.102 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:08:40.260 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T08:08:40.412 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:08:40.952 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:08:45.242 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:09:19.105 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:09:19.146 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:09:19.440 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:09:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T08:09:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T08:09:19.441 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T08:09:20.348 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:09:20.349 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:09:20.349 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:09:20.349 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T08:09:20.349 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T08:09:20.349 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T08:09:20.349 INFO:teuthology.orchestra.run.smithi165.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T08:09:20.349 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:09:20.349 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:09:20.696 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T08:09:20.776 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T08:09:20.776 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T08:09:20.776 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T08:09:20.776 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T08:09:20.776 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T08:09:20.776 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-09-21T08:09:20.777 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi165.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T08:09:20.777 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T08:09:20.835 INFO:teuthology.orchestra.run.smithi165.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T08:09:20.835 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T08:09:20.835 DEBUG:teuthology.orchestra.run.smithi165:> sudo rpm -qi grub2-tools 2024-09-21T08:09:20.868 INFO:teuthology.orchestra.run.smithi165.stdout:Name : grub2-tools 2024-09-21T08:09:20.868 INFO:teuthology.orchestra.run.smithi165.stdout:Epoch : 1 2024-09-21T08:09:20.868 INFO:teuthology.orchestra.run.smithi165.stdout:Version : 2.06 2024-09-21T08:09:20.868 INFO:teuthology.orchestra.run.smithi165.stdout:Release : 82.el9 2024-09-21T08:09:20.868 INFO:teuthology.orchestra.run.smithi165.stdout:Architecture: x86_64 2024-09-21T08:09:20.868 INFO:teuthology.orchestra.run.smithi165.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T08:09:20.869 INFO:teuthology.orchestra.run.smithi165.stdout:Group : Unspecified 2024-09-21T08:09:20.869 INFO:teuthology.orchestra.run.smithi165.stdout:Size : 8274670 2024-09-21T08:09:20.869 INFO:teuthology.orchestra.run.smithi165.stdout:License : GPLv3+ 2024-09-21T08:09:20.869 INFO:teuthology.orchestra.run.smithi165.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T08:09:20.869 INFO:teuthology.orchestra.run.smithi165.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T08:09:20.869 INFO:teuthology.orchestra.run.smithi165.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T08:09:20.869 INFO:teuthology.orchestra.run.smithi165.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T08:09:20.869 INFO:teuthology.orchestra.run.smithi165.stdout:Packager : builder@centos.org 2024-09-21T08:09:20.869 INFO:teuthology.orchestra.run.smithi165.stdout:Vendor : CentOS 2024-09-21T08:09:20.869 INFO:teuthology.orchestra.run.smithi165.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T08:09:20.870 INFO:teuthology.orchestra.run.smithi165.stdout:Summary : Support tools for GRUB. 2024-09-21T08:09:20.870 INFO:teuthology.orchestra.run.smithi165.stdout:Description : 2024-09-21T08:09:20.870 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:09:20.870 INFO:teuthology.orchestra.run.smithi165.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T08:09:20.870 INFO:teuthology.orchestra.run.smithi165.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T08:09:20.870 INFO:teuthology.orchestra.run.smithi165.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T08:09:20.870 INFO:teuthology.orchestra.run.smithi165.stdout:hardware devices. 2024-09-21T08:09:20.870 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:09:20.870 INFO:teuthology.orchestra.run.smithi165.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T08:09:20.871 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T08:09:20.872 INFO:teuthology.task.kernel:Updating grub on smithi165 to boot 5.14.0-508.el9.x86_64 2024-09-21T08:09:20.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T08:09:21.501 INFO:teuthology.orchestra.run.smithi165.stderr:Generating grub configuration file ... 2024-09-21T08:09:22.859 INFO:teuthology.orchestra.run.smithi165.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T08:09:22.884 INFO:teuthology.orchestra.run.smithi165.stderr:done 2024-09-21T08:09:22.886 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-09-21T08:09:22.901 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.VkayOxJKl0 2024-09-21T08:09:22.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VkayOxJKl0 2024-09-21T08:09:22.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.VkayOxJKl0 2024-09-21T08:09:26.332 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:09:26.332 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-modules-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: kernel-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-5.14.0-508.el9.x86_64 1/4 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-core-5.14.0-508.el9.x86_64 2/4 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-5.14.0-508.el9.x86_64 3/4 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : kernel-modules-core-5.14.0-508.el9.x86_64 4/4 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-5.14.0-508.el9.x86_64 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-core-5.14.0-508.el9.x86_64 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-5.14.0-508.el9.x86_64 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: kernel-modules-core-5.14.0-508.el9.x86_64 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:09:26.333 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:09:26.334 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T08:09:26.434 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T08:09:26.435 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T08:09:26.435 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-503.el9.x86_64 Expected: 5.14.0-508.el9.x86_64 2024-09-21T08:09:26.435 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-21T08:09:26.435 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-21T08:09:26.435 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-09-21T08:09:26.435 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi137.front.sepia.ceph.com, path=None, version=distro) 2024-09-21T08:09:26.435 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q kernel | sort -rV | head -n 1 2024-09-21T08:09:26.453 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.VkayOxJKl0 is 6KB 2024-09-21T08:09:26.479 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.VkayOxJKl0 2024-09-21T08:09:26.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T08:09:26.497 INFO:teuthology.orchestra.run.smithi137.stdout:kernel-5.14.0-508.el9.x86_64 2024-09-21T08:09:26.498 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-508.el9.x86_64 2024-09-21T08:09:26.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo rpm -qi grub2-tools 2024-09-21T08:09:26.530 INFO:teuthology.orchestra.run.smithi137.stdout:Name : grub2-tools 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Epoch : 1 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Version : 2.06 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Release : 82.el9 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Architecture: x86_64 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Install Date: Wed 28 Aug 2024 09:06:27 PM UTC 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Group : Unspecified 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Size : 8274670 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:License : GPLv3+ 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Signature : RSA/SHA256, Mon 01 Jul 2024 07:46:42 AM UTC, Key ID 05b555b38483c65d 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Source RPM : grub2-2.06-82.el9.src.rpm 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Build Date : Fri 28 Jun 2024 09:24:44 AM UTC 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Packager : builder@centos.org 2024-09-21T08:09:26.531 INFO:teuthology.orchestra.run.smithi137.stdout:Vendor : CentOS 2024-09-21T08:09:26.532 INFO:teuthology.orchestra.run.smithi137.stdout:URL : http://www.gnu.org/software/grub/ 2024-09-21T08:09:26.532 INFO:teuthology.orchestra.run.smithi137.stdout:Summary : Support tools for GRUB. 2024-09-21T08:09:26.532 INFO:teuthology.orchestra.run.smithi137.stdout:Description : 2024-09-21T08:09:26.532 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:09:26.532 INFO:teuthology.orchestra.run.smithi137.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-09-21T08:09:26.532 INFO:teuthology.orchestra.run.smithi137.stdout:customizable bootloader with modular architecture. It supports a rich 2024-09-21T08:09:26.532 INFO:teuthology.orchestra.run.smithi137.stdout:variety of kernel formats, file systems, computer architectures and 2024-09-21T08:09:26.532 INFO:teuthology.orchestra.run.smithi137.stdout:hardware devices. 2024-09-21T08:09:26.532 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:09:26.532 INFO:teuthology.orchestra.run.smithi137.stdout:This subpackage provides tools for support of all platforms. 2024-09-21T08:09:26.533 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-09-21T08:09:26.533 INFO:teuthology.task.kernel:Updating grub on smithi137 to boot 5.14.0-508.el9.x86_64 2024-09-21T08:09:26.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-09-21T08:09:26.562 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T08:09:26.563 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T08:09:26.563 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T08:09:26.563 INFO:teuthology.orchestra.run.smithi165.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T08:09:26.565 DEBUG:teuthology.orchestra.run.smithi165:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T08:09:26.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo shutdown -r now 2024-09-21T08:09:27.173 INFO:teuthology.orchestra.run.smithi137.stderr:Generating grub configuration file ... 2024-09-21T08:09:28.564 INFO:teuthology.orchestra.run.smithi137.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-09-21T08:09:28.595 INFO:teuthology.orchestra.run.smithi137.stderr:done 2024-09-21T08:09:28.604 DEBUG:teuthology.orchestra.run.smithi137:> mktemp 2024-09-21T08:09:28.620 INFO:teuthology.orchestra.run.smithi137.stdout:/tmp/tmp.TS5Ojrl4dc 2024-09-21T08:09:28.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.TS5Ojrl4dc 2024-09-21T08:09:28.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0666 /tmp/tmp.TS5Ojrl4dc 2024-09-21T08:09:28.836 DEBUG:teuthology.orchestra.remote:smithi137:/tmp/tmp.TS5Ojrl4dc is 6KB 2024-09-21T08:09:28.855 DEBUG:teuthology.orchestra.run.smithi137:> rm -fr /tmp/tmp.TS5Ojrl4dc 2024-09-21T08:09:28.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo /bin/ls /boot/loader/entries || true 2024-09-21T08:09:28.939 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-0-rescue.conf 2024-09-21T08:09:28.939 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-239.el9.x86_64.conf 2024-09-21T08:09:28.939 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-503.el9.x86_64.conf 2024-09-21T08:09:28.939 INFO:teuthology.orchestra.run.smithi137.stdout:fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64.conf 2024-09-21T08:09:28.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo grub2-set-default fbf6b6fa6cf04654a0e563f30ecbf43a-5.14.0-508.el9.x86_64 2024-09-21T08:09:29.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo shutdown -r now 2024-09-21T08:09:56.759 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-09-21T08:09:56.759 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-21T08:09:56.760 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T08:09:59.100 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-09-21T08:09:59.100 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-21T08:09:59.101 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T08:10:31.279 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-21T08:10:40.286 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-21T08:10:40.287 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T08:10:58.671 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.165 2024-09-21T08:10:59.104 DEBUG:teuthology.orchestra.remote:timed out 2024-09-21T08:11:08.112 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-21T08:11:08.113 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T08:11:08.698 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-21T08:11:09.126 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-21T08:11:09.127 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-508.el9.x86_64"... 2024-09-21T08:11:09.127 DEBUG:teuthology.orchestra.run.smithi137:> uname -r 2024-09-21T08:11:09.141 INFO:teuthology.orchestra.run.smithi137.stdout:5.14.0-508.el9.x86_64 2024-09-21T08:11:09.142 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T08:11:09.142 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T08:11:09.142 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T08:11:10.143 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-09-21T08:11:10.143 DEBUG:teuthology.orchestra.run.smithi137:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T08:11:10.255 INFO:teuthology.orchestra.run.smithi137.stdout:ttyS1 2024-09-21T08:11:10.273 DEBUG:teuthology.parallel:result is None 2024-09-21T08:11:10.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-21T08:11:10.672 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T08:11:11.208 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-21T08:11:11.667 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-21T08:11:11.667 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-508.el9.x86_64"... 2024-09-21T08:11:11.667 DEBUG:teuthology.orchestra.run.smithi165:> uname -r 2024-09-21T08:11:11.682 INFO:teuthology.orchestra.run.smithi165.stdout:5.14.0-508.el9.x86_64 2024-09-21T08:11:11.682 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-508.el9.x86_64 vs 5.14.0-508.el9.x86_64 2024-09-21T08:11:11.682 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-21T08:11:11.682 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-09-21T08:11:12.683 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-09-21T08:11:12.683 DEBUG:teuthology.orchestra.run.smithi165:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-21T08:11:12.751 INFO:teuthology.orchestra.run.smithi165.stdout:ttyS1 2024-09-21T08:11:12.774 DEBUG:teuthology.parallel:result is None 2024-09-21T08:11:12.774 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-21T08:11:12.781 INFO:teuthology.task.internal:Creating test directory... 2024-09-21T08:11:12.781 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T08:11:12.784 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-21T08:11:12.799 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-21T08:11:12.805 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-21T08:11:12.816 INFO:teuthology.task.internal:Creating archive directory... 2024-09-21T08:11:12.817 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T08:11:12.836 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-21T08:11:12.886 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-21T08:11:12.893 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-21T08:11:12.893 DEBUG:teuthology.orchestra.run.smithi137:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T08:11:12.895 DEBUG:teuthology.orchestra.run.smithi165:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-21T08:11:12.923 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T08:11:12.933 INFO:teuthology.orchestra.run.smithi137.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T08:11:12.937 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T08:11:12.947 INFO:teuthology.orchestra.run.smithi165.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-21T08:11:12.949 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-21T08:11:12.957 INFO:teuthology.task.internal:Configuring sudo... 2024-09-21T08:11:12.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T08:11:12.973 DEBUG:teuthology.orchestra.run.smithi165:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-21T08:11:13.015 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-21T08:11:13.027 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-21T08:11:13.028 DEBUG:teuthology.orchestra.run.smithi137:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T08:11:13.036 DEBUG:teuthology.orchestra.run.smithi165:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-21T08:11:13.070 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T08:11:13.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T08:11:13.181 DEBUG:teuthology.orchestra.run.smithi137:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T08:11:13.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T08:11:13.302 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:11:13.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T08:11:13.367 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T08:11:13.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-21T08:11:13.460 DEBUG:teuthology.orchestra.run.smithi165:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T08:11:13.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-21T08:11:13.583 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:11:13.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-21T08:11:13.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo service rsyslog restart 2024-09-21T08:11:13.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo service rsyslog restart 2024-09-21T08:11:13.699 INFO:teuthology.orchestra.run.smithi137.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T08:11:13.747 INFO:teuthology.orchestra.run.smithi165.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-09-21T08:11:14.031 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-21T08:11:14.046 INFO:teuthology.task.internal:Starting timer... 2024-09-21T08:11:14.046 INFO:teuthology.run_tasks:Running task pcp... 2024-09-21T08:11:14.058 INFO:teuthology.run_tasks:Running task selinux... 2024-09-21T08:11:14.092 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-09-21T08:11:14.092 DEBUG:teuthology.orchestra.run.smithi137:> sudo service auditd rotate 2024-09-21T08:11:14.147 INFO:teuthology.orchestra.run.smithi137.stdout:Rotating logs: 2024-09-21T08:11:14.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo service auditd rotate 2024-09-21T08:11:14.204 INFO:teuthology.orchestra.run.smithi165.stdout:Rotating logs: 2024-09-21T08:11:14.206 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-21T08:11:14.206 DEBUG:teuthology.orchestra.run.smithi137:> /usr/sbin/getenforce 2024-09-21T08:11:14.223 INFO:teuthology.orchestra.run.smithi137.stdout:Permissive 2024-09-21T08:11:14.223 DEBUG:teuthology.orchestra.run.smithi165:> /usr/sbin/getenforce 2024-09-21T08:11:14.243 INFO:teuthology.orchestra.run.smithi165.stdout:Permissive 2024-09-21T08:11:14.243 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi137.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi165.front.sepia.ceph.com': 'permissive'} 2024-09-21T08:11:14.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T08:11:14.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:11:14.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e context=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 2024-09-21T08:11:14.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:11:14.314 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-21T08:11:14.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo /usr/sbin/setenforce permissive 2024-09-21T08:11:14.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo /usr/sbin/setenforce permissive 2024-09-21T08:11:14.390 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-21T08:11:14.401 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-21T08:11:14.413 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-21T08:11:14.414 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi137.front.sepia.ceph.com,smithi165.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-21T08:14:55.492 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi137.front.sepia.ceph.com'), Remote(name='ubuntu@smithi165.front.sepia.ceph.com')] 2024-09-21T08:14:55.493 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-21T08:14:55.494 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi137.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T08:14:55.568 DEBUG:teuthology.orchestra.run.smithi137:> true 2024-09-21T08:14:55.600 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi137.front.sepia.ceph.com' 2024-09-21T08:14:55.601 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-21T08:14:55.601 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi165.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-21T08:14:55.676 DEBUG:teuthology.orchestra.run.smithi165:> true 2024-09-21T08:14:55.754 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi165.front.sepia.ceph.com' 2024-09-21T08:14:55.754 INFO:teuthology.run_tasks:Running task clock... 2024-09-21T08:14:55.764 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-21T08:14:55.764 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T08:14:55.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T08:14:55.767 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-21T08:14:55.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-21T08:14:55.795 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T08:14:55.811 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T08:14:55.823 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-09-21T08:14:55.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: ntpd: command not found 2024-09-21T08:14:55.837 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-09-21T08:14:55.847 INFO:teuthology.orchestra.run.smithi137.stdout:506 Cannot talk to daemon 2024-09-21T08:14:55.861 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T08:14:55.862 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: ntpd: command not found 2024-09-21T08:14:55.873 INFO:teuthology.orchestra.run.smithi165.stdout:506 Cannot talk to daemon 2024-09-21T08:14:55.876 INFO:teuthology.orchestra.run.smithi137.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T08:14:55.887 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-09-21T08:14:55.901 INFO:teuthology.orchestra.run.smithi165.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-09-21T08:14:55.932 INFO:teuthology.orchestra.run.smithi137.stderr:bash: line 1: ntpq: command not found 2024-09-21T08:14:55.936 INFO:teuthology.orchestra.run.smithi137.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T08:14:55.936 INFO:teuthology.orchestra.run.smithi137.stdout:=============================================================================== 2024-09-21T08:14:55.936 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T08:14:55.936 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T08:14:55.936 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T08:14:55.936 INFO:teuthology.orchestra.run.smithi137.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T08:14:55.957 INFO:teuthology.orchestra.run.smithi165.stderr:bash: line 1: ntpq: command not found 2024-09-21T08:14:55.960 INFO:teuthology.orchestra.run.smithi165.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-09-21T08:14:55.960 INFO:teuthology.orchestra.run.smithi165.stdout:=============================================================================== 2024-09-21T08:14:55.960 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T08:14:55.960 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T08:14:55.960 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T08:14:55.960 INFO:teuthology.orchestra.run.smithi165.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-09-21T08:14:55.961 INFO:teuthology.run_tasks:Running task pexec... 2024-09-21T08:14:55.976 INFO:teuthology.task.pexec:Executing custom commands... 2024-09-21T08:14:55.976 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi137.front.sepia.ceph.com 2024-09-21T08:14:55.977 DEBUG:teuthology.orchestra.run.smithi137:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T08:14:55.977 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi165.front.sepia.ceph.com 2024-09-21T08:14:55.977 DEBUG:teuthology.orchestra.run.smithi165:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-09-21T08:14:56.326 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:14:56.342 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:14:56.342 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-21T08:14:56.342 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:14:56.342 INFO:teuthology.orchestra.run.smithi137.stdout:Removing: 2024-09-21T08:14:56.342 INFO:teuthology.orchestra.run.smithi137.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T08:14:56.342 INFO:teuthology.orchestra.run.smithi137.stdout:Removing unused dependencies: 2024-09-21T08:14:56.342 INFO:teuthology.orchestra.run.smithi137.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T08:14:56.342 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:14:56.342 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:14:56.342 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:14:56.343 INFO:teuthology.orchestra.run.smithi137.stdout:Remove 2 Packages 2024-09-21T08:14:56.343 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:14:56.343 INFO:teuthology.orchestra.run.smithi137.stdout:Freed space: 5.5 M 2024-09-21T08:14:56.343 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:14:56.346 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:14:56.346 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:14:56.351 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:14:56.359 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout:Removing: 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout: nvme-cli x86_64 2.9.1-6.el9 @baseos 5.3 M 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout:Removing unused dependencies: 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout: libnvme x86_64 1.9-3.el9 @baseos 273 k 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout:Remove 2 Packages 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout:Freed space: 5.5 M 2024-09-21T08:14:56.360 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:14:56.367 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:14:56.367 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:14:56.408 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:14:56.409 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:14:56.417 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:14:56.417 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:14:56.554 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:14:56.574 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:14:56.633 INFO:teuthology.orchestra.run.smithi137.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T08:14:56.635 INFO:teuthology.orchestra.run.smithi165.stdout: Erasing : nvme-cli-2.9.1-6.el9.x86_64 1/2 2024-09-21T08:14:56.695 INFO:teuthology.orchestra.run.smithi137.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T08:14:56.702 INFO:teuthology.orchestra.run.smithi165.stdout: Erasing : libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T08:14:57.204 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T08:14:57.205 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T08:14:57.234 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libnvme-1.9-3.el9.x86_64 2/2 2024-09-21T08:14:57.234 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/2 2024-09-21T08:14:57.471 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T08:14:57.471 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:14:57.471 INFO:teuthology.orchestra.run.smithi137.stdout:Removed: 2024-09-21T08:14:57.471 INFO:teuthology.orchestra.run.smithi137.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T08:14:57.471 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:14:57.471 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:14:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/2 2024-09-21T08:14:57.519 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:14:57.519 INFO:teuthology.orchestra.run.smithi165.stdout:Removed: 2024-09-21T08:14:57.520 INFO:teuthology.orchestra.run.smithi165.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T08:14:57.520 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:14:57.520 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:14:58.312 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:46 ago on Sat 21 Sep 2024 08:14:12 AM UTC. 2024-09-21T08:14:58.383 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:46 ago on Sat 21 Sep 2024 08:14:12 AM UTC. 2024-09-21T08:14:58.446 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:14:58.447 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:14:58.447 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:14:58.447 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:14:58.447 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:14:58.447 INFO:teuthology.orchestra.run.smithi137.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T08:14:58.447 INFO:teuthology.orchestra.run.smithi137.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T08:14:58.447 INFO:teuthology.orchestra.run.smithi137.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T08:14:58.447 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-21T08:14:58.447 INFO:teuthology.orchestra.run.smithi137.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T08:14:58.448 INFO:teuthology.orchestra.run.smithi137.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T08:14:58.448 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T08:14:58.448 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T08:14:58.448 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:14:58.448 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:14:58.448 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:14:58.448 INFO:teuthology.orchestra.run.smithi137.stdout:Install 7 Packages 2024-09-21T08:14:58.448 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:14:58.464 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 5.2 M 2024-09-21T08:14:58.464 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 19 M 2024-09-21T08:14:58.464 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:14:58.517 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:14:58.518 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:14:58.518 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:14:58.518 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:14:58.518 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:14:58.518 INFO:teuthology.orchestra.run.smithi165.stdout: nvme-cli x86_64 2.9.1-6.el9 baseos 906 k 2024-09-21T08:14:58.518 INFO:teuthology.orchestra.run.smithi165.stdout: nvmetcli noarch 0.7-3.el9 baseos 44 k 2024-09-21T08:14:58.518 INFO:teuthology.orchestra.run.smithi165.stdout: runc x86_64 4:1.1.13-4.el9 appstream 3.2 M 2024-09-21T08:14:58.519 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-21T08:14:58.519 INFO:teuthology.orchestra.run.smithi165.stdout: libnvme x86_64 1.9-3.el9 baseos 106 k 2024-09-21T08:14:58.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-configshell noarch 1:1.1.30-1.el9 baseos 72 k 2024-09-21T08:14:58.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kmod x86_64 0.9-32.el9 baseos 84 k 2024-09-21T08:14:58.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urwid x86_64 2.1.2-4.el9 baseos 837 k 2024-09-21T08:14:58.519 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:14:58.519 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:14:58.519 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:14:58.519 INFO:teuthology.orchestra.run.smithi165.stdout:Install 7 Packages 2024-09-21T08:14:58.519 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:14:58.531 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 5.2 M 2024-09-21T08:14:58.531 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 19 M 2024-09-21T08:14:58.531 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:14:58.725 INFO:teuthology.orchestra.run.smithi137.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 506 kB/s | 44 kB 00:00 2024-09-21T08:14:58.750 INFO:teuthology.orchestra.run.smithi137.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 935 kB/s | 106 kB 00:00 2024-09-21T08:14:58.784 INFO:teuthology.orchestra.run.smithi137.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 1.2 MB/s | 72 kB 00:00 2024-09-21T08:14:58.810 INFO:teuthology.orchestra.run.smithi137.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.4 MB/s | 84 kB 00:00 2024-09-21T08:14:58.843 INFO:teuthology.orchestra.run.smithi137.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 4.3 MB/s | 906 kB 00:00 2024-09-21T08:14:58.910 INFO:teuthology.orchestra.run.smithi137.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 6.5 MB/s | 837 kB 00:00 2024-09-21T08:14:58.951 INFO:teuthology.orchestra.run.smithi165.stdout:(1/7): nvmetcli-0.7-3.el9.noarch.rpm 383 kB/s | 44 kB 00:00 2024-09-21T08:14:58.993 INFO:teuthology.orchestra.run.smithi165.stdout:(2/7): libnvme-1.9-3.el9.x86_64.rpm 670 kB/s | 106 kB 00:00 2024-09-21T08:14:59.026 INFO:teuthology.orchestra.run.smithi165.stdout:(3/7): python3-configshell-1.1.30-1.el9.noarch. 952 kB/s | 72 kB 00:00 2024-09-21T08:14:59.068 INFO:teuthology.orchestra.run.smithi165.stdout:(4/7): python3-kmod-0.9-32.el9.x86_64.rpm 1.1 MB/s | 84 kB 00:00 2024-09-21T08:14:59.218 INFO:teuthology.orchestra.run.smithi165.stdout:(5/7): nvme-cli-2.9.1-6.el9.x86_64.rpm 2.3 MB/s | 906 kB 00:00 2024-09-21T08:14:59.344 INFO:teuthology.orchestra.run.smithi165.stdout:(6/7): python3-urwid-2.1.2-4.el9.x86_64.rpm 2.6 MB/s | 837 kB 00:00 2024-09-21T08:14:59.444 INFO:teuthology.orchestra.run.smithi165.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 8.4 MB/s | 3.2 MB 00:00 2024-09-21T08:14:59.444 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:14:59.445 INFO:teuthology.orchestra.run.smithi165.stdout:Total 5.7 MB/s | 5.2 MB 00:00 2024-09-21T08:14:59.557 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:14:59.572 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:14:59.572 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:14:59.769 INFO:teuthology.orchestra.run.smithi137.stdout:(7/7): runc-1.1.13-4.el9.x86_64.rpm 3.3 MB/s | 3.2 MB 00:00 2024-09-21T08:14:59.769 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:14:59.769 INFO:teuthology.orchestra.run.smithi137.stdout:Total 4.0 MB/s | 5.2 MB 00:01 2024-09-21T08:14:59.845 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:14:59.845 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:14:59.879 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:14:59.894 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:14:59.894 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:15:00.194 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:15:00.194 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:15:00.348 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:15:00.431 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T08:15:00.469 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T08:15:00.510 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T08:15:00.666 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:15:00.718 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T08:15:00.746 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urwid-2.1.2-4.el9.x86_64 1/7 2024-09-21T08:15:00.749 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T08:15:00.780 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-configshell-1:1.1.30-1.el9.noarch 2/7 2024-09-21T08:15:00.809 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kmod-0.9-32.el9.x86_64 3/7 2024-09-21T08:15:01.014 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libnvme-1.9-3.el9.x86_64 4/7 2024-09-21T08:15:01.042 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T08:15:01.191 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T08:15:01.229 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T08:15:01.348 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T08:15:01.423 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T08:15:01.490 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: nvme-cli-2.9.1-6.el9.x86_64 5/7 2024-09-21T08:15:01.520 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T08:15:01.640 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: nvmetcli-0.7-3.el9.noarch 6/7 2024-09-21T08:15:01.707 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T08:15:02.370 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T08:15:02.370 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T08:15:02.370 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T08:15:02.371 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T08:15:02.371 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T08:15:02.371 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T08:15:02.371 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T08:15:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T08:15:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libnvme-1.9-3.el9.x86_64 1/7 2024-09-21T08:15:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : nvme-cli-2.9.1-6.el9.x86_64 2/7 2024-09-21T08:15:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : nvmetcli-0.7-3.el9.noarch 3/7 2024-09-21T08:15:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-configshell-1:1.1.30-1.el9.noarch 4/7 2024-09-21T08:15:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kmod-0.9-32.el9.x86_64 5/7 2024-09-21T08:15:02.693 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urwid-2.1.2-4.el9.x86_64 6/7 2024-09-21T08:15:02.739 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T08:15:02.739 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:02.739 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:15:02.739 INFO:teuthology.orchestra.run.smithi165.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T08:15:02.740 INFO:teuthology.orchestra.run.smithi165.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T08:15:02.740 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T08:15:02.740 INFO:teuthology.orchestra.run.smithi165.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T08:15:02.740 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:02.740 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:15:03.062 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : runc-4:1.1.13-4.el9.x86_64 7/7 2024-09-21T08:15:03.062 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:03.062 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:15:03.062 INFO:teuthology.orchestra.run.smithi137.stdout: libnvme-1.9-3.el9.x86_64 nvme-cli-2.9.1-6.el9.x86_64 2024-09-21T08:15:03.062 INFO:teuthology.orchestra.run.smithi137.stdout: nvmetcli-0.7-3.el9.noarch python3-configshell-1:1.1.30-1.el9.noarch 2024-09-21T08:15:03.062 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kmod-0.9-32.el9.x86_64 python3-urwid-2.1.2-4.el9.x86_64 2024-09-21T08:15:03.062 INFO:teuthology.orchestra.run.smithi137.stdout: runc-4:1.1.13-4.el9.x86_64 2024-09-21T08:15:03.062 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:03.062 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:15:03.128 DEBUG:teuthology.parallel:result is None 2024-09-21T08:15:03.434 DEBUG:teuthology.parallel:result is None 2024-09-21T08:15:03.435 INFO:teuthology.run_tasks:Running task install... 2024-09-21T08:15:03.449 DEBUG:teuthology.task.install:project ceph 2024-09-21T08:15:03.449 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736'}} 2024-09-21T08:15:03.449 DEBUG:teuthology.task.install:config {'flavor': 'default', 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736'} 2024-09-21T08:15:03.450 INFO:teuthology.task.install:Using flavor: default 2024-09-21T08:15:03.455 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-21T08:15:03.455 INFO:teuthology.task.install:extra packages: [] 2024-09-21T08:15:03.455 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736', 'tag': None, 'wait_for_package': False} 2024-09-21T08:15:03.456 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-21T08:15:03.457 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736', 'tag': None, 'wait_for_package': False} 2024-09-21T08:15:03.457 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-21T08:15:03.601 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/ 2024-09-21T08:15:03.602 INFO:teuthology.task.install.rpm:Package version is 19.1.1-367.g9e2ac70b 2024-09-21T08:15:03.606 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/ 2024-09-21T08:15:03.606 INFO:teuthology.task.install.rpm:Package version is 19.1.1-367.g9e2ac70b 2024-09-21T08:15:03.712 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T08:15:03.712 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:15:03.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T08:15:03.717 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-09-21T08:15:03.717 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:15:03.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-09-21T08:15:03.739 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T08:15:03.739 DEBUG:teuthology.orchestra.run.smithi165:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9e2ac70bb7aea67d34f0d50002344abbe7803736/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T08:15:03.743 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-09-21T08:15:03.744 DEBUG:teuthology.orchestra.run.smithi137:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/9e2ac70bb7aea67d34f0d50002344abbe7803736/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-09-21T08:15:03.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T08:15:03.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-09-21T08:15:03.881 DEBUG:teuthology.orchestra.run.smithi165:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T08:15:03.896 DEBUG:teuthology.orchestra.run.smithi137:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-09-21T08:15:03.932 INFO:teuthology.orchestra.run.smithi137.stdout:check_obsoletes = 1 2024-09-21T08:15:03.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum clean all 2024-09-21T08:15:03.946 INFO:teuthology.orchestra.run.smithi165.stdout:check_obsoletes = 1 2024-09-21T08:15:03.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum clean all 2024-09-21T08:15:04.162 INFO:teuthology.orchestra.run.smithi137.stdout:45 files removed 2024-09-21T08:15:04.184 INFO:teuthology.orchestra.run.smithi165.stdout:45 files removed 2024-09-21T08:15:04.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-radosgw 2024-09-21T08:15:04.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-radosgw 2024-09-21T08:15:04.743 INFO:teuthology.orchestra.run.smithi137.stdout:ceph packages for x86_64 308 kB/s | 84 kB 00:00 2024-09-21T08:15:04.777 INFO:teuthology.orchestra.run.smithi165.stdout:ceph packages for x86_64 303 kB/s | 84 kB 00:00 2024-09-21T08:15:05.028 INFO:teuthology.orchestra.run.smithi137.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-21T08:15:05.061 INFO:teuthology.orchestra.run.smithi165.stdout:ceph noarch packages 49 kB/s | 12 kB 00:00 2024-09-21T08:15:05.303 INFO:teuthology.orchestra.run.smithi137.stdout:ceph source packages 7.9 kB/s | 1.9 kB 00:00 2024-09-21T08:15:05.321 INFO:teuthology.orchestra.run.smithi165.stdout:ceph source packages 8.5 kB/s | 1.9 kB 00:00 2024-09-21T08:15:05.937 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.3 MB 00:00 2024-09-21T08:15:07.236 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - BaseOS 4.3 MB/s | 8.3 MB 00:01 2024-09-21T08:15:10.838 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - AppStream 5.3 MB/s | 20 MB 00:03 2024-09-21T08:15:16.153 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - AppStream 2.6 MB/s | 20 MB 00:07 2024-09-21T08:15:16.463 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.5 MB 00:01 2024-09-21T08:15:17.856 INFO:teuthology.orchestra.run.smithi165.stdout:CentOS Stream 9 - Extras packages 121 kB/s | 19 kB 00:00 2024-09-21T08:15:18.758 INFO:teuthology.orchestra.run.smithi165.stdout:Extra Packages for Enterprise Linux 29 MB/s | 23 MB 00:00 2024-09-21T08:15:20.545 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.5 MB 00:00 2024-09-21T08:15:22.838 INFO:teuthology.orchestra.run.smithi137.stdout:CentOS Stream 9 - Extras packages 18 kB/s | 19 kB 00:01 2024-09-21T08:15:23.950 INFO:teuthology.orchestra.run.smithi137.stdout:Extra Packages for Enterprise Linux 23 MB/s | 23 MB 00:00 2024-09-21T08:15:24.249 INFO:teuthology.orchestra.run.smithi165.stdout:lab-extras 27 kB/s | 1.7 kB 00:00 2024-09-21T08:15:26.441 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:15:26.444 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:15:26.444 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:15:26.444 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:15:26.444 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:15:26.444 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 11 M 2024-09-21T08:15:26.444 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrading: 2024-09-21T08:15:26.444 INFO:teuthology.orchestra.run.smithi165.stdout: librados2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.4 M 2024-09-21T08:15:26.444 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.2 M 2024-09-21T08:15:26.444 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-21T08:15:26.444 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 5.4 M 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 21 M 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 25 k 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 752 k 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 506 k 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T08:15:26.445 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 5.3 M 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 45 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 136 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 161 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 320 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 301 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 99 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T08:15:26.446 INFO:teuthology.orchestra.run.smithi165.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T08:15:26.447 INFO:teuthology.orchestra.run.smithi165.stdout:Installing weak dependencies: 2024-09-21T08:15:26.447 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T08:15:26.447 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:26.447 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:15:26.447 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:15:26.447 INFO:teuthology.orchestra.run.smithi165.stdout:Install 30 Packages 2024-09-21T08:15:26.447 INFO:teuthology.orchestra.run.smithi165.stdout:Upgrade 2 Packages 2024-09-21T08:15:26.447 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:26.447 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 60 M 2024-09-21T08:15:26.448 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:15:27.451 INFO:teuthology.orchestra.run.smithi165.stdout:(1/32): ceph-radosgw-19.1.1-367.g9e2ac70b.el9.x 14 MB/s | 11 MB 00:00 2024-09-21T08:15:27.534 INFO:teuthology.orchestra.run.smithi165.stdout:(2/32): ceph-base-19.1.1-367.g9e2ac70b.el9.x86_ 6.6 MB/s | 5.4 MB 00:00 2024-09-21T08:15:27.560 INFO:teuthology.orchestra.run.smithi165.stdout:(3/32): ceph-selinux-19.1.1-367.g9e2ac70b.el9.x 232 kB/s | 25 kB 00:00 2024-09-21T08:15:27.610 INFO:teuthology.orchestra.run.smithi165.stdout:(4/32): libcephfs2-19.1.1-367.g9e2ac70b.el9.x86 9.8 MB/s | 752 kB 00:00 2024-09-21T08:15:27.635 INFO:teuthology.orchestra.run.smithi165.stdout:(5/32): libradosstriper1-19.1.1-367.g9e2ac70b.e 6.6 MB/s | 506 kB 00:00 2024-09-21T08:15:27.685 INFO:teuthology.orchestra.run.smithi165.stdout:(6/32): python3-ceph-argparse-19.1.1-367.g9e2ac 901 kB/s | 45 kB 00:00 2024-09-21T08:15:27.902 INFO:teuthology.orchestra.run.smithi165.stdout:(7/32): ceph-common-19.1.1-367.g9e2ac70b.el9.x8 18 MB/s | 21 MB 00:01 2024-09-21T08:15:27.927 INFO:teuthology.orchestra.run.smithi165.stdout:(8/32): python3-ceph-common-19.1.1-367.g9e2ac70 562 kB/s | 136 kB 00:00 2024-09-21T08:15:27.953 INFO:teuthology.orchestra.run.smithi165.stdout:(9/32): python3-cephfs-19.1.1-367.g9e2ac70b.el9 3.2 MB/s | 161 kB 00:00 2024-09-21T08:15:27.986 INFO:teuthology.orchestra.run.smithi165.stdout:(10/32): python3-rados-19.1.1-367.g9e2ac70b.el9 5.3 MB/s | 320 kB 00:00 2024-09-21T08:15:28.012 INFO:teuthology.orchestra.run.smithi165.stdout:(11/32): python3-rbd-19.1.1-367.g9e2ac70b.el9.x 5.0 MB/s | 301 kB 00:00 2024-09-21T08:15:28.037 INFO:teuthology.orchestra.run.smithi165.stdout:(12/32): python3-rgw-19.1.1-367.g9e2ac70b.el9.x 1.9 MB/s | 99 kB 00:00 2024-09-21T08:15:28.121 INFO:teuthology.orchestra.run.smithi165.stdout:(13/32): librgw2-19.1.1-367.g9e2ac70b.el9.x86_6 10 MB/s | 5.3 MB 00:00 2024-09-21T08:15:28.204 INFO:teuthology.orchestra.run.smithi165.stdout:(14/32): boost-program-options-1.75.0-8.el9.x86 557 kB/s | 107 kB 00:00 2024-09-21T08:15:28.254 INFO:teuthology.orchestra.run.smithi165.stdout:(15/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 339 kB/s | 45 kB 00:00 2024-09-21T08:15:28.280 INFO:teuthology.orchestra.run.smithi165.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 660 kB/s | 160 kB 00:00 2024-09-21T08:15:28.322 INFO:teuthology.orchestra.run.smithi165.stdout:(17/32): python3-prettytable-0.7.2-27.el9.noarc 1.0 MB/s | 42 kB 00:00 2024-09-21T08:15:28.355 INFO:teuthology.orchestra.run.smithi165.stdout:(18/32): librdkafka-1.6.1-102.el9.x86_64.rpm 4.3 MB/s | 662 kB 00:00 2024-09-21T08:15:28.380 INFO:teuthology.orchestra.run.smithi165.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.3 MB/s | 292 kB 00:00 2024-09-21T08:15:28.406 INFO:teuthology.orchestra.run.smithi165.stdout:(20/32): qatlib-24.02.0-1.el9.x86_64.rpm 2.6 MB/s | 221 kB 00:00 2024-09-21T08:15:28.431 INFO:teuthology.orchestra.run.smithi165.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 476 kB/s | 36 kB 00:00 2024-09-21T08:15:28.457 INFO:teuthology.orchestra.run.smithi165.stdout:(22/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 626 kB/s | 47 kB 00:00 2024-09-21T08:15:28.507 INFO:teuthology.orchestra.run.smithi165.stdout:(23/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 3.0 MB/s | 308 kB 00:00 2024-09-21T08:15:28.532 INFO:teuthology.orchestra.run.smithi165.stdout:(24/32): liboath-2.6.7-2.el9.x86_64.rpm 2.0 MB/s | 49 kB 00:00 2024-09-21T08:15:28.557 INFO:teuthology.orchestra.run.smithi165.stdout:(25/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 300 kB/s | 26 kB 00:00 2024-09-21T08:15:28.583 INFO:teuthology.orchestra.run.smithi165.stdout:(26/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-09-21T08:15:28.616 INFO:teuthology.orchestra.run.smithi165.stdout:(27/32): re2-20211101-20.el9.x86_64.rpm 5.7 MB/s | 191 kB 00:00 2024-09-21T08:15:28.675 INFO:teuthology.orchestra.run.smithi165.stdout:(28/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 7.0 MB/s | 839 kB 00:00 2024-09-21T08:15:28.734 INFO:teuthology.orchestra.run.smithi165.stdout:(29/32): thrift-0.15.0-2.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-09-21T08:15:28.825 INFO:teuthology.orchestra.run.smithi165.stdout:(30/32): libarrow-9.0.0-11.el9.x86_64.rpm 11 MB/s | 4.4 MB 00:00 2024-09-21T08:15:28.951 INFO:teuthology.orchestra.run.smithi165.stdout:(31/32): librbd1-19.1.1-367.g9e2ac70b.el9.x86_6 15 MB/s | 3.2 MB 00:00 2024-09-21T08:15:29.009 INFO:teuthology.orchestra.run.smithi165.stdout:(32/32): librados2-19.1.1-367.g9e2ac70b.el9.x86 10 MB/s | 3.4 MB 00:00 2024-09-21T08:15:29.013 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:15:29.014 INFO:teuthology.orchestra.run.smithi165.stdout:Total 24 MB/s | 60 MB 00:02 2024-09-21T08:15:29.110 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:15:29.145 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:15:29.145 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:15:29.364 INFO:teuthology.orchestra.run.smithi137.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-09-21T08:15:29.690 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:15:29.690 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:15:31.120 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:15:31.205 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-21T08:15:31.240 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-21T08:15:31.273 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-21T08:15:31.295 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-21T08:15:31.319 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-21T08:15:31.507 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-21T08:15:31.556 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-21T08:15:31.609 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:15:31.611 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 11 M 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrading: 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout: librados2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.4 M 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.2 M 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 5.4 M 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 21 M 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 25 k 2024-09-21T08:15:31.612 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs x86_64 2.9.1-3.el9 epel 308 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 752 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 506 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2 x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 5.3 M 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 45 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 136 k 2024-09-21T08:15:31.613 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 161 k 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 320 k 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 301 k 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 99 k 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout: re2 x86_64 1:20211101-20.el9 epel 191 k 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout:Installing weak dependencies: 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout:Install 30 Packages 2024-09-21T08:15:31.614 INFO:teuthology.orchestra.run.smithi137.stdout:Upgrade 2 Packages 2024-09-21T08:15:31.615 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:31.615 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 60 M 2024-09-21T08:15:31.615 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:15:31.917 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-21T08:15:32.012 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-21T08:15:32.082 INFO:teuthology.orchestra.run.smithi137.stdout:(1/32): ceph-base-19.1.1-367.g9e2ac70b.el9.x86_ 13 MB/s | 5.4 MB 00:00 2024-09-21T08:15:32.105 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-21T08:15:32.132 INFO:teuthology.orchestra.run.smithi137.stdout:(2/32): ceph-selinux-19.1.1-367.g9e2ac70b.el9.x 504 kB/s | 25 kB 00:00 2024-09-21T08:15:32.191 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 8/34 2024-09-21T08:15:32.199 INFO:teuthology.orchestra.run.smithi137.stdout:(3/32): libcephfs2-19.1.1-367.g9e2ac70b.el9.x86 11 MB/s | 752 kB 00:00 2024-09-21T08:15:32.222 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 9/34 2024-09-21T08:15:32.249 INFO:teuthology.orchestra.run.smithi137.stdout:(4/32): libradosstriper1-19.1.1-367.g9e2ac70b.e 9.9 MB/s | 506 kB 00:00 2024-09-21T08:15:32.268 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 9/34 2024-09-21T08:15:32.305 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-21T08:15:32.349 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x 11/34 2024-09-21T08:15:32.434 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 12/34 2024-09-21T08:15:32.458 INFO:teuthology.orchestra.run.smithi137.stdout:(5/32): librgw2-19.1.1-367.g9e2ac70b.el9.x86_64 26 MB/s | 5.3 MB 00:00 2024-09-21T08:15:32.458 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-21T08:15:32.508 INFO:teuthology.orchestra.run.smithi137.stdout:(6/32): python3-ceph-argparse-19.1.1-367.g9e2ac 905 kB/s | 45 kB 00:00 2024-09-21T08:15:32.530 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-21T08:15:32.559 INFO:teuthology.orchestra.run.smithi137.stdout:(7/32): python3-ceph-common-19.1.1-367.g9e2ac70 2.7 MB/s | 136 kB 00:00 2024-09-21T08:15:32.566 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-21T08:15:32.609 INFO:teuthology.orchestra.run.smithi137.stdout:(8/32): python3-cephfs-19.1.1-367.g9e2ac70b.el9 3.2 MB/s | 161 kB 00:00 2024-09-21T08:15:32.667 INFO:teuthology.orchestra.run.smithi137.stdout:(9/32): python3-rados-19.1.1-367.g9e2ac70b.el9. 5.4 MB/s | 320 kB 00:00 2024-09-21T08:15:32.679 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-21T08:15:32.725 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-21T08:15:32.726 INFO:teuthology.orchestra.run.smithi137.stdout:(10/32): python3-rbd-19.1.1-367.g9e2ac70b.el9.x 5.0 MB/s | 301 kB 00:00 2024-09-21T08:15:32.781 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-21T08:15:32.784 INFO:teuthology.orchestra.run.smithi137.stdout:(11/32): python3-rgw-19.1.1-367.g9e2ac70b.el9.x 1.7 MB/s | 99 kB 00:00 2024-09-21T08:15:32.823 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-21T08:15:32.853 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-21T08:15:32.886 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-21T08:15:32.977 INFO:teuthology.orchestra.run.smithi137.stdout:(12/32): ceph-radosgw-19.1.1-367.g9e2ac70b.el9. 8.1 MB/s | 11 MB 00:01 2024-09-21T08:15:33.035 INFO:teuthology.orchestra.run.smithi137.stdout:(13/32): boost-program-options-1.75.0-8.el9.x86 427 kB/s | 107 kB 00:00 2024-09-21T08:15:33.110 INFO:teuthology.orchestra.run.smithi137.stdout:(14/32): librabbitmq-0.11.0-7.el9.x86_64.rpm 602 kB/s | 45 kB 00:00 2024-09-21T08:15:33.222 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-21T08:15:33.273 INFO:teuthology.orchestra.run.smithi165.stdout: Upgrading : librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 22/34 2024-09-21T08:15:33.369 INFO:teuthology.orchestra.run.smithi137.stdout:(15/32): ceph-common-19.1.1-367.g9e2ac70b.el9.x 12 MB/s | 21 MB 00:01 2024-09-21T08:15:33.379 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 22/34 2024-09-21T08:15:33.394 INFO:teuthology.orchestra.run.smithi137.stdout:(16/32): libpmemobj-1.12.1-1.el9.x86_64.rpm 384 kB/s | 160 kB 00:00 2024-09-21T08:15:33.411 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 23/34 2024-09-21T08:15:33.428 INFO:teuthology.orchestra.run.smithi137.stdout:(17/32): librdkafka-1.6.1-102.el9.x86_64.rpm 2.0 MB/s | 662 kB 00:00 2024-09-21T08:15:33.448 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-21T08:15:33.495 INFO:teuthology.orchestra.run.smithi137.stdout:(18/32): qatlib-24.02.0-1.el9.x86_64.rpm 3.2 MB/s | 221 kB 00:00 2024-09-21T08:15:33.521 INFO:teuthology.orchestra.run.smithi137.stdout:(19/32): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.9 MB/s | 292 kB 00:00 2024-09-21T08:15:33.545 INFO:teuthology.orchestra.run.smithi137.stdout:(20/32): python3-prettytable-0.7.2-27.el9.noarc 280 kB/s | 42 kB 00:00 2024-09-21T08:15:33.571 INFO:teuthology.orchestra.run.smithi137.stdout:(21/32): qatlib-service-24.02.0-1.el9.x86_64.rp 473 kB/s | 36 kB 00:00 2024-09-21T08:15:33.621 INFO:teuthology.orchestra.run.smithi137.stdout:(22/32): gperftools-libs-2.9.1-3.el9.x86_64.rpm 4.0 MB/s | 308 kB 00:00 2024-09-21T08:15:33.646 INFO:teuthology.orchestra.run.smithi137.stdout:(23/32): qatzip-libs-1.2.0-1.el9.x86_64.rpm 377 kB/s | 47 kB 00:00 2024-09-21T08:15:33.671 INFO:teuthology.orchestra.run.smithi137.stdout:(24/32): libarrow-doc-9.0.0-11.el9.noarch.rpm 535 kB/s | 26 kB 00:00 2024-09-21T08:15:33.722 INFO:teuthology.orchestra.run.smithi137.stdout:(25/32): libunwind-1.6.2-1.el9.x86_64.rpm 1.5 MB/s | 67 kB 00:00 2024-09-21T08:15:33.747 INFO:teuthology.orchestra.run.smithi137.stdout:(26/32): liboath-2.6.7-2.el9.x86_64.rpm 484 kB/s | 49 kB 00:00 2024-09-21T08:15:33.797 INFO:teuthology.orchestra.run.smithi137.stdout:(27/32): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-09-21T08:15:33.823 INFO:teuthology.orchestra.run.smithi137.stdout:(28/32): re2-20211101-20.el9.x86_64.rpm 2.5 MB/s | 191 kB 00:00 2024-09-21T08:15:33.915 INFO:teuthology.orchestra.run.smithi137.stdout:(29/32): libarrow-9.0.0-11.el9.x86_64.rpm 13 MB/s | 4.4 MB 00:00 2024-09-21T08:15:33.945 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-21T08:15:33.956 INFO:teuthology.orchestra.run.smithi137.stdout:(30/32): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-09-21T08:15:33.984 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 26/34 2024-09-21T08:15:34.055 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 26/34 2024-09-21T08:15:34.073 INFO:teuthology.orchestra.run.smithi137.stdout:(31/32): librados2-19.1.1-367.g9e2ac70b.el9.x86 14 MB/s | 3.4 MB 00:00 2024-09-21T08:15:34.125 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 27/34 2024-09-21T08:15:34.154 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86 28/34 2024-09-21T08:15:34.157 INFO:teuthology.orchestra.run.smithi137.stdout:(32/32): librbd1-19.1.1-367.g9e2ac70b.el9.x86_6 13 MB/s | 3.2 MB 00:00 2024-09-21T08:15:34.162 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:15:34.162 INFO:teuthology.orchestra.run.smithi137.stdout:Total 24 MB/s | 60 MB 00:02 2024-09-21T08:15:34.267 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:15:34.302 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:15:34.302 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:15:34.804 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:15:34.805 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:15:36.269 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:15:36.349 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/34 2024-09-21T08:15:36.386 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 2/34 2024-09-21T08:15:36.412 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 3/34 2024-09-21T08:15:36.434 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-21T08:15:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-21T08:15:36.460 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 4/34 2024-09-21T08:15:36.516 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-21T08:15:36.629 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-21T08:15:36.682 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 5/34 2024-09-21T08:15:37.027 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/34 2024-09-21T08:15:37.076 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-21T08:15:37.112 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 30/34 2024-09-21T08:15:37.123 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-21T08:15:37.206 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 30/34 2024-09-21T08:15:37.206 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T08:15:37.206 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T08:15:37.206 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:37.207 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-21T08:15:37.238 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-21T08:15:37.296 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 8/34 2024-09-21T08:15:37.333 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 9/34 2024-09-21T08:15:37.387 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 9/34 2024-09-21T08:15:37.415 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : liboath-2.6.7-2.el9.x86_64 10/34 2024-09-21T08:15:37.451 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x 11/34 2024-09-21T08:15:37.521 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 12/34 2024-09-21T08:15:37.543 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-21T08:15:37.607 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-21T08:15:37.641 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 14/34 2024-09-21T08:15:37.752 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : re2-1:20211101-20.el9.x86_64 15/34 2024-09-21T08:15:37.784 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 16/34 2024-09-21T08:15:37.825 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 17/34 2024-09-21T08:15:37.857 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : gperftools-libs-2.9.1-3.el9.x86_64 18/34 2024-09-21T08:15:37.887 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 19/34 2024-09-21T08:15:37.920 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 20/34 2024-09-21T08:15:38.253 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 21/34 2024-09-21T08:15:38.291 INFO:teuthology.orchestra.run.smithi137.stdout: Upgrading : librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 22/34 2024-09-21T08:15:38.365 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 22/34 2024-09-21T08:15:38.400 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 23/34 2024-09-21T08:15:38.439 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 24/34 2024-09-21T08:15:38.911 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 25/34 2024-09-21T08:15:38.943 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 26/34 2024-09-21T08:15:39.015 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 26/34 2024-09-21T08:15:39.091 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 27/34 2024-09-21T08:15:39.120 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86 28/34 2024-09-21T08:15:41.435 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-21T08:15:41.542 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-21T08:15:42.095 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 29/34 2024-09-21T08:15:42.136 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 30/34 2024-09-21T08:15:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 30/34 2024-09-21T08:15:42.228 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-09-21T08:15:42.228 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-09-21T08:15:42.228 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:42.263 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-21T08:15:47.386 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-21T08:15:47.386 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /sys 2024-09-21T08:15:47.386 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /proc 2024-09-21T08:15:47.386 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /mnt 2024-09-21T08:15:47.386 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /var/tmp 2024-09-21T08:15:47.386 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /home 2024-09-21T08:15:47.386 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /root 2024-09-21T08:15:47.387 INFO:teuthology.orchestra.run.smithi165.stdout:skipping the directory /tmp 2024-09-21T08:15:47.387 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:48.348 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 32/34 2024-09-21T08:15:48.374 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 32/34 2024-09-21T08:15:48.374 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:15:48.374 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T08:15:48.374 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T08:15:48.374 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T08:15:48.374 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:48.376 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-21T08:15:48.459 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-21T08:15:48.459 INFO:teuthology.orchestra.run.smithi165.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T08:15:49.577 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T08:15:49.577 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 5/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 6/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x 8/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86 9/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 10/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 11/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 12/34 2024-09-21T08:15:49.578 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-21T08:15:49.579 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-21T08:15:49.580 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-21T08:15:49.580 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-21T08:15:49.580 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-21T08:15:49.580 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-21T08:15:49.580 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/34 2024-09-21T08:15:50.378 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T08:15:50.378 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:50.378 INFO:teuthology.orchestra.run.smithi165.stdout:Upgraded: 2024-09-21T08:15:50.378 INFO:teuthology.orchestra.run.smithi165.stdout: librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.378 INFO:teuthology.orchestra.run.smithi165.stdout: librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.378 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:15:50.378 INFO:teuthology.orchestra.run.smithi165.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T08:15:50.379 INFO:teuthology.orchestra.run.smithi165.stdout: librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T08:15:50.380 INFO:teuthology.orchestra.run.smithi165.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T08:15:50.381 INFO:teuthology.orchestra.run.smithi165.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T08:15:50.381 INFO:teuthology.orchestra.run.smithi165.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T08:15:50.381 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:50.381 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:15:50.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-test 2024-09-21T08:15:51.365 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:15:51.498 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:15:51.499 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:15:51.499 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:15:51.499 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:15:51.499 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:15:51.499 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 49 M 2024-09-21T08:15:51.499 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-21T08:15:51.499 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 163 k 2024-09-21T08:15:51.499 INFO:teuthology.orchestra.run.smithi165.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T08:15:51.499 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T08:15:51.499 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:51.500 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:15:51.500 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:15:51.500 INFO:teuthology.orchestra.run.smithi165.stdout:Install 4 Packages 2024-09-21T08:15:51.500 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:15:51.500 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 49 M 2024-09-21T08:15:51.500 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 209 M 2024-09-21T08:15:51.500 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:15:51.817 INFO:teuthology.orchestra.run.smithi165.stdout:(1/4): socat-1.7.4.1-6.el9.x86_64.rpm 1.5 MB/s | 304 kB 00:00 2024-09-21T08:15:51.842 INFO:teuthology.orchestra.run.smithi165.stdout:(2/4): libcephsqlite-19.1.1-367.g9e2ac70b.el9.x 732 kB/s | 163 kB 00:00 2024-09-21T08:15:51.867 INFO:teuthology.orchestra.run.smithi165.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 64 kB 00:00 2024-09-21T08:15:52.368 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-21T08:15:52.368 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /sys 2024-09-21T08:15:52.368 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /proc 2024-09-21T08:15:52.368 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /mnt 2024-09-21T08:15:52.368 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /var/tmp 2024-09-21T08:15:52.368 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /home 2024-09-21T08:15:52.368 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /root 2024-09-21T08:15:52.368 INFO:teuthology.orchestra.run.smithi137.stdout:skipping the directory /tmp 2024-09-21T08:15:52.368 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:53.135 INFO:teuthology.orchestra.run.smithi165.stdout:(4/4): ceph-test-19.1.1-367.g9e2ac70b.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-21T08:15:53.137 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:15:53.137 INFO:teuthology.orchestra.run.smithi165.stdout:Total 30 MB/s | 49 MB 00:01 2024-09-21T08:15:53.165 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:15:53.185 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:15:53.185 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:15:53.354 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 32/34 2024-09-21T08:15:53.380 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 32/34 2024-09-21T08:15:53.380 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:15:53.380 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-09-21T08:15:53.380 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T08:15:53.380 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-09-21T08:15:53.380 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:53.382 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-21T08:15:53.474 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 33/34 2024-09-21T08:15:53.474 INFO:teuthology.orchestra.run.smithi137.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T08:15:53.567 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:15:53.567 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:15:54.100 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:15:54.169 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T08:15:54.208 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T08:15:54.236 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/4 2024-09-21T08:15:54.630 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 5/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 6/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 7/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x 8/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86 9/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 10/34 2024-09-21T08:15:54.631 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 11/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 12/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 13/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 14/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 15/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 16/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 17/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 18/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 19/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 20/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 21/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 22/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : gperftools-libs-2.9.1-3.el9.x86_64 23/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 24/34 2024-09-21T08:15:54.632 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 25/34 2024-09-21T08:15:54.633 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 26/34 2024-09-21T08:15:54.633 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 27/34 2024-09-21T08:15:54.633 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 28/34 2024-09-21T08:15:54.633 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : re2-1:20211101-20.el9.x86_64 29/34 2024-09-21T08:15:54.633 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 30/34 2024-09-21T08:15:54.633 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 31/34 2024-09-21T08:15:54.633 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 32/34 2024-09-21T08:15:54.633 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/34 2024-09-21T08:15:55.784 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 34/34 2024-09-21T08:15:55.784 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:55.784 INFO:teuthology.orchestra.run.smithi137.stdout:Upgraded: 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-radosgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-selinux-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: gperftools-libs-2.9.1-3.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-9.0.0-11.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: liboath-2.6.7-2.el9.x86_64 2024-09-21T08:15:55.785 INFO:teuthology.orchestra.run.smithi137.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: libradosstriper1-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: librgw2-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: libunwind-1.6.2-1.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-argparse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ceph-common-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.786 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.787 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:15:55.787 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-24.02.0-1.el9.x86_64 2024-09-21T08:15:55.787 INFO:teuthology.orchestra.run.smithi137.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-09-21T08:15:55.787 INFO:teuthology.orchestra.run.smithi137.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-09-21T08:15:55.787 INFO:teuthology.orchestra.run.smithi137.stdout: re2-1:20211101-20.el9.x86_64 2024-09-21T08:15:55.787 INFO:teuthology.orchestra.run.smithi137.stdout: thrift-0.15.0-2.el9.x86_64 2024-09-21T08:15:55.787 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:55.787 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:15:56.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-test 2024-09-21T08:15:56.726 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:27 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:15:56.862 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 49 M 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 163 k 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:15:56.863 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:15:56.864 INFO:teuthology.orchestra.run.smithi137.stdout:Install 4 Packages 2024-09-21T08:15:56.864 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:15:56.864 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 49 M 2024-09-21T08:15:56.864 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 209 M 2024-09-21T08:15:56.864 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:15:57.339 INFO:teuthology.orchestra.run.smithi137.stdout:(1/4): libcephsqlite-19.1.1-367.g9e2ac70b.el9.x 746 kB/s | 163 kB 00:00 2024-09-21T08:15:57.964 INFO:teuthology.orchestra.run.smithi137.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 360 kB/s | 304 kB 00:00 2024-09-21T08:15:57.998 INFO:teuthology.orchestra.run.smithi137.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 97 kB/s | 64 kB 00:00 2024-09-21T08:15:58.648 INFO:teuthology.orchestra.run.smithi137.stdout:(4/4): ceph-test-19.1.1-367.g9e2ac70b.el9.x86_6 32 MB/s | 49 MB 00:01 2024-09-21T08:15:58.650 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:15:58.651 INFO:teuthology.orchestra.run.smithi137.stdout:Total 28 MB/s | 49 MB 00:01 2024-09-21T08:15:58.678 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:15:58.699 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:15:58.699 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:15:58.837 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/4 2024-09-21T08:15:58.927 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/4 2024-09-21T08:15:59.081 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:15:59.081 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:15:59.633 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:15:59.700 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-09-21T08:15:59.747 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-09-21T08:15:59.774 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/4 2024-09-21T08:16:01.171 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/4 2024-09-21T08:16:01.171 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/4 2024-09-21T08:16:01.171 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/4 2024-09-21T08:16:01.171 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T08:16:01.570 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T08:16:01.570 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:01.570 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:01.570 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:01.571 INFO:teuthology.orchestra.run.smithi165.stdout: libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:01.571 INFO:teuthology.orchestra.run.smithi165.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T08:16:01.571 INFO:teuthology.orchestra.run.smithi165.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T08:16:01.571 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:01.571 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:01.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph 2024-09-21T08:16:02.371 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:38 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:02.514 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:02.517 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:16:02.517 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:02.517 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: ceph x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 6.5 k 2024-09-21T08:16:02.517 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-21T08:16:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 2.4 M 2024-09-21T08:16:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 1.5 M 2024-09-21T08:16:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 249 k 2024-09-21T08:16:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 4.7 M 2024-09-21T08:16:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 17 M 2024-09-21T08:16:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T08:16:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T08:16:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T08:16:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T08:16:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T08:16:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T08:16:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T08:16:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T08:16:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T08:16:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T08:16:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T08:16:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T08:16:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T08:16:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T08:16:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T08:16:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T08:16:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T08:16:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T08:16:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T08:16:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T08:16:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T08:16:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T08:16:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T08:16:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T08:16:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T08:16:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T08:16:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T08:16:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T08:16:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T08:16:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T08:16:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T08:16:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:02.523 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:02.524 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:02.524 INFO:teuthology.orchestra.run.smithi165.stdout:Install 37 Packages 2024-09-21T08:16:02.524 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:02.525 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 30 M 2024-09-21T08:16:02.525 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 108 M 2024-09-21T08:16:02.525 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:03.352 INFO:teuthology.orchestra.run.smithi165.stdout:(1/37): ceph-19.1.1-367.g9e2ac70b.el9.x86_64.rp 56 kB/s | 6.5 kB 00:00 2024-09-21T08:16:03.543 INFO:teuthology.orchestra.run.smithi165.stdout:(2/37): ceph-mgr-19.1.1-367.g9e2ac70b.el9.x86_6 4.9 MB/s | 1.5 MB 00:00 2024-09-21T08:16:03.594 INFO:teuthology.orchestra.run.smithi165.stdout:(3/37): ceph-mds-19.1.1-367.g9e2ac70b.el9.x86_6 6.6 MB/s | 2.4 MB 00:00 2024-09-21T08:16:03.669 INFO:teuthology.orchestra.run.smithi165.stdout:(4/37): ceph-mgr-modules-core-19.1.1-367.g9e2ac 3.3 MB/s | 249 kB 00:00 2024-09-21T08:16:03.811 INFO:teuthology.orchestra.run.smithi165.stdout:(5/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.7 MB/s | 253 kB 00:00 2024-09-21T08:16:03.903 INFO:teuthology.orchestra.run.smithi165.stdout:(6/37): ceph-mon-19.1.1-367.g9e2ac70b.el9.x86_6 8.5 MB/s | 4.7 MB 00:00 2024-09-21T08:16:03.945 INFO:teuthology.orchestra.run.smithi165.stdout:(7/37): python3-cryptography-36.0.1-4.el9.x86_6 9.2 MB/s | 1.2 MB 00:00 2024-09-21T08:16:04.129 INFO:teuthology.orchestra.run.smithi165.stdout:(8/37): ceph-osd-19.1.1-367.g9e2ac70b.el9.x86_6 29 MB/s | 17 MB 00:00 2024-09-21T08:16:04.154 INFO:teuthology.orchestra.run.smithi165.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 648 kB/s | 135 kB 00:00 2024-09-21T08:16:04.179 INFO:teuthology.orchestra.run.smithi165.stdout:(10/37): python3-ply-3.11-14.el9.noarch.rpm 386 kB/s | 106 kB 00:00 2024-09-21T08:16:04.221 INFO:teuthology.orchestra.run.smithi165.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 3.1 MB/s | 215 kB 00:00 2024-09-21T08:16:04.279 INFO:teuthology.orchestra.run.smithi165.stdout:(12/37): python3-requests-2.25.1-8.el9.noarch.r 829 kB/s | 125 kB 00:00 2024-09-21T08:16:04.355 INFO:teuthology.orchestra.run.smithi165.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 1.0 MB/s | 188 kB 00:00 2024-09-21T08:16:04.380 INFO:teuthology.orchestra.run.smithi165.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 346 kB/s | 35 kB 00:00 2024-09-21T08:16:04.383 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/4 2024-09-21T08:16:04.405 INFO:teuthology.orchestra.run.smithi165.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.5 MB/s | 77 kB 00:00 2024-09-21T08:16:04.431 INFO:teuthology.orchestra.run.smithi165.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 822 kB/s | 172 kB 00:00 2024-09-21T08:16:04.456 INFO:teuthology.orchestra.run.smithi165.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 674 kB/s | 51 kB 00:00 2024-09-21T08:16:04.465 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/4 2024-09-21T08:16:04.481 INFO:teuthology.orchestra.run.smithi165.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 552 kB/s | 42 kB 00:00 2024-09-21T08:16:04.523 INFO:teuthology.orchestra.run.smithi165.stdout:(19/37): luarocks-3.9.2-5.el9.noarch.rpm 3.6 MB/s | 151 kB 00:00 2024-09-21T08:16:04.548 INFO:teuthology.orchestra.run.smithi165.stdout:(20/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 7.3 MB/s | 172 kB 00:00 2024-09-21T08:16:04.574 INFO:teuthology.orchestra.run.smithi165.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 472 kB/s | 43 kB 00:00 2024-09-21T08:16:04.599 INFO:teuthology.orchestra.run.smithi165.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 429 kB/s | 11 kB 00:00 2024-09-21T08:16:04.624 INFO:teuthology.orchestra.run.smithi165.stdout:(23/37): lua-devel-5.4.4-4.el9.x86_64.rpm 115 kB/s | 22 kB 00:00 2024-09-21T08:16:04.658 INFO:teuthology.orchestra.run.smithi165.stdout:(24/37): python3-jaraco-classes-3.2.1-5.el9.noa 302 kB/s | 18 kB 00:00 2024-09-21T08:16:04.708 INFO:teuthology.orchestra.run.smithi165.stdout:(25/37): python3-jaraco-functools-3.5.0-2.el9.n 401 kB/s | 19 kB 00:00 2024-09-21T08:16:04.733 INFO:teuthology.orchestra.run.smithi165.stdout:(26/37): python3-jaraco-collections-3.0.0-8.el9 212 kB/s | 23 kB 00:00 2024-09-21T08:16:04.759 INFO:teuthology.orchestra.run.smithi165.stdout:(27/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.7 MB/s | 358 kB 00:00 2024-09-21T08:16:04.784 INFO:teuthology.orchestra.run.smithi165.stdout:(28/37): python3-jaraco-text-3.2.0-6.el9.noarch 259 kB/s | 20 kB 00:00 2024-09-21T08:16:04.810 INFO:teuthology.orchestra.run.smithi165.stdout:(29/37): python3-logutils-0.3.5-21.el9.noarch.r 610 kB/s | 46 kB 00:00 2024-09-21T08:16:04.835 INFO:teuthology.orchestra.run.smithi165.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-21T08:16:04.860 INFO:teuthology.orchestra.run.smithi165.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-21T08:16:04.886 INFO:teuthology.orchestra.run.smithi165.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.7 MB/s | 90 kB 00:00 2024-09-21T08:16:04.911 INFO:teuthology.orchestra.run.smithi165.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 710 kB/s | 36 kB 00:00 2024-09-21T08:16:04.936 INFO:teuthology.orchestra.run.smithi165.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-21T08:16:04.961 INFO:teuthology.orchestra.run.smithi165.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 795 kB/s | 20 kB 00:00 2024-09-21T08:16:04.986 INFO:teuthology.orchestra.run.smithi165.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-09-21T08:16:05.020 INFO:teuthology.orchestra.run.smithi165.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.8 MB/s | 427 kB 00:00 2024-09-21T08:16:05.023 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:05.023 INFO:teuthology.orchestra.run.smithi165.stdout:Total 12 MB/s | 30 MB 00:02 2024-09-21T08:16:05.167 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:05.207 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:05.208 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:05.658 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:05.658 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:06.865 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/4 2024-09-21T08:16:06.865 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/4 2024-09-21T08:16:06.865 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/4 2024-09-21T08:16:06.865 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-09-21T08:16:07.015 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:07.073 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T08:16:07.109 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T08:16:07.150 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T08:16:07.223 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T08:16:07.266 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T08:16:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-09-21T08:16:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:07.294 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:16:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-test-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: libcephsqlite-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: socat-1.7.4.1-6.el9.x86_64 2024-09-21T08:16:07.294 INFO:teuthology.orchestra.run.smithi137.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-09-21T08:16:07.295 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:07.295 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:16:07.298 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T08:16:07.391 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T08:16:07.461 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T08:16:07.501 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T08:16:07.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph 2024-09-21T08:16:07.545 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T08:16:07.575 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T08:16:07.609 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T08:16:07.642 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T08:16:07.683 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T08:16:07.718 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T08:16:07.763 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T08:16:07.809 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T08:16:07.841 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T08:16:07.915 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T08:16:08.079 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T08:16:08.119 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:39 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:16:08.158 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T08:16:08.211 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T08:16:08.250 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T08:16:08.263 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:16:08.265 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:16:08.265 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:08.265 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:16:08.265 INFO:teuthology.orchestra.run.smithi137.stdout: ceph x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 6.5 k 2024-09-21T08:16:08.265 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 2.4 M 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 1.5 M 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 249 k 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 4.7 M 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 17 M 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks noarch 3.9.2-5.el9 epel 151 k 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-09-21T08:16:08.266 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-09-21T08:16:08.267 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3 noarch 1.26.5-6.el9 baseos 215 k 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob noarch 1.8.8-2.el9 epel 230 k 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout:Install 37 Packages 2024-09-21T08:16:08.268 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:08.269 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 30 M 2024-09-21T08:16:08.269 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 108 M 2024-09-21T08:16:08.269 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:16:08.316 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T08:16:08.371 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T08:16:08.561 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T08:16:08.612 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T08:16:08.683 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T08:16:08.766 INFO:teuthology.orchestra.run.smithi137.stdout:(1/37): ceph-19.1.1-367.g9e2ac70b.el9.x86_64.rp 51 kB/s | 6.5 kB 00:00 2024-09-21T08:16:08.772 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T08:16:08.811 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T08:16:08.924 INFO:teuthology.orchestra.run.smithi137.stdout:(2/37): ceph-mgr-19.1.1-367.g9e2ac70b.el9.x86_6 5.3 MB/s | 1.5 MB 00:00 2024-09-21T08:16:08.928 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T08:16:09.008 INFO:teuthology.orchestra.run.smithi137.stdout:(3/37): ceph-mds-19.1.1-367.g9e2ac70b.el9.x86_6 6.4 MB/s | 2.4 MB 00:00 2024-09-21T08:16:09.087 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.n 32/37 2024-09-21T08:16:09.109 INFO:teuthology.orchestra.run.smithi137.stdout:(4/37): ceph-mgr-modules-core-19.1.1-367.g9e2ac 2.4 MB/s | 249 kB 00:00 2024-09-21T08:16:09.114 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/37 2024-09-21T08:16:09.140 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/37 2024-09-21T08:16:09.140 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:09.140 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T08:16:09.140 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T08:16:09.140 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T08:16:09.140 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:09.175 INFO:teuthology.orchestra.run.smithi137.stdout:(5/37): ceph-mon-19.1.1-367.g9e2ac70b.el9.x86_6 12 MB/s | 4.7 MB 00:00 2024-09-21T08:16:09.284 INFO:teuthology.orchestra.run.smithi137.stdout:(6/37): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.4 MB/s | 253 kB 00:00 2024-09-21T08:16:09.326 INFO:teuthology.orchestra.run.smithi137.stdout:(7/37): python3-ply-3.11-14.el9.noarch.rpm 2.5 MB/s | 106 kB 00:00 2024-09-21T08:16:09.360 INFO:teuthology.orchestra.run.smithi137.stdout:(8/37): python3-cryptography-36.0.1-4.el9.x86_6 6.7 MB/s | 1.2 MB 00:00 2024-09-21T08:16:09.401 INFO:teuthology.orchestra.run.smithi137.stdout:(9/37): python3-pycparser-2.20-6.el9.noarch.rpm 1.8 MB/s | 135 kB 00:00 2024-09-21T08:16:09.427 INFO:teuthology.orchestra.run.smithi137.stdout:(10/37): python3-requests-2.25.1-8.el9.noarch.r 1.8 MB/s | 125 kB 00:00 2024-09-21T08:16:09.452 INFO:teuthology.orchestra.run.smithi137.stdout:(11/37): python3-urllib3-1.26.5-6.el9.noarch.rp 4.2 MB/s | 215 kB 00:00 2024-09-21T08:16:09.652 INFO:teuthology.orchestra.run.smithi137.stdout:(12/37): ceph-osd-19.1.1-367.g9e2ac70b.el9.x86_ 23 MB/s | 17 MB 00:00 2024-09-21T08:16:09.727 INFO:teuthology.orchestra.run.smithi137.stdout:(13/37): lua-5.4.4-4.el9.x86_64.rpm 627 kB/s | 188 kB 00:00 2024-09-21T08:16:09.753 INFO:teuthology.orchestra.run.smithi137.stdout:(14/37): python3-markupsafe-1.1.1-12.el9.x86_64 348 kB/s | 35 kB 00:00 2024-09-21T08:16:09.787 INFO:teuthology.orchestra.run.smithi137.stdout:(15/37): python3-packaging-20.9-5.el9.noarch.rp 1.3 MB/s | 77 kB 00:00 2024-09-21T08:16:09.812 INFO:teuthology.orchestra.run.smithi137.stdout:(16/37): python3-mako-1.1.4-6.el9.noarch.rpm 478 kB/s | 172 kB 00:00 2024-09-21T08:16:09.837 INFO:teuthology.orchestra.run.smithi137.stdout:(17/37): python3-pytz-2021.1-5.el9.noarch.rpm 605 kB/s | 51 kB 00:00 2024-09-21T08:16:09.862 INFO:teuthology.orchestra.run.smithi137.stdout:(18/37): python3-toml-0.10.2-6.el9.noarch.rpm 551 kB/s | 42 kB 00:00 2024-09-21T08:16:09.887 INFO:teuthology.orchestra.run.smithi137.stdout:(19/37): lua-devel-5.4.4-4.el9.x86_64.rpm 296 kB/s | 22 kB 00:00 2024-09-21T08:16:09.938 INFO:teuthology.orchestra.run.smithi137.stdout:(20/37): luarocks-3.9.2-5.el9.noarch.rpm 1.5 MB/s | 151 kB 00:00 2024-09-21T08:16:09.963 INFO:teuthology.orchestra.run.smithi137.stdout:(21/37): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 432 kB/s | 43 kB 00:00 2024-09-21T08:16:09.989 INFO:teuthology.orchestra.run.smithi137.stdout:(22/37): python3-jaraco-8.2.1-3.el9.noarch.rpm 431 kB/s | 11 kB 00:00 2024-09-21T08:16:10.013 INFO:teuthology.orchestra.run.smithi137.stdout:(23/37): python3-jaraco-classes-3.2.1-5.el9.noa 714 kB/s | 18 kB 00:00 2024-09-21T08:16:10.039 INFO:teuthology.orchestra.run.smithi137.stdout:(24/37): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.2 MB/s | 172 kB 00:00 2024-09-21T08:16:10.066 INFO:teuthology.orchestra.run.smithi137.stdout:(25/37): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-09-21T08:16:10.090 INFO:teuthology.orchestra.run.smithi137.stdout:(26/37): python3-jaraco-functools-3.5.0-2.el9.n 388 kB/s | 19 kB 00:00 2024-09-21T08:16:10.115 INFO:teuthology.orchestra.run.smithi137.stdout:(27/37): python3-jaraco-text-3.2.0-6.el9.noarch 391 kB/s | 20 kB 00:00 2024-09-21T08:16:10.140 INFO:teuthology.orchestra.run.smithi137.stdout:(28/37): python3-logutils-0.3.5-21.el9.noarch.r 925 kB/s | 46 kB 00:00 2024-09-21T08:16:10.165 INFO:teuthology.orchestra.run.smithi137.stdout:(29/37): python3-cherrypy-18.6.1-2.el9.noarch.r 1.6 MB/s | 358 kB 00:00 2024-09-21T08:16:10.191 INFO:teuthology.orchestra.run.smithi137.stdout:(30/37): python3-more-itertools-8.12.0-2.el9.no 1.0 MB/s | 79 kB 00:00 2024-09-21T08:16:10.216 INFO:teuthology.orchestra.run.smithi137.stdout:(31/37): python3-portend-3.1.0-2.el9.noarch.rpm 327 kB/s | 16 kB 00:00 2024-09-21T08:16:10.241 INFO:teuthology.orchestra.run.smithi137.stdout:(32/37): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-09-21T08:16:10.266 INFO:teuthology.orchestra.run.smithi137.stdout:(33/37): python3-tempora-5.0.0-2.el9.noarch.rpm 713 kB/s | 36 kB 00:00 2024-09-21T08:16:10.292 INFO:teuthology.orchestra.run.smithi137.stdout:(34/37): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-09-21T08:16:10.317 INFO:teuthology.orchestra.run.smithi137.stdout:(35/37): python3-zc-lockfile-2.0-10.el9.noarch. 804 kB/s | 20 kB 00:00 2024-09-21T08:16:10.342 INFO:teuthology.orchestra.run.smithi137.stdout:(36/37): python3-webob-1.8.8-2.el9.noarch.rpm 3.0 MB/s | 230 kB 00:00 2024-09-21T08:16:10.367 INFO:teuthology.orchestra.run.smithi137.stdout:(37/37): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.2 MB/s | 427 kB 00:00 2024-09-21T08:16:10.372 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:10.372 INFO:teuthology.orchestra.run.smithi137.stdout:Total 14 MB/s | 30 MB 00:02 2024-09-21T08:16:10.529 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:16:10.570 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:16:10.571 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:16:10.598 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 34/37 2024-09-21T08:16:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 34/37 2024-09-21T08:16:10.625 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:10.625 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T08:16:10.625 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T08:16:10.625 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T08:16:10.625 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:11.040 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:16:11.041 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:16:11.074 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 35/37 2024-09-21T08:16:11.097 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 35/37 2024-09-21T08:16:11.097 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:11.097 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T08:16:11.098 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T08:16:11.098 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T08:16:11.098 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:11.324 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 36/37 2024-09-21T08:16:11.347 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 36/37 2024-09-21T08:16:11.348 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:11.348 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T08:16:11.348 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T08:16:11.348 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T08:16:11.348 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:11.412 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 37/37 2024-09-21T08:16:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:16:12.428 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/37 2024-09-21T08:16:12.475 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/37 2024-09-21T08:16:12.505 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-5.4.4-4.el9.x86_64 3/37 2024-09-21T08:16:12.577 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 4/37 2024-09-21T08:16:12.638 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : luarocks-3.9.2-5.el9.noarch 5/37 2024-09-21T08:16:12.671 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 6/37 2024-09-21T08:16:12.763 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 7/37 2024-09-21T08:16:12.841 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 8/37 2024-09-21T08:16:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-webob-1.8.8-2.el9.noarch 9/37 2024-09-21T08:16:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 10/37 2024-09-21T08:16:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 11/37 2024-09-21T08:16:12.981 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 12/37 2024-09-21T08:16:13.014 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 13/37 2024-09-21T08:16:13.055 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 14/37 2024-09-21T08:16:13.098 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 15/37 2024-09-21T08:16:13.140 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 16/37 2024-09-21T08:16:13.189 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 17/37 2024-09-21T08:16:13.221 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-packaging-20.9-5.el9.noarch 18/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 37/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 5/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.n 6/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T08:16:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T08:16:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T08:16:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T08:16:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T08:16:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T08:16:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T08:16:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T08:16:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T08:16:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T08:16:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T08:16:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T08:16:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T08:16:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T08:16:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T08:16:13.294 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 19/37 2024-09-21T08:16:13.431 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 20/37 2024-09-21T08:16:13.528 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 21/37 2024-09-21T08:16:13.590 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-urllib3-1.26.5-6.el9.noarch 22/37 2024-09-21T08:16:13.629 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 23/37 2024-09-21T08:16:13.686 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-ply-3.11-14.el9.noarch 24/37 2024-09-21T08:16:13.741 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 25/37 2024-09-21T08:16:13.934 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 26/37 2024-09-21T08:16:13.992 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 27/37 2024-09-21T08:16:14.068 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 28/37 2024-09-21T08:16:14.157 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 29/37 2024-09-21T08:16:14.199 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 30/37 2024-09-21T08:16:14.310 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T08:16:14.310 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:14.310 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:14.310 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:14.310 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:14.310 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T08:16:14.311 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T08:16:14.312 INFO:teuthology.orchestra.run.smithi165.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T08:16:14.313 INFO:teuthology.orchestra.run.smithi165.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T08:16:14.313 INFO:teuthology.orchestra.run.smithi165.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T08:16:14.313 INFO:teuthology.orchestra.run.smithi165.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T08:16:14.313 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:14.313 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:14.314 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 31/37 2024-09-21T08:16:14.470 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.n 32/37 2024-09-21T08:16:14.501 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/37 2024-09-21T08:16:14.527 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 33/37 2024-09-21T08:16:14.527 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:14.527 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-09-21T08:16:14.527 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T08:16:14.528 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-09-21T08:16:14.528 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:14.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-base 2024-09-21T08:16:15.293 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:15.383 INFO:teuthology.orchestra.run.smithi165.stdout:Package ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:16:15.425 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:15.426 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-21T08:16:15.427 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:15.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install cephadm 2024-09-21T08:16:15.960 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 34/37 2024-09-21T08:16:15.988 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 34/37 2024-09-21T08:16:15.988 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:15.988 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-09-21T08:16:15.988 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T08:16:15.988 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-09-21T08:16:15.989 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:16.085 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:16.216 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:16.217 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:16.217 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:16:16.217 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:16.217 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:16.217 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 765 k 2024-09-21T08:16:16.217 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:16.217 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:16.217 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:16.218 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-21T08:16:16.218 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:16.218 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 765 k 2024-09-21T08:16:16.218 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 771 k 2024-09-21T08:16:16.218 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:16.428 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 35/37 2024-09-21T08:16:16.452 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 35/37 2024-09-21T08:16:16.452 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:16.452 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-09-21T08:16:16.452 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T08:16:16.452 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-09-21T08:16:16.452 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:16.457 INFO:teuthology.orchestra.run.smithi165.stdout:cephadm-19.1.1-367.g9e2ac70b.el9.noarch.rpm 3.1 MB/s | 765 kB 00:00 2024-09-21T08:16:16.459 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:16.459 INFO:teuthology.orchestra.run.smithi165.stdout:Total 3.1 MB/s | 765 kB 00:00 2024-09-21T08:16:16.459 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:16.464 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:16.464 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:16.470 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:16.470 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:16.547 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:16.686 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 36/37 2024-09-21T08:16:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 36/37 2024-09-21T08:16:16.711 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:16.711 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-09-21T08:16:16.711 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T08:16:16.711 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-09-21T08:16:16.711 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:16.775 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 37/37 2024-09-21T08:16:16.990 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:17.079 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:17.444 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:17.702 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:17.703 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:17.703 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:17.703 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:17.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T08:16:18.504 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 37/37 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/37 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/37 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 3/37 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 4/37 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 5/37 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.n 6/37 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/37 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/37 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/37 2024-09-21T08:16:18.613 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-urllib3-1.26.5-6.el9.noarch 12/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-5.4.4-4.el9.x86_64 13/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 14/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 15/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 16/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 17/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 18/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 19/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : luarocks-3.9.2-5.el9.noarch 20/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 21/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 22/37 2024-09-21T08:16:18.614 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 23/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 24/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 25/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 26/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 27/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 28/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 29/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 30/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 31/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 32/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 33/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 34/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-webob-1.8.8-2.el9.noarch 35/37 2024-09-21T08:16:18.615 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 36/37 2024-09-21T08:16:18.639 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:18.640 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-21T08:16:18.640 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:18.640 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 145 k 2024-09-21T08:16:18.640 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:18.640 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:18.640 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:18.641 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-21T08:16:18.641 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:18.641 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 145 k 2024-09-21T08:16:18.641 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 436 k 2024-09-21T08:16:18.641 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:18.858 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-immutable-object-cache-19.1.1-367.g9e2ac70 677 kB/s | 145 kB 00:00 2024-09-21T08:16:18.858 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:18.858 INFO:teuthology.orchestra.run.smithi165.stdout:Total 671 kB/s | 145 kB 00:00 2024-09-21T08:16:18.859 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:18.864 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:18.865 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:18.927 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:18.927 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:19.081 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:19.174 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-21T08:16:19.199 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-21T08:16:19.199 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:19.199 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T08:16:19.199 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:19.574 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 37/37 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mds-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-modules-core-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mon-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-osd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: lua-5.4.4-4.el9.x86_64 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: luarocks-3.9.2-5.el9.noarch 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-09-21T08:16:19.575 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-mako-1.1.4-6.el9.noarch 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-09-21T08:16:19.576 INFO:teuthology.orchestra.run.smithi137.stdout: python3-packaging-20.9-5.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-ply-3.11-14.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-portend-3.1.0-2.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pycparser-2.20-6.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pytz-2021.1-5.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-2.25.1-8.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-toml-0.10.2-6.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-urllib3-1.26.5-6.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-webob-1.8.8-2.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-09-21T08:16:19.577 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:19.578 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:16:19.947 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-base 2024-09-21T08:16:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-21T08:16:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:20.158 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:20.158 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:20.158 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:20.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr 2024-09-21T08:16:20.531 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:51 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:16:20.622 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-base-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:16:20.664 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:16:20.664 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-21T08:16:20.664 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:16:20.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install cephadm 2024-09-21T08:16:20.956 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:56 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:21.047 INFO:teuthology.orchestra.run.smithi165.stdout:Package ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:16:21.091 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:21.092 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-21T08:16:21.092 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:21.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T08:16:21.452 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:52 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:16:21.583 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:16:21.583 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:21.583 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:16:21.583 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:21.583 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:16:21.584 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 765 k 2024-09-21T08:16:21.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:21.584 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:16:21.584 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:21.584 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-21T08:16:21.584 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:21.584 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 765 k 2024-09-21T08:16:21.584 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 771 k 2024-09-21T08:16:21.584 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:16:21.760 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:21.845 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm-19.1.1-367.g9e2ac70b.el9.noarch.rpm 2.9 MB/s | 765 kB 00:00 2024-09-21T08:16:21.846 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:21.846 INFO:teuthology.orchestra.run.smithi137.stdout:Total 2.8 MB/s | 765 kB 00:00 2024-09-21T08:16:21.847 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:16:21.852 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:16:21.852 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:16:21.858 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:16:21.858 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:16:21.893 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:21.894 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:21.894 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:16:21.894 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:21.894 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:21.894 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 3.6 M 2024-09-21T08:16:21.894 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-21T08:16:21.894 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T08:16:21.894 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 32 k 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 16 k 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout:Install 12 Packages 2024-09-21T08:16:21.895 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:21.896 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 8.7 M 2024-09-21T08:16:21.896 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 105 M 2024-09-21T08:16:21.896 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:21.934 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:16:22.386 INFO:teuthology.orchestra.run.smithi165.stdout:(1/12): ceph-prometheus-alerts-19.1.1-367.g9e2a 103 kB/s | 16 kB 00:00 2024-09-21T08:16:22.403 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:22.403 INFO:teuthology.orchestra.run.smithi165.stdout:(2/12): ceph-grafana-dashboards-19.1.1-367.g9e2 179 kB/s | 32 kB 00:00 2024-09-21T08:16:22.503 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:22.578 INFO:teuthology.orchestra.run.smithi165.stdout:(3/12): ceph-mgr-dashboard-19.1.1-367.g9e2ac70b 10 MB/s | 3.6 MB 00:00 2024-09-21T08:16:22.603 INFO:teuthology.orchestra.run.smithi165.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.3 MB/s | 269 kB 00:00 2024-09-21T08:16:22.646 INFO:teuthology.orchestra.run.smithi165.stdout:(5/12): protobuf-3.14.0-13.el9.x86_64.rpm 3.9 MB/s | 1.0 MB 00:00 2024-09-21T08:16:22.688 INFO:teuthology.orchestra.run.smithi165.stdout:(6/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 6.4 MB/s | 548 kB 00:00 2024-09-21T08:16:22.704 INFO:teuthology.orchestra.run.smithi165.stdout:(7/12): grpc-data-1.46.7-10.el9.noarch.rpm 342 kB/s | 19 kB 00:00 2024-09-21T08:16:22.763 INFO:teuthology.orchestra.run.smithi165.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 2.4 MB/s | 144 kB 00:00 2024-09-21T08:16:22.780 INFO:teuthology.orchestra.run.smithi165.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.8 MB/s | 31 kB 00:00 2024-09-21T08:16:22.805 INFO:teuthology.orchestra.run.smithi165.stdout:(10/12): python3-routes-2.5.1-5.el9.noarch.rpm 7.4 MB/s | 188 kB 00:00 2024-09-21T08:16:22.830 INFO:teuthology.orchestra.run.smithi165.stdout:(11/12): protobuf-compiler-3.14.0-13.el9.x86_64 3.3 MB/s | 863 kB 00:00 2024-09-21T08:16:22.859 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:22.873 INFO:teuthology.orchestra.run.smithi165.stdout:(12/12): python3-grpcio-1.46.7-10.el9.x86_64.rp 11 MB/s | 2.0 MB 00:00 2024-09-21T08:16:22.876 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:22.876 INFO:teuthology.orchestra.run.smithi165.stdout:Total 8.8 MB/s | 8.7 MB 00:00 2024-09-21T08:16:22.925 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:22.942 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:22.942 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:23.100 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:16:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:23.100 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:23.100 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:16:23.158 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:23.159 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:23.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-immutable-object-cache 2024-09-21T08:16:23.699 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:23.782 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T08:16:23.817 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T08:16:23.864 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T08:16:23.894 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T08:16:23.898 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:54 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:16:24.001 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T08:16:24.030 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:16:24.031 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:24.031 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-21T08:16:24.031 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:24.031 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:16:24.031 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 145 k 2024-09-21T08:16:24.031 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:24.031 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:16:24.031 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:24.031 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-21T08:16:24.032 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:24.032 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 145 k 2024-09-21T08:16:24.032 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 436 k 2024-09-21T08:16:24.032 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:16:24.135 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T08:16:24.221 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T08:16:24.235 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-immutable-object-cache-19.1.1-367.g9e2ac70 721 kB/s | 145 kB 00:00 2024-09-21T08:16:24.235 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:24.236 INFO:teuthology.orchestra.run.smithi137.stdout:Total 713 kB/s | 145 kB 00:00 2024-09-21T08:16:24.236 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:16:24.243 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:16:24.243 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:16:24.266 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T08:16:24.294 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T08:16:24.305 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:16:24.305 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:16:24.326 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9. 10/12 2024-09-21T08:16:24.455 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:16:24.519 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-21T08:16:24.544 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-21T08:16:24.544 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:24.544 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-09-21T08:16:24.544 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:25.384 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9 11/12 2024-09-21T08:16:25.419 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 12/12 2024-09-21T08:16:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.e 1/1 2024-09-21T08:16:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:25.491 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:16:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-immutable-object-cache-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:25.492 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:16:25.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr 2024-09-21T08:16:26.290 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:57 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:16:26.383 INFO:teuthology.orchestra.run.smithi137.stdout:Package ceph-mgr-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:16:26.425 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:16:26.426 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-21T08:16:26.426 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:16:26.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-dashboard 2024-09-21T08:16:26.690 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 12/12 2024-09-21T08:16:26.690 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9 1/12 2024-09-21T08:16:26.690 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 2/12 2024-09-21T08:16:26.690 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9. 3/12 2024-09-21T08:16:26.690 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T08:16:26.690 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T08:16:26.690 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T08:16:26.691 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T08:16:26.691 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T08:16:26.691 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T08:16:26.691 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T08:16:26.691 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T08:16:27.008 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T08:16:27.008 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:27.008 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:27.008 INFO:teuthology.orchestra.run.smithi165.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T08:16:27.008 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:27.008 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:27.008 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:27.008 INFO:teuthology.orchestra.run.smithi165.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T08:16:27.008 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T08:16:27.008 INFO:teuthology.orchestra.run.smithi165.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T08:16:27.009 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T08:16:27.009 INFO:teuthology.orchestra.run.smithi165.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T08:16:27.009 INFO:teuthology.orchestra.run.smithi165.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T08:16:27.009 INFO:teuthology.orchestra.run.smithi165.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T08:16:27.009 INFO:teuthology.orchestra.run.smithi165.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T08:16:27.009 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:27.009 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:27.073 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:00:58 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:16:27.205 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:16:27.206 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:27.206 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:16:27.206 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 3.6 M 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 32 k 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 16 k 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-09-21T08:16:27.207 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-09-21T08:16:27.208 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:27.208 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:16:27.208 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:27.208 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-09-21T08:16:27.208 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:27.208 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 8.7 M 2024-09-21T08:16:27.208 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 105 M 2024-09-21T08:16:27.209 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:16:27.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T08:16:27.463 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): ceph-grafana-dashboards-19.1.1-367.g9e2 211 kB/s | 32 kB 00:00 2024-09-21T08:16:27.488 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): ceph-prometheus-alerts-19.1.1-367.g9e2a 94 kB/s | 16 kB 00:00 2024-09-21T08:16:27.713 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): ceph-mgr-dashboard-19.1.1-367.g9e2ac70b 8.9 MB/s | 3.6 MB 00:00 2024-09-21T08:16:27.739 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): python3-protobuf-3.14.0-13.el9.noarch.r 1.1 MB/s | 269 kB 00:00 2024-09-21T08:16:27.789 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): abseil-cpp-20211102.0-3.el9.x86_64.rpm 11 MB/s | 548 kB 00:00 2024-09-21T08:16:27.814 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): grpc-data-1.46.7-10.el9.noarch.rpm 774 kB/s | 19 kB 00:00 2024-09-21T08:16:27.825 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:03 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:27.873 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): python3-grpcio-1.46.7-10.el9.x86_64.rpm 35 MB/s | 2.0 MB 00:00 2024-09-21T08:16:27.898 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): python3-grpcio-tools-1.46.7-10.el9.x86_ 5.7 MB/s | 144 kB 00:00 2024-09-21T08:16:27.923 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): python3-repoze-lru-0.7-16.el9.noarch.rp 1.2 MB/s | 31 kB 00:00 2024-09-21T08:16:27.949 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): protobuf-compiler-3.14.0-13.el9.x86_64 3.6 MB/s | 863 kB 00:00 2024-09-21T08:16:27.958 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:27.959 INFO:teuthology.orchestra.run.smithi165.stdout:====================================================================================== 2024-09-21T08:16:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:16:27.959 INFO:teuthology.orchestra.run.smithi165.stdout:====================================================================================== 2024-09-21T08:16:27.959 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:27.959 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 7.4 M 2024-09-21T08:16:27.959 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:27.960 INFO:teuthology.orchestra.run.smithi165.stdout:====================================================================================== 2024-09-21T08:16:27.961 INFO:teuthology.orchestra.run.smithi165.stdout:Install 12 Packages 2024-09-21T08:16:27.961 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:27.961 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 38 M 2024-09-21T08:16:27.961 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 212 M 2024-09-21T08:16:27.961 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:27.974 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-09-21T08:16:28.007 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): protobuf-3.14.0-13.el9.x86_64.rpm 1.9 MB/s | 1.0 MB 00:00 2024-09-21T08:16:28.010 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:28.010 INFO:teuthology.orchestra.run.smithi137.stdout:Total 11 MB/s | 8.7 MB 00:00 2024-09-21T08:16:28.058 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:16:28.075 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:16:28.075 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:16:28.292 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:16:28.293 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:16:28.314 INFO:teuthology.orchestra.run.smithi165.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.0 MB/s | 191 kB 00:00 2024-09-21T08:16:28.356 INFO:teuthology.orchestra.run.smithi165.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 3.4 MB/s | 801 kB 00:00 2024-09-21T08:16:28.407 INFO:teuthology.orchestra.run.smithi165.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 356 kB/s | 33 kB 00:00 2024-09-21T08:16:28.449 INFO:teuthology.orchestra.run.smithi165.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 437 kB/s | 18 kB 00:00 2024-09-21T08:16:28.507 INFO:teuthology.orchestra.run.smithi165.stdout:(5/12): openblas-0.3.26-2.el9.x86_64.rpm 658 kB/s | 38 kB 00:00 2024-09-21T08:16:28.650 INFO:teuthology.orchestra.run.smithi165.stdout:(6/12): ceph-mgr-diskprediction-local-19.1.1-36 14 MB/s | 7.4 MB 00:00 2024-09-21T08:16:28.700 INFO:teuthology.orchestra.run.smithi165.stdout:(7/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 8.7 MB/s | 3.0 MB 00:00 2024-09-21T08:16:28.816 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:16:28.875 INFO:teuthology.orchestra.run.smithi165.stdout:(8/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 5.1 MB 00:00 2024-09-21T08:16:28.883 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 1/12 2024-09-21T08:16:28.918 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 2/12 2024-09-21T08:16:28.942 INFO:teuthology.orchestra.run.smithi165.stdout:(9/12): openblas-openmp-0.3.26-2.el9.x86_64.rpm 11 MB/s | 4.9 MB 00:00 2024-09-21T08:16:28.964 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/12 2024-09-21T08:16:28.967 INFO:teuthology.orchestra.run.smithi165.stdout:(10/12): python3-devel-3.9.19-8.el9.x86_64.rpm 771 kB/s | 245 kB 00:00 2024-09-21T08:16:28.993 INFO:teuthology.orchestra.run.smithi165.stdout:(11/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.1 MB/s | 250 kB 00:00 2024-09-21T08:16:28.995 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/12 2024-09-21T08:16:29.092 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 5/12 2024-09-21T08:16:29.221 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 6/12 2024-09-21T08:16:29.301 INFO:teuthology.orchestra.run.smithi165.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 45 MB/s | 16 MB 00:00 2024-09-21T08:16:29.302 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:29.302 INFO:teuthology.orchestra.run.smithi165.stdout:Total 28 MB/s | 38 MB 00:01 2024-09-21T08:16:29.323 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 7/12 2024-09-21T08:16:29.367 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 8/12 2024-09-21T08:16:29.395 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 9/12 2024-09-21T08:16:29.427 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9. 10/12 2024-09-21T08:16:29.654 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:29.680 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:29.680 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:30.103 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:30.103 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:30.488 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9 11/12 2024-09-21T08:16:30.536 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 12/12 2024-09-21T08:16:30.870 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:30.929 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T08:16:30.984 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T08:16:31.090 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T08:16:31.134 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T08:16:31.321 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T08:16:31.358 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T08:16:31.435 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T08:16:31.805 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 12/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9 1/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noar 2/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9. 3/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 4/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 5/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 6/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 7/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 8/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 9/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 10/12 2024-09-21T08:16:31.806 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 11/12 2024-09-21T08:16:32.101 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T08:16:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 12/12 2024-09-21T08:16:32.141 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-grafana-dashboards-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-dashboard-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-prometheus-alerts-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: grpc-data-1.46.7-10.el9.noarch 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-3.14.0-13.el9.x86_64 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-09-21T08:16:32.142 INFO:teuthology.orchestra.run.smithi137.stdout: python3-routes-2.5.1-5.el9.noarch 2024-09-21T08:16:32.143 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:32.143 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:16:32.222 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T08:16:32.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-diskprediction-local 2024-09-21T08:16:32.958 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:03 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:16:33.089 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:16:33.090 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-09-21T08:16:33.090 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:16:33.090 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-09-21T08:16:33.090 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:16:33.090 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 7.4 M 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran x86_64 11.5.0-2.el9 baseos 801 k 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath x86_64 11.5.0-2.el9 baseos 191 k 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: openblas x86_64 0.3.26-2.el9 appstream 38 k 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp x86_64 0.3.26-2.el9 appstream 4.9 M 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel x86_64 3.9.19-8.el9 appstream 245 k 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout:====================================================================================== 2024-09-21T08:16:33.091 INFO:teuthology.orchestra.run.smithi137.stdout:Install 12 Packages 2024-09-21T08:16:33.092 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:33.092 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 38 M 2024-09-21T08:16:33.092 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 212 M 2024-09-21T08:16:33.092 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:16:33.398 INFO:teuthology.orchestra.run.smithi137.stdout:(1/12): libquadmath-11.5.0-2.el9.x86_64.rpm 1.4 MB/s | 191 kB 00:00 2024-09-21T08:16:33.440 INFO:teuthology.orchestra.run.smithi137.stdout:(2/12): libgfortran-11.5.0-2.el9.x86_64.rpm 4.6 MB/s | 801 kB 00:00 2024-09-21T08:16:33.467 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T08:16:33.615 INFO:teuthology.orchestra.run.smithi137.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 151 kB/s | 33 kB 00:00 2024-09-21T08:16:33.724 INFO:teuthology.orchestra.run.smithi137.stdout:(4/12): ceph-mgr-diskprediction-local-19.1.1-36 16 MB/s | 7.4 MB 00:00 2024-09-21T08:16:33.749 INFO:teuthology.orchestra.run.smithi137.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 136 kB/s | 18 kB 00:00 2024-09-21T08:16:33.933 INFO:teuthology.orchestra.run.smithi137.stdout:(6/12): openblas-0.3.26-2.el9.x86_64.rpm 185 kB/s | 38 kB 00:00 2024-09-21T08:16:34.233 INFO:teuthology.orchestra.run.smithi137.stdout:(7/12): python3-devel-3.9.19-8.el9.x86_64.rpm 816 kB/s | 245 kB 00:00 2024-09-21T08:16:34.536 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T08:16:34.568 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 12/12 2024-09-21T08:16:34.808 INFO:teuthology.orchestra.run.smithi137.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.2 MB/s | 3.0 MB 00:01 2024-09-21T08:16:34.942 INFO:teuthology.orchestra.run.smithi137.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.8 MB/s | 250 kB 00:00 2024-09-21T08:16:35.534 INFO:teuthology.orchestra.run.smithi137.stdout:(10/12): openblas-openmp-0.3.26-2.el9.x86_64.rp 2.8 MB/s | 4.9 MB 00:01 2024-09-21T08:16:35.976 INFO:teuthology.orchestra.run.smithi137.stdout:(11/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 2.9 MB/s | 5.1 MB 00:01 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 12/12 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 1/12 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T08:16:37.194 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T08:16:37.195 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T08:16:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T08:16:37.676 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:37.677 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:38.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-rook 2024-09-21T08:16:38.667 INFO:teuthology.orchestra.run.smithi137.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 4.4 MB/s | 16 MB 00:03 2024-09-21T08:16:38.668 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:38.669 INFO:teuthology.orchestra.run.smithi137.stdout:Total 6.8 MB/s | 38 MB 00:05 2024-09-21T08:16:38.688 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:14 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:38.822 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:38.824 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:38.824 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:16:38.824 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:38.824 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 49 k 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T08:16:38.825 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T08:16:38.826 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T08:16:38.826 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T08:16:38.826 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:38.826 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:38.826 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:38.826 INFO:teuthology.orchestra.run.smithi165.stdout:Install 13 Packages 2024-09-21T08:16:38.826 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:38.826 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 2.2 M 2024-09-21T08:16:38.826 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 26 M 2024-09-21T08:16:38.826 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:39.030 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:16:39.056 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:16:39.056 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:16:39.303 INFO:teuthology.orchestra.run.smithi165.stdout:(1/13): ceph-mgr-rook-19.1.1-367.g9e2ac70b.el9. 286 kB/s | 49 kB 00:00 2024-09-21T08:16:39.420 INFO:teuthology.orchestra.run.smithi165.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 66 kB/s | 19 kB 00:00 2024-09-21T08:16:39.445 INFO:teuthology.orchestra.run.smithi165.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 81 kB/s | 26 kB 00:00 2024-09-21T08:16:39.479 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:16:39.479 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:16:39.812 INFO:teuthology.orchestra.run.smithi165.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 405 kB/s | 159 kB 00:00 2024-09-21T08:16:39.887 INFO:teuthology.orchestra.run.smithi165.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 380 kB/s | 222 kB 00:00 2024-09-21T08:16:39.929 INFO:teuthology.orchestra.run.smithi165.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 771 kB/s | 32 kB 00:00 2024-09-21T08:16:39.957 INFO:teuthology.orchestra.run.smithi165.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 377 kB/s | 54 kB 00:00 2024-09-21T08:16:39.980 INFO:teuthology.orchestra.run.smithi165.stdout:(8/13): python3-certifi-2023.05.07-4.el9.noarch 281 kB/s | 14 kB 00:00 2024-09-21T08:16:40.005 INFO:teuthology.orchestra.run.smithi165.stdout:(9/13): python3-pyasn1-modules-0.4.8-6.el9.noar 498 kB/s | 279 kB 00:00 2024-09-21T08:16:40.047 INFO:teuthology.orchestra.run.smithi165.stdout:(10/13): python3-google-auth-2.34.0-1.el9.noarc 2.3 MB/s | 220 kB 00:00 2024-09-21T08:16:40.073 INFO:teuthology.orchestra.run.smithi165.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 3.8 MB/s | 90 kB 00:00 2024-09-21T08:16:40.098 INFO:teuthology.orchestra.run.smithi165.stdout:(12/13): python3-rsa-4.9-2.el9.noarch.rpm 653 kB/s | 59 kB 00:00 2024-09-21T08:16:40.140 INFO:teuthology.orchestra.run.smithi165.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.7 MB/s | 1.0 MB 00:00 2024-09-21T08:16:40.142 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:40.142 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.7 MB/s | 2.2 MB 00:01 2024-09-21T08:16:40.207 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:40.222 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:40.222 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:40.254 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:16:40.311 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:40.312 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:40.326 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-09-21T08:16:40.375 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libquadmath-11.5.0-2.el9.x86_64 2/12 2024-09-21T08:16:40.471 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libgfortran-11.5.0-2.el9.x86_64 3/12 2024-09-21T08:16:40.508 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-devel-3.9.19-8.el9.x86_64 4/12 2024-09-21T08:16:40.691 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-0.3.26-2.el9.x86_64 5/12 2024-09-21T08:16:40.723 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : openblas-openmp-0.3.26-2.el9.x86_64 6/12 2024-09-21T08:16:40.800 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-09-21T08:16:40.816 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:40.985 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T08:16:41.056 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T08:16:41.114 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T08:16:41.151 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T08:16:41.191 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T08:16:41.293 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T08:16:41.395 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T08:16:41.443 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-09-21T08:16:41.443 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T08:16:41.560 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-09-21T08:16:42.152 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T08:16:42.200 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T08:16:42.233 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T08:16:42.281 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T08:16:42.303 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 13/13 2024-09-21T08:16:42.794 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T08:16:43.035 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 13/13 2024-09-21T08:16:43.035 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 1/13 2024-09-21T08:16:43.035 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T08:16:43.035 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T08:16:43.035 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T08:16:43.035 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T08:16:43.035 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T08:16:43.035 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T08:16:43.036 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T08:16:43.036 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T08:16:43.036 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T08:16:43.036 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T08:16:43.036 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T08:16:43.406 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T08:16:43.406 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T08:16:43.407 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:43.408 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:43.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T08:16:43.817 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-09-21T08:16:43.856 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 12/12 2024-09-21T08:16:44.285 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:20 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:44.421 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:44.421 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:44.421 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:16:44.421 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:44.421 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 146 k 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout:Install 6 Packages 2024-09-21T08:16:44.422 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:44.423 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 7.0 M 2024-09-21T08:16:44.423 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 33 M 2024-09-21T08:16:44.423 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:44.735 INFO:teuthology.orchestra.run.smithi165.stdout:(1/6): ceph-mgr-cephadm-19.1.1-367.g9e2ac70b.el 628 kB/s | 146 kB 00:00 2024-09-21T08:16:44.768 INFO:teuthology.orchestra.run.smithi165.stdout:(2/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 932 kB/s | 248 kB 00:00 2024-09-21T08:16:44.818 INFO:teuthology.orchestra.run.smithi165.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 6.4 MB/s | 548 kB 00:00 2024-09-21T08:16:44.844 INFO:teuthology.orchestra.run.smithi165.stdout:(4/6): python3-typing-extensions-4.12.2-2.el9.n 3.1 MB/s | 74 kB 00:00 2024-09-21T08:16:44.869 INFO:teuthology.orchestra.run.smithi165.stdout:(5/6): python3-natsort-7.1.1-5.el9.noarch.rpm 584 kB/s | 58 kB 00:00 2024-09-21T08:16:46.344 INFO:teuthology.orchestra.run.smithi165.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 3.2 MB/s | 6.0 MB 00:01 2024-09-21T08:16:46.346 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:46.347 INFO:teuthology.orchestra.run.smithi165.stdout:Total 3.6 MB/s | 7.0 MB 00:01 2024-09-21T08:16:46.417 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:46.428 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:46.429 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 12/12 2024-09-21T08:16:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac7 1/12 2024-09-21T08:16:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libgfortran-11.5.0-2.el9.x86_64 2/12 2024-09-21T08:16:46.480 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libquadmath-11.5.0-2.el9.x86_64 3/12 2024-09-21T08:16:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-09-21T08:16:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-09-21T08:16:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-09-21T08:16:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-0.3.26-2.el9.x86_64 7/12 2024-09-21T08:16:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : openblas-openmp-0.3.26-2.el9.x86_64 8/12 2024-09-21T08:16:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-devel-3.9.19-8.el9.x86_64 9/12 2024-09-21T08:16:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-09-21T08:16:46.481 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-09-21T08:16:46.528 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:46.529 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:46.837 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:46.864 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-09-21T08:16:46.864 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:46.864 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:16:46.864 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-diskprediction-local-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:46.864 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-09-21T08:16:46.864 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-09-21T08:16:46.864 INFO:teuthology.orchestra.run.smithi137.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-09-21T08:16:46.865 INFO:teuthology.orchestra.run.smithi137.stdout: libgfortran-11.5.0-2.el9.x86_64 2024-09-21T08:16:46.865 INFO:teuthology.orchestra.run.smithi137.stdout: libquadmath-11.5.0-2.el9.x86_64 2024-09-21T08:16:46.865 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-0.3.26-2.el9.x86_64 2024-09-21T08:16:46.865 INFO:teuthology.orchestra.run.smithi137.stdout: openblas-openmp-0.3.26-2.el9.x86_64 2024-09-21T08:16:46.865 INFO:teuthology.orchestra.run.smithi137.stdout: python3-devel-3.9.19-8.el9.x86_64 2024-09-21T08:16:46.865 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-09-21T08:16:46.865 INFO:teuthology.orchestra.run.smithi137.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-09-21T08:16:46.865 INFO:teuthology.orchestra.run.smithi137.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-09-21T08:16:46.865 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:46.865 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:16:47.001 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T08:16:47.054 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T08:16:47.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-rook 2024-09-21T08:16:47.566 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T08:16:47.657 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T08:16:47.721 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T08:16:47.750 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 6/6 2024-09-21T08:16:47.860 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:18 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:16:47.992 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:16:47.993 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 49 k 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi noarch 2023.05.07-4.el9 epel 14 k 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth noarch 1:2.34.0-1.el9 epel 220 k 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-09-21T08:16:47.994 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-09-21T08:16:47.995 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-09-21T08:16:47.995 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-09-21T08:16:47.995 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-09-21T08:16:47.995 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-09-21T08:16:47.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:47.995 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:16:47.995 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:47.995 INFO:teuthology.orchestra.run.smithi137.stdout:Install 13 Packages 2024-09-21T08:16:47.995 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:47.995 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 2.2 M 2024-09-21T08:16:47.996 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 26 M 2024-09-21T08:16:47.996 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:16:48.432 INFO:teuthology.orchestra.run.smithi137.stdout:(1/13): ceph-mgr-rook-19.1.1-367.g9e2ac70b.el9. 296 kB/s | 49 kB 00:00 2024-09-21T08:16:48.517 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 6/6 2024-09-21T08:16:48.517 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/6 2024-09-21T08:16:48.517 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T08:16:48.517 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T08:16:48.517 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T08:16:48.517 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T08:16:48.773 INFO:teuthology.orchestra.run.smithi137.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 38 kB/s | 19 kB 00:00 2024-09-21T08:16:48.796 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T08:16:48.797 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:48.797 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:48.797 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:48.797 INFO:teuthology.orchestra.run.smithi165.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T08:16:48.797 INFO:teuthology.orchestra.run.smithi165.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T08:16:48.797 INFO:teuthology.orchestra.run.smithi165.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T08:16:48.797 INFO:teuthology.orchestra.run.smithi165.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T08:16:48.797 INFO:teuthology.orchestra.run.smithi165.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T08:16:48.797 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:48.797 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:48.874 INFO:teuthology.orchestra.run.smithi137.stdout:(3/13): python3-jsonpatch-1.21-16.el9.noarch.rp 42 kB/s | 26 kB 00:00 2024-09-21T08:16:49.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-fuse 2024-09-21T08:16:49.191 INFO:teuthology.orchestra.run.smithi137.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 293 kB/s | 222 kB 00:00 2024-09-21T08:16:49.541 INFO:teuthology.orchestra.run.smithi137.stdout:(5/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 207 kB/s | 159 kB 00:00 2024-09-21T08:16:49.583 INFO:teuthology.orchestra.run.smithi137.stdout:(6/13): python3-cachetools-4.2.4-1.el9.noarch.r 781 kB/s | 32 kB 00:00 2024-09-21T08:16:49.600 INFO:teuthology.orchestra.run.smithi137.stdout:(7/13): python3-certifi-2023.05.07-4.el9.noarch 853 kB/s | 14 kB 00:00 2024-09-21T08:16:49.625 INFO:teuthology.orchestra.run.smithi137.stdout:(8/13): python3-google-auth-2.34.0-1.el9.noarch 8.7 MB/s | 220 kB 00:00 2024-09-21T08:16:49.664 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:25 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:49.692 INFO:teuthology.orchestra.run.smithi137.stdout:(9/13): python3-kubernetes-26.1.0-2.el9.noarch. 15 MB/s | 1.0 MB 00:00 2024-09-21T08:16:49.717 INFO:teuthology.orchestra.run.smithi137.stdout:(10/13): python3-pyasn1-modules-0.4.8-6.el9.noa 331 kB/s | 279 kB 00:00 2024-09-21T08:16:49.742 INFO:teuthology.orchestra.run.smithi137.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-09-21T08:16:49.784 INFO:teuthology.orchestra.run.smithi137.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.3 MB/s | 90 kB 00:00 2024-09-21T08:16:49.798 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:49.798 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:49.798 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:16:49.798 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:49.798 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 896 k 2024-09-21T08:16:49.799 INFO:teuthology.orchestra.run.smithi165.stdout:Installing dependencies: 2024-09-21T08:16:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T08:16:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:49.799 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:49.799 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:49.799 INFO:teuthology.orchestra.run.smithi165.stdout:Install 2 Packages 2024-09-21T08:16:49.799 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:49.800 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 976 k 2024-09-21T08:16:49.800 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 2.9 M 2024-09-21T08:16:49.800 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:49.810 INFO:teuthology.orchestra.run.smithi137.stdout:(13/13): python3-requests-oauthlib-1.3.0-12.el9 87 kB/s | 54 kB 00:00 2024-09-21T08:16:49.813 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:49.813 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.2 MB/s | 2.2 MB 00:01 2024-09-21T08:16:49.885 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:16:49.901 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:16:49.901 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:16:49.980 INFO:teuthology.orchestra.run.smithi165.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 616 kB/s | 80 kB 00:00 2024-09-21T08:16:49.990 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:16:49.991 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:16:50.138 INFO:teuthology.orchestra.run.smithi165.stdout:(2/2): ceph-fuse-19.1.1-367.g9e2ac70b.el9.x86_6 3.0 MB/s | 896 kB 00:00 2024-09-21T08:16:50.139 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:50.139 INFO:teuthology.orchestra.run.smithi165.stdout:Total 2.8 MB/s | 976 kB 00:00 2024-09-21T08:16:50.150 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:50.172 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:50.172 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:50.253 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:50.253 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:50.450 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:50.496 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:16:50.590 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T08:16:50.663 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/2 2024-09-21T08:16:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-09-21T08:16:50.711 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-09-21T08:16:50.766 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-09-21T08:16:50.803 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-09-21T08:16:50.845 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-certifi-2023.05.07-4.el9.noarch 5/13 2024-09-21T08:16:50.947 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-09-21T08:16:51.048 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-google-auth-1:2.34.0-1.el9.noarch 7/13 2024-09-21T08:16:51.096 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-09-21T08:16:51.323 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/2 2024-09-21T08:16:51.323 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/2 2024-09-21T08:16:51.597 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T08:16:51.634 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:51.634 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:51.635 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T08:16:51.635 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:51.635 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:51.804 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-09-21T08:16:51.860 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-09-21T08:16:51.894 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-09-21T08:16:51.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install ceph-volume 2024-09-21T08:16:51.941 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-09-21T08:16:51.972 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 13/13 2024-09-21T08:16:52.514 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:28 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:52.650 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:52.650 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:52.650 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repository Size 2024-09-21T08:16:52.650 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:52.650 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:52.650 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 285 k 2024-09-21T08:16:52.650 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:52.650 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:52.651 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:52.651 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-21T08:16:52.651 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:52.651 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 285 k 2024-09-21T08:16:52.651 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 1.3 M 2024-09-21T08:16:52.651 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:52.717 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 13/13 2024-09-21T08:16:52.717 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 1/13 2024-09-21T08:16:52.717 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-09-21T08:16:52.717 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-09-21T08:16:52.718 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-09-21T08:16:52.718 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-09-21T08:16:52.718 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-09-21T08:16:52.718 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-09-21T08:16:52.718 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-09-21T08:16:52.718 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-certifi-2023.05.07-4.el9.noarch 9/13 2024-09-21T08:16:52.718 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-google-auth-1:2.34.0-1.el9.noarch 10/13 2024-09-21T08:16:52.718 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-09-21T08:16:52.718 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-09-21T08:16:52.860 INFO:teuthology.orchestra.run.smithi165.stdout:ceph-volume-19.1.1-367.g9e2ac70b.el9.noarch.rpm 1.3 MB/s | 285 kB 00:00 2024-09-21T08:16:52.861 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:52.861 INFO:teuthology.orchestra.run.smithi165.stdout:Total 1.3 MB/s | 285 kB 00:00 2024-09-21T08:16:52.861 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:52.870 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:52.870 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:52.903 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:52.904 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-rook-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout: python3-certifi-2023.05.07-4.el9.noarch 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout: python3-google-auth-1:2.34.0-1.el9.noarch 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-09-21T08:16:53.045 INFO:teuthology.orchestra.run.smithi137.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-09-21T08:16:53.046 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-09-21T08:16:53.046 INFO:teuthology.orchestra.run.smithi137.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-09-21T08:16:53.046 INFO:teuthology.orchestra.run.smithi137.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-09-21T08:16:53.046 INFO:teuthology.orchestra.run.smithi137.stdout: python3-rsa-4.9-2.el9.noarch 2024-09-21T08:16:53.046 INFO:teuthology.orchestra.run.smithi137.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-09-21T08:16:53.046 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:53.046 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:16:53.123 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:53.196 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:53.212 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:53.212 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:16:53.213 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T08:16:53.213 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:53.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-mgr-cephadm 2024-09-21T08:16:53.885 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:24 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:16:54.019 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:16:54.020 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 146 k 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions noarch 4.12.2-2.el9 epel 74 k 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:16:54.021 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:54.022 INFO:teuthology.orchestra.run.smithi137.stdout:Install 6 Packages 2024-09-21T08:16:54.022 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:54.023 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 7.0 M 2024-09-21T08:16:54.023 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 33 M 2024-09-21T08:16:54.023 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:16:54.203 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:16:54.203 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:54.204 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:54.204 INFO:teuthology.orchestra.run.smithi165.stdout: ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:54.204 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:54.204 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:54.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librados-devel 2024-09-21T08:16:54.611 INFO:teuthology.orchestra.run.smithi137.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.7 MB/s | 248 kB 00:00 2024-09-21T08:16:54.662 INFO:teuthology.orchestra.run.smithi137.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 11 MB/s | 548 kB 00:00 2024-09-21T08:16:54.678 INFO:teuthology.orchestra.run.smithi137.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 3.4 MB/s | 58 kB 00:00 2024-09-21T08:16:54.704 INFO:teuthology.orchestra.run.smithi137.stdout:(4/6): ceph-mgr-cephadm-19.1.1-367.g9e2ac70b.el 612 kB/s | 146 kB 00:00 2024-09-21T08:16:54.729 INFO:teuthology.orchestra.run.smithi137.stdout:(5/6): python3-typing-extensions-4.12.2-2.el9.n 1.4 MB/s | 74 kB 00:00 2024-09-21T08:16:54.821 INFO:teuthology.orchestra.run.smithi137.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-09-21T08:16:54.823 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:54.823 INFO:teuthology.orchestra.run.smithi137.stdout:Total 8.7 MB/s | 7.0 MB 00:00 2024-09-21T08:16:54.893 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:16:54.904 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:16:54.904 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:16:55.007 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:16:55.007 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:16:55.117 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:55.252 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:55.252 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:55.252 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-21T08:16:55.253 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:55.253 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:55.253 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 127 k 2024-09-21T08:16:55.253 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:55.253 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:55.253 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:55.253 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-21T08:16:55.253 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:55.254 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 127 k 2024-09-21T08:16:55.254 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 456 k 2024-09-21T08:16:55.254 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:55.315 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:16:55.458 INFO:teuthology.orchestra.run.smithi165.stdout:librados-devel-19.1.1-367.g9e2ac70b.el9.x86_64. 620 kB/s | 127 kB 00:00 2024-09-21T08:16:55.458 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:55.459 INFO:teuthology.orchestra.run.smithi165.stdout:Total 616 kB/s | 127 kB 00:00 2024-09-21T08:16:55.459 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:55.464 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:55.464 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:55.500 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-typing-extensions-4.12.2-2.el9.noarch 1/6 2024-09-21T08:16:55.529 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:55.529 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:55.559 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-09-21T08:16:55.679 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:55.764 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:16:56.087 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-09-21T08:16:56.175 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-09-21T08:16:56.240 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-09-21T08:16:56.267 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:16:56.268 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 6/6 2024-09-21T08:16:56.536 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:16:56.537 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:56.537 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:56.537 INFO:teuthology.orchestra.run.smithi165.stdout: librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:56.537 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:56.537 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:56.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install libcephfs2 2024-09-21T08:16:57.085 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 6/6 2024-09-21T08:16:57.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 1/6 2024-09-21T08:16:57.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-09-21T08:16:57.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-09-21T08:16:57.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-09-21T08:16:57.085 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-09-21T08:16:57.353 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : python3-typing-extensions-4.12.2-2.el9.noarch 6/6 2024-09-21T08:16:57.353 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:57.353 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:16:57.353 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-mgr-cephadm-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:16:57.353 INFO:teuthology.orchestra.run.smithi137.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-09-21T08:16:57.353 INFO:teuthology.orchestra.run.smithi137.stdout: python3-babel-2.9.1-2.el9.noarch 2024-09-21T08:16:57.353 INFO:teuthology.orchestra.run.smithi137.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-09-21T08:16:57.353 INFO:teuthology.orchestra.run.smithi137.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-09-21T08:16:57.353 INFO:teuthology.orchestra.run.smithi137.stdout: python3-typing-extensions-4.12.2-2.el9.noarch 2024-09-21T08:16:57.353 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:57.354 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:16:57.394 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:33 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:57.485 INFO:teuthology.orchestra.run.smithi165.stdout:Package libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:16:57.529 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:57.530 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-21T08:16:57.530 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:57.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-fuse 2024-09-21T08:16:57.622 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install libcephfs-devel 2024-09-21T08:16:58.186 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:29 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:16:58.219 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:16:58.320 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:16:58.321 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:16:58.321 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:58.321 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:16:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 896 k 2024-09-21T08:16:58.321 INFO:teuthology.orchestra.run.smithi137.stdout:Installing dependencies: 2024-09-21T08:16:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: fuse x86_64 2.9.9-16.el9 baseos 80 k 2024-09-21T08:16:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:58.322 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:16:58.322 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:16:58.322 INFO:teuthology.orchestra.run.smithi137.stdout:Install 2 Packages 2024-09-21T08:16:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:16:58.322 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 976 k 2024-09-21T08:16:58.322 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 2.9 M 2024-09-21T08:16:58.322 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:16:58.354 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 32 k 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 32 k 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 142 k 2024-09-21T08:16:58.355 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:16:58.520 INFO:teuthology.orchestra.run.smithi165.stdout:libcephfs-devel-19.1.1-367.g9e2ac70b.el9.x86_64 192 kB/s | 32 kB 00:00 2024-09-21T08:16:58.521 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:58.521 INFO:teuthology.orchestra.run.smithi165.stdout:Total 191 kB/s | 32 kB 00:00 2024-09-21T08:16:58.522 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:16:58.523 INFO:teuthology.orchestra.run.smithi137.stdout:(1/2): fuse-2.9.9-16.el9.x86_64.rpm 443 kB/s | 80 kB 00:00 2024-09-21T08:16:58.525 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:16:58.525 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:16:58.547 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:16:58.547 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:16:58.590 INFO:teuthology.orchestra.run.smithi137.stdout:(2/2): ceph-fuse-19.1.1-367.g9e2ac70b.el9.x86_6 3.5 MB/s | 896 kB 00:00 2024-09-21T08:16:58.590 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:16:58.591 INFO:teuthology.orchestra.run.smithi137.stdout:Total 3.5 MB/s | 976 kB 00:00 2024-09-21T08:16:58.602 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:16:58.623 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:16:58.624 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:16:58.635 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:16:58.704 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:16:58.704 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:16:58.744 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:16:58.890 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:16:59.033 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : fuse-2.9.9-16.el9.x86_64 1/2 2024-09-21T08:16:59.097 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/2 2024-09-21T08:16:59.151 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:16:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:16:59.409 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:59.409 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:16:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:16:59.410 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:16:59.410 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:16:59.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librados2 2024-09-21T08:16:59.754 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2/2 2024-09-21T08:16:59.754 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/2 2024-09-21T08:17:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : fuse-2.9.9-16.el9.x86_64 2/2 2024-09-21T08:17:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:00.036 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:17:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 fuse-2.9.9-16.el9.x86_64 2024-09-21T08:17:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:00.036 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:00.174 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:17:00.263 INFO:teuthology.orchestra.run.smithi165.stdout:Package librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:00.306 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:17:00.308 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-21T08:17:00.308 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:17:00.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install ceph-volume 2024-09-21T08:17:00.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install librbd1 2024-09-21T08:17:00.926 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:31 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:00.973 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:17:01.061 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:01.062 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:01.062 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repository Size 2024-09-21T08:17:01.062 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:01.062 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:17:01.063 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume noarch 2:19.1.1-367.g9e2ac70b.el9 ceph-noarch 285 k 2024-09-21T08:17:01.063 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:01.063 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:17:01.063 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:01.063 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-21T08:17:01.063 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:01.063 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 285 k 2024-09-21T08:17:01.063 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 1.3 M 2024-09-21T08:17:01.063 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:17:01.064 INFO:teuthology.orchestra.run.smithi165.stdout:Package librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:01.109 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:17:01.110 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-21T08:17:01.110 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:17:01.200 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rados 2024-09-21T08:17:01.286 INFO:teuthology.orchestra.run.smithi137.stdout:ceph-volume-19.1.1-367.g9e2ac70b.el9.noarch.rpm 1.2 MB/s | 285 kB 00:00 2024-09-21T08:17:01.287 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:17:01.287 INFO:teuthology.orchestra.run.smithi137.stdout:Total 1.2 MB/s | 285 kB 00:00 2024-09-21T08:17:01.287 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:17:01.295 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:17:01.295 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:17:01.326 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:17:01.327 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:17:01.549 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:17:01.604 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:17:01.620 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:17:01.620 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:17:01.620 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-09-21T08:17:01.620 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:01.778 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:17:01.868 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:01.911 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:17:01.912 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-21T08:17:01.912 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:17:02.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rgw 2024-09-21T08:17:02.592 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:38 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:17:02.611 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 1/1 2024-09-21T08:17:02.611 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:02.612 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:17:02.612 INFO:teuthology.orchestra.run.smithi137.stdout: ceph-volume-2:19.1.1-367.g9e2ac70b.el9.noarch 2024-09-21T08:17:02.612 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:02.612 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:02.683 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:02.726 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:17:02.727 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-21T08:17:02.727 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:17:02.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-cephfs 2024-09-21T08:17:02.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados-devel 2024-09-21T08:17:03.398 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:17:03.483 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:34 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:03.489 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:03.533 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:17:03.534 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-21T08:17:03.534 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:17:03.621 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:03.621 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:03.621 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-21T08:17:03.621 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:03.621 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:17:03.622 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 127 k 2024-09-21T08:17:03.622 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:03.622 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:17:03.622 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:03.622 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-21T08:17:03.622 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:03.622 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 127 k 2024-09-21T08:17:03.622 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 456 k 2024-09-21T08:17:03.623 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:17:03.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install python3-rbd 2024-09-21T08:17:03.815 INFO:teuthology.orchestra.run.smithi137.stdout:librados-devel-19.1.1-367.g9e2ac70b.el9.x86_64. 657 kB/s | 127 kB 00:00 2024-09-21T08:17:03.816 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:17:03.816 INFO:teuthology.orchestra.run.smithi137.stdout:Total 652 kB/s | 127 kB 00:00 2024-09-21T08:17:03.816 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:17:03.822 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:17:03.822 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:17:03.886 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:17:03.886 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:17:04.039 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:17:04.138 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:04.219 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:17:04.316 INFO:teuthology.orchestra.run.smithi165.stdout:Package python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:04.362 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:17:04.363 INFO:teuthology.orchestra.run.smithi165.stdout:Nothing to do. 2024-09-21T08:17:04.363 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:17:04.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-fuse 2024-09-21T08:17:04.583 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:04.861 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:17:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: librados-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:17:04.861 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:04.861 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:05.066 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:17:05.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs2 2024-09-21T08:17:05.206 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:17:05.206 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:17:05.207 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-21T08:17:05.207 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:17:05.207 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:17:05.207 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 85 k 2024-09-21T08:17:05.207 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:05.207 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:17:05.207 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:17:05.207 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-21T08:17:05.207 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:05.208 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 85 k 2024-09-21T08:17:05.208 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 231 k 2024-09-21T08:17:05.208 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:17:05.399 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-fuse-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 445 kB/s | 85 kB 00:00 2024-09-21T08:17:05.400 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:17:05.401 INFO:teuthology.orchestra.run.smithi165.stdout:Total 441 kB/s | 85 kB 00:00 2024-09-21T08:17:05.401 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:17:05.408 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:17:05.409 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:17:05.473 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:17:05.473 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:17:05.616 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:17:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:05.764 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:36 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:05.855 INFO:teuthology.orchestra.run.smithi137.stdout:Package libcephfs2-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:05.901 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:05.902 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-21T08:17:05.903 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:05.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install libcephfs-devel 2024-09-21T08:17:06.133 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:06.385 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:06.386 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:06.386 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:17:06.386 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:17:06.386 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:06.386 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:17:06.553 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:37 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:06.617 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-mirror 2024-09-21T08:17:06.686 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 32 k 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 32 k 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 142 k 2024-09-21T08:17:06.688 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:17:06.835 INFO:teuthology.orchestra.run.smithi137.stdout:libcephfs-devel-19.1.1-367.g9e2ac70b.el9.x86_64 215 kB/s | 32 kB 00:00 2024-09-21T08:17:06.836 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:17:06.836 INFO:teuthology.orchestra.run.smithi137.stdout:Total 213 kB/s | 32 kB 00:00 2024-09-21T08:17:06.836 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:17:06.840 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:17:06.840 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:17:06.853 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:17:06.854 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:17:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:17:07.066 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:07.208 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:17:07.345 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:17:07.346 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:17:07.346 INFO:teuthology.orchestra.run.smithi165.stdout: Package Arch Version Repo Size 2024-09-21T08:17:07.346 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:17:07.346 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:17:07.346 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.1 M 2024-09-21T08:17:07.347 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:07.347 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:17:07.347 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:17:07.347 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-21T08:17:07.347 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:07.347 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 3.1 M 2024-09-21T08:17:07.347 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 13 M 2024-09-21T08:17:07.347 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:17:07.456 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:07.675 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:07.675 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:07.675 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:17:07.675 INFO:teuthology.orchestra.run.smithi137.stdout: libcephfs-devel-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:17:07.675 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:07.675 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:07.679 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-mirror-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 9.4 MB/s | 3.1 MB 00:00 2024-09-21T08:17:07.679 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:17:07.679 INFO:teuthology.orchestra.run.smithi165.stdout:Total 9.3 MB/s | 3.1 MB 00:00 2024-09-21T08:17:07.680 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:17:07.689 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:17:07.689 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:17:07.771 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:17:07.771 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:17:07.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librados2 2024-09-21T08:17:08.260 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:17:08.338 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:08.364 INFO:teuthology.orchestra.run.smithi165.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:17:08.364 INFO:teuthology.orchestra.run.smithi165.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T08:17:08.364 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T08:17:08.364 INFO:teuthology.orchestra.run.smithi165.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T08:17:08.364 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:08.429 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:39 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:08.519 INFO:teuthology.orchestra.run.smithi137.stdout:Package librados2-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:08.563 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:08.564 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-21T08:17:08.564 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:08.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install librbd1 2024-09-21T08:17:09.210 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:09.300 INFO:teuthology.orchestra.run.smithi137.stdout:Package librbd1-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:09.344 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:09.345 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-21T08:17:09.345 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:09.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rados 2024-09-21T08:17:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:09.519 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:17:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:17:09.519 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:09.519 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:17:09.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo yum -y install rbd-nbd 2024-09-21T08:17:09.995 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:40 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:10.085 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rados-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:10.128 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:10.129 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-21T08:17:10.129 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:10.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rgw 2024-09-21T08:17:10.400 INFO:teuthology.orchestra.run.smithi165.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 08:15:24 AM UTC. 2024-09-21T08:17:10.533 INFO:teuthology.orchestra.run.smithi165.stdout:Dependencies resolved. 2024-09-21T08:17:10.533 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:17:10.534 INFO:teuthology.orchestra.run.smithi165.stdout: Package Architecture Version Repository Size 2024-09-21T08:17:10.534 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:17:10.534 INFO:teuthology.orchestra.run.smithi165.stdout:Installing: 2024-09-21T08:17:10.534 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 170 k 2024-09-21T08:17:10.534 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:10.534 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction Summary 2024-09-21T08:17:10.534 INFO:teuthology.orchestra.run.smithi165.stdout:================================================================================ 2024-09-21T08:17:10.534 INFO:teuthology.orchestra.run.smithi165.stdout:Install 1 Package 2024-09-21T08:17:10.534 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:10.535 INFO:teuthology.orchestra.run.smithi165.stdout:Total download size: 170 k 2024-09-21T08:17:10.535 INFO:teuthology.orchestra.run.smithi165.stdout:Installed size: 494 k 2024-09-21T08:17:10.535 INFO:teuthology.orchestra.run.smithi165.stdout:Downloading Packages: 2024-09-21T08:17:10.758 INFO:teuthology.orchestra.run.smithi165.stdout:rbd-nbd-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 764 kB/s | 170 kB 00:00 2024-09-21T08:17:10.758 INFO:teuthology.orchestra.run.smithi165.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:17:10.759 INFO:teuthology.orchestra.run.smithi165.stdout:Total 759 kB/s | 170 kB 00:00 2024-09-21T08:17:10.759 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction check 2024-09-21T08:17:10.767 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction check succeeded. 2024-09-21T08:17:10.767 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction test 2024-09-21T08:17:10.777 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:41 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:10.833 INFO:teuthology.orchestra.run.smithi165.stdout:Transaction test succeeded. 2024-09-21T08:17:10.834 INFO:teuthology.orchestra.run.smithi165.stdout:Running transaction 2024-09-21T08:17:10.866 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rgw-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:10.909 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:10.910 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-21T08:17:10.910 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:10.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-cephfs 2024-09-21T08:17:10.992 INFO:teuthology.orchestra.run.smithi165.stdout: Preparing : 1/1 2024-09-21T08:17:11.115 INFO:teuthology.orchestra.run.smithi165.stdout: Installing : rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:11.547 INFO:teuthology.orchestra.run.smithi165.stdout: Running scriptlet: rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:11.555 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:42 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:11.644 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-cephfs-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:11.687 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:11.688 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-21T08:17:11.688 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:11.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install python3-rbd 2024-09-21T08:17:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: Verifying : rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:11.817 INFO:teuthology.orchestra.run.smithi165.stdout:Installed: 2024-09-21T08:17:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:17:11.817 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:17:11.818 INFO:teuthology.orchestra.run.smithi165.stdout:Complete! 2024-09-21T08:17:12.053 DEBUG:teuthology.parallel:result is None 2024-09-21T08:17:12.340 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:43 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:12.431 INFO:teuthology.orchestra.run.smithi137.stdout:Package python3-rbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 is already installed. 2024-09-21T08:17:12.474 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:12.475 INFO:teuthology.orchestra.run.smithi137.stdout:Nothing to do. 2024-09-21T08:17:12.475 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:12.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-fuse 2024-09-21T08:17:13.126 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:44 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:13.259 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 85 k 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 85 k 2024-09-21T08:17:13.260 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 231 k 2024-09-21T08:17:13.261 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:17:13.500 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-fuse-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 357 kB/s | 85 kB 00:00 2024-09-21T08:17:13.500 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:17:13.501 INFO:teuthology.orchestra.run.smithi137.stdout:Total 354 kB/s | 85 kB 00:00 2024-09-21T08:17:13.501 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:17:13.508 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:17:13.508 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:17:13.572 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:17:13.572 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:17:13.715 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:17:13.819 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:14.191 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:14.453 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:14.453 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:14.453 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:17:14.453 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-fuse-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:17:14.454 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:14.454 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:14.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-mirror 2024-09-21T08:17:15.303 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:46 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:15.445 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:15.446 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: Package Arch Version Repo Size 2024-09-21T08:17:15.446 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:15.446 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:17:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 3.1 M 2024-09-21T08:17:15.446 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:15.446 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:17:15.447 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:15.447 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-21T08:17:15.447 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:15.447 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 3.1 M 2024-09-21T08:17:15.447 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 13 M 2024-09-21T08:17:15.447 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:17:15.775 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-mirror-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 9.5 MB/s | 3.1 MB 00:00 2024-09-21T08:17:15.776 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:17:15.776 INFO:teuthology.orchestra.run.smithi137.stdout:Total 9.4 MB/s | 3.1 MB 00:00 2024-09-21T08:17:15.777 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:17:15.786 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:17:15.786 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:17:15.872 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:17:15.872 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:17:16.361 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:17:16.418 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:16.445 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:16.445 INFO:teuthology.orchestra.run.smithi137.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-09-21T08:17:16.445 INFO:teuthology.orchestra.run.smithi137.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-09-21T08:17:16.445 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T08:17:16.445 INFO:teuthology.orchestra.run.smithi137.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-09-21T08:17:16.446 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:17.593 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:17.593 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:17.594 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:17:17.594 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-mirror-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:17:17.594 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:17.594 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:17.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo yum -y install rbd-nbd 2024-09-21T08:17:18.419 INFO:teuthology.orchestra.run.smithi137.stdout:Last metadata expiration check: 0:01:49 ago on Sat 21 Sep 2024 08:15:29 AM UTC. 2024-09-21T08:17:18.552 INFO:teuthology.orchestra.run.smithi137.stdout:Dependencies resolved. 2024-09-21T08:17:18.553 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:18.553 INFO:teuthology.orchestra.run.smithi137.stdout: Package Architecture Version Repository Size 2024-09-21T08:17:18.553 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:18.553 INFO:teuthology.orchestra.run.smithi137.stdout:Installing: 2024-09-21T08:17:18.553 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd x86_64 2:19.1.1-367.g9e2ac70b.el9 ceph 170 k 2024-09-21T08:17:18.553 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:18.553 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction Summary 2024-09-21T08:17:18.553 INFO:teuthology.orchestra.run.smithi137.stdout:================================================================================ 2024-09-21T08:17:18.553 INFO:teuthology.orchestra.run.smithi137.stdout:Install 1 Package 2024-09-21T08:17:18.553 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:18.554 INFO:teuthology.orchestra.run.smithi137.stdout:Total download size: 170 k 2024-09-21T08:17:18.554 INFO:teuthology.orchestra.run.smithi137.stdout:Installed size: 494 k 2024-09-21T08:17:18.554 INFO:teuthology.orchestra.run.smithi137.stdout:Downloading Packages: 2024-09-21T08:17:18.771 INFO:teuthology.orchestra.run.smithi137.stdout:rbd-nbd-19.1.1-367.g9e2ac70b.el9.x86_64.rpm 787 kB/s | 170 kB 00:00 2024-09-21T08:17:18.772 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:17:18.772 INFO:teuthology.orchestra.run.smithi137.stdout:Total 782 kB/s | 170 kB 00:00 2024-09-21T08:17:18.772 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction check 2024-09-21T08:17:18.779 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction check succeeded. 2024-09-21T08:17:18.779 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction test 2024-09-21T08:17:18.843 INFO:teuthology.orchestra.run.smithi137.stdout:Transaction test succeeded. 2024-09-21T08:17:18.844 INFO:teuthology.orchestra.run.smithi137.stdout:Running transaction 2024-09-21T08:17:18.994 INFO:teuthology.orchestra.run.smithi137.stdout: Preparing : 1/1 2024-09-21T08:17:19.093 INFO:teuthology.orchestra.run.smithi137.stdout: Installing : rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:19.536 INFO:teuthology.orchestra.run.smithi137.stdout: Running scriptlet: rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:19.784 INFO:teuthology.orchestra.run.smithi137.stdout: Verifying : rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 1/1 2024-09-21T08:17:19.784 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:19.784 INFO:teuthology.orchestra.run.smithi137.stdout:Installed: 2024-09-21T08:17:19.784 INFO:teuthology.orchestra.run.smithi137.stdout: rbd-nbd-2:19.1.1-367.g9e2ac70b.el9.x86_64 2024-09-21T08:17:19.784 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:17:19.784 INFO:teuthology.orchestra.run.smithi137.stdout:Complete! 2024-09-21T08:17:20.049 DEBUG:teuthology.parallel:result is None 2024-09-21T08:17:20.050 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-21T08:17:20.184 DEBUG:teuthology.orchestra.run.smithi137:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T08:17:20.205 INFO:teuthology.orchestra.run.smithi137.stdout:19.1.1-367.g9e2ac70b.el9 2024-09-21T08:17:20.206 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-367.g9e2ac70b.el9 2024-09-21T08:17:20.206 INFO:teuthology.task.install:The correct ceph version 19.1.1-367.g9e2ac70b is installed. 2024-09-21T08:17:20.208 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-21T08:17:20.340 DEBUG:teuthology.orchestra.run.smithi165:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-09-21T08:17:20.367 INFO:teuthology.orchestra.run.smithi165.stdout:19.1.1-367.g9e2ac70b.el9 2024-09-21T08:17:20.367 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-367.g9e2ac70b.el9 2024-09-21T08:17:20.367 INFO:teuthology.task.install:The correct ceph version 19.1.1-367.g9e2ac70b is installed. 2024-09-21T08:17:20.369 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-21T08:17:20.369 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:17:20.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T08:17:20.395 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:17:20.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-21T08:17:20.437 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-21T08:17:20.437 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:17:20.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T08:17:20.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T08:17:20.528 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:17:20.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/daemon-helper 2024-09-21T08:17:20.556 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-21T08:17:20.623 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-21T08:17:20.623 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:17:20.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T08:17:20.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T08:17:20.712 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:17:20.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-21T08:17:20.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-21T08:17:20.805 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-21T08:17:20.805 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:17:20.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T08:17:20.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T08:17:20.894 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:17:20.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/usr/bin/stdin-killer 2024-09-21T08:17:20.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-21T08:17:20.985 INFO:teuthology.run_tasks:Running task cephadm... 2024-09-21T08:17:21.075 INFO:tasks.cephadm:Config: {'conf': {'mgr': {'debug mgr': 20, 'debug ms': 1}, 'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736', 'cephadm_mode': 'root'} 2024-09-21T08:17:21.076 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-21T08:17:21.076 INFO:tasks.cephadm:Cluster fsid is ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:17:21.076 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-09-21T08:17:21.076 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '[v1:172.21.15.137:6789]', 'mon.c': '[v1:172.21.15.137:6790]', 'mon.b': '[v1:172.21.15.165:6789]'} 2024-09-21T08:17:21.076 INFO:tasks.cephadm:First mon is mon.a on smithi137 2024-09-21T08:17:21.077 INFO:tasks.cephadm:First mgr is y 2024-09-21T08:17:21.077 INFO:tasks.cephadm:Normalizing hostnames... 2024-09-21T08:17:21.077 DEBUG:teuthology.orchestra.run.smithi137:> sudo hostname $(hostname -s) 2024-09-21T08:17:21.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo hostname $(hostname -s) 2024-09-21T08:17:21.127 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-09-21T08:17:21.127 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-21T08:17:21.273 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/56118/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.1.1-367-g9e2ac70b', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.1.1-367.g9e2ac70b'}, 'url': 'https://1.chacra.ceph.com/r/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-09-19 19:23:07.775447', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-09-21T08:17:21.411 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref squid, sha1 9e2ac70bb7aea67d34f0d50002344abbe7803736 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-21T08:17:21.413 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/x86_64/flavors/default/cephadm 2024-09-21T08:17:21.413 INFO:tasks.cephadm:Downloading cephadm from url: https://1.chacra.ceph.com/binaries/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/x86_64/flavors/default/cephadm 2024-09-21T08:17:21.413 DEBUG:teuthology.orchestra.run.smithi137:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T08:17:21.756 INFO:teuthology.orchestra.run.smithi137.stdout:-rw-r--r--. 1 ubuntu ubuntu 784381 Sep 21 08:17 /home/ubuntu/cephtest/cephadm 2024-09-21T08:17:21.757 DEBUG:teuthology.orchestra.run.smithi165:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/squid/9e2ac70bb7aea67d34f0d50002344abbe7803736/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-09-21T08:17:22.139 INFO:teuthology.orchestra.run.smithi165.stdout:-rw-r--r--. 1 ubuntu ubuntu 784381 Sep 21 08:17 /home/ubuntu/cephtest/cephadm 2024-09-21T08:17:22.139 DEBUG:teuthology.orchestra.run.smithi137:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T08:17:22.156 DEBUG:teuthology.orchestra.run.smithi165:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-09-21T08:17:22.186 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 on all hosts... 2024-09-21T08:17:22.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 pull 2024-09-21T08:17:22.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 pull 2024-09-21T08:17:23.764 INFO:teuthology.orchestra.run.smithi137.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736... 2024-09-21T08:17:23.802 INFO:teuthology.orchestra.run.smithi165.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736... 2024-09-21T08:18:08.388 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:18:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "ceph_version": "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)", 2024-09-21T08:18:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "image_id": "cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd", 2024-09-21T08:18:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "repo_digests": [ 2024-09-21T08:18:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5" 2024-09-21T08:18:08.388 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:18:08.389 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:18:12.511 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:18:12.511 INFO:teuthology.orchestra.run.smithi137.stdout: "ceph_version": "ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)", 2024-09-21T08:18:12.511 INFO:teuthology.orchestra.run.smithi137.stdout: "image_id": "cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd", 2024-09-21T08:18:12.511 INFO:teuthology.orchestra.run.smithi137.stdout: "repo_digests": [ 2024-09-21T08:18:12.512 INFO:teuthology.orchestra.run.smithi137.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5" 2024-09-21T08:18:12.512 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:18:12.512 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:18:12.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo mkdir -p /etc/ceph 2024-09-21T08:18:12.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo mkdir -p /etc/ceph 2024-09-21T08:18:12.632 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /etc/ceph 2024-09-21T08:18:12.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /etc/ceph 2024-09-21T08:18:12.700 INFO:tasks.cephadm:Writing seed config... 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [global] ms bind msgr2 = False 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [global] ms type = async 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [mon] mon min osdmap epochs = 50 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [mon] mon osdmap full prune interval = 2 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [mon] mon osdmap full prune min = 15 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [mon] mon osdmap full prune txsize = 2 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [mon] paxos service trim min = 10 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-09-21T08:18:12.702 INFO:tasks.cephadm: override: [osd] osd debug reject backfill probability = 0.3 2024-09-21T08:18:12.703 INFO:tasks.cephadm: override: [osd] osd delete sleep = 1 2024-09-21T08:18:12.703 INFO:tasks.cephadm: override: [osd] osd max backfills = 3 2024-09-21T08:18:12.703 INFO:tasks.cephadm: override: [osd] osd scrub max interval = 120 2024-09-21T08:18:12.703 INFO:tasks.cephadm: override: [osd] osd scrub min interval = 60 2024-09-21T08:18:12.703 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-09-21T08:18:12.703 INFO:tasks.cephadm: override: [osd] osd snap trim sleep = 2 2024-09-21T08:18:12.703 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:18:12.703 DEBUG:teuthology.orchestra.run.smithi137:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-09-21T08:18:12.719 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = ec8566d6-77f1-11ef-baf6-efdc52797490 ms bind msgr2 = False ms type = async [osd] osd scrub load threshold = 5.0 osd scrub max interval = 120 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true debug ms = 1 debug osd = 20 osd debug reject backfill probability = 0.3 osd delete sleep = 1 osd max backfills = 3 osd scrub min interval = 60 osd snap trim sleep = 2 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon min osdmap epochs = 50 mon osdmap full prune interval = 2 mon osdmap full prune min = 15 mon osdmap full prune txsize = 2 paxos service trim min = 10 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true 2024-09-21T08:18:12.719 DEBUG:teuthology.orchestra.run.smithi137:mon.a> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mon.a.service 2024-09-21T08:18:12.761 DEBUG:teuthology.orchestra.run.smithi137:mgr.y> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mgr.y.service 2024-09-21T08:18:12.803 INFO:tasks.cephadm:Bootstrapping... 2024-09-21T08:18:12.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 -v bootstrap --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id y --orphan-initial-daemons --skip-monitoring-stack --mon-addrv '[v1:172.21.15.137:6789]' --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-09-21T08:18:13.003 INFO:teuthology.orchestra.run.smithi137.stdout:-------------------------------------------------------------------------------- 2024-09-21T08:18:13.003 INFO:teuthology.orchestra.run.smithi137.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736', '-v', 'bootstrap', '--fsid', 'ec8566d6-77f1-11ef-baf6-efdc52797490', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'y', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-addrv', '[v1:172.21.15.137:6789]', '--skip-admin-label'] 2024-09-21T08:18:13.003 INFO:teuthology.orchestra.run.smithi137.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-09-21T08:18:13.004 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying podman|docker is present... 2024-09-21T08:18:13.027 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stdout 5.2.2 2024-09-21T08:18:13.027 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying lvm2 is present... 2024-09-21T08:18:13.027 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying time synchronization is in place... 2024-09-21T08:18:13.035 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T08:18:13.035 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T08:18:13.041 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T08:18:13.041 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-21T08:18:13.048 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout enabled 2024-09-21T08:18:13.054 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout active 2024-09-21T08:18:13.054 INFO:teuthology.orchestra.run.smithi137.stdout:Unit chronyd.service is enabled and running 2024-09-21T08:18:13.054 INFO:teuthology.orchestra.run.smithi137.stdout:Repeating the final host check... 2024-09-21T08:18:13.076 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stdout 5.2.2 2024-09-21T08:18:13.076 INFO:teuthology.orchestra.run.smithi137.stdout:podman (/bin/podman) version 5.2.2 is present 2024-09-21T08:18:13.076 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl is present 2024-09-21T08:18:13.076 INFO:teuthology.orchestra.run.smithi137.stdout:lvcreate is present 2024-09-21T08:18:13.083 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-09-21T08:18:13.083 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-09-21T08:18:13.089 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-09-21T08:18:13.089 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-21T08:18:13.095 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout enabled 2024-09-21T08:18:13.102 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout active 2024-09-21T08:18:13.102 INFO:teuthology.orchestra.run.smithi137.stdout:Unit chronyd.service is enabled and running 2024-09-21T08:18:13.102 INFO:teuthology.orchestra.run.smithi137.stdout:Host looks OK 2024-09-21T08:18:13.102 INFO:teuthology.orchestra.run.smithi137.stdout:Cluster fsid: ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:13.102 INFO:teuthology.orchestra.run.smithi137.stdout:Acquiring lock 140078518797312 on /run/cephadm/ec8566d6-77f1-11ef-baf6-efdc52797490.lock 2024-09-21T08:18:13.102 INFO:teuthology.orchestra.run.smithi137.stdout:Lock 140078518797312 acquired on /run/cephadm/ec8566d6-77f1-11ef-baf6-efdc52797490.lock 2024-09-21T08:18:13.103 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying IP 172.21.15.137 port 6789 ... 2024-09-21T08:18:13.103 INFO:teuthology.orchestra.run.smithi137.stdout:Base mon IP(s) is [172.21.15.137:6789], mon addrv is [v1:172.21.15.137:6789] 2024-09-21T08:18:13.106 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.137 metric 100 2024-09-21T08:18:13.106 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.137 metric 100 2024-09-21T08:18:13.109 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-09-21T08:18:13.109 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 1024 pref medium 2024-09-21T08:18:13.109 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 100 pref medium 2024-09-21T08:18:13.111 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-09-21T08:18:13.111 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-09-21T08:18:13.111 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T08:18:13.111 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-09-21T08:18:13.111 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe88:8137/64 scope link noprefixroute 2024-09-21T08:18:13.111 INFO:teuthology.orchestra.run.smithi137.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-09-21T08:18:13.112 INFO:teuthology.orchestra.run.smithi137.stdout:Mon IP `172.21.15.137` is in CIDR network `172.21.0.0/20` 2024-09-21T08:18:13.112 INFO:teuthology.orchestra.run.smithi137.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20'] 2024-09-21T08:18:13.112 INFO:teuthology.orchestra.run.smithi137.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-09-21T08:18:13.113 INFO:teuthology.orchestra.run.smithi137.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736... 2024-09-21T08:18:13.548 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stdout cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd 2024-09-21T08:18:13.549 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736... 2024-09-21T08:18:13.549 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Getting image source signatures 2024-09-21T08:18:13.549 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Copying blob sha256:0031407101cecb3ed6ba394cb16e83fe515e11366eae1f45bd4759e577ea92fc 2024-09-21T08:18:13.549 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Copying blob sha256:e8b54c863393b7a8216a71737771b73b16a8e43ec7acf927c7faa175c255e551 2024-09-21T08:18:13.549 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Copying config sha256:cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd 2024-09-21T08:18:13.549 INFO:teuthology.orchestra.run.smithi137.stdout:/bin/podman: stderr Writing manifest to image destination 2024-09-21T08:18:14.485 INFO:teuthology.orchestra.run.smithi137.stdout:ceph: stdout ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable) 2024-09-21T08:18:14.485 INFO:teuthology.orchestra.run.smithi137.stdout:Ceph version: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable) 2024-09-21T08:18:14.485 INFO:teuthology.orchestra.run.smithi137.stdout:Extracting ceph user uid/gid from container image... 2024-09-21T08:18:15.532 INFO:teuthology.orchestra.run.smithi137.stdout:stat: stdout 167 167 2024-09-21T08:18:15.533 INFO:teuthology.orchestra.run.smithi137.stdout:Creating initial keys... 2024-09-21T08:18:16.513 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph-authtool: stdout AQBHge5mjcyfMxAAo8M2nJqQ2sjer0exY8TFnQ== 2024-09-21T08:18:17.560 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph-authtool: stdout AQBIge5mLdH0NBAA+lrr0VljOX+alYK8dV8PmQ== 2024-09-21T08:18:18.566 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph-authtool: stdout AQBJge5mmk/ANBAAtcubPGXgr2qafV5FEKQBdw== 2024-09-21T08:18:18.567 INFO:teuthology.orchestra.run.smithi137.stdout:Creating initial monmap... 2024-09-21T08:18:19.488 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T08:18:19.488 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-09-21T08:18:19.488 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:19.488 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T08:18:19.489 INFO:teuthology.orchestra.run.smithi137.stdout:monmaptool for a [v1:172.21.15.137:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-09-21T08:18:19.489 INFO:teuthology.orchestra.run.smithi137.stdout:setting min_mon_release = quincy 2024-09-21T08:18:19.489 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: set fsid to ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:19.489 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-09-21T08:18:19.489 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:18:19.489 INFO:teuthology.orchestra.run.smithi137.stdout:Creating mon... 2024-09-21T08:18:20.427 INFO:teuthology.orchestra.run.smithi137.stdout:create mon.a on 2024-09-21T08:18:20.621 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-09-21T08:18:20.758 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-09-21T08:18:20.950 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-ec8566d6-77f1-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-ec8566d6-77f1-11ef-baf6-efdc52797490.target. 2024-09-21T08:18:20.950 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-ec8566d6-77f1-11ef-baf6-efdc52797490.target → /etc/systemd/system/ceph-ec8566d6-77f1-11ef-baf6-efdc52797490.target. 2024-09-21T08:18:21.174 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mon.a 2024-09-21T08:18:21.174 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mon.a.service: Unit ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mon.a.service not loaded. 2024-09-21T08:18:21.334 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ec8566d6-77f1-11ef-baf6-efdc52797490.target.wants/ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mon.a.service → /etc/systemd/system/ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@.service. 2024-09-21T08:18:21.838 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:21 smithi137 ceph-mon[29161]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T08:18:21.838 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:21 smithi137 bash[29122]: 9fbd1ba37bb1d213678d4717c5090fc1c83d789b4fad865a8d939456e095646b 2024-09-21T08:18:21.850 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T08:18:21.851 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout disabled 2024-09-21T08:18:21.857 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T08:18:21.858 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-21T08:18:21.858 INFO:teuthology.orchestra.run.smithi137.stdout:firewalld.service is not enabled 2024-09-21T08:18:21.858 INFO:teuthology.orchestra.run.smithi137.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T08:18:21.858 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mon to start... 2024-09-21T08:18:21.858 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mon... 2024-09-21T08:18:22.089 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:21 smithi137 systemd[1]: Started Ceph mon.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:18:22.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout cluster: 2024-09-21T08:18:22.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout id: ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:22.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-09-21T08:18:22.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:22.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout services: 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.523404s) 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout data: 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout pgs: 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:mon is available 2024-09-21T08:18:22.823 INFO:teuthology.orchestra.run.smithi137.stdout:Assimilating anything we can from ceph.conf... 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: monmap epoch 1 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: last_changed 2024-09-21T08:18:18.964811+0000 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: created 2024-09-21T08:18:18.964811+0000 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: min_mon_release 19 (squid) 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: election_strategy: 1 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: fsmap 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: mgrmap e1: no daemons active 2024-09-21T08:18:22.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:22 smithi137 ceph-mon[29161]: from='client.? v1:172.21.15.137:0/3957555954' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-09-21T08:18:23.845 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:23.845 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T08:18:23.845 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout fsid = ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:23.845 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.137:6789] 2024-09-21T08:18:23.845 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T08:18:23.845 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T08:18:23.845 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T08:18:23.845 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T08:18:23.846 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:23.846 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T08:18:23.846 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T08:18:23.846 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:23.846 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T08:18:23.846 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T08:18:23.846 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T08:18:23.846 INFO:teuthology.orchestra.run.smithi137.stdout:Generating new minimal ceph.conf... 2024-09-21T08:18:23.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:23 smithi137 ceph-mon[29161]: from='client.? v1:172.21.15.137:0/43721947' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T08:18:23.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:23 smithi137 ceph-mon[29161]: from='client.? v1:172.21.15.137:0/43721947' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-09-21T08:18:24.768 INFO:teuthology.orchestra.run.smithi137.stdout:Restarting the monitor... 2024-09-21T08:18:25.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:24 smithi137 systemd[1]: Stopping Ceph mon.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:18:25.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:24 smithi137 ceph-mon[29161]: from='client.? v1:172.21.15.137:0/2647119995' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:18:25.483 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:25 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29138]: 2024-09-21T08:18:25.051+0000 7fb8535de640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T08:18:25.484 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:25 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29138]: 2024-09-21T08:18:25.051+0000 7fb8535de640 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-09-21T08:18:25.484 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:25 smithi137 podman[29480]: 2024-09-21 08:18:25.235535347 +0000 UTC m=+0.293942757 container died 9fbd1ba37bb1d213678d4717c5090fc1c83d789b4fad865a8d939456e095646b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T08:18:25.759 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:25 smithi137 podman[29480]: 2024-09-21 08:18:25.574036068 +0000 UTC m=+0.632443475 container cleanup 9fbd1ba37bb1d213678d4717c5090fc1c83d789b4fad865a8d939456e095646b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731) 2024-09-21T08:18:25.759 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:25 smithi137 bash[29480]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a 2024-09-21T08:18:26.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:25 smithi137 podman[29498]: 2024-09-21 08:18:25.758404063 +0000 UTC m=+0.521343819 container remove 9fbd1ba37bb1d213678d4717c5090fc1c83d789b4fad865a8d939456e095646b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-09-21T08:18:26.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:25 smithi137 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mon.a.service: Deactivated successfully. 2024-09-21T08:18:26.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:25 smithi137 systemd[1]: Stopped Ceph mon.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:18:26.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:25 smithi137 systemd[1]: Starting Ceph mon.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:18:26.445 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 podman[29597]: 2024-09-21 08:18:26.074457175 +0000 UTC m=+0.018569866 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-21T08:18:26.445 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 podman[29597]: 2024-09-21 08:18:26.184329544 +0000 UTC m=+0.128442266 container create 5bccbe3cdb395138674940aa78ff24d031055bfb53b1641f7a30518c51f77ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-09-21T08:18:26.446 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 podman[29597]: 2024-09-21 08:18:26.36975519 +0000 UTC m=+0.313867866 container init 5bccbe3cdb395138674940aa78ff24d031055bfb53b1641f7a30518c51f77ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T08:18:26.446 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 podman[29597]: 2024-09-21 08:18:26.374431414 +0000 UTC m=+0.318544092 container start 5bccbe3cdb395138674940aa78ff24d031055bfb53b1641f7a30518c51f77ce1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70) 2024-09-21T08:18:26.446 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T08:18:26.446 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable), process ceph-mon, pid 7 2024-09-21T08:18:26.446 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: pidfile_write: ignore empty --pid-file 2024-09-21T08:18:26.446 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: load: jerasure load: lrc 2024-09-21T08:18:26.446 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: RocksDB version: 7.9.2 2024-09-21T08:18:26.446 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Git sha 0 2024-09-21T08:18:26.446 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Compile date 2024-09-19 18:20:48 2024-09-21T08:18:26.446 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: DB SUMMARY 2024-09-21T08:18:26.447 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: DB Session ID: 6JOWLQ901NLGNTAQI4LW 2024-09-21T08:18:26.447 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: CURRENT file: CURRENT 2024-09-21T08:18:26.447 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T08:18:26.447 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-09-21T08:18:26.447 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-09-21T08:18:26.447 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 91553 ; 2024-09-21T08:18:26.447 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.error_if_exists: 0 2024-09-21T08:18:26.447 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.create_if_missing: 0 2024-09-21T08:18:26.447 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.paranoid_checks: 1 2024-09-21T08:18:26.447 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.env: 0x5599df542940 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.info_log: 0x5599e13df680 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.statistics: (nil) 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.use_fsync: 0 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_log_file_size: 0 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T08:18:26.448 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.allow_fallocate: 1 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.use_direct_reads: 0 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.db_log_dir: 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.wal_dir: 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T08:18:26.449 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T08:18:26.450 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T08:18:26.450 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T08:18:26.450 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T08:18:26.450 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.write_buffer_manager: 0x5599e13eedc0 2024-09-21T08:18:26.450 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T08:18:26.450 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T08:18:26.450 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T08:18:26.450 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T08:18:26.450 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T08:18:26.450 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T08:18:26.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T08:18:26.451 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.unordered_write: 0 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.row_cache: None 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.wal_filter: None 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.two_write_queues: 0 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.wal_compression: 0 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.atomic_flush: 0 2024-09-21T08:18:26.452 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.log_readahead_size: 0 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_background_jobs: 2 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_background_compactions: -1 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_subcompactions: 1 2024-09-21T08:18:26.453 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_open_files: -1 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_background_flushes: -1 2024-09-21T08:18:26.454 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Compression algorithms supported: 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: kZSTD supported: 0 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: kXpressCompression supported: 0 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: kBZip2Compression supported: 0 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: kLZ4Compression supported: 1 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: kZlibCompression supported: 1 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: kSnappyCompression supported: 1 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.merge_operator: 2024-09-21T08:18:26.455 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_filter: None 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5599e13df2c0) 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout: cache_index_and_filter_blocks: 1 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout: pin_top_level_index_and_filter: 1 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout: index_type: 0 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout: data_block_index_type: 0 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout: index_shortening: 1 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T08:18:26.456 INFO:journalctl@ceph.mon.a.smithi137.stdout: checksum: 4 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: no_block_cache: 0 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_cache: 0x5599e13d31f0 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_cache_name: BinnedLRUCache 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_cache_options: 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: capacity : 536870912 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: num_shard_bits : 4 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: strict_capacity_limit : 0 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: high_pri_pool_ratio: 0.000 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_cache_compressed: (nil) 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: persistent_cache: (nil) 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_size: 4096 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_size_deviation: 10 2024-09-21T08:18:26.457 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_restart_interval: 16 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: index_block_restart_interval: 1 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: metadata_block_size: 4096 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: partition_filters: 0 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: use_delta_encoding: 1 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: filter_policy: bloomfilter 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: whole_key_filtering: 1 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: verify_compression: 0 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: read_amp_bytes_per_bit: 0 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: format_version: 5 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: enable_index_compression: 1 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: block_align: 0 2024-09-21T08:18:26.458 INFO:journalctl@ceph.mon.a.smithi137.stdout: max_auto_readahead_size: 262144 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout: prepopulate_block_cache: 0 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout: initial_auto_readahead_size: 8192 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compression: NoCompression 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.num_levels: 7 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T08:18:26.459 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T08:18:26.460 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T08:18:26.461 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T08:18:26.462 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.inplace_update_support: 0 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.bloom_locality: 0 2024-09-21T08:18:26.463 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.max_successive_merges: 0 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.ttl: 2592000 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.enable_blob_files: false 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.min_blob_size: 0 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T08:18:26.464 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9ea5804b-5339-4f2a-8f8c-e0e0e1d00f15 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726906706400567, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-09-21T08:18:26.465 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726906706401459, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 87627, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 280, "table_properties": {"data_size": 85704, "index_size": 243, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 645, "raw_key_size": 11532, "raw_average_key_size": 48, "raw_value_size": 79309, "raw_average_value_size": 331, "num_data_blocks": 10, "num_entries": 239, "num_filter_entries": 239, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726906706, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9ea5804b-5339-4f2a-8f8c-e0e0e1d00f15", "db_session_id": "6JOWLQ901NLGNTAQI4LW", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726906706401554, "job": 1, "event": "recovery_finished"} 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5599e1400e00 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: DB pointer 0x5599e14fc000 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout: ** DB Stats ** 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T08:18:26.466 INFO:journalctl@ceph.mon.a.smithi137.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: L0 2/0 87.40 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: Sum 2/0 87.40 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 149.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 149.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-09-21T08:18:26.467 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: Cumulative compaction: 0.00 GB write, 27.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: Interval compaction: 0.00 GB write, 27.32 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: Block cache BinnedLRUCache@0x5599e13d31f0#7 capacity: 512.00 MB usage: 1.27 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T08:18:26.468 INFO:journalctl@ceph.mon.a.smithi137.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.83 KB,0.000157952%) IndexBlock(2,0.44 KB,8.34465e-05%) Misc(1,0.00 KB,0%) 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: starting mon.a rank 0 at public addrs v1:172.21.15.137:6789/0 at bind addrs v1:172.21.15.137:6789/0 mon_data /var/lib/ceph/mon/ceph-a fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: mon.a@-1(???) e1 preinit fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: mon.a@-1(???).mds e1 new map 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: mon.a@-1(???).mds e1 print_map 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout: e1 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout: btime 2024-09-21T08:18:21:794533+0000 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout: legacy client fscid: -1 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout: 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout: No filesystems configured 2024-09-21T08:18:26.469 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T08:18:26.470 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T08:18:26.470 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T08:18:26.470 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T08:18:26.470 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-09-21T08:18:26.470 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 bash[29597]: 5bccbe3cdb395138674940aa78ff24d031055bfb53b1641f7a30518c51f77ce1 2024-09-21T08:18:26.470 INFO:teuthology.orchestra.run.smithi137.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-09-21T08:18:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 systemd[1]: Started Ceph mon.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:18:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: mon.a is new leader, mons a in quorum (ranks 0) 2024-09-21T08:18:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: monmap epoch 1 2024-09-21T08:18:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: last_changed 2024-09-21T08:18:18.964811+0000 2024-09-21T08:18:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: created 2024-09-21T08:18:18.964811+0000 2024-09-21T08:18:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: min_mon_release 19 (squid) 2024-09-21T08:18:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: election_strategy: 1 2024-09-21T08:18:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-21T08:18:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: fsmap 2024-09-21T08:18:26.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: osdmap e1: 0 total, 0 up, 0 in 2024-09-21T08:18:26.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:26 smithi137 ceph-mon[29634]: mgrmap e1: no daemons active 2024-09-21T08:18:27.412 INFO:teuthology.orchestra.run.smithi137.stdout:Wrote config to /etc/ceph/ceph.conf 2024-09-21T08:18:27.434 INFO:teuthology.orchestra.run.smithi137.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-09-21T08:18:27.434 INFO:teuthology.orchestra.run.smithi137.stdout:Creating mgr... 2024-09-21T08:18:27.435 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying port 0.0.0.0:9283 ... 2024-09-21T08:18:27.435 INFO:teuthology.orchestra.run.smithi137.stdout:Verifying port 0.0.0.0:8765 ... 2024-09-21T08:18:27.685 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mgr.y 2024-09-21T08:18:27.685 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Failed to reset failed state of unit ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mgr.y.service: Unit ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mgr.y.service not loaded. 2024-09-21T08:18:27.824 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-ec8566d6-77f1-11ef-baf6-efdc52797490.target.wants/ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mgr.y.service → /etc/systemd/system/ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@.service. 2024-09-21T08:18:28.155 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:27 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3993675581' entity='client.admin' 2024-09-21T08:18:28.155 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:28 smithi137 podman[29872]: 2024-09-21 08:18:27.955521933 +0000 UTC m=+0.016528267 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 2024-09-21T08:18:28.677 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:28 smithi137 podman[29872]: 2024-09-21 08:18:28.310974084 +0000 UTC m=+0.371980418 container init fffa2d34afdfb91dab2eb27fa58fa2d36fc2fba29f4e698741280b7b02062197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-09-21T08:18:28.678 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:28 smithi137 podman[29872]: 2024-09-21 08:18:28.315591608 +0000 UTC m=+0.376597933 container start fffa2d34afdfb91dab2eb27fa58fa2d36fc2fba29f4e698741280b7b02062197 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T08:18:28.678 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:28 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:28.471+0000 7f9bc3ae1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T08:18:28.678 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:28 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:28.528+0000 7f9bc3ae1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T08:18:28.678 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:28 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:28.676+0000 7f9bc3ae1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T08:18:29.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:28 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:28.742+0000 7f9bc3ae1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T08:18:29.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:28 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:28.889+0000 7f9bc3ae1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:18:29.323 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:29 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:29.028+0000 7f9bc3ae1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T08:18:29.337 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T08:18:29.337 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout disabled 2024-09-21T08:18:29.345 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T08:18:29.345 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-21T08:18:29.345 INFO:teuthology.orchestra.run.smithi137.stdout:firewalld.service is not enabled 2024-09-21T08:18:29.345 INFO:teuthology.orchestra.run.smithi137.stdout:Not possible to enable service . firewalld.service is not available 2024-09-21T08:18:29.353 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T08:18:29.353 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout disabled 2024-09-21T08:18:29.360 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T08:18:29.361 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-21T08:18:29.361 INFO:teuthology.orchestra.run.smithi137.stdout:firewalld.service is not enabled 2024-09-21T08:18:29.361 INFO:teuthology.orchestra.run.smithi137.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-09-21T08:18:29.361 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mgr to start... 2024-09-21T08:18:29.361 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mgr... 2024-09-21T08:18:29.597 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:29 smithi137 bash[29872]: fffa2d34afdfb91dab2eb27fa58fa2d36fc2fba29f4e698741280b7b02062197 2024-09-21T08:18:29.597 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:29 smithi137 systemd[1]: Started Ceph mgr.y for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:18:29.597 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:29 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:29.345+0000 7f9bc3ae1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T08:18:29.597 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:29 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:29.411+0000 7f9bc3ae1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T08:18:29.597 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:29 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:29.595+0000 7f9bc3ae1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T08:18:30.767 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:30.474+0000 7f9bc3ae1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T08:18:30.768 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:30.543+0000 7f9bc3ae1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T08:18:31.518 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:31 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:31.262+0000 7f9bc3ae1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T08:18:31.518 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:31 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:31.377+0000 7f9bc3ae1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T08:18:31.807 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:31 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:31.530+0000 7f9bc3ae1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:18:31.807 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:31 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:31.590+0000 7f9bc3ae1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T08:18:31.807 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:31 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:31.738+0000 7f9bc3ae1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T08:18:31.807 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:31 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:31.805+0000 7f9bc3ae1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T08:18:32.092 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:31 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:31.865+0000 7f9bc3ae1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T08:18:32.092 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:32 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:32.024+0000 7f9bc3ae1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T08:18:32.092 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:32 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:32.091+0000 7f9bc3ae1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T08:18:32.356 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:32 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:32.156+0000 7f9bc3ae1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T08:18:32.356 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:32 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:32.239+0000 7f9bc3ae1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T08:18:32.356 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:32 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:32.297+0000 7f9bc3ae1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T08:18:32.356 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:32 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:32.355+0000 7f9bc3ae1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T08:18:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:32 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2805193590' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T08:18:33.518 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:33.163+0000 7f9bc3ae1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T08:18:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: Activating manager daemon y 2024-09-21T08:18:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: mgrmap e2: y(active, starting, since 0.00283022s) 2024-09-21T08:18:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T08:18:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T08:18:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T08:18:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T08:18:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T08:18:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T08:18:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:18:34.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T08:18:34.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: Manager daemon y is now available 2024-09-21T08:18:34.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T08:18:34.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T08:18:34.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' 2024-09-21T08:18:34.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' 2024-09-21T08:18:34.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:33 smithi137 ceph-mon[29634]: from='mgr.14100 v1:172.21.15.137:0/710538599' entity='mgr.y' 2024-09-21T08:18:35.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:35 smithi137 ceph-mon[29634]: mgrmap e3: y(active, since 1.00757s) 2024-09-21T08:18:36.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:36 smithi137 ceph-mon[29634]: mgrmap e4: y(active, since 2s) 2024-09-21T08:18:38.711 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:38.711 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-21T08:18:38.711 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "fsid": "ec8566d6-77f1-11ef-baf6-efdc52797490", 2024-09-21T08:18:38.711 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T08:18:38.711 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T08:18:38.711 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T08:18:38.711 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T08:18:38.711 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:38.711 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 0 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum_age": 6, 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T08:18:38.712 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T08:18:38.713 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T08:18:38.713 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T08:18:38.713 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T08:18:38.713 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T08:18:38.716 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T08:18:21:794533+0000", 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "available": false, 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T08:18:38.717 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T08:18:38.718 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T08:18:21.795287+0000", 2024-09-21T08:18:38.718 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T08:18:38.718 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:38.718 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T08:18:38.718 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-21T08:18:38.718 INFO:teuthology.orchestra.run.smithi137.stdout:mgr not available, waiting (1/15)... 2024-09-21T08:18:41.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:41 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1943714371' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-09-21T08:18:41.927 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:41.927 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-21T08:18:41.927 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "fsid": "ec8566d6-77f1-11ef-baf6-efdc52797490", 2024-09-21T08:18:41.927 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "health": { 2024-09-21T08:18:41.927 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-09-21T08:18:41.927 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-09-21T08:18:41.927 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 0 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "a" 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "monmap": { 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:41.928 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-09-21T08:18:41.929 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-09-21T08:18:41.931 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "btime": "2024-09-21T08:18:21:794533+0000", 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "modules": [ 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "iostat", 2024-09-21T08:18:41.932 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "nfs", 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "restful" 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ], 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "modified": "2024-09-21T08:18:21.795287+0000", 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "services": {} 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout }, 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-21T08:18:41.933 INFO:teuthology.orchestra.run.smithi137.stdout:mgr is available 2024-09-21T08:18:42.830 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:42 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4087997377' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-09-21T08:18:43.067 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:43.067 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [global] 2024-09-21T08:18:43.067 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout fsid = ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:43.067 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_host = [v1:172.21.15.137:6789] 2024-09-21T08:18:43.067 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-09-21T08:18:43.067 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-09-21T08:18:43.067 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-09-21T08:18:43.067 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-09-21T08:18:43.067 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:43.068 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [mgr] 2024-09-21T08:18:43.068 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-09-21T08:18:43.068 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 2024-09-21T08:18:43.068 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout [osd] 2024-09-21T08:18:43.068 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-09-21T08:18:43.068 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-09-21T08:18:43.068 INFO:teuthology.orchestra.run.smithi137.stdout:Enabling cephadm module... 2024-09-21T08:18:44.170 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:43 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1933741819' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-09-21T08:18:44.171 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:43 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ignoring --setuser ceph since I am not root 2024-09-21T08:18:44.171 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:43 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ignoring --setgroup ceph since I am not root 2024-09-21T08:18:44.171 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:43 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:43.966+0000 7fed77c60040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T08:18:44.171 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:44 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:44.022+0000 7fed77c60040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T08:18:44.171 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:44 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:44.168+0000 7fed77c60040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T08:18:44.517 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:44 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:44.235+0000 7fed77c60040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T08:18:44.518 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:44 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:44.380+0000 7fed77c60040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:18:44.827 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:44 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:44.518+0000 7fed77c60040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T08:18:45.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:44 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1933741819' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-09-21T08:18:45.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:44 smithi137 ceph-mon[29634]: mgrmap e5: y(active, since 10s) 2024-09-21T08:18:45.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:44 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:44.825+0000 7fed77c60040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T08:18:45.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:44 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:44.887+0000 7fed77c60040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T08:18:45.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:45 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:45.071+0000 7fed77c60040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T08:18:46.267 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:45 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:45.942+0000 7fed77c60040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T08:18:46.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:46 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:46.009+0000 7fed77c60040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T08:18:46.864 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:46 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:46.758+0000 7fed77c60040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T08:18:47.215 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:46 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2717301340' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T08:18:47.215 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:46 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:46.885+0000 7fed77c60040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T08:18:47.216 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.028+0000 7fed77c60040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:18:47.216 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.095+0000 7fed77c60040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T08:18:47.418 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-21T08:18:47.418 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-09-21T08:18:47.419 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T08:18:47.419 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T08:18:47.419 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T08:18:47.419 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-21T08:18:47.419 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for the mgr to restart... 2024-09-21T08:18:47.419 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mgr epoch 5... 2024-09-21T08:18:47.517 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.246+0000 7fed77c60040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T08:18:47.518 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.314+0000 7fed77c60040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T08:18:47.518 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.375+0000 7fed77c60040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T08:18:47.802 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.539+0000 7fed77c60040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T08:18:47.802 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.605+0000 7fed77c60040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T08:18:47.802 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.675+0000 7fed77c60040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T08:18:47.802 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.761+0000 7fed77c60040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T08:18:48.267 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.821+0000 7fed77c60040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T08:18:48.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:47.884+0000 7fed77c60040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T08:18:49.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:48.671+0000 7fed77c60040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T08:18:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: Active manager daemon y restarted 2024-09-21T08:18:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: Activating manager daemon y 2024-09-21T08:18:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: osdmap e2: 0 total, 0 up, 0 in 2024-09-21T08:18:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: mgrmap e6: y(active, starting, since 0.0033696s) 2024-09-21T08:18:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:18:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T08:18:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T08:18:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T08:18:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T08:18:49.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: Manager daemon y is now available 2024-09-21T08:18:49.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:18:49.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:18:49.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:18:49.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:18:49.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T08:18:49.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:48 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T08:18:50.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:49 smithi137 ceph-mon[29634]: Found migration_current of "None". Setting to last migration. 2024-09-21T08:18:50.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:49 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:18:50.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:49 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:18:50.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:49 smithi137 ceph-mon[29634]: mgrmap e7: y(active, since 1.0053s) 2024-09-21T08:18:50.691 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-21T08:18:50.691 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-09-21T08:18:50.692 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T08:18:50.692 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-21T08:18:50.692 INFO:teuthology.orchestra.run.smithi137.stdout:mgr epoch 5 is available 2024-09-21T08:18:50.692 INFO:teuthology.orchestra.run.smithi137.stdout:Setting orchestrator backend to cephadm... 2024-09-21T08:18:51.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:50 smithi137 ceph-mon[29634]: from='client.14120 v1:172.21.15.137:0/633164148' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T08:18:51.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:50 smithi137 ceph-mon[29634]: from='client.14120 v1:172.21.15.137:0/633164148' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T08:18:51.853 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-mon[29634]: mgrmap e8: y(active, since 2s) 2024-09-21T08:18:51.854 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:18:51.854 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:18:51.854 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:18:51.854 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:51.606+0000 7fece4785640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:08:18:51] ENGINE Error in HTTPServer.serve 2024-09-21T08:18:51.854 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Traceback (most recent call last): 2024-09-21T08:18:51.854 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T08:18:51.854 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self._connections.run(self.expiration_interval) 2024-09-21T08:18:51.854 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self._run(expiration_interval) 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: s = self.context.wrap_socket( 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: return self.sslsocket_class._create( 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self.do_handshake() 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self._sslobj.do_handshake() 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T08:18:51.855 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:51 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:18:52.963 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout value unchanged 2024-09-21T08:18:52.963 INFO:teuthology.orchestra.run.smithi137.stdout:Generating ssh key... 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:52 smithi137 ceph-mon[29634]: from='client.14128 v1:172.21.15.137:0/3144418980' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:52 smithi137 ceph-mon[29634]: [21/Sep/2024:08:18:51] ENGINE Bus STARTING 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:52 smithi137 ceph-mon[29634]: [21/Sep/2024:08:18:51] ENGINE Serving on https://172.21.15.137:7150 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:52 smithi137 ceph-mon[29634]: [21/Sep/2024:08:18:51] ENGINE Error in HTTPServer.serve 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout: Traceback (most recent call last): 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._run(expiration_interval) 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T08:18:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: s = self.context.wrap_socket( 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: return self.sslsocket_class._create( 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: self.do_handshake() 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._sslobj.do_handshake() 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:52 smithi137 ceph-mon[29634]: [21/Sep/2024:08:18:51] ENGINE Serving on http://172.21.15.137:8765 2024-09-21T08:18:53.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:52 smithi137 ceph-mon[29634]: [21/Sep/2024:08:18:51] ENGINE Bus STARTED 2024-09-21T08:18:54.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-mon[29634]: from='client.14130 v1:172.21.15.137:0/171885590' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:18:54.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Generating public/private rsa key pair. 2024-09-21T08:18:54.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Your identification has been saved in /tmp/tmpdi3h2jt3/key 2024-09-21T08:18:54.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Your public key has been saved in /tmp/tmpdi3h2jt3/key.pub 2024-09-21T08:18:54.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: The key fingerprint is: 2024-09-21T08:18:54.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: SHA256:PMj2ADbaTkLj+tvNezymPHx0mx64o6dxdalboUMoqCY ceph-ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:54.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: The key's randomart image is: 2024-09-21T08:18:54.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: +---[RSA 3072]----+ 2024-09-21T08:18:54.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: | | 2024-09-21T08:18:54.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: | | 2024-09-21T08:18:54.019 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: | o + | 2024-09-21T08:18:54.019 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: | o = = o . . | 2024-09-21T08:18:54.019 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: | + + * S o + | 2024-09-21T08:18:54.019 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: | . = . +.=.+ . | 2024-09-21T08:18:54.019 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: |E o ...o+.=o. | 2024-09-21T08:18:54.019 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: | + . +ooO.o= | 2024-09-21T08:18:54.019 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: | o.. O@.+o | 2024-09-21T08:18:54.019 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:18:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: +----[SHA256]-----+ 2024-09-21T08:18:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:54 smithi137 ceph-mon[29634]: from='client.14132 v1:172.21.15.137:0/3250582151' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:18:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:54 smithi137 ceph-mon[29634]: Generating ssh key... 2024-09-21T08:18:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:54 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:18:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:54 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:18:55.483 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQPhUs270xg9U3NAEqq3tFdm+nmYQf5PAhy84JcZKvWVHPRUYxMc9nnnxswApQ83PmKghLBMdNNVFGLahfQfdvqosxD0Qt063Y+00bQ/ZXPiB5AgQ1/EZGNKUO98/CHPOFy4xbiKhtEQEbHAO8v6N7sKnQoUPmRGkBBVL5rOgXppQUUrQMljI3NVjj3BP73fDZLwVRbih2ipu0Piw0I8RozSHuD9gQhN3RCwcTH4+pbTOCSKHWRwZVYZUuyilKyPDxj0wfFXcycMO2vwK3fMNbNyImgtajohc//+SlLpRUqIhA1/hAkdlgbyFBNe/gG10Y7/B0OiTO1qaw6eKSpEEMl70zxP43bbShJrmoSp0KkEWAHxO/rGLuOj9c1TJ0G5m7G3xv7M4u0w4bU/u3tVCx97glgX2OKzWFq1c5fUMD9e6biI0wVqdzcYIZ8RnrVVBb2Sd12VUDFVsNX9q3Xfjl8yglS8GhKxk0zAyuRCI6tVvrKcbJ7RqNqU4ZG0NduHU= ceph-ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:18:55.483 INFO:teuthology.orchestra.run.smithi137.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-09-21T08:18:55.483 INFO:teuthology.orchestra.run.smithi137.stdout:Adding key to root@localhost authorized_keys... 2024-09-21T08:18:55.498 INFO:teuthology.orchestra.run.smithi137.stdout:Adding host smithi137... 2024-09-21T08:18:56.254 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:55 smithi137 ceph-mon[29634]: from='client.14134 v1:172.21.15.137:0/2650671798' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:18:57.187 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:56 smithi137 ceph-mon[29634]: from='client.14136 v1:172.21.15.137:0/1768060425' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi137", "addr": "172.21.15.137", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:18:58.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:18:57 smithi137 ceph-mon[29634]: Deploying cephadm binary to smithi137 2024-09-21T08:18:59.848 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout Added host 'smithi137' with addr '172.21.15.137' 2024-09-21T08:18:59.849 INFO:teuthology.orchestra.run.smithi137.stdout:Deploying unmanaged mon service... 2024-09-21T08:19:00.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:00 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:19:00.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:00 smithi137 ceph-mon[29634]: Added host smithi137 2024-09-21T08:19:00.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:00 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:01.877 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:01 smithi137 ceph-mon[29634]: from='client.14138 v1:172.21.15.137:0/3760922231' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:19:01.877 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:01 smithi137 ceph-mon[29634]: Saving service mon spec with placement count:5 2024-09-21T08:19:01.877 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:01 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:19:02.031 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-09-21T08:19:02.031 INFO:teuthology.orchestra.run.smithi137.stdout:Deploying unmanaged mgr service... 2024-09-21T08:19:03.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:02 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:19:03.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:02 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:19:03.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:02 smithi137 ceph-mon[29634]: from='client.14140 v1:172.21.15.137:0/2916612329' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:19:03.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:02 smithi137 ceph-mon[29634]: Saving service mgr spec with placement count:2 2024-09-21T08:19:03.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:02 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:19:03.890 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-09-21T08:19:04.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:04 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:19:04.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:04 smithi137 ceph-mon[29634]: from='mgr.14116 v1:172.21.15.137:0/3030255384' entity='mgr.y' 2024-09-21T08:19:04.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:04 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2445537379' entity='client.admin' 2024-09-21T08:19:07.084 INFO:teuthology.orchestra.run.smithi137.stdout:Enabling the dashboard module... 2024-09-21T08:19:07.181 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:06 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/371973522' entity='client.admin' 2024-09-21T08:19:08.240 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:07 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3581769454' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-09-21T08:19:08.240 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:07 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ignoring --setuser ceph since I am not root 2024-09-21T08:19:08.240 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:07 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ignoring --setgroup ceph since I am not root 2024-09-21T08:19:08.240 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:08 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:08.033+0000 7fdbed287040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T08:19:08.240 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:08 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:08.089+0000 7fdbed287040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T08:19:08.240 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:08 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:08.238+0000 7fdbed287040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T08:19:08.512 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:08 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:08.306+0000 7fdbed287040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T08:19:08.513 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:08 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:08.456+0000 7fdbed287040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:19:08.767 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:08 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:08.600+0000 7fdbed287040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T08:19:09.180 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:08 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3581769454' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-09-21T08:19:09.180 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:08 smithi137 ceph-mon[29634]: mgrmap e9: y(active, since 19s) 2024-09-21T08:19:09.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:08 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:08.923+0000 7fdbed287040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T08:19:09.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:08 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:08.987+0000 7fdbed287040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T08:19:09.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:09.178+0000 7fdbed287040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T08:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-21T08:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-09-21T08:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "available": true, 2024-09-21T08:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "active_name": "y", 2024-09-21T08:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-09-21T08:19:09.822 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-21T08:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for the mgr to restart... 2024-09-21T08:19:09.823 INFO:teuthology.orchestra.run.smithi137.stdout:Waiting for mgr epoch 9... 2024-09-21T08:19:10.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:09 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1049911679' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-09-21T08:19:10.147 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:10.077+0000 7fdbed287040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T08:19:10.517 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:10.145+0000 7fdbed287040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T08:19:11.104 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:10.849+0000 7fdbed287040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T08:19:11.104 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:10.963+0000 7fdbed287040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T08:19:11.377 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.103+0000 7fdbed287040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:19:11.377 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.163+0000 7fdbed287040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T08:19:11.377 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.309+0000 7fdbed287040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T08:19:11.377 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.375+0000 7fdbed287040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T08:19:11.655 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.435+0000 7fdbed287040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T08:19:11.655 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.590+0000 7fdbed287040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T08:19:11.655 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.653+0000 7fdbed287040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T08:19:11.916 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.719+0000 7fdbed287040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T08:19:11.917 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.799+0000 7fdbed287040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T08:19:11.917 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.858+0000 7fdbed287040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T08:19:11.917 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:11.915+0000 7fdbed287040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T08:19:13.017 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:12.724+0000 7fdbed287040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: Active manager daemon y restarted 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: Activating manager daemon y 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: osdmap e3: 0 total, 0 up, 0 in 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: mgrmap e10: y(active, starting, since 0.00428795s) 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: Manager daemon y is now available 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:13.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T08:19:13.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:12 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T08:19:14.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:14.166+0000 7fdb5859b640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:08:19:14] ENGINE Error in HTTPServer.serve 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Traceback (most recent call last): 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self._connections.run(self.expiration_interval) 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self._run(expiration_interval) 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: s = self.context.wrap_socket( 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T08:19:14.269 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: return self.sslsocket_class._create( 2024-09-21T08:19:14.270 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T08:19:14.270 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self.do_handshake() 2024-09-21T08:19:14.270 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T08:19:14.270 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self._sslobj.do_handshake() 2024-09-21T08:19:14.270 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T08:19:14.270 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:19:14.557 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout { 2024-09-21T08:19:14.557 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-09-21T08:19:14.557 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout "initialized": true 2024-09-21T08:19:14.557 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout } 2024-09-21T08:19:14.557 INFO:teuthology.orchestra.run.smithi137.stdout:mgr epoch 9 is available 2024-09-21T08:19:14.557 INFO:teuthology.orchestra.run.smithi137.stdout:Generating a dashboard self-signed certificate... 2024-09-21T08:19:14.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-mon[29634]: from='client.14152 v1:172.21.15.137:0/3111744810' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-09-21T08:19:14.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-mon[29634]: mgrmap e11: y(active, since 1.00566s) 2024-09-21T08:19:14.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-mon[29634]: from='client.14152 v1:172.21.15.137:0/3111744810' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-09-21T08:19:14.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:14 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:15.827 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:15 smithi137 ceph-mon[29634]: [21/Sep/2024:08:19:14] ENGINE Bus STARTING 2024-09-21T08:19:15.827 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:15 smithi137 ceph-mon[29634]: [21/Sep/2024:08:19:14] ENGINE Serving on https://172.21.15.137:7150 2024-09-21T08:19:15.827 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:15 smithi137 ceph-mon[29634]: [21/Sep/2024:08:19:14] ENGINE Error in HTTPServer.serve 2024-09-21T08:19:15.827 INFO:journalctl@ceph.mon.a.smithi137.stdout: Traceback (most recent call last): 2024-09-21T08:19:15.827 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._run(expiration_interval) 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: s = self.context.wrap_socket( 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: return self.sslsocket_class._create( 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: self.do_handshake() 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._sslobj.do_handshake() 2024-09-21T08:19:15.828 INFO:journalctl@ceph.mon.a.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T08:19:15.829 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:15 smithi137 ceph-mon[29634]: [21/Sep/2024:08:19:14] ENGINE Serving on http://172.21.15.137:8765 2024-09-21T08:19:15.829 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:15 smithi137 ceph-mon[29634]: [21/Sep/2024:08:19:14] ENGINE Bus STARTED 2024-09-21T08:19:15.829 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:15 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:15.829 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:15 smithi137 ceph-mon[29634]: mgrmap e12: y(active, since 2s) 2024-09-21T08:19:15.829 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:15 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:15.829 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:15 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:15.829 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-09-21T08:19:15.829 INFO:teuthology.orchestra.run.smithi137.stdout:Creating initial admin user... 2024-09-21T08:19:17.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:16 smithi137 ceph-mon[29634]: from='client.14160 v1:172.21.15.137:0/1183017337' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:19:17.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:17.759 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$8ztiz6IOh0sj3MkynCjvye8iQlWiSmgO1tFhkN.Q87bH7Qs21Rtuq", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1726906756, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-09-21T08:19:17.759 INFO:teuthology.orchestra.run.smithi137.stdout:Fetching dashboard port number... 2024-09-21T08:19:17.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:17 smithi137 ceph-mon[29634]: from='client.14162 v1:172.21.15.137:0/1503395380' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:19:18.939 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stdout 8443 2024-09-21T08:19:18.949 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-09-21T08:19:18.949 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout disabled 2024-09-21T08:19:18.956 INFO:teuthology.orchestra.run.smithi137.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-09-21T08:19:18.956 INFO:teuthology.orchestra.run.smithi137.stdout:systemctl: stdout inactive 2024-09-21T08:19:18.956 INFO:teuthology.orchestra.run.smithi137.stdout:firewalld.service is not enabled 2024-09-21T08:19:18.956 INFO:teuthology.orchestra.run.smithi137.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-09-21T08:19:18.958 INFO:teuthology.orchestra.run.smithi137.stdout:Ceph Dashboard is now available at: 2024-09-21T08:19:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: URL: https://smithi137.front.sepia.ceph.com:8443/ 2024-09-21T08:19:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: User: admin 2024-09-21T08:19:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: Password: 39gc9m8jby 2024-09-21T08:19:18.958 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:18.958 INFO:teuthology.orchestra.run.smithi137.stdout:Saving cluster configuration to /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config directory 2024-09-21T08:19:19.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:18 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3404924604' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-09-21T08:19:19.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:19 smithi137 ceph-mon[29634]: mgrmap e13: y(active, since 6s) 2024-09-21T08:19:19.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:19 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4163916859' entity='client.admin' 2024-09-21T08:19:20.188 INFO:teuthology.orchestra.run.smithi137.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-09-21T08:19:20.188 INFO:teuthology.orchestra.run.smithi137.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-09-21T08:19:20.188 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:20.188 INFO:teuthology.orchestra.run.smithi137.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout:Or, if you are only running a single cluster on this host: 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout:Please consider enabling telemetry to help improve Ceph: 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout: ceph telemetry on 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout:For more information see: 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:20.189 INFO:teuthology.orchestra.run.smithi137.stdout:Bootstrap complete. 2024-09-21T08:19:20.229 INFO:tasks.cephadm:Fetching config... 2024-09-21T08:19:20.229 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:19:20.230 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-09-21T08:19:20.245 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-09-21T08:19:20.245 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:19:20.245 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-09-21T08:19:20.301 INFO:tasks.cephadm:Fetching mon keyring... 2024-09-21T08:19:20.301 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:19:20.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/keyring of=/dev/stdout 2024-09-21T08:19:20.365 INFO:tasks.cephadm:Fetching pub ssh key... 2024-09-21T08:19:20.365 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:19:20.365 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-09-21T08:19:20.421 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-09-21T08:19:20.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQPhUs270xg9U3NAEqq3tFdm+nmYQf5PAhy84JcZKvWVHPRUYxMc9nnnxswApQ83PmKghLBMdNNVFGLahfQfdvqosxD0Qt063Y+00bQ/ZXPiB5AgQ1/EZGNKUO98/CHPOFy4xbiKhtEQEbHAO8v6N7sKnQoUPmRGkBBVL5rOgXppQUUrQMljI3NVjj3BP73fDZLwVRbih2ipu0Piw0I8RozSHuD9gQhN3RCwcTH4+pbTOCSKHWRwZVYZUuyilKyPDxj0wfFXcycMO2vwK3fMNbNyImgtajohc//+SlLpRUqIhA1/hAkdlgbyFBNe/gG10Y7/B0OiTO1qaw6eKSpEEMl70zxP43bbShJrmoSp0KkEWAHxO/rGLuOj9c1TJ0G5m7G3xv7M4u0w4bU/u3tVCx97glgX2OKzWFq1c5fUMD9e6biI0wVqdzcYIZ8RnrVVBb2Sd12VUDFVsNX9q3Xfjl8yglS8GhKxk0zAyuRCI6tVvrKcbJ7RqNqU4ZG0NduHU= ceph-ec8566d6-77f1-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T08:19:20.497 INFO:teuthology.orchestra.run.smithi137.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQPhUs270xg9U3NAEqq3tFdm+nmYQf5PAhy84JcZKvWVHPRUYxMc9nnnxswApQ83PmKghLBMdNNVFGLahfQfdvqosxD0Qt063Y+00bQ/ZXPiB5AgQ1/EZGNKUO98/CHPOFy4xbiKhtEQEbHAO8v6N7sKnQoUPmRGkBBVL5rOgXppQUUrQMljI3NVjj3BP73fDZLwVRbih2ipu0Piw0I8RozSHuD9gQhN3RCwcTH4+pbTOCSKHWRwZVYZUuyilKyPDxj0wfFXcycMO2vwK3fMNbNyImgtajohc//+SlLpRUqIhA1/hAkdlgbyFBNe/gG10Y7/B0OiTO1qaw6eKSpEEMl70zxP43bbShJrmoSp0KkEWAHxO/rGLuOj9c1TJ0G5m7G3xv7M4u0w4bU/u3tVCx97glgX2OKzWFq1c5fUMD9e6biI0wVqdzcYIZ8RnrVVBb2Sd12VUDFVsNX9q3Xfjl8yglS8GhKxk0zAyuRCI6tVvrKcbJ7RqNqU4ZG0NduHU= ceph-ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:20.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQPhUs270xg9U3NAEqq3tFdm+nmYQf5PAhy84JcZKvWVHPRUYxMc9nnnxswApQ83PmKghLBMdNNVFGLahfQfdvqosxD0Qt063Y+00bQ/ZXPiB5AgQ1/EZGNKUO98/CHPOFy4xbiKhtEQEbHAO8v6N7sKnQoUPmRGkBBVL5rOgXppQUUrQMljI3NVjj3BP73fDZLwVRbih2ipu0Piw0I8RozSHuD9gQhN3RCwcTH4+pbTOCSKHWRwZVYZUuyilKyPDxj0wfFXcycMO2vwK3fMNbNyImgtajohc//+SlLpRUqIhA1/hAkdlgbyFBNe/gG10Y7/B0OiTO1qaw6eKSpEEMl70zxP43bbShJrmoSp0KkEWAHxO/rGLuOj9c1TJ0G5m7G3xv7M4u0w4bU/u3tVCx97glgX2OKzWFq1c5fUMD9e6biI0wVqdzcYIZ8RnrVVBb2Sd12VUDFVsNX9q3Xfjl8yglS8GhKxk0zAyuRCI6tVvrKcbJ7RqNqU4ZG0NduHU= ceph-ec8566d6-77f1-11ef-baf6-efdc52797490' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-09-21T08:19:20.567 INFO:teuthology.orchestra.run.smithi165.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDQPhUs270xg9U3NAEqq3tFdm+nmYQf5PAhy84JcZKvWVHPRUYxMc9nnnxswApQ83PmKghLBMdNNVFGLahfQfdvqosxD0Qt063Y+00bQ/ZXPiB5AgQ1/EZGNKUO98/CHPOFy4xbiKhtEQEbHAO8v6N7sKnQoUPmRGkBBVL5rOgXppQUUrQMljI3NVjj3BP73fDZLwVRbih2ipu0Piw0I8RozSHuD9gQhN3RCwcTH4+pbTOCSKHWRwZVYZUuyilKyPDxj0wfFXcycMO2vwK3fMNbNyImgtajohc//+SlLpRUqIhA1/hAkdlgbyFBNe/gG10Y7/B0OiTO1qaw6eKSpEEMl70zxP43bbShJrmoSp0KkEWAHxO/rGLuOj9c1TJ0G5m7G3xv7M4u0w4bU/u3tVCx97glgX2OKzWFq1c5fUMD9e6biI0wVqdzcYIZ8RnrVVBb2Sd12VUDFVsNX9q3Xfjl8yglS8GhKxk0zAyuRCI6tVvrKcbJ7RqNqU4ZG0NduHU= ceph-ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:20.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-09-21T08:19:20.834 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:19:21.998 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-09-21T08:19:21.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-09-21T08:19:22.427 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:19:22.600 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:22 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3173296446' entity='client.admin' 2024-09-21T08:19:22.600 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:22 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:22.600 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:22 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:22.600 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:22 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi137", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:19:22.600 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:22 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:22.600 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:22 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:22.600 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:22 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:24.419 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi165 2024-09-21T08:19:24.419 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:19:24.419 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.conf 2024-09-21T08:19:24.438 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:19:24.438 DEBUG:teuthology.orchestra.run.smithi165:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:19:24.496 INFO:tasks.cephadm:Adding host smithi165 to orchestrator... 2024-09-21T08:19:24.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch host add smithi165 2024-09-21T08:19:24.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:24 smithi137 ceph-mon[29634]: from='client.14170 v1:172.21.15.137:0/2802566419' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:19:24.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:24.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:24.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:24.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:19:24.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:24 smithi137 ceph-mon[29634]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-21T08:19:24.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:24 smithi137 ceph-mon[29634]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:19:24.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:24.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:24.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:24.841 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:19:26.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:25 smithi137 ceph-mon[29634]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:19:26.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:25 smithi137 ceph-mon[29634]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T08:19:27.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:26 smithi137 ceph-mon[29634]: from='client.14172 v1:172.21.15.137:0/4231084284' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi165", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:19:28.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:27 smithi137 ceph-mon[29634]: Deploying cephadm binary to smithi165 2024-09-21T08:19:28.603 INFO:teuthology.orchestra.run.smithi137.stdout:Added host 'smithi165' with addr '172.21.15.165' 2024-09-21T08:19:29.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch host ls --format=json 2024-09-21T08:19:29.403 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:19:29.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:29.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:29 smithi137 ceph-mon[29634]: Added host smithi165 2024-09-21T08:19:29.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:29.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:29.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:30.004 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:30.004 INFO:teuthology.orchestra.run.smithi137.stdout:[{"addr": "172.21.15.137", "hostname": "smithi137", "labels": [], "status": ""}, {"addr": "172.21.15.165", "hostname": "smithi165", "labels": [], "status": ""}] 2024-09-21T08:19:30.557 INFO:tasks.cephadm:Setting crush tunables to default 2024-09-21T08:19:30.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd crush tunables default 2024-09-21T08:19:30.793 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:19:31.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:30 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:31.945 INFO:teuthology.orchestra.run.smithi137.stderr:adjusted tunables profile to default 2024-09-21T08:19:32.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:31 smithi137 ceph-mon[29634]: from='client.14174 v1:172.21.15.137:0/1697495033' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:19:32.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:31 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/556357350' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-09-21T08:19:32.557 INFO:tasks.cephadm:Adding mon.a on smithi137 2024-09-21T08:19:32.557 INFO:tasks.cephadm:Adding mon.c on smithi137 2024-09-21T08:19:32.557 INFO:tasks.cephadm:Adding mon.b on smithi165 2024-09-21T08:19:32.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch apply mon '3;smithi137:[v1:172.21.15.137:6789]=a;smithi137:[v1:172.21.15.137:6790]=c;smithi165:[v1:172.21.15.165:6789]=b' 2024-09-21T08:19:32.874 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T08:19:32.921 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /etc/ceph/ceph.conf 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/556357350' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd/host:smithi165", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:19:33.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:32 smithi137 ceph-mon[29634]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-21T08:19:33.629 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mon update... 2024-09-21T08:19:34.295 DEBUG:teuthology.orchestra.run.smithi137:mon.c> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mon.c.service 2024-09-21T08:19:34.297 DEBUG:teuthology.orchestra.run.smithi165:mon.b> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mon.b.service 2024-09-21T08:19:34.300 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T08:19:34.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T08:19:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:19:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:19:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T08:19:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: from='client.14178 v1:172.21.15.165:0/11562813' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi137:[v1:172.21.15.137:6789]=a;smithi137:[v1:172.21.15.137:6790]=c;smithi165:[v1:172.21.15.165:6789]=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:19:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: Saving service mon spec with placement smithi137:[v1:172.21.15.137:6789]=a;smithi137:[v1:172.21.15.137:6790]=c;smithi165:[v1:172.21.15.165:6789]=b;count:3 2024-09-21T08:19:35.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:35.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:35.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:35.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:35.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:35.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:35.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:34 smithi137 ceph-mon[29634]: Deploying daemon mon.b on smithi165 2024-09-21T08:19:35.060 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:19:35.698 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:19:35.698 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":1,"fsid":"ec8566d6-77f1-11ef-baf6-efdc52797490","modified":"2024-09-21T08:18:18.964811Z","created":"2024-09-21T08:18:18.964811Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-09-21T08:19:35.698 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 1 2024-09-21T08:19:36.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:35 smithi137 ceph-mon[29634]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:37.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:36 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/2597368534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:37 smithi137 ceph-mon[29634]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:38.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T08:19:38.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T08:19:38.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T08:19:38.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T08:19:38.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T08:19:38.429 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T08:19:38.637 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T08:19:38.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T08:19:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T08:19:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T08:19:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T08:19:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T08:19:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/2597368534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:38.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:38.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 ceph-mon[32893]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-09-21T08:19:38.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 bash[32854]: 273a353a5f97434725c242b84f661a06295d6cda68f271e56864db1044fe148a 2024-09-21T08:19:38.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:38 smithi165 systemd[1]: Started Ceph mon.b for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:19:38.895 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:19:43.700 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:19:43.700 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":2,"fsid":"ec8566d6-77f1-11ef-baf6-efdc52797490","modified":"2024-09-21T08:19:38.433021Z","created":"2024-09-21T08:18:18.964811Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T08:19:43.700 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 2 2024-09-21T08:19:43.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:19:43.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: mon.a calling monitor election 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: mon.b calling monitor election 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: monmap epoch 2 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: last_changed 2024-09-21T08:19:38.433021+0000 2024-09-21T08:19:43.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: created 2024-09-21T08:18:18.964811+0000 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: min_mon_release 19 (squid) 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: election_strategy: 1 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: fsmap 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: mgrmap e13: y(active, since 30s) 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: overall HEALTH_OK 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:43.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:43.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:19:43.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: mon.a calling monitor election 2024-09-21T08:19:43.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:43.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: mon.b calling monitor election 2024-09-21T08:19:43.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:43.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: monmap epoch 2 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: last_changed 2024-09-21T08:19:38.433021+0000 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: created 2024-09-21T08:18:18.964811+0000 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: min_mon_release 19 (squid) 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: election_strategy: 1 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: fsmap 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T08:19:43.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: mgrmap e13: y(active, since 30s) 2024-09-21T08:19:43.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: overall HEALTH_OK 2024-09-21T08:19:43.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:43.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:43.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:43.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:43.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:44.723 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:44 smithi137 ceph-mon[29634]: Deploying daemon mon.c on smithi137 2024-09-21T08:19:44.723 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:44 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/2022132416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:44.723 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:44 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:44.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:44 smithi165 ceph-mon[32893]: Deploying daemon mon.c on smithi137 2024-09-21T08:19:44.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:44 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/2022132416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:44.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:44 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:45.245 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T08:19:45.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T08:19:45.495 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:19:45.616 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:45 smithi165 ceph-mon[32893]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:45.747 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:45 smithi137 ceph-mon[29634]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:46.184 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:19:46.184 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":2,"fsid":"ec8566d6-77f1-11ef-baf6-efdc52797490","modified":"2024-09-21T08:19:38.433021Z","created":"2024-09-21T08:18:18.964811Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-09-21T08:19:46.184 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 2 2024-09-21T08:19:46.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 systemd[1]: Starting Ceph mon.c for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:19:46.643 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/4218616004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:46.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 podman[41006]: 2024-09-21 08:19:46.381639382 +0000 UTC m=+0.105906499 container create b466e3843528ef2acb66c6489dabd45c5cfe08df2df64c385e3f116fb5361716 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-c, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T08:19:46.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 podman[41006]: 2024-09-21 08:19:46.291744925 +0000 UTC m=+0.016012042 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T08:19:46.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 podman[41006]: 2024-09-21 08:19:46.592350265 +0000 UTC m=+0.316617382 container init b466e3843528ef2acb66c6489dabd45c5cfe08df2df64c385e3f116fb5361716 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, io.buildah.version=1.37.2) 2024-09-21T08:19:46.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 podman[41006]: 2024-09-21 08:19:46.597189453 +0000 UTC m=+0.321456570 container start b466e3843528ef2acb66c6489dabd45c5cfe08df2df64c385e3f116fb5361716 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-c, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS) 2024-09-21T08:19:46.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: set uid:gid to 167:167 (ceph:ceph) 2024-09-21T08:19:46.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable), process ceph-mon, pid 7 2024-09-21T08:19:46.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: pidfile_write: ignore empty --pid-file 2024-09-21T08:19:46.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: load: jerasure load: lrc 2024-09-21T08:19:46.644 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: RocksDB version: 7.9.2 2024-09-21T08:19:46.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Git sha 0 2024-09-21T08:19:46.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Compile date 2024-09-19 18:20:48 2024-09-21T08:19:46.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: DB SUMMARY 2024-09-21T08:19:46.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: DB Session ID: CRC7ZQ24HD9AG0DELX1D 2024-09-21T08:19:46.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: CURRENT file: CURRENT 2024-09-21T08:19:46.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: IDENTITY file: IDENTITY 2024-09-21T08:19:46.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-09-21T08:19:46.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-09-21T08:19:46.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 566 ; 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.error_if_exists: 0 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.create_if_missing: 0 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.paranoid_checks: 1 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.flush_verify_memtable_count: 1 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.env: 0x564a893c9940 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.fs: PosixFileSystem 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.info_log: 0x564a89c296a0 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_file_opening_threads: 16 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.statistics: (nil) 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.use_fsync: 0 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_log_file_size: 0 2024-09-21T08:19:46.647 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.log_file_time_to_roll: 0 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.keep_log_file_num: 1000 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.recycle_log_file_num: 0 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.allow_fallocate: 1 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.allow_mmap_reads: 0 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.allow_mmap_writes: 0 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.use_direct_reads: 0 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.create_missing_column_families: 0 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.db_log_dir: 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.wal_dir: 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.table_cache_numshardbits: 6 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.WAL_ttl_seconds: 0 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.WAL_size_limit_MB: 0 2024-09-21T08:19:46.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.is_fd_close_on_exec: 1 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.advise_random_on_open: 1 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.db_write_buffer_size: 0 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.write_buffer_manager: 0x564a89c38dc0 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.use_adaptive_mutex: 0 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.rate_limiter: (nil) 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.wal_recovery_mode: 2 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.enable_thread_tracking: 0 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.enable_pipelined_write: 0 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.unordered_write: 0 2024-09-21T08:19:46.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.row_cache: None 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.wal_filter: None 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.allow_ingest_behind: 0 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.two_write_queues: 0 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.manual_wal_flush: 0 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.wal_compression: 0 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.atomic_flush: 0 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.persist_stats_to_disk: 0 2024-09-21T08:19:46.650 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.write_dbid_to_manifest: 0 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.log_readahead_size: 0 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.best_efforts_recovery: 0 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.allow_data_in_errors: 0 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.db_host_id: __hostname__ 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.enforce_single_del_contracts: true 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_background_jobs: 2 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_background_compactions: -1 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_subcompactions: 1 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.delayed_write_rate : 16777216 2024-09-21T08:19:46.651 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_total_wal_size: 0 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.stats_dump_period_sec: 600 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.stats_persist_period_sec: 600 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_open_files: -1 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bytes_per_sync: 0 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.wal_bytes_per_sync: 0 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.strict_bytes_per_sync: 0 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_readahead_size: 0 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_background_flushes: -1 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Compression algorithms supported: 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: kZSTD supported: 0 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: kXpressCompression supported: 0 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: kBZip2Compression supported: 0 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-09-21T08:19:46.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: kLZ4Compression supported: 1 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: kZlibCompression supported: 1 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: kLZ4HCCompression supported: 1 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: kSnappyCompression supported: 1 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Fast CRC32 supported: Supported on x86 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: DMutex implementation: pthread_mutex_t 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.merge_operator: 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_filter: None 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_filter_factory: None 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.sst_partitioner_factory: None 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.memtable_factory: SkipListFactory 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.table_factory: BlockBasedTable 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x564a89c29300) 2024-09-21T08:19:46.653 INFO:journalctl@ceph.mon.c.smithi137.stdout: cache_index_and_filter_blocks: 1 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: pin_top_level_index_and_filter: 1 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: index_type: 0 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: data_block_index_type: 0 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: index_shortening: 1 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: data_block_hash_table_util_ratio: 0.750000 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: checksum: 4 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: no_block_cache: 0 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_cache: 0x564a89c1d1f0 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_cache_name: BinnedLRUCache 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_cache_options: 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: capacity : 536870912 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: num_shard_bits : 4 2024-09-21T08:19:46.654 INFO:journalctl@ceph.mon.c.smithi137.stdout: strict_capacity_limit : 0 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: high_pri_pool_ratio: 0.000 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_cache_compressed: (nil) 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: persistent_cache: (nil) 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_size: 4096 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_size_deviation: 10 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_restart_interval: 16 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: index_block_restart_interval: 1 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: metadata_block_size: 4096 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: partition_filters: 0 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: use_delta_encoding: 1 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: filter_policy: bloomfilter 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: whole_key_filtering: 1 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: verify_compression: 0 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: read_amp_bytes_per_bit: 0 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: format_version: 5 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: enable_index_compression: 1 2024-09-21T08:19:46.655 INFO:journalctl@ceph.mon.c.smithi137.stdout: block_align: 0 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout: max_auto_readahead_size: 262144 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout: prepopulate_block_cache: 0 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout: initial_auto_readahead_size: 8192 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout: num_file_reads_for_auto_readahead: 2 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.write_buffer_size: 33554432 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_write_buffer_number: 2 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compression: NoCompression 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bottommost_compression: Disabled 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.prefix_extractor: nullptr 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.num_levels: 7 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-09-21T08:19:46.656 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compression_opts.window_bits: -14 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compression_opts.level: 32767 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compression_opts.strategy: 0 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compression_opts.enabled: false 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-09-21T08:19:46.657 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.target_file_size_base: 67108864 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.target_file_size_multiplier: 1 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-09-21T08:19:46.658 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.arena_block_size: 1048576 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.disable_auto_compactions: 0 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-09-21T08:19:46.659 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.inplace_update_support: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.inplace_update_num_locks: 10000 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.memtable_huge_page_size: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.bloom_locality: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.max_successive_merges: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.optimize_filters_for_hits: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.paranoid_file_checks: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.force_consistency_checks: 1 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.report_bg_io_stats: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.ttl: 2592000 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.periodic_compaction_seconds: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-09-21T08:19:46.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.enable_blob_files: false 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.min_blob_size: 0 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.blob_file_size: 268435456 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.blob_compression_type: NoCompression 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.enable_blob_garbage_collection: false 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.blob_file_starting_level: 0 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 6ca34571-147b-4063-adae-bfd8068486a7 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726906786623843, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-09-21T08:19:46.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726906786624234, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1703, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 578, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 456, "raw_average_value_size": 91, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1726906786, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "6ca34571-147b-4063-adae-bfd8068486a7", "db_session_id": "CRC7ZQ24HD9AG0DELX1D", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: EVENT_LOG_v1 {"time_micros": 1726906786624325, "job": 1, "event": "recovery_finished"} 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x564a89c4ae00 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: DB pointer 0x564a89d46000 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: ** DB Stats ** 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-09-21T08:19:46.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: L0 1/0 1.66 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: Sum 1/0 1.66 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: ** Compaction Stats [default] ** 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-21T08:19:46.663 INFO:journalctl@ceph.mon.c.smithi137.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: AddFile(Keys): cumulative 0, interval 0 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: Block cache BinnedLRUCache@0x564a89c1d1f0#7 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-21T08:19:46.664 INFO:journalctl@ceph.mon.c.smithi137.stdout: ** File Read Latency Histogram By Level [default] ** 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: using public_addrv v1:172.21.15.137:6790/0 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: starting mon.c rank -1 at public addrs v1:172.21.15.137:6790/0 at bind addrs v1:172.21.15.137:6790/0 mon_data /var/lib/ceph/mon/ceph-c fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(???) e0 preinit fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).mds e1 new map 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).mds e1 print_map 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout: e1 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout: btime 2024-09-21T08:18:21:794533+0000 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout: legacy client fscid: -1 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout: 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout: No filesystems configured 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-09-21T08:19:46.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/2597368534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.a calling monitor election 2024-09-21T08:19:46.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.b calling monitor election 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.a is new leader, mons a,b in quorum (ranks 0,1) 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: monmap epoch 2 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: last_changed 2024-09-21T08:19:38.433021+0000 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: created 2024-09-21T08:18:18.964811+0000 2024-09-21T08:19:46.667 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: min_mon_release 19 (squid) 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: election_strategy: 1 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: fsmap 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mgrmap e13: y(active, since 30s) 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: overall HEALTH_OK 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: Deploying daemon mon.c on smithi137 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/2022132416' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:46.668 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/4218616004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:46.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:46 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/4218616004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:47.017 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 ceph-mon[41044]: mon.c@-1(synchronizing).paxosservice(auth 1..5) refresh upgraded, format 0 -> 3 2024-09-21T08:19:47.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 bash[41006]: b466e3843528ef2acb66c6489dabd45c5cfe08df2df64c385e3f116fb5361716 2024-09-21T08:19:47.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:46 smithi137 systemd[1]: Started Ceph mon.c for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:19:47.770 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-09-21T08:19:47.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph mon dump -f json 2024-09-21T08:19:48.022 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:19:51.932 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:19:51.932 INFO:teuthology.orchestra.run.smithi165.stdout:{"epoch":3,"fsid":"ec8566d6-77f1-11ef-baf6-efdc52797490","modified":"2024-09-21T08:19:46.652397Z","created":"2024-09-21T08:18:18.964811Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6789","nonce":0}]},"addr":"172.21.15.137:6789/0","public_addr":"172.21.15.137:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6789","nonce":0}]},"addr":"172.21.15.165:6789/0","public_addr":"172.21.15.165:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6790","nonce":0}]},"addr":"172.21.15.137:6790/0","public_addr":"172.21.15.137:6790/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-09-21T08:19:51.932 INFO:teuthology.orchestra.run.smithi165.stderr:dumped monmap epoch 3 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: mon.a calling monitor election 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: mon.b calling monitor election 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: mon.c calling monitor election 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: monmap epoch 3 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: last_changed 2024-09-21T08:19:46.652397+0000 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: created 2024-09-21T08:18:18.964811+0000 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: min_mon_release 19 (squid) 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: election_strategy: 1 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: 2: v1:172.21.15.137:6790/0 mon.c 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: fsmap 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: mgrmap e13: y(active, since 38s) 2024-09-21T08:19:52.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: overall HEALTH_OK 2024-09-21T08:19:52.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:19:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: mon.a calling monitor election 2024-09-21T08:19:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: mon.b calling monitor election 2024-09-21T08:19:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: mon.c calling monitor election 2024-09-21T08:19:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: monmap epoch 3 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: last_changed 2024-09-21T08:19:46.652397+0000 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: created 2024-09-21T08:18:18.964811+0000 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: min_mon_release 19 (squid) 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: election_strategy: 1 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: 2: v1:172.21.15.137:6790/0 mon.c 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: fsmap 2024-09-21T08:19:52.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: mgrmap e13: y(active, since 38s) 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: overall HEALTH_OK 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: mon.a calling monitor election 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: mon.b calling monitor election 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:52.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: mon.c calling monitor election 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: mon.a is new leader, mons a,b,c in quorum (ranks 0,1,2) 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: monmap epoch 3 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: last_changed 2024-09-21T08:19:46.652397+0000 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: created 2024-09-21T08:18:18.964811+0000 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: min_mon_release 19 (squid) 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: election_strategy: 1 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: 0: v1:172.21.15.137:6789/0 mon.a 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: 1: v1:172.21.15.165:6789/0 mon.b 2024-09-21T08:19:52.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: 2: v1:172.21.15.137:6790/0 mon.c 2024-09-21T08:19:52.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: fsmap 2024-09-21T08:19:52.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: osdmap e4: 0 total, 0 up, 0 in 2024-09-21T08:19:52.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: mgrmap e13: y(active, since 38s) 2024-09-21T08:19:52.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: overall HEALTH_OK 2024-09-21T08:19:52.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:52.671 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-09-21T08:19:52.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph config generate-minimal-conf 2024-09-21T08:19:52.933 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:19:53.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:52 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:53.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:52 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/3855307517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:53.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:52 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:53.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:52 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:53.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:52 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/3855307517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:52 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:53.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:52 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:53.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:52 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/3855307517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:19:53.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:52 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:19:53.521 INFO:teuthology.orchestra.run.smithi137.stdout:# minimal ceph.conf for ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:53.521 INFO:teuthology.orchestra.run.smithi137.stdout:[global] 2024-09-21T08:19:53.521 INFO:teuthology.orchestra.run.smithi137.stdout: fsid = ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:19:53.521 INFO:teuthology.orchestra.run.smithi137.stdout: mon_host = 172.21.15.137:6789/0 172.21.15.165:6789/0 v1:172.21.15.137:6790/0 2024-09-21T08:19:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:53 smithi165 ceph-mon[32893]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:53 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:53 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:53 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:53 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:53 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:53 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:53 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:19:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:53 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2741999606' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:54.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[29634]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:54.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2741999606' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[41044]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:19:54.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:53 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2741999606' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:54.176 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-09-21T08:19:54.177 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:19:54.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T08:19:54.203 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:19:54.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:19:54.268 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:19:54.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.conf 2024-09-21T08:19:54.295 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:19:54.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:19:54.361 INFO:tasks.cephadm:Adding mgr.y on smithi137 2024-09-21T08:19:54.361 INFO:tasks.cephadm:Adding mgr.x on smithi165 2024-09-21T08:19:54.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch apply mgr '2;smithi137=y;smithi165=x' 2024-09-21T08:19:54.649 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.003 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.004 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.004 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T08:19:55.004 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:55.004 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T08:19:55.004 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:55.004 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:54 smithi165 ceph-mon[32893]: Reconfiguring daemon mon.a on smithi137 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[29634]: Reconfiguring daemon mon.a on smithi137 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:55.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: Reconfiguring mon.a (unknown last config time)... 2024-09-21T08:19:55.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:55.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T08:19:55.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:55.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:54 smithi137 ceph-mon[41044]: Reconfiguring daemon mon.a on smithi137 2024-09-21T08:19:55.276 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled mgr update... 2024-09-21T08:19:55.900 DEBUG:teuthology.orchestra.run.smithi165:mgr.x> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@mgr.x.service 2024-09-21T08:19:55.902 INFO:tasks.cephadm:Deploying OSDs... 2024-09-21T08:19:55.902 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:19:55.902 DEBUG:teuthology.orchestra.run.smithi137:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T08:19:55.918 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T08:19:55.918 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_1 2024-09-21T08:19:55.973 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T08:19:55.973 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T08:19:55.973 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 979 Links: 1 2024-09-21T08:19:55.974 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T08:19:55.974 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T08:19:55.974 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-21 08:19:46.118168243 +0000 2024-09-21T08:19:55.974 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-21 08:15:01.596281833 +0000 2024-09-21T08:19:55.974 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-21 08:15:01.596281833 +0000 2024-09-21T08:19:55.974 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-21 08:15:01.596281833 +0000 2024-09-21T08:19:55.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T08:19:56.038 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-21T08:19:56.038 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-21T08:19:56.038 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000250147 s, 2.0 MB/s 2024-09-21T08:19:56.039 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T08:19:56.094 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_2 2024-09-21T08:19:56.150 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T08:19:56.150 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T08:19:56.150 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 967 Links: 1 2024-09-21T08:19:56.150 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T08:19:56.150 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T08:19:56.150 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-21 08:19:46.118168243 +0000 2024-09-21T08:19:56.150 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-21 08:15:01.595281851 +0000 2024-09-21T08:19:56.150 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-21 08:15:01.595281851 +0000 2024-09-21T08:19:56.150 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-21 08:15:01.595281851 +0000 2024-09-21T08:19:56.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[29634]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[41044]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:56.213 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T08:19:56.214 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:56.214 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:55 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:56.215 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-21T08:19:56.215 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-21T08:19:56.215 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.0001725 s, 3.0 MB/s 2024-09-21T08:19:56.216 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T08:19:56.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:55 smithi165 ceph-mon[32893]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:56.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:55 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:56.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:55 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:56.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:55 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:56.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:55 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T08:19:56.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:55 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:56.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:55 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:56.273 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_3 2024-09-21T08:19:56.327 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T08:19:56.327 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T08:19:56.327 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 982 Links: 1 2024-09-21T08:19:56.328 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T08:19:56.328 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T08:19:56.328 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-21 08:19:46.119168225 +0000 2024-09-21T08:19:56.328 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-21 08:15:01.597281815 +0000 2024-09-21T08:19:56.328 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-21 08:15:01.597281815 +0000 2024-09-21T08:19:56.328 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-21 08:15:01.597281815 +0000 2024-09-21T08:19:56.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T08:19:56.390 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-21T08:19:56.391 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-21T08:19:56.391 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000197616 s, 2.6 MB/s 2024-09-21T08:19:56.392 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T08:19:56.447 DEBUG:teuthology.orchestra.run.smithi137:> stat /dev/vg_nvme/lv_4 2024-09-21T08:19:56.503 INFO:teuthology.orchestra.run.smithi137.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T08:19:56.503 INFO:teuthology.orchestra.run.smithi137.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T08:19:56.503 INFO:teuthology.orchestra.run.smithi137.stdout:Device: 5h/5d Inode: 973 Links: 1 2024-09-21T08:19:56.503 INFO:teuthology.orchestra.run.smithi137.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T08:19:56.503 INFO:teuthology.orchestra.run.smithi137.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T08:19:56.503 INFO:teuthology.orchestra.run.smithi137.stdout:Access: 2024-09-21 08:19:46.119168225 +0000 2024-09-21T08:19:56.503 INFO:teuthology.orchestra.run.smithi137.stdout:Modify: 2024-09-21 08:15:01.596281833 +0000 2024-09-21T08:19:56.503 INFO:teuthology.orchestra.run.smithi137.stdout:Change: 2024-09-21 08:15:01.596281833 +0000 2024-09-21T08:19:56.503 INFO:teuthology.orchestra.run.smithi137.stdout: Birth: 2024-09-21 08:15:01.596281833 +0000 2024-09-21T08:19:56.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T08:19:56.566 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records in 2024-09-21T08:19:56.566 INFO:teuthology.orchestra.run.smithi137.stderr:1+0 records out 2024-09-21T08:19:56.566 INFO:teuthology.orchestra.run.smithi137.stderr:512 bytes copied, 0.000177815 s, 2.9 MB/s 2024-09-21T08:19:56.567 DEBUG:teuthology.orchestra.run.smithi137:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T08:19:56.623 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:19:56.623 DEBUG:teuthology.orchestra.run.smithi165:> dd if=/scratch_devs of=/dev/stdout 2024-09-21T08:19:56.642 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-21T08:19:56.642 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_1 2024-09-21T08:19:56.699 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-21T08:19:56.699 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T08:19:56.699 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 974 Links: 1 2024-09-21T08:19:56.699 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T08:19:56.699 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T08:19:56.699 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-21 08:19:37.874800024 +0000 2024-09-21T08:19:56.699 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-21 08:15:01.300886857 +0000 2024-09-21T08:19:56.699 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-21 08:15:01.300886857 +0000 2024-09-21T08:19:56.699 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-21 08:15:01.300886857 +0000 2024-09-21T08:19:56.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-21T08:19:56.764 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-21T08:19:56.764 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-21T08:19:56.764 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000194954 s, 2.6 MB/s 2024-09-21T08:19:56.765 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-21T08:19:56.822 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_2 2024-09-21T08:19:56.878 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-21T08:19:56.878 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T08:19:56.878 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 981 Links: 1 2024-09-21T08:19:56.879 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T08:19:56.879 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T08:19:56.879 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-21 08:19:37.874800024 +0000 2024-09-21T08:19:56.879 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-21 08:15:01.300886857 +0000 2024-09-21T08:19:56.879 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-21 08:15:01.300886857 +0000 2024-09-21T08:19:56.879 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-21 08:15:01.300886857 +0000 2024-09-21T08:19:56.879 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-21T08:19:56.942 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-21T08:19:56.942 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-21T08:19:56.942 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000208448 s, 2.5 MB/s 2024-09-21T08:19:56.943 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-21T08:19:56.999 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_3 2024-09-21T08:19:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:56 smithi165 ceph-mon[32893]: Reconfiguring mon.c (monmap changed)... 2024-09-21T08:19:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:56 smithi165 ceph-mon[32893]: Reconfiguring daemon mon.c on smithi137 2024-09-21T08:19:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:56 smithi165 ceph-mon[32893]: from='client.24109 v1:172.21.15.165:0/749716489' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi137=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:19:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:56 smithi165 ceph-mon[32893]: Saving service mgr spec with placement smithi137=y;smithi165=x;count:2 2024-09-21T08:19:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T08:19:57.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:57.028 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-21T08:19:57.028 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T08:19:57.028 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 978 Links: 1 2024-09-21T08:19:57.028 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T08:19:57.028 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T08:19:57.028 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-21 08:19:37.875800009 +0000 2024-09-21T08:19:57.028 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-21 08:15:01.300886857 +0000 2024-09-21T08:19:57.028 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-21 08:15:01.300886857 +0000 2024-09-21T08:19:57.028 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-21 08:15:01.300886857 +0000 2024-09-21T08:19:57.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-21T08:19:57.091 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-21T08:19:57.091 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-21T08:19:57.091 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.00021549 s, 2.4 MB/s 2024-09-21T08:19:57.093 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-21T08:19:57.149 DEBUG:teuthology.orchestra.run.smithi165:> stat /dev/vg_nvme/lv_4 2024-09-21T08:19:57.204 INFO:teuthology.orchestra.run.smithi165.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-21T08:19:57.204 INFO:teuthology.orchestra.run.smithi165.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-21T08:19:57.204 INFO:teuthology.orchestra.run.smithi165.stdout:Device: 5h/5d Inode: 966 Links: 1 2024-09-21T08:19:57.205 INFO:teuthology.orchestra.run.smithi165.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-21T08:19:57.205 INFO:teuthology.orchestra.run.smithi165.stdout:Context: system_u:object_r:device_t:s0 2024-09-21T08:19:57.205 INFO:teuthology.orchestra.run.smithi165.stdout:Access: 2024-09-21 08:19:37.875800009 +0000 2024-09-21T08:19:57.205 INFO:teuthology.orchestra.run.smithi165.stdout:Modify: 2024-09-21 08:15:01.299886884 +0000 2024-09-21T08:19:57.205 INFO:teuthology.orchestra.run.smithi165.stdout:Change: 2024-09-21 08:15:01.299886884 +0000 2024-09-21T08:19:57.205 INFO:teuthology.orchestra.run.smithi165.stdout: Birth: 2024-09-21 08:15:01.299886884 +0000 2024-09-21T08:19:57.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-21T08:19:57.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[41044]: Reconfiguring mon.c (monmap changed)... 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[41044]: Reconfiguring daemon mon.c on smithi137 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[41044]: from='client.24109 v1:172.21.15.165:0/749716489' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi137=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[41044]: Saving service mgr spec with placement smithi137=y;smithi165=x;count:2 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[29634]: Reconfiguring mon.c (monmap changed)... 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[29634]: Reconfiguring daemon mon.c on smithi137 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[29634]: from='client.24109 v1:172.21.15.165:0/749716489' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "2;smithi137=y;smithi165=x", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:19:57.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[29634]: Saving service mgr spec with placement smithi137=y;smithi165=x;count:2 2024-09-21T08:19:57.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:57.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:57.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-09-21T08:19:57.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-09-21T08:19:57.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:57.270 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records in 2024-09-21T08:19:57.270 INFO:teuthology.orchestra.run.smithi165.stderr:1+0 records out 2024-09-21T08:19:57.270 INFO:teuthology.orchestra.run.smithi165.stderr:512 bytes copied, 0.000171816 s, 3.0 MB/s 2024-09-21T08:19:57.270 DEBUG:teuthology.orchestra.run.smithi165:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-21T08:19:57.327 INFO:tasks.cephadm:Deploying osd.0 on smithi137 with /dev/vg_nvme/lv_4... 2024-09-21T08:19:57.327 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T08:19:57.566 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:19:57.988 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:57 smithi137 ceph-mon[29634]: Reconfiguring mon.b (monmap changed)... 2024-09-21T08:19:57.988 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:57 smithi137 ceph-mon[29634]: Reconfiguring daemon mon.b on smithi165 2024-09-21T08:19:57.988 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:57 smithi137 ceph-mon[29634]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:57.988 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:57 smithi137 ceph-mon[41044]: Reconfiguring mon.b (monmap changed)... 2024-09-21T08:19:57.989 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:57 smithi137 ceph-mon[41044]: Reconfiguring daemon mon.b on smithi165 2024-09-21T08:19:57.989 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:57 smithi137 ceph-mon[41044]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:58.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:57 smithi165 ceph-mon[32893]: Reconfiguring mon.b (monmap changed)... 2024-09-21T08:19:58.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:57 smithi165 ceph-mon[32893]: Reconfiguring daemon mon.b on smithi165 2024-09-21T08:19:58.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:57 smithi165 ceph-mon[32893]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:19:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T08:19:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T08:19:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T08:19:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: Deploying daemon mgr.x on smithi165 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[29634]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: Deploying daemon mgr.x on smithi165 2024-09-21T08:19:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:19:58 smithi137 ceph-mon[41044]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:59.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:59.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:59.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:19:59.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:59.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:19:59.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:19:59.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T08:19:59.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-09-21T08:19:59.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T08:19:59.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:19:59.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: Deploying daemon mgr.x on smithi165 2024-09-21T08:19:59.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:19:58 smithi165 ceph-mon[32893]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:19:59.853 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:19:59.881 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi137:vg_nvme/lv_4 2024-09-21T08:19:59.998 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:19:59 smithi165 systemd[1]: Starting Ceph mgr.x for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:20:00.122 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:20:00.226 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:00 smithi137 ceph-mon[29634]: overall HEALTH_OK 2024-09-21T08:20:00.226 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:00 smithi137 ceph-mon[41044]: overall HEALTH_OK 2024-09-21T08:20:00.253 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:00 smithi165 ceph-mon[32893]: overall HEALTH_OK 2024-09-21T08:20:00.254 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:19:59 smithi165 podman[34861]: 2024-09-21 08:19:59.900375999 +0000 UTC m=+0.015559331 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T08:20:00.254 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 podman[34861]: 2024-09-21 08:20:00.002167013 +0000 UTC m=+0.117350348 container create aa7e3bfad9dff6b32cb6566b6ba3fcacca98835c0d05c2355c899d103125ab9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, GIT_CLEAN=True, RELEASE=squid-9e2ac70, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-09-21T08:20:00.254 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 podman[34861]: 2024-09-21 08:20:00.24661142 +0000 UTC m=+0.361794751 container init aa7e3bfad9dff6b32cb6566b6ba3fcacca98835c0d05c2355c899d103125ab9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-09-21T08:20:00.254 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 podman[34861]: 2024-09-21 08:20:00.251482112 +0000 UTC m=+0.366665446 container start aa7e3bfad9dff6b32cb6566b6ba3fcacca98835c0d05c2355c899d103125ab9e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-09-21T08:20:00.513 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 bash[34861]: aa7e3bfad9dff6b32cb6566b6ba3fcacca98835c0d05c2355c899d103125ab9e 2024-09-21T08:20:00.513 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 systemd[1]: Started Ceph mgr.x for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:20:00.513 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:00.370+0000 7fa17349b040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T08:20:00.513 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:00.431+0000 7fa17349b040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T08:20:00.973 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:00.586+0000 7fa17349b040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T08:20:00.973 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:00.655+0000 7fa17349b040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T08:20:00.973 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:00.818+0000 7fa17349b040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:20:00.973 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:00 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:00.971+0000 7fa17349b040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T08:20:01.473 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.473 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.473 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.473 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.473 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[41044]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[41044]: from='client.14217 v1:172.21.15.137:0/1538505600' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[29634]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:01.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[29634]: from='client.14217 v1:172.21.15.137:0/1538505600' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:20:01.475 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:20:01.475 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:20:01.475 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:01.559 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:01.297+0000 7fa17349b040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T08:20:01.559 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:01.357+0000 7fa17349b040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T08:20:01.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:01.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:01.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-mon[32893]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:01.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-mon[32893]: from='client.14217 v1:172.21.15.137:0/1538505600' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:20:01.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:20:01.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:20:01.560 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:02.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:01.557+0000 7fa17349b040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T08:20:02.702 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:02.703 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:02.703 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:02.703 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:02.703 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:02.703 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[29634]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T08:20:02.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T08:20:02.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T08:20:02.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:02.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[29634]: Reconfiguring daemon mgr.y on smithi137 2024-09-21T08:20:02.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:02.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:02.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:02.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:02.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:02.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[41044]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T08:20:02.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T08:20:02.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T08:20:02.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:02.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:02 smithi137 ceph-mon[41044]: Reconfiguring daemon mgr.y on smithi137 2024-09-21T08:20:02.763 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:02.439+0000 7fa17349b040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T08:20:02.763 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:02.505+0000 7fa17349b040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T08:20:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:02.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-mon[32893]: Reconfiguring mgr.y (unknown last config time)... 2024-09-21T08:20:02.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-09-21T08:20:02.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr services"}]: dispatch 2024-09-21T08:20:02.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:02.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:02 smithi165 ceph-mon[32893]: Reconfiguring daemon mgr.y on smithi137 2024-09-21T08:20:03.478 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:03.221+0000 7fa17349b040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T08:20:03.479 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:03.337+0000 7fa17349b040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T08:20:03.479 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:03.477+0000 7fa17349b040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:20:03.756 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:03.537+0000 7fa17349b040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T08:20:03.756 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:03.687+0000 7fa17349b040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T08:20:03.756 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:03.754+0000 7fa17349b040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T08:20:04.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:03.816+0000 7fa17349b040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T08:20:04.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:03.973+0000 7fa17349b040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T08:20:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-mon[32893]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3789343116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977"}]: dispatch 2024-09-21T08:20:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-mon[32893]: from='client.44120 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977"}]: dispatch 2024-09-21T08:20:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-mon[32893]: from='client.44120 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977"}]': finished 2024-09-21T08:20:04.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-mon[32893]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T08:20:04.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:04.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:03 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1648844820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:20:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[29634]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3789343116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977"}]: dispatch 2024-09-21T08:20:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[29634]: from='client.44120 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977"}]: dispatch 2024-09-21T08:20:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[29634]: from='client.44120 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977"}]': finished 2024-09-21T08:20:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[29634]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T08:20:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1648844820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:20:04.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[41044]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:04.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:04.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3789343116' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977"}]: dispatch 2024-09-21T08:20:04.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[41044]: from='client.44120 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977"}]: dispatch 2024-09-21T08:20:04.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[41044]: from='client.44120 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977"}]': finished 2024-09-21T08:20:04.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[41044]: osdmap e5: 1 total, 0 up, 1 in 2024-09-21T08:20:04.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:04.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:03 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1648844820' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:20:04.327 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:04 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:04.038+0000 7fa17349b040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T08:20:04.327 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:04 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:04.104+0000 7fa17349b040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T08:20:04.327 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:04 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:04.193+0000 7fa17349b040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T08:20:04.327 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:04 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:04.259+0000 7fa17349b040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T08:20:04.327 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:04 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:04.325+0000 7fa17349b040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T08:20:05.513 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:20:05 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:20:05.122+0000 7fa17349b040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T08:20:06.263 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[29634]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:06.423 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[29634]: Standby manager daemon x started 2024-09-21T08:20:06.423 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[29634]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T08:20:06.423 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[29634]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T08:20:06.423 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[29634]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T08:20:06.423 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[29634]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T08:20:06.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[41044]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:06.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[41044]: Standby manager daemon x started 2024-09-21T08:20:06.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[41044]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T08:20:06.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[41044]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T08:20:06.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[41044]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T08:20:06.424 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:05 smithi137 ceph-mon[41044]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T08:20:06.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:05 smithi165 ceph-mon[32893]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:06.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:05 smithi165 ceph-mon[32893]: Standby manager daemon x started 2024-09-21T08:20:06.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:05 smithi165 ceph-mon[32893]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T08:20:06.424 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:05 smithi165 ceph-mon[32893]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T08:20:06.425 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:05 smithi165 ceph-mon[32893]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T08:20:06.425 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:05 smithi165 ceph-mon[32893]: from='mgr.? v1:172.21.15.165:0/41606960' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T08:20:06.876 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:06 smithi137 ceph-mon[29634]: mgrmap e14: y(active, since 53s), standbys: x 2024-09-21T08:20:06.876 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:06 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T08:20:06.876 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:06 smithi137 ceph-mon[41044]: mgrmap e14: y(active, since 53s), standbys: x 2024-09-21T08:20:06.876 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:06 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T08:20:07.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:06 smithi165 ceph-mon[32893]: mgrmap e14: y(active, since 53s), standbys: x 2024-09-21T08:20:07.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:06 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T08:20:07.813 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:07 smithi137 ceph-mon[29634]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:07.813 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:07 smithi137 ceph-mon[41044]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:08.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:07 smithi165 ceph-mon[32893]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[29634]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[41044]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:09 smithi165 ceph-mon[32893]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:10.985 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:10 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T08:20:10.986 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:10 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:10.986 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:10 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T08:20:10.986 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:10 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:11.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:10 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-09-21T08:20:11.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:10 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:12.067 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:11 smithi137 ceph-mon[29634]: Deploying daemon osd.0 on smithi137 2024-09-21T08:20:12.067 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:11 smithi137 ceph-mon[29634]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:12.067 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:11 smithi137 ceph-mon[41044]: Deploying daemon osd.0 on smithi137 2024-09-21T08:20:12.068 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:11 smithi137 ceph-mon[41044]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:12.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:11 smithi165 ceph-mon[32893]: Deploying daemon osd.0 on smithi137 2024-09-21T08:20:12.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:11 smithi165 ceph-mon[32893]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:13.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:13 smithi137 ceph-mon[29634]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:13.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:13 smithi137 ceph-mon[41044]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:14.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:13 smithi165 ceph-mon[32893]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:16.058 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:15 smithi137 ceph-mon[29634]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:16.058 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:15 smithi137 ceph-mon[41044]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:16.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:15 smithi165 ceph-mon[32893]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:17.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:17.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:17.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:17.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:17.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:17.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:17.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:17.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:17.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:18.126 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:17 smithi137 ceph-mon[29634]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:18.126 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:17 smithi137 ceph-mon[41044]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:18.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:17 smithi165 ceph-mon[32893]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:20.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[29634]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:20.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:20.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:20.086 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[41044]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:20.086 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.087 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.087 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:20.087 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:20.087 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:19 smithi165 ceph-mon[32893]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.393 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 0 on host 'smithi137' 2024-09-21T08:20:20.957 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:20 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:20.957 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:20 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.957 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:20 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.957 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:20 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:20.958 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:20 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.958 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:20 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:20.984 DEBUG:teuthology.orchestra.run.smithi137:osd.0> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T08:20:20.986 INFO:tasks.cephadm:Deploying osd.1 on smithi137 with /dev/vg_nvme/lv_3... 2024-09-21T08:20:20.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T08:20:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:20 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:20 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:20 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:21.750 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:20:21.767 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:20:21 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0[45185]: 2024-09-21T08:20:21.514+0000 7fd24a87d640 -1 osd.0 0 log_to_monitors true 2024-09-21T08:20:22.098 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:21 smithi137 ceph-mon[29634]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:22.098 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:21 smithi137 ceph-mon[29634]: from='osd.0 v1:172.21.15.137:6801/3938795530' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T08:20:22.098 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:21 smithi137 ceph-mon[29634]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T08:20:22.099 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:21 smithi137 ceph-mon[41044]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:22.099 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:21 smithi137 ceph-mon[41044]: from='osd.0 v1:172.21.15.137:6801/3938795530' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T08:20:22.099 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:21 smithi137 ceph-mon[41044]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T08:20:22.164 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:21 smithi165 ceph-mon[32893]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:22.164 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:21 smithi165 ceph-mon[32893]: from='osd.0 v1:172.21.15.137:6801/3938795530' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T08:20:22.164 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:21 smithi165 ceph-mon[32893]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T08:20:23.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[29634]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T08:20:23.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[29634]: from='osd.0 v1:172.21.15.137:6801/3938795530' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:23.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[29634]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T08:20:23.716 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:23.716 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[29634]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:23.716 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[29634]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:23.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[41044]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T08:20:23.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[41044]: from='osd.0 v1:172.21.15.137:6801/3938795530' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:23.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[41044]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T08:20:23.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:23.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[41044]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:23.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-mon[41044]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:23.716 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0[45185]: 2024-09-21T08:20:23.433+0000 7fd246800640 -1 osd.0 0 waiting for initial osdmap 2024-09-21T08:20:23.716 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:20:23 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0[45185]: 2024-09-21T08:20:23.437+0000 7fd242628640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T08:20:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:23 smithi165 ceph-mon[32893]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T08:20:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:23 smithi165 ceph-mon[32893]: from='osd.0 v1:172.21.15.137:6801/3938795530' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:23 smithi165 ceph-mon[32893]: osdmap e6: 1 total, 0 up, 1 in 2024-09-21T08:20:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:23 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:23 smithi165 ceph-mon[32893]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:23 smithi165 ceph-mon[32893]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:24.755 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:20:24.756 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T08:20:24.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:24.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:24.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:20:24.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:24.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:24.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:20:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: osdmap e7: 1 total, 0 up, 1 in 2024-09-21T08:20:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:20:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:24.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:25.621 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:20:25.644 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:20:25.644 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[29634]: Detected new or changed devices on smithi137 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[29634]: Adjusting osd_memory_target on smithi137 to 16064M 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[29634]: osd.0 v1:172.21.15.137:6801/3938795530 boot 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[29634]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[29634]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[41044]: Detected new or changed devices on smithi137 2024-09-21T08:20:25.645 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[41044]: Adjusting osd_memory_target on smithi137 to 16064M 2024-09-21T08:20:25.646 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:25.646 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[41044]: osd.0 v1:172.21.15.137:6801/3938795530 boot 2024-09-21T08:20:25.646 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[41044]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T08:20:25.646 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:25.646 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:25 smithi137 ceph-mon[41044]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:25.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi137:vg_nvme/lv_3 2024-09-21T08:20:25.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:25 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:20:25.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:25 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:20:25.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:25 smithi165 ceph-mon[32893]: Detected new or changed devices on smithi137 2024-09-21T08:20:25.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:25 smithi165 ceph-mon[32893]: Adjusting osd_memory_target on smithi137 to 16064M 2024-09-21T08:20:25.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:25 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:25.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:25 smithi165 ceph-mon[32893]: osd.0 v1:172.21.15.137:6801/3938795530 boot 2024-09-21T08:20:25.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:25 smithi165 ceph-mon[32893]: osdmap e8: 1 total, 1 up, 1 in 2024-09-21T08:20:25.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:25 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:20:25.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:25 smithi165 ceph-mon[32893]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-09-21T08:20:25.886 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:20:26.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:26 smithi165 ceph-mon[32893]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T08:20:26.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:26 smithi137 ceph-mon[29634]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T08:20:26.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:26 smithi137 ceph-mon[41044]: osdmap e9: 1 total, 1 up, 1 in 2024-09-21T08:20:27.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:27 smithi137 ceph-mon[29634]: from='client.44141 v1:172.21.15.137:0/670234180' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:20:27.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:20:27.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:20:27.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:27.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:27 smithi137 ceph-mon[29634]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:27.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:27 smithi137 ceph-mon[41044]: from='client.44141 v1:172.21.15.137:0/670234180' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:20:27.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:20:27.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:20:27.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:27.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:27 smithi137 ceph-mon[41044]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:27 smithi165 ceph-mon[32893]: from='client.44141 v1:172.21.15.137:0/670234180' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:20:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:20:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:20:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:27 smithi165 ceph-mon[32893]: pgmap v36: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:30.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:29 smithi165 ceph-mon[32893]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:30.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:29 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4012918269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d"}]: dispatch 2024-09-21T08:20:30.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:29 smithi165 ceph-mon[32893]: from='client.44147 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d"}]: dispatch 2024-09-21T08:20:30.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:29 smithi165 ceph-mon[32893]: from='client.44147 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d"}]': finished 2024-09-21T08:20:30.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:29 smithi165 ceph-mon[32893]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T08:20:30.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:29 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:30.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:29 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3026246464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:20:30.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[29634]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:30.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4012918269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d"}]: dispatch 2024-09-21T08:20:30.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[29634]: from='client.44147 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d"}]: dispatch 2024-09-21T08:20:30.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[29634]: from='client.44147 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d"}]': finished 2024-09-21T08:20:30.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[29634]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T08:20:30.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:30.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3026246464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:20:30.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[41044]: pgmap v37: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:30.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4012918269' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d"}]: dispatch 2024-09-21T08:20:30.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[41044]: from='client.44147 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d"}]: dispatch 2024-09-21T08:20:30.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[41044]: from='client.44147 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d"}]': finished 2024-09-21T08:20:30.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[41044]: osdmap e10: 2 total, 1 up, 2 in 2024-09-21T08:20:30.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:30.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:29 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3026246464' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:20:32.127 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:31 smithi137 ceph-mon[29634]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:32.128 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:31 smithi137 ceph-mon[41044]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:32.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:31 smithi165 ceph-mon[32893]: pgmap v39: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:34.133 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:33 smithi137 ceph-mon[29634]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:34.133 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:33 smithi137 ceph-mon[41044]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:34.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:33 smithi165 ceph-mon[32893]: pgmap v40: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:35 smithi137 ceph-mon[29634]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:35 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T08:20:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:35 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:36.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:35 smithi137 ceph-mon[41044]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:36.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:35 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T08:20:36.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:35 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:36.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:35 smithi165 ceph-mon[32893]: pgmap v41: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:36.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:35 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-09-21T08:20:36.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:35 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:36.990 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:36 smithi137 ceph-mon[29634]: Deploying daemon osd.1 on smithi137 2024-09-21T08:20:36.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:36 smithi137 ceph-mon[41044]: Deploying daemon osd.1 on smithi137 2024-09-21T08:20:37.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:36 smithi165 ceph-mon[32893]: Deploying daemon osd.1 on smithi137 2024-09-21T08:20:38.136 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:37 smithi137 ceph-mon[29634]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:38.136 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:37 smithi137 ceph-mon[41044]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:38.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:37 smithi165 ceph-mon[32893]: pgmap v42: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:40.175 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:39 smithi137 ceph-mon[29634]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:40.175 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:39 smithi137 ceph-mon[41044]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:40.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:39 smithi165 ceph-mon[32893]: pgmap v43: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:42.123 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:41 smithi137 ceph-mon[29634]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:42.159 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:41 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:42.159 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:41 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:42.159 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:41 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:42.159 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:41 smithi137 ceph-mon[41044]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:42.159 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:41 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:42.159 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:41 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:42.159 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:41 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:42.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:41 smithi165 ceph-mon[32893]: pgmap v44: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:42.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:41 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:42.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:41 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:42.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:41 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:43.858 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:43 smithi137 ceph-mon[29634]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:43.859 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:43 smithi137 ceph-mon[41044]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:44.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:43 smithi165 ceph-mon[32893]: pgmap v45: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:45.431 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 1 on host 'smithi137' 2024-09-21T08:20:45.431 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[29634]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:45.432 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:45 smithi137 ceph-mon[41044]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:45.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:45 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:45.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:45 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:45.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:45 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:45.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:45 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:45.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:45 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:45.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:45 smithi165 ceph-mon[32893]: pgmap v46: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:45.979 DEBUG:teuthology.orchestra.run.smithi137:osd.1> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.1.service 2024-09-21T08:20:45.981 INFO:tasks.cephadm:Deploying osd.2 on smithi137 with /dev/vg_nvme/lv_2... 2024-09-21T08:20:45.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T08:20:46.262 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:46.263 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:46.263 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:46 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:46.263 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:46 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:46.263 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:46 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:46.263 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:46 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:46.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:46 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:20:46.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:46 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:46.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:46 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:46.702 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:20:46.768 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:20:46 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-1[51140]: 2024-09-21T08:20:46.620+0000 7fe268ecc640 -1 osd.1 0 log_to_monitors true 2024-09-21T08:20:47.762 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:47 smithi137 ceph-mon[29634]: from='osd.1 v1:172.21.15.137:6805/2743268361' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T08:20:47.763 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:47 smithi137 ceph-mon[29634]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T08:20:47.763 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:47 smithi137 ceph-mon[29634]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:47.763 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:47 smithi137 ceph-mon[41044]: from='osd.1 v1:172.21.15.137:6805/2743268361' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T08:20:47.763 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:47 smithi137 ceph-mon[41044]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T08:20:47.763 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:47 smithi137 ceph-mon[41044]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:47 smithi165 ceph-mon[32893]: from='osd.1 v1:172.21.15.137:6805/2743268361' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T08:20:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:47 smithi165 ceph-mon[32893]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-09-21T08:20:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:47 smithi165 ceph-mon[32893]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:48 smithi165 ceph-mon[32893]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T08:20:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:48 smithi165 ceph-mon[32893]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T08:20:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:48 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:48 smithi165 ceph-mon[32893]: from='osd.1 v1:172.21.15.137:6805/2743268361' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:48 smithi165 ceph-mon[32893]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:48.767 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-1[51140]: 2024-09-21T08:20:48.446+0000 7fe264e4f640 -1 osd.1 0 waiting for initial osdmap 2024-09-21T08:20:48.768 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-1[51140]: 2024-09-21T08:20:48.448+0000 7fe260c77640 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T08:20:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-mon[29634]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T08:20:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-mon[29634]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T08:20:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-mon[29634]: from='osd.1 v1:172.21.15.137:6805/2743268361' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-mon[29634]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-mon[41044]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-09-21T08:20:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-mon[41044]: osdmap e11: 2 total, 1 up, 2 in 2024-09-21T08:20:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-mon[41044]: from='osd.1 v1:172.21.15.137:6805/2743268361' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:48.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:48 smithi137 ceph-mon[41044]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:20:49.569 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:49 smithi137 ceph-mon[29634]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:20:49.569 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:49 smithi137 ceph-mon[29634]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T08:20:49.569 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:49.569 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:49.569 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:49 smithi137 ceph-mon[29634]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:49.569 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:49 smithi137 ceph-mon[41044]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:20:49.569 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:49 smithi137 ceph-mon[41044]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T08:20:49.570 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:49.570 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:49.570 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:49 smithi137 ceph-mon[41044]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:49.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:49 smithi165 ceph-mon[32893]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:20:49.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:49 smithi165 ceph-mon[32893]: osdmap e12: 2 total, 1 up, 2 in 2024-09-21T08:20:49.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:49.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:49.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:49 smithi165 ceph-mon[32893]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-09-21T08:20:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:20:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:20:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: osd.1 v1:172.21.15.137:6805/2743268361 boot 2024-09-21T08:20:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T08:20:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: Detected new or changed devices on smithi137 2024-09-21T08:20:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: Adjusting osd_memory_target on smithi137 to 8032M 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: osd.1 v1:172.21.15.137:6805/2743268361 boot 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T08:20:50.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:50.520 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: Detected new or changed devices on smithi137 2024-09-21T08:20:50.520 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.520 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.520 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:20:50.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:20:50.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: Adjusting osd_memory_target on smithi137 to 8032M 2024-09-21T08:20:50.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:50.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:50.521 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:50 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:20:50.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:20:50.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:50.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: osd.1 v1:172.21.15.137:6805/2743268361 boot 2024-09-21T08:20:50.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: osdmap e13: 2 total, 2 up, 2 in 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: Detected new or changed devices on smithi137 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: Adjusting osd_memory_target on smithi137 to 8032M 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:20:50.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:50 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:20:50.887 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:20:50.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi137:vg_nvme/lv_2 2024-09-21T08:20:51.155 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:20:51.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:51 smithi137 ceph-mon[29634]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T08:20:51.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:51 smithi137 ceph-mon[29634]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:51.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:51 smithi137 ceph-mon[41044]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T08:20:51.555 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:51 smithi137 ceph-mon[41044]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:51.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:51 smithi165 ceph-mon[32893]: osdmap e14: 2 total, 2 up, 2 in 2024-09-21T08:20:51.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:51 smithi165 ceph-mon[32893]: pgmap v53: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:52.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:52 smithi165 ceph-mon[32893]: from='client.44168 v1:172.21.15.137:0/727123671' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:20:52.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:52 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:20:52.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:52 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:20:52.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:52 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:52 smithi137 ceph-mon[29634]: from='client.44168 v1:172.21.15.137:0/727123671' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:20:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:52 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:20:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:52 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:20:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:52 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:52 smithi137 ceph-mon[41044]: from='client.44168 v1:172.21.15.137:0/727123671' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:20:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:52 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:20:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:52 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:20:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:52 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:20:53.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:53 smithi137 ceph-mon[29634]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:53.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:53 smithi137 ceph-mon[41044]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:53.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:53 smithi165 ceph-mon[32893]: pgmap v54: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:54.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:54 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2345886561' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d"}]: dispatch 2024-09-21T08:20:54.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:54 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2345886561' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d"}]': finished 2024-09-21T08:20:54.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:54 smithi137 ceph-mon[29634]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T08:20:54.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:20:54.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:54 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2345886561' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d"}]: dispatch 2024-09-21T08:20:54.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:54 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2345886561' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d"}]': finished 2024-09-21T08:20:54.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:54 smithi137 ceph-mon[41044]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T08:20:54.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:20:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:54 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2345886561' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d"}]: dispatch 2024-09-21T08:20:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:54 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2345886561' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d"}]': finished 2024-09-21T08:20:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:54 smithi165 ceph-mon[32893]: osdmap e15: 3 total, 2 up, 3 in 2024-09-21T08:20:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:20:55.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:55 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2874444170' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:20:55.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:55 smithi165 ceph-mon[32893]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:55.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:55 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2874444170' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:20:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:55 smithi137 ceph-mon[29634]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:55.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:55 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2874444170' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:20:55.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:55 smithi137 ceph-mon[41044]: pgmap v56: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:57.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:57 smithi165 ceph-mon[32893]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:57.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:57 smithi137 ceph-mon[29634]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:57.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:57 smithi137 ceph-mon[41044]: pgmap v57: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:59.901 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:20:59 smithi137 ceph-mon[41044]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:20:59.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:20:59 smithi137 ceph-mon[29634]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:00.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:20:59 smithi165 ceph-mon[32893]: pgmap v58: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:01 smithi137 ceph-mon[29634]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T08:21:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:02.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:01 smithi137 ceph-mon[41044]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:02.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T08:21:02.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:02.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:01 smithi165 ceph-mon[32893]: pgmap v59: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:02.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-09-21T08:21:02.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:03.221 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:02 smithi137 ceph-mon[29634]: Deploying daemon osd.2 on smithi137 2024-09-21T08:21:03.221 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:02 smithi137 ceph-mon[41044]: Deploying daemon osd.2 on smithi137 2024-09-21T08:21:03.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:02 smithi165 ceph-mon[32893]: Deploying daemon osd.2 on smithi137 2024-09-21T08:21:04.207 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:03 smithi137 ceph-mon[29634]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:04.207 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:03 smithi137 ceph-mon[41044]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:04.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:03 smithi165 ceph-mon[32893]: pgmap v60: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:05.912 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:05 smithi137 ceph-mon[29634]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:05.913 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:05 smithi137 ceph-mon[41044]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:06.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:05 smithi165 ceph-mon[32893]: pgmap v61: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:07.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:06 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:07.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:06 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:07.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:06 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:07.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:06 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:07.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:06 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:07.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:06 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:07.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:06 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:07.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:06 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:07.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:06 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:08.078 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:07 smithi137 ceph-mon[29634]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:08.078 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:07 smithi137 ceph-mon[41044]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:08.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:07 smithi165 ceph-mon[32893]: pgmap v62: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:10.090 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[29634]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:10.090 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.090 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.091 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:10.091 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:10.091 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.091 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[41044]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:10.091 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.091 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.091 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:10.091 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:10.091 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:09 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:09 smithi165 ceph-mon[32893]: pgmap v63: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:09 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.389 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 2 on host 'smithi137' 2024-09-21T08:21:10.969 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:10 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:10.969 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:10 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.969 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:10 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.969 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:10 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:10.970 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:10 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.970 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:10 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:10.997 DEBUG:teuthology.orchestra.run.smithi137:osd.2> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.2.service 2024-09-21T08:21:10.999 INFO:tasks.cephadm:Deploying osd.3 on smithi137 with /dev/vg_nvme/lv_1... 2024-09-21T08:21:11.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T08:21:11.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:10 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:11.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:10 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:11.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:10 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:11.821 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-2[57518]: 2024-09-21T08:21:11.726+0000 7ff22b59d640 -1 osd.2 0 log_to_monitors true 2024-09-21T08:21:11.849 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:21:12.077 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:11 smithi137 ceph-mon[29634]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:12.077 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:11 smithi137 ceph-mon[29634]: from='osd.2 v1:172.21.15.137:6809/1500397557' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T08:21:12.077 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:11 smithi137 ceph-mon[29634]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T08:21:12.077 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:11 smithi137 ceph-mon[41044]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:12.077 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:11 smithi137 ceph-mon[41044]: from='osd.2 v1:172.21.15.137:6809/1500397557' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T08:21:12.077 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:11 smithi137 ceph-mon[41044]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T08:21:12.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:11 smithi165 ceph-mon[32893]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:12.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:11 smithi165 ceph-mon[32893]: from='osd.2 v1:172.21.15.137:6809/1500397557' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T08:21:12.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:11 smithi165 ceph-mon[32893]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-09-21T08:21:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:13 smithi165 ceph-mon[32893]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T08:21:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:13 smithi165 ceph-mon[32893]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T08:21:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:13 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:13 smithi165 ceph-mon[32893]: from='osd.2 v1:172.21.15.137:6809/1500397557' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:13 smithi165 ceph-mon[32893]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:13 smithi165 ceph-mon[32893]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[41044]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T08:21:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[41044]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T08:21:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[41044]: from='osd.2 v1:172.21.15.137:6809/1500397557' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[41044]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[41044]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[29634]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-09-21T08:21:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[29634]: osdmap e16: 3 total, 2 up, 3 in 2024-09-21T08:21:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:13.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[29634]: from='osd.2 v1:172.21.15.137:6809/1500397557' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:13.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[29634]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:13.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-mon[29634]: pgmap v66: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:13.769 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-2[57518]: 2024-09-21T08:21:13.446+0000 7ff227520640 -1 osd.2 0 waiting for initial osdmap 2024-09-21T08:21:13.769 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:13 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-2[57518]: 2024-09-21T08:21:13.448+0000 7ff223348640 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T08:21:14.492 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:21:14.515 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:14 smithi137 ceph-mon[29634]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:21:14.516 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:14 smithi137 ceph-mon[29634]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T08:21:14.516 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:14 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:14.516 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:14 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:14.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:14 smithi137 ceph-mon[41044]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:21:14.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:14 smithi137 ceph-mon[41044]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T08:21:14.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:14 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:14.516 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:14 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:14.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi137:vg_nvme/lv_1 2024-09-21T08:21:14.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:14 smithi165 ceph-mon[32893]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:21:14.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:14 smithi165 ceph-mon[32893]: osdmap e17: 3 total, 2 up, 3 in 2024-09-21T08:21:14.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:14 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:14.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:14 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:15.156 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:21:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:15 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:21:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:15 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:21:15.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:15 smithi165 ceph-mon[32893]: osd.2 v1:172.21.15.137:6809/1500397557 boot 2024-09-21T08:21:15.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:15 smithi165 ceph-mon[32893]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T08:21:15.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:15 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:15.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:15 smithi165 ceph-mon[32893]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:15.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:15 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[29634]: osd.2 v1:172.21.15.137:6809/1500397557 boot 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[29634]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[29634]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[41044]: osd.2 v1:172.21.15.137:6809/1500397557 boot 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[41044]: osdmap e18: 3 total, 3 up, 3 in 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:21:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[41044]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-09-21T08:21:15.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:15 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='client.24202 v1:172.21.15.137:0/3174385803' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:16.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='client.24202 v1:172.21.15.137:0/3174385803' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:16.520 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:16.520 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:16.520 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.520 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:16.520 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:16.520 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:16 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-09-21T08:21:16.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: osdmap e19: 3 total, 3 up, 3 in 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='client.24202 v1:172.21.15.137:0/3174385803' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi137:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:16.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:16 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:17.324 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61599]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T08:21:17.324 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61599]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:17.324 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61599]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:17.324 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61599]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:17.605 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61602]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T08:21:17.605 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61602]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:17.605 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61602]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:17.605 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61602]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:17.605 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[41044]: Detected new or changed devices on smithi137 2024-09-21T08:21:17.605 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[41044]: Adjusting osd_memory_target on smithi137 to 5354M 2024-09-21T08:21:17.605 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T08:21:17.606 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[41044]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T08:21:17.606 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[41044]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:17.606 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:17.606 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[29634]: Detected new or changed devices on smithi137 2024-09-21T08:21:17.606 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[29634]: Adjusting osd_memory_target on smithi137 to 5354M 2024-09-21T08:21:17.606 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T08:21:17.606 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[29634]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T08:21:17.606 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[29634]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:17.606 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:17 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:17 smithi165 ceph-mon[32893]: Detected new or changed devices on smithi137 2024-09-21T08:21:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:17 smithi165 ceph-mon[32893]: Adjusting osd_memory_target on smithi137 to 5354M 2024-09-21T08:21:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:17 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-09-21T08:21:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:17 smithi165 ceph-mon[32893]: osdmap e20: 3 total, 3 up, 3 in 2024-09-21T08:21:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:17 smithi165 ceph-mon[32893]: pgmap v72: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:17.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:17 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:17.869 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61626]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T08:21:17.869 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61626]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:17.869 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61626]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:17.869 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61626]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:18.124 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61656]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T08:21:18.124 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61656]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:18.125 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61656]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:18.125 INFO:journalctl@ceph.osd.1.smithi137.stdout:Sep 21 08:21:17 smithi137 sudo[61656]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:18.403 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61682]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T08:21:18.403 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61682]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:18.404 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61682]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:18.404 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61682]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:18.654 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61708]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T08:21:18.654 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61708]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:18.655 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61708]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:18.655 INFO:journalctl@ceph.osd.2.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61708]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:18.917 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61719]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T08:21:18.917 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61719]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:18.917 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61719]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:18.918 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61719]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:18 smithi165 ceph-mon[32893]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:18 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3882023856' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73f21606-38be-4578-a86b-bb417217c0e0"}]: dispatch 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:18 smithi165 ceph-mon[32893]: from='client.44207 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73f21606-38be-4578-a86b-bb417217c0e0"}]: dispatch 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:18 smithi165 ceph-mon[32893]: from='client.44207 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73f21606-38be-4578-a86b-bb417217c0e0"}]': finished 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:18 smithi165 ceph-mon[32893]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:18 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:18 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:18 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:18 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 sudo[35562]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 sudo[35562]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 sudo[35562]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:19.184 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 sudo[35562]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61741]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61741]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61741]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 sudo[61741]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[29634]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3882023856' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73f21606-38be-4578-a86b-bb417217c0e0"}]: dispatch 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[29634]: from='client.44207 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73f21606-38be-4578-a86b-bb417217c0e0"}]: dispatch 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[29634]: from='client.44207 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73f21606-38be-4578-a86b-bb417217c0e0"}]': finished 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[29634]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:19.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[41044]: osdmap e21: 3 total, 3 up, 3 in 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3882023856' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73f21606-38be-4578-a86b-bb417217c0e0"}]: dispatch 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[41044]: from='client.44207 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "73f21606-38be-4578-a86b-bb417217c0e0"}]: dispatch 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[41044]: from='client.44207 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "73f21606-38be-4578-a86b-bb417217c0e0"}]': finished 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[41044]: osdmap e22: 4 total, 3 up, 4 in 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:21:19.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:18 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:21:19.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 sudo[35565]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T08:21:19.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 sudo[35565]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:19.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 sudo[35565]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:19.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 sudo[35565]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:19.945 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 sudo[61791]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-09-21T08:21:19.945 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 sudo[61791]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:19.945 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 sudo[61791]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:19.945 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 sudo[61791]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T08:21:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T08:21:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:21:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:21:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:21:20.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T08:21:20.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2292540417' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:21:20.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T08:21:20.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T08:21:20.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:20.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:21:20.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:21:20.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2292540417' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 sudo[61796]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 sudo[61796]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 sudo[61796]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 sudo[61796]: pam_unix(sudo:session): session closed for user root 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: pgmap v75: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:21:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T08:21:20.270 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2292540417' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:21:20.270 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T08:21:20.270 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: mgrmap e15: y(active, since 2m), standbys: x 2024-09-21T08:21:20.270 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:20.270 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:21:20.270 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:21:20.270 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:21:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:20 smithi165 ceph-mon[32893]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T08:21:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:20 smithi165 ceph-mon[32893]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T08:21:21.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:20 smithi137 ceph-mon[29634]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T08:21:21.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:20 smithi137 ceph-mon[29634]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T08:21:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:20 smithi137 ceph-mon[41044]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-09-21T08:21:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:20 smithi137 ceph-mon[41044]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-09-21T08:21:22.249 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:21 smithi137 ceph-mon[29634]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:22.250 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:21 smithi137 ceph-mon[41044]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:22.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:21 smithi165 ceph-mon[32893]: pgmap v76: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:23.266 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:23 smithi137 ceph-mon[29634]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:23.266 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:23 smithi137 ceph-mon[41044]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:23.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:23 smithi165 ceph-mon[32893]: pgmap v77: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:26.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:25 smithi137 ceph-mon[41044]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:26.140 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:25 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T08:21:26.141 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:25 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:26.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:25 smithi137 ceph-mon[29634]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:26.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:25 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T08:21:26.141 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:25 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:26.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:25 smithi165 ceph-mon[32893]: pgmap v78: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:26.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:25 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-09-21T08:21:26.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:25 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:27.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:26 smithi137 ceph-mon[29634]: Deploying daemon osd.3 on smithi137 2024-09-21T08:21:27.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:26 smithi137 ceph-mon[41044]: Deploying daemon osd.3 on smithi137 2024-09-21T08:21:27.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:26 smithi165 ceph-mon[32893]: Deploying daemon osd.3 on smithi137 2024-09-21T08:21:28.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:27 smithi137 ceph-mon[29634]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:28.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:27 smithi137 ceph-mon[41044]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:28.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:27 smithi165 ceph-mon[32893]: pgmap v79: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:30.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:29 smithi137 ceph-mon[29634]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:30.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:29 smithi137 ceph-mon[41044]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:30.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:29 smithi165 ceph-mon[32893]: pgmap v80: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:31.017 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:30 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:30 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:30 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:30 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:30 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:30 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:31.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:30 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:31.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:30 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:31.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:30 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:32.185 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:31 smithi137 ceph-mon[29634]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:32.185 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:31 smithi137 ceph-mon[41044]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:32.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:31 smithi165 ceph-mon[32893]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:33.973 INFO:teuthology.orchestra.run.smithi137.stdout:Created osd(s) 3 on host 'smithi137' 2024-09-21T08:21:34.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:33 smithi165 ceph-mon[32893]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:34.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:33 smithi137 ceph-mon[29634]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:34.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:33 smithi137 ceph-mon[41044]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:34.593 DEBUG:teuthology.orchestra.run.smithi137:osd.3> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.3.service 2024-09-21T08:21:34.595 INFO:tasks.cephadm:Deploying osd.4 on smithi165 with /dev/vg_nvme/lv_4... 2024-09-21T08:21:34.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_4 2024-09-21T08:21:34.847 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:21:35.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:34 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:34 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:34 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:34 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.235 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:34 smithi165 ceph-mon[32893]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:35.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:34 smithi137 ceph-mon[29634]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:35.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:34 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:34 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:34 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:34 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:35.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:34 smithi137 ceph-mon[41044]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:36.414 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 21 08:21:36 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-3[63522]: 2024-09-21T08:21:36.035+0000 7fcaa9cf4640 -1 osd.3 0 log_to_monitors true 2024-09-21T08:21:36.665 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:36 smithi137 ceph-mon[29634]: from='osd.3 v1:172.21.15.137:6813/617155021' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T08:21:36.665 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:36 smithi137 ceph-mon[29634]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T08:21:36.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:36 smithi137 ceph-mon[41044]: from='osd.3 v1:172.21.15.137:6813/617155021' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T08:21:36.665 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:36 smithi137 ceph-mon[41044]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T08:21:36.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:36 smithi165 ceph-mon[32893]: from='osd.3 v1:172.21.15.137:6813/617155021' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T08:21:36.951 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:36 smithi165 ceph-mon[32893]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-09-21T08:21:37.247 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:21:37.272 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi165:vg_nvme/lv_4 2024-09-21T08:21:37.513 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:21:37.880 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:37 smithi165 ceph-mon[32893]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T08:21:37.880 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:37 smithi165 ceph-mon[32893]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T08:21:37.880 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:37 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:37.880 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:37 smithi165 ceph-mon[32893]: from='osd.3 v1:172.21.15.137:6813/617155021' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:37.880 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:37 smithi165 ceph-mon[32893]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:37.880 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:37 smithi165 ceph-mon[32893]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:37.954 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[29634]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T08:21:37.954 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[29634]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T08:21:37.955 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:37.955 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[29634]: from='osd.3 v1:172.21.15.137:6813/617155021' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:37.955 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[29634]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:37.955 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[29634]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:37.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[41044]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-09-21T08:21:37.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[41044]: osdmap e23: 4 total, 3 up, 4 in 2024-09-21T08:21:37.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:37.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[41044]: from='osd.3 v1:172.21.15.137:6813/617155021' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:37.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[41044]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:21:37.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-mon[41044]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:37.955 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-3[63522]: 2024-09-21T08:21:37.669+0000 7fcaa5c77640 -1 osd.3 0 waiting for initial osdmap 2024-09-21T08:21:37.956 INFO:journalctl@ceph.osd.3.smithi137.stdout:Sep 21 08:21:37 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-3[63522]: 2024-09-21T08:21:37.672+0000 7fcaa1a9f640 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T08:21:38.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:21:38.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T08:21:38.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:38.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:38.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:38.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:38.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:38.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:38.858 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:38.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:21:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T08:21:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]': finished 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: osdmap e24: 4 total, 3 up, 4 in 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:21:39.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:39.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:39.762 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:39 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:21:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:39 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:21:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:39 smithi165 ceph-mon[32893]: Detected new or changed devices on smithi137 2024-09-21T08:21:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:39 smithi165 ceph-mon[32893]: Adjusting osd_memory_target on smithi137 to 4016M 2024-09-21T08:21:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:39 smithi165 ceph-mon[32893]: from='client.24235 v1:172.21.15.165:0/4100572540' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:21:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:39 smithi165 ceph-mon[32893]: osd.3 v1:172.21.15.137:6813/617155021 boot 2024-09-21T08:21:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:39 smithi165 ceph-mon[32893]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T08:21:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:39 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:39 smithi165 ceph-mon[32893]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:21:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:21:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[29634]: Detected new or changed devices on smithi137 2024-09-21T08:21:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[29634]: Adjusting osd_memory_target on smithi137 to 4016M 2024-09-21T08:21:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[29634]: from='client.24235 v1:172.21.15.165:0/4100572540' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:21:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[29634]: osd.3 v1:172.21.15.137:6813/617155021 boot 2024-09-21T08:21:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[29634]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T08:21:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:40.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[29634]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:21:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:21:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[41044]: Detected new or changed devices on smithi137 2024-09-21T08:21:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[41044]: Adjusting osd_memory_target on smithi137 to 4016M 2024-09-21T08:21:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[41044]: from='client.24235 v1:172.21.15.165:0/4100572540' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:21:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[41044]: osd.3 v1:172.21.15.137:6813/617155021 boot 2024-09-21T08:21:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[41044]: osdmap e25: 4 total, 4 up, 4 in 2024-09-21T08:21:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:21:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:39 smithi137 ceph-mon[41044]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 82 MiB used, 268 GiB / 268 GiB avail 2024-09-21T08:21:40.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[29634]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T08:21:40.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/1512931845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae"}]: dispatch 2024-09-21T08:21:40.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[29634]: from='client.24241 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae"}]: dispatch 2024-09-21T08:21:40.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[29634]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae"}]': finished 2024-09-21T08:21:40.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[29634]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T08:21:40.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:21:40.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[41044]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T08:21:40.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/1512931845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae"}]: dispatch 2024-09-21T08:21:40.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[41044]: from='client.24241 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae"}]: dispatch 2024-09-21T08:21:40.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[41044]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae"}]': finished 2024-09-21T08:21:40.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[41044]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T08:21:40.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:40 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:21:40.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:40 smithi165 ceph-mon[32893]: osdmap e26: 4 total, 4 up, 4 in 2024-09-21T08:21:40.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:40 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/1512931845' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae"}]: dispatch 2024-09-21T08:21:40.965 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:40 smithi165 ceph-mon[32893]: from='client.24241 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae"}]: dispatch 2024-09-21T08:21:40.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:40 smithi165 ceph-mon[32893]: from='client.24241 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae"}]': finished 2024-09-21T08:21:40.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:40 smithi165 ceph-mon[32893]: osdmap e27: 5 total, 4 up, 5 in 2024-09-21T08:21:40.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:40 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:21:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:41 smithi165 ceph-mon[32893]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:41 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/3874491247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:21:42.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:41 smithi137 ceph-mon[29634]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:41 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/3874491247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:21:42.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:41 smithi137 ceph-mon[41044]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:42.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:41 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/3874491247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:21:44.088 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:43 smithi137 ceph-mon[29634]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:44.088 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:43 smithi137 ceph-mon[41044]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:44.157 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:43 smithi165 ceph-mon[32893]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:46.215 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:45 smithi165 ceph-mon[32893]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:46.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:45 smithi137 ceph-mon[29634]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:46.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:45 smithi137 ceph-mon[41044]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:46 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:21:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:46 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:21:47.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:46 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:21:47.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:46 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:21:47.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:46 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:21:47.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:46 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:21:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:47 smithi165 ceph-mon[32893]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:47 smithi165 ceph-mon[32893]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T08:21:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:47 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:21:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:47 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T08:21:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:47 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:48.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:47 smithi137 ceph-mon[29634]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:48.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:47 smithi137 ceph-mon[29634]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T08:21:48.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:47 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:21:48.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:47 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T08:21:48.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:47 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:48.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:47 smithi137 ceph-mon[41044]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:48.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:47 smithi137 ceph-mon[41044]: osdmap e28: 5 total, 4 up, 5 in 2024-09-21T08:21:48.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:47 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:21:48.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:47 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-09-21T08:21:48.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:47 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:49.027 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:48 smithi165 ceph-mon[32893]: Deploying daemon osd.4 on smithi165 2024-09-21T08:21:49.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:48 smithi137 ceph-mon[29634]: Deploying daemon osd.4 on smithi165 2024-09-21T08:21:49.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:48 smithi137 ceph-mon[41044]: Deploying daemon osd.4 on smithi165 2024-09-21T08:21:49.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:49 smithi165 ceph-mon[32893]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:50.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:49 smithi137 ceph-mon[29634]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:50.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:49 smithi137 ceph-mon[41044]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:52.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:51 smithi165 ceph-mon[32893]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:52.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:51 smithi137 ceph-mon[29634]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:52.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:51 smithi137 ceph-mon[41044]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:54.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:53 smithi165 ceph-mon[32893]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:54.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:53 smithi137 ceph-mon[41044]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:54.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:53 smithi137 ceph-mon[29634]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:55.118 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:55.118 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:55.118 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:55.118 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:55.118 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:55.119 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:55.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:56.178 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:55 smithi165 ceph-mon[32893]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:56.179 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:55 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:56.179 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:55 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:56.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:55 smithi137 ceph-mon[29634]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:56.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:55 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:56.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:55 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:56.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:55 smithi137 ceph-mon[41044]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:56.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:55 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:56.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:55 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:57.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:57.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:57.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:57.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:57.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:57.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:21:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:21:57.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:58.100 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:57 smithi165 ceph-mon[32893]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:58.140 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 4 on host 'smithi165' 2024-09-21T08:21:58.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:57 smithi137 ceph-mon[29634]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:58.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:57 smithi137 ceph-mon[41044]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:21:58.732 DEBUG:teuthology.orchestra.run.smithi165:osd.4> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.4.service 2024-09-21T08:21:58.734 INFO:tasks.cephadm:Deploying osd.5 on smithi165 with /dev/vg_nvme/lv_3... 2024-09-21T08:21:58.735 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_3 2024-09-21T08:21:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:59.025 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:59.025 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:58 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:59.025 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:58 smithi165 ceph-mon[32893]: from='osd.4 v1:172.21.15.165:6800/70360958' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T08:21:59.025 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:58 smithi165 ceph-mon[32893]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T08:21:59.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:59.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:59.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:58 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:59.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:58 smithi137 ceph-mon[29634]: from='osd.4 v1:172.21.15.165:6800/70360958' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T08:21:59.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:58 smithi137 ceph-mon[29634]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T08:21:59.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:21:59.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:59.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:58 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:21:59.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:58 smithi137 ceph-mon[41044]: from='osd.4 v1:172.21.15.165:6800/70360958' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T08:21:59.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:58 smithi137 ceph-mon[41044]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-09-21T08:21:59.461 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:22:00.095 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:59 smithi165 ceph-mon[32893]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:22:00.095 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:59 smithi165 ceph-mon[32893]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T08:22:00.095 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:59 smithi165 ceph-mon[32893]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T08:22:00.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:59 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:00.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:59 smithi165 ceph-mon[32893]: from='osd.4 v1:172.21.15.165:6800/70360958' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:00.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:21:59 smithi165 ceph-mon[32893]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:00.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[29634]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:22:00.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[29634]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T08:22:00.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[29634]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T08:22:00.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:00.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[29634]: from='osd.4 v1:172.21.15.165:6800/70360958' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:00.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[29634]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:00.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[41044]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:22:00.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[41044]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-09-21T08:22:00.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[41044]: osdmap e29: 5 total, 4 up, 5 in 2024-09-21T08:22:00.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:00.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[41044]: from='osd.4 v1:172.21.15.165:6800/70360958' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:00.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:21:59 smithi137 ceph-mon[41044]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:00.763 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 21 08:22:00 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-4[38072]: 2024-09-21T08:22:00.576+0000 7fd041096640 -1 osd.4 0 waiting for initial osdmap 2024-09-21T08:22:00.763 INFO:journalctl@ceph.osd.4.smithi165.stdout:Sep 21 08:22:00 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-4[38072]: 2024-09-21T08:22:00.580+0000 7fd038eb6640 -1 osd.4 30 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T08:22:01.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:01 smithi165 ceph-mon[32893]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:22:01.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:01 smithi165 ceph-mon[32893]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T08:22:01.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:01.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:01.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:01 smithi165 ceph-mon[32893]: from='osd.4 ' entity='osd.4' 2024-09-21T08:22:01.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:01 smithi165 ceph-mon[32893]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:22:01.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[29634]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:22:01.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[29634]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T08:22:01.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:01.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:01.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[29634]: from='osd.4 ' entity='osd.4' 2024-09-21T08:22:01.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[29634]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:22:01.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[41044]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:22:01.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[41044]: osdmap e30: 5 total, 4 up, 5 in 2024-09-21T08:22:01.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:01.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:01.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[41044]: from='osd.4 ' entity='osd.4' 2024-09-21T08:22:01.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:01 smithi137 ceph-mon[41044]: pgmap v104: 1 pgs: 1 peering; 577 KiB data, 108 MiB used, 358 GiB / 358 GiB avail 2024-09-21T08:22:02.239 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:02 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:22:02.240 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:02 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:22:02.240 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:02 smithi165 ceph-mon[32893]: osd.4 v1:172.21.15.165:6800/70360958 boot 2024-09-21T08:22:02.240 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:02 smithi165 ceph-mon[32893]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T08:22:02.240 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:02 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:02.240 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:02 smithi165 ceph-mon[32893]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T08:22:02.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:22:02.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:22:02.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[29634]: osd.4 v1:172.21.15.165:6800/70360958 boot 2024-09-21T08:22:02.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[29634]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T08:22:02.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:02.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[29634]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T08:22:02.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:22:02.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:22:02.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[41044]: osd.4 v1:172.21.15.165:6800/70360958 boot 2024-09-21T08:22:02.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[41044]: osdmap e31: 5 total, 5 up, 5 in 2024-09-21T08:22:02.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:22:02.519 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:02 smithi137 ceph-mon[41044]: osdmap e32: 5 total, 5 up, 5 in 2024-09-21T08:22:03.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: Detected new or changed devices on smithi165 2024-09-21T08:22:03.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:03.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: Adjusting osd_memory_target on smithi165 to 17088M 2024-09-21T08:22:03.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.567 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:03.568 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:03.568 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.568 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T08:22:03.568 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:03 smithi165 ceph-mon[32893]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:03.570 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:22:03.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi165:vg_nvme/lv_3 2024-09-21T08:22:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: Detected new or changed devices on smithi165 2024-09-21T08:22:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: Adjusting osd_memory_target on smithi165 to 17088M 2024-09-21T08:22:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[29634]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: Detected new or changed devices on smithi165 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: Adjusting osd_memory_target on smithi165 to 17088M 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: osdmap e33: 5 total, 5 up, 5 in 2024-09-21T08:22:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:03 smithi137 ceph-mon[41044]: pgmap v108: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:03.842 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:22:04.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:04 smithi165 ceph-mon[32893]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T08:22:04.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:04 smithi137 ceph-mon[29634]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T08:22:04.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:04 smithi137 ceph-mon[41044]: osdmap e34: 5 total, 5 up, 5 in 2024-09-21T08:22:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:05 smithi165 ceph-mon[32893]: osdmap e35: 5 total, 5 up, 5 in 2024-09-21T08:22:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:05 smithi165 ceph-mon[32893]: from='client.24262 v1:172.21.15.165:0/3147639697' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:22:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:05 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:22:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:05 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:22:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:05 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:05 smithi165 ceph-mon[32893]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[29634]: osdmap e35: 5 total, 5 up, 5 in 2024-09-21T08:22:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[29634]: from='client.24262 v1:172.21.15.165:0/3147639697' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:22:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:22:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:22:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[29634]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[41044]: osdmap e35: 5 total, 5 up, 5 in 2024-09-21T08:22:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[41044]: from='client.24262 v1:172.21.15.165:0/3147639697' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:22:05.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:22:05.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:22:05.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:05.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:05 smithi137 ceph-mon[41044]: pgmap v111: 1 pgs: 1 peering; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:07.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:06 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/1754922686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd"}]: dispatch 2024-09-21T08:22:07.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:06 smithi165 ceph-mon[32893]: from='client.24268 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd"}]: dispatch 2024-09-21T08:22:07.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:06 smithi165 ceph-mon[32893]: from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd"}]': finished 2024-09-21T08:22:07.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:06 smithi165 ceph-mon[32893]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T08:22:07.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:06 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:07.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:06 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/1754922686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd"}]: dispatch 2024-09-21T08:22:07.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:06 smithi137 ceph-mon[29634]: from='client.24268 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd"}]: dispatch 2024-09-21T08:22:07.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:06 smithi137 ceph-mon[29634]: from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd"}]': finished 2024-09-21T08:22:07.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:06 smithi137 ceph-mon[29634]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T08:22:07.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:06 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:07.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:06 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/1754922686' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd"}]: dispatch 2024-09-21T08:22:07.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:06 smithi137 ceph-mon[41044]: from='client.24268 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd"}]: dispatch 2024-09-21T08:22:07.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:06 smithi137 ceph-mon[41044]: from='client.24268 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd"}]': finished 2024-09-21T08:22:07.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:06 smithi137 ceph-mon[41044]: osdmap e36: 6 total, 5 up, 6 in 2024-09-21T08:22:07.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:06 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:08.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:07 smithi165 ceph-mon[32893]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-09-21T08:22:08.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:07 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/768020770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:22:08.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:07 smithi137 ceph-mon[41044]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-09-21T08:22:08.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:07 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/768020770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:22:08.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:07 smithi137 ceph-mon[29634]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 108 KiB/s, 0 objects/s recovering 2024-09-21T08:22:08.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:07 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/768020770' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:22:10.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:09 smithi165 ceph-mon[32893]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T08:22:10.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:09 smithi137 ceph-mon[29634]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T08:22:10.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:09 smithi137 ceph-mon[41044]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 136 MiB used, 447 GiB / 447 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T08:22:12.006 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:11 smithi165 ceph-mon[32893]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T08:22:12.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:11 smithi137 ceph-mon[29634]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T08:22:12.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:11 smithi137 ceph-mon[41044]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-09-21T08:22:13.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:12 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:22:13.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:12 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:22:13.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:12 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:22:13.169 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:12 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:22:13.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:12 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:22:13.169 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:12 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:22:14.216 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:13 smithi165 ceph-mon[32893]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-21T08:22:14.216 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:13 smithi165 ceph-mon[32893]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T08:22:14.216 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:13 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:14.216 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:13 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T08:22:14.216 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:13 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:14.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:13 smithi137 ceph-mon[29634]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-21T08:22:14.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:13 smithi137 ceph-mon[29634]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T08:22:14.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:13 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:14.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:13 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T08:22:14.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:13 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:14.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:13 smithi137 ceph-mon[41044]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-09-21T08:22:14.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:13 smithi137 ceph-mon[41044]: osdmap e37: 6 total, 5 up, 6 in 2024-09-21T08:22:14.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:13 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:14.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:13 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-09-21T08:22:14.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:13 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:15.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:14 smithi165 ceph-mon[32893]: Deploying daemon osd.5 on smithi165 2024-09-21T08:22:15.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:14 smithi137 ceph-mon[41044]: Deploying daemon osd.5 on smithi165 2024-09-21T08:22:15.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:14 smithi137 ceph-mon[29634]: Deploying daemon osd.5 on smithi165 2024-09-21T08:22:16.193 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:15 smithi165 ceph-mon[32893]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:16.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:15 smithi137 ceph-mon[41044]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:16.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:15 smithi137 ceph-mon[29634]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:17 smithi165 ceph-mon[32893]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:17.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:17 smithi137 ceph-mon[41044]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:17 smithi137 ceph-mon[29634]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:19 smithi165 ceph-mon[32893]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:19 smithi137 ceph-mon[29634]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:20.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:19 smithi137 ceph-mon[41044]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:20.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:20.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:20.269 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:20 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:22:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:20 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:22:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:20 smithi165 ceph-mon[32893]: osdmap e38: 6 total, 5 up, 6 in 2024-09-21T08:22:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:20 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:21.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:20 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:22:21.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:20 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:22:21.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:20 smithi137 ceph-mon[29634]: osdmap e38: 6 total, 5 up, 6 in 2024-09-21T08:22:21.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:20 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:20 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:22:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:20 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:22:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:20 smithi137 ceph-mon[41044]: osdmap e38: 6 total, 5 up, 6 in 2024-09-21T08:22:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:20 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:22.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:21 smithi165 ceph-mon[32893]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:22.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:21 smithi137 ceph-mon[29634]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:22.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:21 smithi137 ceph-mon[41044]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:23.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:23 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:23.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:23 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:23.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:23 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:23.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:23 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:23.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:23 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:23.663 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:23 smithi165 ceph-mon[32893]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:23.706 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 5 on host 'smithi165' 2024-09-21T08:22:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[41044]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:23.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:23.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:23.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:23.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:23 smithi137 ceph-mon[29634]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:24.283 DEBUG:teuthology.orchestra.run.smithi165:osd.5> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.5.service 2024-09-21T08:22:24.285 INFO:tasks.cephadm:Deploying osd.6 on smithi165 with /dev/vg_nvme/lv_2... 2024-09-21T08:22:24.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_2 2024-09-21T08:22:24.550 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:24.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:24.551 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:24 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:24.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:24.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:24.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:24 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:24.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:24.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:24.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:24 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:24.861 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 21 08:22:24 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-5[44301]: 2024-09-21T08:22:24.548+0000 7f95bd6cf640 -1 osd.5 0 log_to_monitors true 2024-09-21T08:22:25.077 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:22:25.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:25 smithi165 ceph-mon[32893]: from='osd.5 v1:172.21.15.165:6804/1835779380' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T08:22:25.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:25 smithi165 ceph-mon[32893]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T08:22:25.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:25 smithi165 ceph-mon[32893]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:25 smithi137 ceph-mon[29634]: from='osd.5 v1:172.21.15.165:6804/1835779380' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T08:22:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:25 smithi137 ceph-mon[29634]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T08:22:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:25 smithi137 ceph-mon[29634]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:25 smithi137 ceph-mon[41044]: from='osd.5 v1:172.21.15.165:6804/1835779380' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T08:22:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:25 smithi137 ceph-mon[41044]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-09-21T08:22:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:25 smithi137 ceph-mon[41044]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:26.661 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 21 08:22:26 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-5[44301]: 2024-09-21T08:22:26.390+0000 7f95b9652640 -1 osd.5 0 waiting for initial osdmap 2024-09-21T08:22:26.661 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 21 08:22:26 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-5[44301]: 2024-09-21T08:22:26.394+0000 7f95b547a640 -1 osd.5 40 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T08:22:26.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:26 smithi165 ceph-mon[32893]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T08:22:26.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:26 smithi165 ceph-mon[32893]: from='osd.5 v1:172.21.15.165:6804/1835779380' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:26.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:26 smithi165 ceph-mon[32893]: osdmap e39: 6 total, 5 up, 6 in 2024-09-21T08:22:26.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:26 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:26.661 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:26 smithi165 ceph-mon[32893]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:26 smithi137 ceph-mon[29634]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T08:22:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:26 smithi137 ceph-mon[29634]: from='osd.5 v1:172.21.15.165:6804/1835779380' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:26 smithi137 ceph-mon[29634]: osdmap e39: 6 total, 5 up, 6 in 2024-09-21T08:22:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:26 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:26 smithi137 ceph-mon[29634]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:26.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:26 smithi137 ceph-mon[41044]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-09-21T08:22:26.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:26 smithi137 ceph-mon[41044]: from='osd.5 v1:172.21.15.165:6804/1835779380' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:26.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:26 smithi137 ceph-mon[41044]: osdmap e39: 6 total, 5 up, 6 in 2024-09-21T08:22:26.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:26 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:26.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:26 smithi137 ceph-mon[41044]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:27.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:27 smithi165 ceph-mon[32893]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:22:27.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:27 smithi165 ceph-mon[32893]: osdmap e40: 6 total, 5 up, 6 in 2024-09-21T08:22:27.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:27.710 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:27 smithi165 ceph-mon[32893]: osd.5 v1:172.21.15.165:6804/1835779380 boot 2024-09-21T08:22:27.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:27 smithi165 ceph-mon[32893]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T08:22:27.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:27.711 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:27 smithi165 ceph-mon[32893]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:27.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[29634]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:22:27.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[29634]: osdmap e40: 6 total, 5 up, 6 in 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[29634]: osd.5 v1:172.21.15.165:6804/1835779380 boot 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[29634]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[29634]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[41044]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[41044]: osdmap e40: 6 total, 5 up, 6 in 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[41044]: osd.5 v1:172.21.15.165:6804/1835779380 boot 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[41044]: osdmap e41: 6 total, 6 up, 6 in 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:22:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:27 smithi137 ceph-mon[41044]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 135 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:22:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:22:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: Detected new or changed devices on smithi165 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: Adjusting osd_memory_target on smithi165 to 8544M 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:28 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: Detected new or changed devices on smithi165 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: Adjusting osd_memory_target on smithi165 to 8544M 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: osdmap e42: 6 total, 6 up, 6 in 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: Detected new or changed devices on smithi165 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: Adjusting osd_memory_target on smithi165 to 8544M 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:28.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:28 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:29.044 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:22:29.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi165:vg_nvme/lv_2 2024-09-21T08:22:29.312 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:22:29.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:29 smithi165 ceph-mon[32893]: osdmap e43: 6 total, 6 up, 6 in 2024-09-21T08:22:29.759 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:29 smithi165 ceph-mon[32893]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:29.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:29 smithi137 ceph-mon[29634]: osdmap e43: 6 total, 6 up, 6 in 2024-09-21T08:22:29.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:29 smithi137 ceph-mon[29634]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:29.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:29 smithi137 ceph-mon[41044]: osdmap e43: 6 total, 6 up, 6 in 2024-09-21T08:22:29.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:29 smithi137 ceph-mon[41044]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:30.596 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:30 smithi165 ceph-mon[32893]: osdmap e44: 6 total, 6 up, 6 in 2024-09-21T08:22:30.596 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:30 smithi165 ceph-mon[32893]: from='client.24289 v1:172.21.15.165:0/302273726' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:22:30.596 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:30 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:22:30.596 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:30 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:22:30.596 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:30 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:30.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:30 smithi137 ceph-mon[29634]: osdmap e44: 6 total, 6 up, 6 in 2024-09-21T08:22:30.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:30 smithi137 ceph-mon[29634]: from='client.24289 v1:172.21.15.165:0/302273726' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:22:30.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:30 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:22:30.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:30 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:22:30.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:30 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:30.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:30 smithi137 ceph-mon[41044]: osdmap e44: 6 total, 6 up, 6 in 2024-09-21T08:22:30.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:30 smithi137 ceph-mon[41044]: from='client.24289 v1:172.21.15.165:0/302273726' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:22:30.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:30 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:22:30.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:30 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:22:30.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:30 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:31.585 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:31 smithi165 ceph-mon[32893]: osdmap e45: 6 total, 6 up, 6 in 2024-09-21T08:22:31.585 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:31 smithi165 ceph-mon[32893]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:31 smithi137 ceph-mon[29634]: osdmap e45: 6 total, 6 up, 6 in 2024-09-21T08:22:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:31 smithi137 ceph-mon[29634]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:31.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:31 smithi137 ceph-mon[41044]: osdmap e45: 6 total, 6 up, 6 in 2024-09-21T08:22:31.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:31 smithi137 ceph-mon[41044]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:32 smithi165 ceph-mon[32893]: 1.0 scrub starts 2024-09-21T08:22:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:32 smithi165 ceph-mon[32893]: 1.0 scrub ok 2024-09-21T08:22:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:32 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/3774779184' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b"}]: dispatch 2024-09-21T08:22:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:32 smithi165 ceph-mon[32893]: from='client.24295 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b"}]: dispatch 2024-09-21T08:22:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:32 smithi165 ceph-mon[32893]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b"}]': finished 2024-09-21T08:22:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:32 smithi165 ceph-mon[32893]: osdmap e46: 7 total, 6 up, 7 in 2024-09-21T08:22:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:32 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:32.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[29634]: 1.0 scrub starts 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[29634]: 1.0 scrub ok 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/3774779184' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b"}]: dispatch 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[29634]: from='client.24295 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b"}]: dispatch 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[29634]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b"}]': finished 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[29634]: osdmap e46: 7 total, 6 up, 7 in 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[41044]: 1.0 scrub starts 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[41044]: 1.0 scrub ok 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/3774779184' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b"}]: dispatch 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[41044]: from='client.24295 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b"}]: dispatch 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[41044]: from='client.24295 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b"}]': finished 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[41044]: osdmap e46: 7 total, 6 up, 7 in 2024-09-21T08:22:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:32 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:33.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:33 smithi165 ceph-mon[32893]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:33.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:33 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/879566231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:22:33.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:33 smithi137 ceph-mon[29634]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:33.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:33 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/879566231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:22:33.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:33 smithi137 ceph-mon[41044]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:33.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:33 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/879566231' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:22:36.096 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:35 smithi165 ceph-mon[32893]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:36.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:35 smithi137 ceph-mon[29634]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:36.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:35 smithi137 ceph-mon[41044]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:37.736 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:37 smithi165 ceph-mon[32893]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:37.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:37 smithi137 ceph-mon[29634]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:37.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:37 smithi137 ceph-mon[41044]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:39 smithi165 ceph-mon[32893]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:39 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T08:22:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:39 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:40.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:39 smithi137 ceph-mon[29634]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:40.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:39 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T08:22:40.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:39 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:40.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:39 smithi137 ceph-mon[41044]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:40.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:39 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-09-21T08:22:40.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:39 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:41.134 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:40 smithi165 ceph-mon[32893]: Deploying daemon osd.6 on smithi165 2024-09-21T08:22:41.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:40 smithi137 ceph-mon[29634]: Deploying daemon osd.6 on smithi165 2024-09-21T08:22:41.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:40 smithi137 ceph-mon[41044]: Deploying daemon osd.6 on smithi165 2024-09-21T08:22:42.008 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:41 smithi165 ceph-mon[32893]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:42.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:41 smithi137 ceph-mon[29634]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:42.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:41 smithi137 ceph-mon[41044]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:44.080 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:43 smithi165 ceph-mon[32893]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:44.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:43 smithi137 ceph-mon[29634]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:44.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:43 smithi137 ceph-mon[41044]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:46.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:45 smithi165 ceph-mon[32893]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:46.174 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:45 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:46.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:45 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:46.175 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:45 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:46.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:45 smithi137 ceph-mon[41044]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:46.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:45 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:46.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:45 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:46.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:45 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:46.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:45 smithi137 ceph-mon[29634]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:46.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:45 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:46.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:45 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:46.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:45 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:48.142 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:47 smithi165 ceph-mon[32893]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:48.267 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:47 smithi137 ceph-mon[41044]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:48.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:47 smithi137 ceph-mon[29634]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:49.352 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 6 on host 'smithi165' 2024-09-21T08:22:49.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:49.695 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:49.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:49 smithi165 ceph-mon[32893]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:49.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:49.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[41044]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:49.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:49.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[29634]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:49.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:22:49.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:49.948 DEBUG:teuthology.orchestra.run.smithi165:osd.6> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.6.service 2024-09-21T08:22:49.950 INFO:tasks.cephadm:Deploying osd.7 on smithi165 with /dev/vg_nvme/lv_1... 2024-09-21T08:22:49.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 ceph-volume -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- lvm zap /dev/vg_nvme/lv_1 2024-09-21T08:22:50.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:50 smithi165 ceph-mon[32893]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T08:22:50.513 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 21 08:22:50 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-6[50634]: 2024-09-21T08:22:50.393+0000 7fb3b8809640 -1 osd.6 0 log_to_monitors true 2024-09-21T08:22:50.681 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:22:50.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:50 smithi137 ceph-mon[29634]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T08:22:50.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:50 smithi137 ceph-mon[41044]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-09-21T08:22:51.513 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 21 08:22:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-6[50634]: 2024-09-21T08:22:51.480+0000 7fb3b478c640 -1 osd.6 0 waiting for initial osdmap 2024-09-21T08:22:51.513 INFO:journalctl@ceph.osd.6.smithi165.stdout:Sep 21 08:22:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-6[50634]: 2024-09-21T08:22:51.488+0000 7fb3b05b4640 -1 osd.6 48 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T08:22:51.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:51 smithi165 ceph-mon[32893]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T08:22:51.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:51 smithi165 ceph-mon[32893]: osdmap e47: 7 total, 6 up, 7 in 2024-09-21T08:22:51.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:51 smithi165 ceph-mon[32893]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:51.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:51 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:51.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:51 smithi165 ceph-mon[32893]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:51.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:51 smithi137 ceph-mon[29634]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T08:22:51.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:51 smithi137 ceph-mon[29634]: osdmap e47: 7 total, 6 up, 7 in 2024-09-21T08:22:51.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:51 smithi137 ceph-mon[29634]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:51.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:51 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:51.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:51 smithi137 ceph-mon[29634]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:51.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:51 smithi137 ceph-mon[41044]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-09-21T08:22:51.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:51 smithi137 ceph-mon[41044]: osdmap e47: 7 total, 6 up, 7 in 2024-09-21T08:22:51.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:51 smithi137 ceph-mon[41044]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:22:51.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:51 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:51.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:51 smithi137 ceph-mon[41044]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:52.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:52 smithi165 ceph-mon[32893]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:22:52.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:52 smithi165 ceph-mon[32893]: osdmap e48: 7 total, 6 up, 7 in 2024-09-21T08:22:52.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:52 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:52.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:52 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:52.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:52 smithi137 ceph-mon[29634]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:22:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:52 smithi137 ceph-mon[29634]: osdmap e48: 7 total, 6 up, 7 in 2024-09-21T08:22:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:52 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:52 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:52 smithi137 ceph-mon[41044]: from='osd.6 v1:172.21.15.165:6808/3820354208' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:22:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:52 smithi137 ceph-mon[41044]: osdmap e48: 7 total, 6 up, 7 in 2024-09-21T08:22:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:52 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:52 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:53.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:53 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:22:53.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:53 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:22:53.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:53 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:53.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:53 smithi165 ceph-mon[32893]: osd.6 v1:172.21.15.165:6808/3820354208 boot 2024-09-21T08:22:53.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:53 smithi165 ceph-mon[32893]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T08:22:53.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:53 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:53.623 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:53 smithi165 ceph-mon[32893]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[41044]: osd.6 v1:172.21.15.165:6808/3820354208 boot 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[41044]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[41044]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:53.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[29634]: osd.6 v1:172.21.15.165:6808/3820354208 boot 2024-09-21T08:22:53.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[29634]: osdmap e49: 7 total, 7 up, 7 in 2024-09-21T08:22:53.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:22:53.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:53 smithi137 ceph-mon[29634]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 162 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:22:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T08:22:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: Detected new or changed devices on smithi165 2024-09-21T08:22:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:54.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:54.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:54.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: Adjusting osd_memory_target on smithi165 to 5696M 2024-09-21T08:22:54.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:54.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:54.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: Detected new or changed devices on smithi165 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: Adjusting osd_memory_target on smithi165 to 5696M 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: osdmap e50: 7 total, 7 up, 7 in 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: Detected new or changed devices on smithi165 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: Adjusting osd_memory_target on smithi165 to 5696M 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:22:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:22:55.341 INFO:teuthology.orchestra.run.smithi165.stdout: 2024-09-21T08:22:55.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch daemon add osd smithi165:vg_nvme/lv_1 2024-09-21T08:22:55.618 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:22:55.739 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:55 smithi165 ceph-mon[32893]: osdmap e51: 7 total, 7 up, 7 in 2024-09-21T08:22:55.739 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:55 smithi165 ceph-mon[32893]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:22:55.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:55 smithi137 ceph-mon[29634]: osdmap e51: 7 total, 7 up, 7 in 2024-09-21T08:22:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:55 smithi137 ceph-mon[29634]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:22:55.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:55 smithi137 ceph-mon[41044]: osdmap e51: 7 total, 7 up, 7 in 2024-09-21T08:22:55.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:55 smithi137 ceph-mon[41044]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:22:56.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:56 smithi165 ceph-mon[32893]: osdmap e52: 7 total, 7 up, 7 in 2024-09-21T08:22:56.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:22:56.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:22:56.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:56 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:56.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:56 smithi137 ceph-mon[29634]: osdmap e52: 7 total, 7 up, 7 in 2024-09-21T08:22:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:22:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:22:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:56 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:56.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:56 smithi137 ceph-mon[41044]: osdmap e52: 7 total, 7 up, 7 in 2024-09-21T08:22:56.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-09-21T08:22:56.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-09-21T08:22:56.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:56 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:22:57.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:57 smithi165 ceph-mon[32893]: from='client.24316 v1:172.21.15.165:0/4014525349' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:22:57.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:57 smithi165 ceph-mon[32893]: osdmap e53: 7 total, 7 up, 7 in 2024-09-21T08:22:57.631 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:57 smithi165 ceph-mon[32893]: pgmap v155: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-21T08:22:57.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:57 smithi137 ceph-mon[29634]: from='client.24316 v1:172.21.15.165:0/4014525349' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:22:57.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:57 smithi137 ceph-mon[29634]: osdmap e53: 7 total, 7 up, 7 in 2024-09-21T08:22:57.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:57 smithi137 ceph-mon[29634]: pgmap v155: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-21T08:22:57.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:57 smithi137 ceph-mon[41044]: from='client.24316 v1:172.21.15.165:0/4014525349' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi165:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:22:57.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:57 smithi137 ceph-mon[41044]: osdmap e53: 7 total, 7 up, 7 in 2024-09-21T08:22:57.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:57 smithi137 ceph-mon[41044]: pgmap v155: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-21T08:23:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:59 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:23:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:59 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:23:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:59 smithi165 ceph-mon[32893]: osdmap e54: 7 total, 7 up, 7 in 2024-09-21T08:23:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:59 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/3645696826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f"}]: dispatch 2024-09-21T08:23:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:59 smithi165 ceph-mon[32893]: from='client.24322 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f"}]: dispatch 2024-09-21T08:23:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:59 smithi165 ceph-mon[32893]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f"}]': finished 2024-09-21T08:23:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:59 smithi165 ceph-mon[32893]: osdmap e55: 8 total, 7 up, 8 in 2024-09-21T08:23:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:59 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:59 smithi165 ceph-mon[32893]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-21T08:23:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:22:59 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/243832643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:23:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:23:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:23:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[29634]: osdmap e54: 7 total, 7 up, 7 in 2024-09-21T08:23:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/3645696826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f"}]: dispatch 2024-09-21T08:23:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[29634]: from='client.24322 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f"}]: dispatch 2024-09-21T08:23:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[29634]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f"}]': finished 2024-09-21T08:23:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[29634]: osdmap e55: 8 total, 7 up, 8 in 2024-09-21T08:23:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[29634]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/243832643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[41044]: osdmap e54: 7 total, 7 up, 7 in 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/3645696826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f"}]: dispatch 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[41044]: from='client.24322 ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f"}]: dispatch 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[41044]: from='client.24322 ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f"}]': finished 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[41044]: osdmap e55: 8 total, 7 up, 8 in 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[41044]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-09-21T08:23:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:22:59 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/243832643' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-09-21T08:23:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:00 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:23:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:00 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:23:01.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:00 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:23:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:00 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:23:01.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:00 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:23:01.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:00 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:23:02.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:01 smithi165 ceph-mon[32893]: osdmap e56: 8 total, 7 up, 8 in 2024-09-21T08:23:02.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:01 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:02.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:01 smithi165 ceph-mon[32893]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:01 smithi137 ceph-mon[29634]: osdmap e56: 8 total, 7 up, 8 in 2024-09-21T08:23:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:01 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:01 smithi137 ceph-mon[29634]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:02.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:01 smithi137 ceph-mon[41044]: osdmap e56: 8 total, 7 up, 8 in 2024-09-21T08:23:02.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:01 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:02.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:01 smithi137 ceph-mon[41044]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 189 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:03 smithi165 ceph-mon[32893]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:04.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:03 smithi137 ceph-mon[29634]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:04.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:03 smithi137 ceph-mon[41044]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:05.960 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:05 smithi165 ceph-mon[32893]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:06.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:05 smithi137 ceph-mon[29634]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:06.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:05 smithi137 ceph-mon[41044]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:06 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T08:23:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:06 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:07.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:06 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T08:23:07.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:06 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:07.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:06 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-09-21T08:23:07.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:06 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:07 smithi165 ceph-mon[32893]: Deploying daemon osd.7 on smithi165 2024-09-21T08:23:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:07 smithi165 ceph-mon[32893]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:08.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:07 smithi137 ceph-mon[29634]: Deploying daemon osd.7 on smithi165 2024-09-21T08:23:08.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:07 smithi137 ceph-mon[29634]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:08.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:07 smithi137 ceph-mon[41044]: Deploying daemon osd.7 on smithi165 2024-09-21T08:23:08.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:07 smithi137 ceph-mon[41044]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:09 smithi165 ceph-mon[32893]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:10.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:09 smithi137 ceph-mon[29634]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:10.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:09 smithi137 ceph-mon[41044]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:12.118 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:11 smithi165 ceph-mon[32893]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:12.118 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:11 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:12.118 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:11 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:12.118 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:11 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:12.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:11 smithi137 ceph-mon[29634]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:12.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:11 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:12.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:11 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:12.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:11 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:12.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:11 smithi137 ceph-mon[41044]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:12.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:11 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:12.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:11 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:12.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:11 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:14.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:13 smithi165 ceph-mon[32893]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:14.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:13 smithi137 ceph-mon[29634]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:14.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:13 smithi137 ceph-mon[41044]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:15.146 INFO:teuthology.orchestra.run.smithi165.stdout:Created osd(s) 7 on host 'smithi165' 2024-09-21T08:23:15.809 DEBUG:teuthology.orchestra.run.smithi165:osd.7> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T08:23:15.811 INFO:tasks.cephadm:Waiting for 8 OSDs to come up... 2024-09-21T08:23:15.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T08:23:16.067 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:16.183 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:15 smithi137 ceph-mon[29634]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:16.184 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:15 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.184 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:15 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.184 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:15 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.184 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:15 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.184 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:15 smithi137 ceph-mon[41044]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:16.184 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:15 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.184 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:15 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.184 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:15 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.184 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:15 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:15 smithi165 ceph-mon[32893]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:16.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:15 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:15 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:15 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:15 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:16.680 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:23:16.897 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:23:16 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7[56695]: 2024-09-21T08:23:16.693+0000 7f09ec493640 -1 osd.7 0 log_to_monitors true 2024-09-21T08:23:17.417 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":56,"num_osds":8,"num_up_osds":7,"osd_up_since":1726906972,"num_in_osds":8,"osd_in_since":1726906978,"num_remapped_pgs":0} 2024-09-21T08:23:17.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:17 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3866676417' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T08:23:17.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:17 smithi165 ceph-mon[32893]: from='osd.7 v1:172.21.15.165:6812/1933524273' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:23:17.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:17 smithi165 ceph-mon[32893]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:23:17.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:17 smithi165 ceph-mon[32893]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:17 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3866676417' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T08:23:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:17 smithi137 ceph-mon[41044]: from='osd.7 v1:172.21.15.165:6812/1933524273' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:23:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:17 smithi137 ceph-mon[41044]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:23:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:17 smithi137 ceph-mon[41044]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:17 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3866676417' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T08:23:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:17 smithi137 ceph-mon[29634]: from='osd.7 v1:172.21.15.165:6812/1933524273' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:23:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:17 smithi137 ceph-mon[29634]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:23:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:17 smithi137 ceph-mon[29634]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:18.418 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T08:23:18.679 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:18.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:18 smithi165 ceph-mon[32893]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T08:23:18.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:18 smithi165 ceph-mon[32893]: osdmap e57: 8 total, 7 up, 8 in 2024-09-21T08:23:18.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:18 smithi165 ceph-mon[32893]: from='osd.7 v1:172.21.15.165:6812/1933524273' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:23:18.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:18 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:18.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:18 smithi165 ceph-mon[32893]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:23:18.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:18 smithi137 ceph-mon[29634]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T08:23:18.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:18 smithi137 ceph-mon[29634]: osdmap e57: 8 total, 7 up, 8 in 2024-09-21T08:23:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:18 smithi137 ceph-mon[29634]: from='osd.7 v1:172.21.15.165:6812/1933524273' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:23:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:18 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:18 smithi137 ceph-mon[29634]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:23:18.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:18 smithi137 ceph-mon[41044]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T08:23:18.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:18 smithi137 ceph-mon[41044]: osdmap e57: 8 total, 7 up, 8 in 2024-09-21T08:23:18.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:18 smithi137 ceph-mon[41044]: from='osd.7 v1:172.21.15.165:6812/1933524273' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:23:18.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:18 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:18.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:18 smithi137 ceph-mon[41044]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:23:19.224 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:23:18 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7[56695]: 2024-09-21T08:23:18.910+0000 7f09e8416640 -1 osd.7 0 waiting for initial osdmap 2024-09-21T08:23:19.224 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:23:18 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7[56695]: 2024-09-21T08:23:18.916+0000 7f09e0236640 -1 osd.7 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T08:23:19.322 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:23:19.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:23:19.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: osdmap e58: 8 total, 7 up, 8 in 2024-09-21T08:23:19.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='osd.7 ' entity='osd.7' 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.622 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2509382478' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: osdmap e58: 8 total, 7 up, 8 in 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='osd.7 ' entity='osd.7' 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2509382478' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:19.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:19 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]': finished 2024-09-21T08:23:19.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: osdmap e58: 8 total, 7 up, 8 in 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 190 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='osd.7 ' entity='osd.7' 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:19.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:19.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:19.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2509382478' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T08:23:19.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:19.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:19.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:19.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:19 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:20.000 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":58,"num_osds":8,"num_up_osds":7,"osd_up_since":1726906972,"num_in_osds":8,"osd_in_since":1726906978,"num_remapped_pgs":0} 2024-09-21T08:23:20.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:20 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:23:20.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:20 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:23:20.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:20 smithi165 ceph-mon[32893]: Detected new or changed devices on smithi165 2024-09-21T08:23:20.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:20 smithi165 ceph-mon[32893]: Adjusting osd_memory_target on smithi165 to 4272M 2024-09-21T08:23:20.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:20 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:20.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:20 smithi165 ceph-mon[32893]: osd.7 v1:172.21.15.165:6812/1933524273 boot 2024-09-21T08:23:20.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:20 smithi165 ceph-mon[32893]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T08:23:20.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:20 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:20.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[29634]: Detected new or changed devices on smithi165 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[29634]: Adjusting osd_memory_target on smithi165 to 4272M 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[29634]: osd.7 v1:172.21.15.165:6812/1933524273 boot 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[29634]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[41044]: Detected new or changed devices on smithi165 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[41044]: Adjusting osd_memory_target on smithi165 to 4272M 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[41044]: osd.7 v1:172.21.15.165:6812/1933524273 boot 2024-09-21T08:23:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[41044]: osdmap e59: 8 total, 8 up, 8 in 2024-09-21T08:23:20.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:20 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:23:21.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd stat -f json 2024-09-21T08:23:21.255 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:21.711 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:21 smithi137 ceph-mon[29634]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T08:23:21.711 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:21 smithi137 ceph-mon[29634]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:23:21.711 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:21 smithi137 ceph-mon[41044]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T08:23:21.711 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:21 smithi137 ceph-mon[41044]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:23:21.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:21 smithi165 ceph-mon[32893]: osdmap e60: 8 total, 8 up, 8 in 2024-09-21T08:23:21.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:21 smithi165 ceph-mon[32893]: pgmap v174: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:23:21.849 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:23:22.557 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":61,"num_osds":8,"num_up_osds":8,"osd_up_since":1726906999,"num_in_osds":8,"osd_in_since":1726906978,"num_remapped_pgs":1} 2024-09-21T08:23:22.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T08:23:22.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:22 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:23:22.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:22 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:23:22.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:22 smithi165 ceph-mon[32893]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T08:23:22.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:22 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/182815315' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T08:23:22.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:22 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:23:22.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:22 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:23:22.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:22 smithi137 ceph-mon[29634]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T08:23:22.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:22 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/182815315' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T08:23:22.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:22 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:23:22.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:22 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:23:22.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:22 smithi137 ceph-mon[41044]: osdmap e61: 8 total, 8 up, 8 in 2024-09-21T08:23:22.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:22 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/182815315' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-09-21T08:23:22.798 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:23.371 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:23:23.371 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":62,"fsid":"ec8566d6-77f1-11ef-baf6-efdc52797490","created":"2024-09-21T08:18:21.794737+0000","modified":"2024-09-21T08:23:22.492608+0000","last_up_change":"2024-09-21T08:23:19.473450+0000","last_in_change":"2024-09-21T08:22:58.590287+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T08:21:15.037508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b007d1d-6c6b-4a68-9e68-2c5fd7d26977","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":61,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":3938795530}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":3938795530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":3938795530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":3938795530}]},"public_addr":"172.21.15.137:6801/3938795530","cluster_addr":"172.21.15.137:6802/3938795530","heartbeat_back_addr":"172.21.15.137:6804/3938795530","heartbeat_front_addr":"172.21.15.137:6803/3938795530","state":["exists","up"]},{"osd":1,"uuid":"8651068e-694f-4bd4-9917-5c55670aca5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":2743268361}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":2743268361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":2743268361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":2743268361}]},"public_addr":"172.21.15.137:6805/2743268361","cluster_addr":"172.21.15.137:6806/2743268361","heartbeat_back_addr":"172.21.15.137:6808/2743268361","heartbeat_front_addr":"172.21.15.137:6807/2743268361","state":["exists","up"]},{"osd":2,"uuid":"2a22f0f7-c610-4995-98a7-d4785efb1e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":1500397557}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":1500397557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":1500397557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":1500397557}]},"public_addr":"172.21.15.137:6809/1500397557","cluster_addr":"172.21.15.137:6810/1500397557","heartbeat_back_addr":"172.21.15.137:6812/1500397557","heartbeat_front_addr":"172.21.15.137:6811/1500397557","state":["exists","up"]},{"osd":3,"uuid":"73f21606-38be-4578-a86b-bb417217c0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":617155021}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":617155021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6816","nonce":617155021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":617155021}]},"public_addr":"172.21.15.137:6813/617155021","cluster_addr":"172.21.15.137:6814/617155021","heartbeat_back_addr":"172.21.15.137:6816/617155021","heartbeat_front_addr":"172.21.15.137:6815/617155021","state":["exists","up"]},{"osd":4,"uuid":"e5f6762b-65f5-4dc7-8e54-9087149ae6ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":70360958}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":70360958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":70360958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":70360958}]},"public_addr":"172.21.15.165:6800/70360958","cluster_addr":"172.21.15.165:6801/70360958","heartbeat_back_addr":"172.21.15.165:6803/70360958","heartbeat_front_addr":"172.21.15.165:6802/70360958","state":["exists","up"]},{"osd":5,"uuid":"f7f8c456-b77f-4435-a860-202e64dda8dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":1835779380}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":1835779380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":1835779380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":1835779380}]},"public_addr":"172.21.15.165:6804/1835779380","cluster_addr":"172.21.15.165:6805/1835779380","heartbeat_back_addr":"172.21.15.165:6807/1835779380","heartbeat_front_addr":"172.21.15.165:6806/1835779380","state":["exists","up"]},{"osd":6,"uuid":"b8b3df16-73fa-41a5-a93f-117c599d488b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":52,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":3820354208}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":3820354208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":3820354208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":3820354208}]},"public_addr":"172.21.15.165:6808/3820354208","cluster_addr":"172.21.15.165:6809/3820354208","heartbeat_back_addr":"172.21.15.165:6811/3820354208","heartbeat_front_addr":"172.21.15.165:6810/3820354208","state":["exists","up"]},{"osd":7,"uuid":"ddf234e4-1494-40bd-a31b-333b01717c3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":1933524273}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":1933524273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":1933524273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":1933524273}]},"public_addr":"172.21.15.165:6812/1933524273","cluster_addr":"172.21.15.165:6813/1933524273","heartbeat_back_addr":"172.21.15.165:6815/1933524273","heartbeat_front_addr":"172.21.15.165:6814/1933524273","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:00.155347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:12.100659+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:19.738828+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:58.117448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:21.051166+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:25.587597+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:51.433176+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:17.735488+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6800/1900990534":"2024-09-22T08:19:12.726560+0000","172.21.15.137:6800/2580000250":"2024-09-22T08:18:48.673163+0000","172.21.15.137:0/223603733":"2024-09-22T08:19:12.726560+0000","172.21.15.137:0/705906958":"2024-09-22T08:19:12.726560+0000","172.21.15.137:0/700535732":"2024-09-22T08:18:48.673163+0000","172.21.15.137:0/3936900881":"2024-09-22T08:19:12.726560+0000","172.21.15.137:0/2643910317":"2024-09-22T08:18:48.673163+0000","172.21.15.137:0/1738156348":"2024-09-22T08:18:48.673163+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T08:23:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:23 smithi165 ceph-mon[32893]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T08:23:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:23 smithi165 ceph-mon[32893]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:23:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:23 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3701942462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T08:23:23.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:23 smithi137 ceph-mon[41044]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T08:23:23.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:23 smithi137 ceph-mon[41044]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:23:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:23 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3701942462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T08:23:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:23 smithi137 ceph-mon[29634]: osdmap e62: 8 total, 8 up, 8 in 2024-09-21T08:23:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:23 smithi137 ceph-mon[29634]: pgmap v177: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:23:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:23 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3701942462' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T08:23:24.085 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T08:21:15.037508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-09-21T08:23:24.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pool get .mgr pg_num 2024-09-21T08:23:24.329 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:24.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:24 smithi137 ceph-mon[29634]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T08:23:24.758 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:24 smithi137 ceph-mon[41044]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T08:23:24.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:24 smithi165 ceph-mon[32893]: osdmap e63: 8 total, 8 up, 8 in 2024-09-21T08:23:24.913 INFO:teuthology.orchestra.run.smithi137.stdout:pg_num: 1 2024-09-21T08:23:25.606 INFO:tasks.cephadm:Adding ceph.rgw.foo.a on smithi137 2024-09-21T08:23:25.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch apply rgw foo.a --placement '1;smithi137=foo.a' 2024-09-21T08:23:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:25 smithi137 ceph-mon[29634]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:23:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:25 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1367351006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T08:23:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:25 smithi137 ceph-mon[41044]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:23:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:25 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1367351006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T08:23:25.865 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:23:25.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:25 smithi165 ceph-mon[32893]: pgmap v179: 1 pgs: 1 remapped+peering; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:23:25.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:25 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1367351006' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-09-21T08:23:26.491 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled rgw.foo.a update... 2024-09-21T08:23:27.061 DEBUG:teuthology.orchestra.run.smithi137:rgw.foo.a> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@rgw.foo.a.service 2024-09-21T08:23:27.063 INFO:tasks.cephadm:Adding ceph.iscsi.iscsi.a on smithi165 2024-09-21T08:23:27.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pool create datapool 3 3 replicated 2024-09-21T08:23:27.313 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: from='client.24373 v1:172.21.15.165:0/2113335149' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi137=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:27.741 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: Deploying daemon rgw.foo.a on smithi137 2024-09-21T08:23:27.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:27 smithi165 ceph-mon[32893]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T08:23:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: from='client.24373 v1:172.21.15.165:0/2113335149' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi137=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-21T08:23:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: Deploying daemon rgw.foo.a on smithi137 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[41044]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: from='client.24373 v1:172.21.15.165:0/2113335149' entity='client.admin' cmd=[{"prefix": "orch apply rgw", "svc_id": "foo.a", "placement": "1;smithi137=foo.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:27.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]: dispatch 2024-09-21T08:23:27.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.rgw.foo.a", "caps": ["mon", "allow *", "mgr", "allow rw", "osd", "allow rwx tag rgw *=*"]}]': finished 2024-09-21T08:23:27.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:27.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:27.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: Deploying daemon rgw.foo.a on smithi137 2024-09-21T08:23:27.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:27 smithi137 ceph-mon[29634]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T08:23:28.518 INFO:teuthology.orchestra.run.smithi165.stderr:pool 'datapool' created 2024-09-21T08:23:28.680 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:28 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/2689239029' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-21T08:23:28.680 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:28 smithi137 ceph-mon[29634]: from='client.24379 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-21T08:23:28.680 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:28 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/2689239029' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-21T08:23:28.680 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:28 smithi137 ceph-mon[41044]: from='client.24379 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-21T08:23:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:28 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/2689239029' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-21T08:23:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:28 smithi165 ceph-mon[32893]: from='client.24379 ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]: dispatch 2024-09-21T08:23:29.018 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 21 08:23:28 smithi137 podman[68535]: 2024-09-21 08:23:28.929138244 +0000 UTC m=+0.357885764 container start b238a596ffd0fc5c405b93ec98123a71eb0ff38a0fe06f19875174fff50a2bca (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-rgw-foo-a, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240731, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux ) 2024-09-21T08:23:29.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- rbd pool init datapool 2024-09-21T08:23:29.352 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:23:29.517 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 21 08:23:29 smithi137 bash[68535]: b238a596ffd0fc5c405b93ec98123a71eb0ff38a0fe06f19875174fff50a2bca 2024-09-21T08:23:29.518 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 21 08:23:29 smithi137 systemd[1]: Started Ceph rgw.foo.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:23:29.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:29 smithi165 ceph-mon[32893]: from='client.24379 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T08:23:29.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:29 smithi165 ceph-mon[32893]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T08:23:29.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:29 smithi165 ceph-mon[32893]: pgmap v182: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T08:23:29.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:29 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:29 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:29 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:29 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:29 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:29 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:29.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[29634]: from='client.24379 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T08:23:29.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[29634]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T08:23:29.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[29634]: pgmap v182: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T08:23:29.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:29.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[41044]: from='client.24379 ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "datapool", "pg_num": 3, "pgp_num": 3, "pool_type": "replicated"}]': finished 2024-09-21T08:23:29.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[41044]: osdmap e64: 8 total, 8 up, 8 in 2024-09-21T08:23:29.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[41044]: pgmap v182: 4 pgs: 3 unknown, 1 active+clean; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 92 KiB/s, 0 objects/s recovering 2024-09-21T08:23:29.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:29.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:29 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:23:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:30 smithi165 ceph-mon[32893]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-21T08:23:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:30 smithi165 ceph-mon[32893]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T08:23:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:30 smithi165 ceph-mon[32893]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T08:23:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:30 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1756555978' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-21T08:23:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:30 smithi165 ceph-mon[32893]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-21T08:23:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:30 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/2482038767' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-21T08:23:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:30 smithi165 ceph-mon[32893]: from='client.24394 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-21T08:23:31.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[29634]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-21T08:23:31.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[29634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[29634]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1756555978' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[29634]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/2482038767' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[29634]: from='client.24394 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[41044]: Saving service rgw.foo.a spec with placement smithi137=foo.a;count:1 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[41044]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[41044]: osdmap e65: 8 total, 8 up, 8 in 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1756555978' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[41044]: from='client.44375 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]: dispatch 2024-09-21T08:23:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/2482038767' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-21T08:23:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:30 smithi137 ceph-mon[41044]: from='client.24394 ' entity='client.admin' cmd=[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]: dispatch 2024-09-21T08:23:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:31 smithi165 ceph-mon[32893]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T08:23:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:31 smithi165 ceph-mon[32893]: from='client.24394 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T08:23:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:31 smithi165 ceph-mon[32893]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T08:23:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:31 smithi165 ceph-mon[32893]: pgmap v185: 36 pgs: 21 creating+peering, 3 active+clean, 12 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T08:23:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:31 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:32.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:31 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:32.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:31 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:32.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:31 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:32.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:31 smithi165 ceph-mon[32893]: Checking dashboard <-> RGW credentials 2024-09-21T08:23:32.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[29634]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[29634]: from='client.24394 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[29634]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[29634]: pgmap v185: 36 pgs: 21 creating+peering, 3 active+clean, 12 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[29634]: Checking dashboard <-> RGW credentials 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[41044]: from='client.44375 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": ".rgw.root","app": "rgw"}]': finished 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[41044]: from='client.24394 ' entity='client.admin' cmd='[{"prefix": "osd pool application enable","pool": "datapool","app": "rbd"}]': finished 2024-09-21T08:23:32.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[41044]: osdmap e66: 8 total, 8 up, 8 in 2024-09-21T08:23:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[41044]: pgmap v185: 36 pgs: 21 creating+peering, 3 active+clean, 12 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-09-21T08:23:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:23:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:31 smithi137 ceph-mon[41044]: Checking dashboard <-> RGW credentials 2024-09-21T08:23:32.900 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:32 smithi165 ceph-mon[32893]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T08:23:32.900 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:32 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:23:32.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:32 smithi165 ceph-mon[32893]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T08:23:32.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:32 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:32.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:32 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:32.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:32 smithi165 ceph-mon[32893]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:32.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:32 smithi165 ceph-mon[32893]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:33.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[29634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[29634]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[29634]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[29634]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[41044]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[41044]: osdmap e67: 8 total, 8 up, 8 in 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:33.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[41044]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:33.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:32 smithi137 ceph-mon[41044]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]: dispatch 2024-09-21T08:23:33.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch apply iscsi datapool admin admin --trusted_ip_list 172.21.15.165 --placement '1;smithi165=iscsi.a' 2024-09-21T08:23:33.455 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:23:33.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:33 smithi165 ceph-mon[32893]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T08:23:33.828 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:33 smithi165 ceph-mon[32893]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T08:23:33.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:33 smithi165 ceph-mon[32893]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T08:23:33.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:33 smithi165 ceph-mon[32893]: pgmap v188: 68 pgs: 23 creating+peering, 13 active+clean, 32 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-21T08:23:33.829 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:33 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:33 smithi137 ceph-mon[29634]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T08:23:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:33 smithi137 ceph-mon[29634]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T08:23:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:33 smithi137 ceph-mon[29634]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T08:23:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:33 smithi137 ceph-mon[29634]: pgmap v188: 68 pgs: 23 creating+peering, 13 active+clean, 32 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-21T08:23:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:33 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:34.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:33 smithi137 ceph-mon[41044]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T08:23:34.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:33 smithi137 ceph-mon[41044]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.log","app": "rgw"}]': finished 2024-09-21T08:23:34.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:33 smithi137 ceph-mon[41044]: osdmap e68: 8 total, 8 up, 8 in 2024-09-21T08:23:34.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:33 smithi137 ceph-mon[41044]: pgmap v188: 68 pgs: 23 creating+peering, 13 active+clean, 32 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-21T08:23:34.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:33 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:34.115 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled iscsi.datapool update... 2024-09-21T08:23:34.749 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:34 smithi165 ceph-mon[32893]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T08:23:34.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:34.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:34.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:34 smithi165 ceph-mon[32893]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:34.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:34 smithi165 ceph-mon[32893]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:34.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:34 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:34.775 INFO:tasks.cephadm:Distributing iscsi-gateway.cfg... 2024-09-21T08:23:34.775 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:23:34.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-21T08:23:34.802 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:23:34.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/iscsi-gateway.cfg 2024-09-21T08:23:34.829 DEBUG:teuthology.orchestra.run.smithi165:iscsi.iscsi.a> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@iscsi.iscsi.a.service 2024-09-21T08:23:34.873 INFO:tasks.cephadm:Adding prometheus.a on smithi165 2024-09-21T08:23:34.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch apply prometheus '1;smithi165=a' 2024-09-21T08:23:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[29634]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T08:23:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[29634]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[29634]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:35.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[41044]: osdmap e69: 8 total, 8 up, 8 in 2024-09-21T08:23:35.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:35.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:35.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[41044]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:35.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[41044]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]: dispatch 2024-09-21T08:23:35.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:34 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:35.153 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:23:35.914 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled prometheus update... 2024-09-21T08:23:35.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:35 smithi165 ceph-mon[32893]: from='client.24439 v1:172.21.15.165:0/3961087617' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.165", "placement": "1;smithi165=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:35.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:35 smithi165 ceph-mon[32893]: Saving service iscsi.datapool spec with placement smithi165=iscsi.a;count:1 2024-09-21T08:23:35.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:35 smithi165 ceph-mon[32893]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T08:23:35.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:35 smithi165 ceph-mon[32893]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T08:23:35.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:35 smithi165 ceph-mon[32893]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T08:23:35.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:35 smithi165 ceph-mon[32893]: pgmap v191: 100 pgs: 17 creating+peering, 35 active+clean, 48 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-21T08:23:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[29634]: from='client.24439 v1:172.21.15.165:0/3961087617' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.165", "placement": "1;smithi165=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[29634]: Saving service iscsi.datapool spec with placement smithi165=iscsi.a;count:1 2024-09-21T08:23:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[29634]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T08:23:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[29634]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T08:23:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[29634]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T08:23:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[29634]: pgmap v191: 100 pgs: 17 creating+peering, 35 active+clean, 48 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-21T08:23:36.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[41044]: from='client.24439 v1:172.21.15.165:0/3961087617' entity='client.admin' cmd=[{"prefix": "orch apply iscsi", "pool": "datapool", "api_user": "admin", "api_password": "admin", "trusted_ip_list": "172.21.15.165", "placement": "1;smithi165=iscsi.a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:36.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[41044]: Saving service iscsi.datapool spec with placement smithi165=iscsi.a;count:1 2024-09-21T08:23:36.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[41044]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T08:23:36.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[41044]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.control","app": "rgw"}]': finished 2024-09-21T08:23:36.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[41044]: osdmap e70: 8 total, 8 up, 8 in 2024-09-21T08:23:36.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:35 smithi137 ceph-mon[41044]: pgmap v191: 100 pgs: 17 creating+peering, 35 active+clean, 48 unknown; 577 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 511 B/s wr, 1 op/s 2024-09-21T08:23:36.541 DEBUG:teuthology.orchestra.run.smithi165:prometheus.a> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@prometheus.a.service 2024-09-21T08:23:36.543 INFO:tasks.cephadm:Adding node-exporter.a on smithi137 2024-09-21T08:23:36.544 INFO:tasks.cephadm:Adding node-exporter.b on smithi165 2024-09-21T08:23:36.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch apply node-exporter '2;smithi137=a;smithi165=b' 2024-09-21T08:23:36.828 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:23:36.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:36 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:23:36.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:36 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:23:36.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:36 smithi165 ceph-mon[32893]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T08:23:36.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:36 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:36.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:36 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:36.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:36 smithi165 ceph-mon[32893]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:36.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:36 smithi165 ceph-mon[32893]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:36.950 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:36 smithi165 ceph-mon[32893]: from='client.24445 v1:172.21.15.165:0/238025628' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:36.950 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:36 smithi165 ceph-mon[32893]: Saving service prometheus spec with placement smithi165=a;count:1 2024-09-21T08:23:36.950 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:36 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:37.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:23:37.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:23:37.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[29634]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T08:23:37.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:37.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:37.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[29634]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:37.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[29634]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:37.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[29634]: from='client.24445 v1:172.21.15.165:0/238025628' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:37.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[29634]: Saving service prometheus spec with placement smithi165=a;count:1 2024-09-21T08:23:37.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:37.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:23:37.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:23:37.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[41044]: osdmap e71: 8 total, 8 up, 8 in 2024-09-21T08:23:37.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:37.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:37.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[41044]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:37.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[41044]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]: dispatch 2024-09-21T08:23:37.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[41044]: from='client.24445 v1:172.21.15.165:0/238025628' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:37.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[41044]: Saving service prometheus spec with placement smithi165=a;count:1 2024-09-21T08:23:37.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:36 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:37.452 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled node-exporter update... 2024-09-21T08:23:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:23:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:23:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T08:23:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T08:23:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T08:23:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: pgmap v194: 132 pgs: 7 creating+peering, 90 active+clean, 35 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-21T08:23:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:37 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:38.017 INFO:journalctl@ceph.rgw.foo.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-rgw-foo-a[68550]: 2024-09-21T08:23:37.644+0000 7f0fe5c4e880 -1 LDAP not started since no server URIs were provided in the configuration. 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: pgmap v194: 132 pgs: 7 creating+peering, 90 active+clean, 35 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-21T08:23:38.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool application enable","pool": "default.rgw.meta","app": "rgw"}]': finished 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: osdmap e72: 8 total, 8 up, 8 in 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3690342859' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1716164338' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: from='client.44387 ' entity='client.rgw.foo.a' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: from='client.44417 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]: dispatch 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: pgmap v194: 132 pgs: 7 creating+peering, 90 active+clean, 35 unknown; 578 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 3.2 KiB/s rd, 1.5 KiB/s wr, 5 op/s 2024-09-21T08:23:38.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:37 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:38.089 DEBUG:teuthology.orchestra.run.smithi137:node-exporter.a> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service 2024-09-21T08:23:38.092 DEBUG:teuthology.orchestra.run.smithi165:node-exporter.b> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.b.service 2024-09-21T08:23:38.094 INFO:tasks.cephadm:Adding alertmanager.a on smithi137 2024-09-21T08:23:38.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch apply alertmanager '1;smithi137=a' 2024-09-21T08:23:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: from='client.24451 v1:172.21.15.165:0/3611677312' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi137=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: Saving service node-exporter spec with placement smithi137=a;smithi165=b;count:2 2024-09-21T08:23:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T08:23:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T08:23:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T08:23:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:38.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:38.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:38.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T08:23:38.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T08:23:38.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:38.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:38 smithi165 ceph-mon[32893]: Deploying daemon iscsi.iscsi.a on smithi165 2024-09-21T08:23:38.851 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:23:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: from='client.24451 v1:172.21.15.165:0/3611677312' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi137=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: Saving service node-exporter spec with placement smithi137=a;smithi165=b;count:2 2024-09-21T08:23:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T08:23:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T08:23:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T08:23:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[29634]: Deploying daemon iscsi.iscsi.a on smithi165 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: from='client.24451 v1:172.21.15.165:0/3611677312' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "placement": "2;smithi137=a;smithi165=b", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: Saving service node-exporter spec with placement smithi137=a;smithi165=b;count:2 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: from='client.44387 ' entity='client.rgw.foo.a' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: from='client.44417 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "default.rgw.meta", "var": "pg_autoscale_bias", "val": "4"}]': finished 2024-09-21T08:23:39.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: osdmap e73: 8 total, 8 up, 8 in 2024-09-21T08:23:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]: dispatch 2024-09-21T08:23:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "auth get-or-create", "entity": "client.iscsi.iscsi.a", "caps": ["mon", "profile rbd, allow command \"osd blocklist\", allow command \"config-key get\" with \"key\" prefix \"iscsi/\"", "mgr", "allow command \"service status\"", "osd", "allow rwx"]}]': finished 2024-09-21T08:23:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:23:39.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:38 smithi137 ceph-mon[41044]: Deploying daemon iscsi.iscsi.a on smithi165 2024-09-21T08:23:39.857 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:39 smithi165 ceph-mon[32893]: pgmap v196: 132 pgs: 5 creating+peering, 104 active+clean, 23 unknown; 579 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 34 op/s 2024-09-21T08:23:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:39 smithi137 ceph-mon[29634]: pgmap v196: 132 pgs: 5 creating+peering, 104 active+clean, 23 unknown; 579 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 34 op/s 2024-09-21T08:23:40.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:39 smithi137 ceph-mon[41044]: pgmap v196: 132 pgs: 5 creating+peering, 104 active+clean, 23 unknown; 579 KiB data, 217 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 2.4 KiB/s wr, 34 op/s 2024-09-21T08:23:40.134 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled alertmanager update... 2024-09-21T08:23:40.438 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:40 smithi165 systemd[1]: Starting Ceph iscsi.iscsi.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:23:40.757 DEBUG:teuthology.orchestra.run.smithi137:alertmanager.a> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@alertmanager.a.service 2024-09-21T08:23:40.760 INFO:tasks.cephadm:Adding grafana.a on smithi165 2024-09-21T08:23:40.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph orch apply grafana '1;smithi165=a' 2024-09-21T08:23:41.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:40 smithi165 podman[61707]: 2024-09-21 08:23:40.84683341 +0000 UTC m=+0.023200362 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T08:23:41.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:40 smithi165 podman[61707]: 2024-09-21 08:23:40.954932482 +0000 UTC m=+0.131299420 container create 6167c4420300e7b12aa726e69fefaa8430938876a5604994745b7c9666f9610c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-09-21T08:23:41.331 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:23:41.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-mon[32893]: from='client.24472 v1:172.21.15.165:0/1582673152' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:41.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-mon[32893]: Saving service alertmanager spec with placement smithi137=a;count:1 2024-09-21T08:23:41.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:41.455 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-mon[32893]: pgmap v197: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 5.2 KiB/s wr, 132 op/s 2024-09-21T08:23:41.456 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 podman[61707]: 2024-09-21 08:23:41.174125181 +0000 UTC m=+0.350492123 container init 6167c4420300e7b12aa726e69fefaa8430938876a5604994745b7c9666f9610c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T08:23:41.456 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 podman[61707]: 2024-09-21 08:23:41.178768901 +0000 UTC m=+0.355135840 container start 6167c4420300e7b12aa726e69fefaa8430938876a5604994745b7c9666f9610c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2) 2024-09-21T08:23:41.456 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 bash[61707]: 6167c4420300e7b12aa726e69fefaa8430938876a5604994745b7c9666f9610c 2024-09-21T08:23:41.456 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 systemd[1]: Started Ceph iscsi.iscsi.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:23:41.517 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:41 smithi137 ceph-mon[41044]: from='client.24472 v1:172.21.15.165:0/1582673152' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:41.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:41 smithi137 ceph-mon[41044]: Saving service alertmanager spec with placement smithi137=a;count:1 2024-09-21T08:23:41.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:41 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:41.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:41 smithi137 ceph-mon[41044]: pgmap v197: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 5.2 KiB/s wr, 132 op/s 2024-09-21T08:23:41.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:41 smithi137 ceph-mon[29634]: from='client.24472 v1:172.21.15.165:0/1582673152' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "placement": "1;smithi137=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:41.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:41 smithi137 ceph-mon[29634]: Saving service alertmanager spec with placement smithi137=a;count:1 2024-09-21T08:23:41.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:41 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:41.519 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:41 smithi137 ceph-mon[29634]: pgmap v197: 132 pgs: 5 creating+peering, 127 active+clean; 581 KiB data, 222 MiB used, 715 GiB / 715 GiB avail; 55 KiB/s rd, 5.2 KiB/s wr, 132 op/s 2024-09-21T08:23:41.763 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug Started the configuration object watcher 2024-09-21T08:23:41.763 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug Checking for config object changes every 1s 2024-09-21T08:23:41.763 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug Processing osd blocklist entries for this node 2024-09-21T08:23:42.181 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug Reading the configuration object to update local LIO configuration 2024-09-21T08:23:42.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug Configuration does not have an entry for this host(smithi165.front.sepia.ceph.com) - nothing to define to LIO 2024-09-21T08:23:42.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: * Serving Flask app 'rbd-target-api' (lazy loading) 2024-09-21T08:23:42.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: * Environment: production 2024-09-21T08:23:42.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T08:23:42.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: Use a production WSGI server instead. 2024-09-21T08:23:42.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: * Debug mode: off 2024-09-21T08:23:42.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug * Running on all addresses. 2024-09-21T08:23:42.182 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T08:23:42.183 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: * Running on all addresses. 2024-09-21T08:23:42.183 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: WARNING: This is a development server. Do not use it in a production deployment. 2024-09-21T08:23:42.183 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug * Running on http://[fe80::ec4:7aff:fe88:7f63]:5000/ (Press CTRL+C to quit) 2024-09-21T08:23:42.183 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: * Running on http://[fe80::ec4:7aff:fe88:7f63]:5000/ (Press CTRL+C to quit) 2024-09-21T08:23:42.221 INFO:teuthology.orchestra.run.smithi165.stdout:Scheduled grafana update... 2024-09-21T08:23:42.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:42 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:42 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:42 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:42 smithi165 ceph-mon[32893]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T08:23:42.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:42 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:42 smithi165 ceph-mon[32893]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T08:23:42.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:42 smithi165 ceph-mon[32893]: Deploying daemon prometheus.a on smithi165 2024-09-21T08:23:42.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:42 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/2309642474' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T08:23:42.514 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:42 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[29634]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T08:23:42.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[29634]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T08:23:42.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[29634]: Deploying daemon prometheus.a on smithi165 2024-09-21T08:23:42.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/2309642474' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T08:23:42.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[41044]: Checking pool "datapool" exists for service iscsi.datapool 2024-09-21T08:23:42.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:42.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[41044]: osdmap e74: 8 total, 8 up, 8 in 2024-09-21T08:23:42.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[41044]: Deploying daemon prometheus.a on smithi165 2024-09-21T08:23:42.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/2309642474' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-09-21T08:23:42.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:42 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:43.081 DEBUG:teuthology.orchestra.run.smithi165:grafana.a> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@grafana.a.service 2024-09-21T08:23:43.083 INFO:tasks.cephadm:Setting up client nodes... 2024-09-21T08:23:43.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T08:23:43.337 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:43.723 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:43 smithi137 ceph-mon[29634]: from='client.24502 v1:172.21.15.165:0/2694371091' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:43.723 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:43 smithi137 ceph-mon[29634]: Saving service grafana spec with placement smithi165=a;count:1 2024-09-21T08:23:43.723 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:43 smithi137 ceph-mon[29634]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.8 KiB/s wr, 180 op/s 2024-09-21T08:23:43.723 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:43 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:43.724 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:43 smithi137 ceph-mon[41044]: from='client.24502 v1:172.21.15.165:0/2694371091' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:43.724 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:43 smithi137 ceph-mon[41044]: Saving service grafana spec with placement smithi165=a;count:1 2024-09-21T08:23:43.724 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:43 smithi137 ceph-mon[41044]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.8 KiB/s wr, 180 op/s 2024-09-21T08:23:43.724 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:43 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:43.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:43 smithi165 ceph-mon[32893]: from='client.24502 v1:172.21.15.165:0/2694371091' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "placement": "1;smithi165=a", "target": ["mon-mgr", ""]}]: dispatch 2024-09-21T08:23:43.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:43 smithi165 ceph-mon[32893]: Saving service grafana spec with placement smithi165=a;count:1 2024-09-21T08:23:43.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:43 smithi165 ceph-mon[32893]: pgmap v199: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 74 KiB/s rd, 5.8 KiB/s wr, 180 op/s 2024-09-21T08:23:43.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:43 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:44.052 INFO:teuthology.orchestra.run.smithi137.stdout:[client.0] 2024-09-21T08:23:44.052 INFO:teuthology.orchestra.run.smithi137.stdout: key = AQCQgu5m5cj8AhAAPsybZOXUuo84AiLUsBTV2g== 2024-09-21T08:23:44.691 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:44 smithi165 ceph-mon[32893]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T08:23:44.691 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:44 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1226972718' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T08:23:44.691 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:44 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1226972718' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T08:23:44.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:44 smithi137 ceph-mon[29634]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T08:23:44.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:44 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1226972718' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T08:23:44.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:44 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1226972718' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T08:23:44.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:44 smithi137 ceph-mon[41044]: mgrmap e16: y(active, since 4m), standbys: x 2024-09-21T08:23:44.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:44 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1226972718' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T08:23:44.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:44 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1226972718' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T08:23:44.795 DEBUG:teuthology.orchestra.run.smithi137:> set -ex 2024-09-21T08:23:44.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-09-21T08:23:44.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-09-21T08:23:44.834 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-09-21T08:23:45.101 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:23:45.479 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:45 smithi165 ceph-mon[32893]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-09-21T08:23:45.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:45 smithi137 ceph-mon[29634]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-09-21T08:23:45.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:45 smithi137 ceph-mon[41044]: pgmap v200: 132 pgs: 132 active+clean; 582 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 58 KiB/s rd, 4.5 KiB/s wr, 140 op/s 2024-09-21T08:23:45.990 INFO:teuthology.orchestra.run.smithi165.stdout:[client.1] 2024-09-21T08:23:45.990 INFO:teuthology.orchestra.run.smithi165.stdout: key = AQCRgu5mO8fHOhAA5fEvG3NNSs4wze3T2DATig== 2024-09-21T08:23:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:46 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.165:0/1458189455' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T08:23:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:46 smithi165 ceph-mon[32893]: from='client.24514 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T08:23:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:46 smithi165 ceph-mon[32893]: from='client.24514 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T08:23:46.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:46 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.165:0/1458189455' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T08:23:46.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:46 smithi137 ceph-mon[29634]: from='client.24514 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T08:23:46.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:46 smithi137 ceph-mon[29634]: from='client.24514 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T08:23:46.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:46 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.165:0/1458189455' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T08:23:46.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:46 smithi137 ceph-mon[41044]: from='client.24514 ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-09-21T08:23:46.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:46 smithi137 ceph-mon[41044]: from='client.24514 ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-09-21T08:23:47.284 DEBUG:teuthology.orchestra.run.smithi165:> set -ex 2024-09-21T08:23:47.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-09-21T08:23:47.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-09-21T08:23:47.326 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-09-21T08:23:47.326 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-09-21T08:23:47.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph mgr dump --format=json 2024-09-21T08:23:47.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:47 smithi165 ceph-mon[32893]: pgmap v201: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 3.9 KiB/s wr, 123 op/s 2024-09-21T08:23:47.586 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:47.707 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:47 smithi137 ceph-mon[29634]: pgmap v201: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 3.9 KiB/s wr, 123 op/s 2024-09-21T08:23:47.708 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:47 smithi137 ceph-mon[41044]: pgmap v201: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 51 KiB/s rd, 3.9 KiB/s wr, 123 op/s 2024-09-21T08:23:48.286 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:23:49.013 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":16,"flags":0,"active_gid":14148,"active_name":"y","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6800","nonce":1825289826}]},"active_addr":"172.21.15.137:6800/1825289826","active_change":"2024-09-21T08:19:12.726695+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"x","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"secondary_zone_period_retry_limit":{"name":"secondary_zone_period_retry_limit","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"RGW module period update retry limit for secondary site","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.137:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"cephadm","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":3166961425}]},{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":3442439903}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1718301516}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":3010106803}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.137:0","nonce":1478868457}]}]} 2024-09-21T08:23:49.016 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-09-21T08:23:49.016 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-09-21T08:23:49.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T08:23:49.263 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:49.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:49 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:49.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:49 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3162497760' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T08:23:49.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:49 smithi137 ceph-mon[29634]: pgmap v202: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.3 KiB/s wr, 101 op/s 2024-09-21T08:23:49.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:49 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:49.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:49 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3162497760' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T08:23:49.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:49 smithi137 ceph-mon[41044]: pgmap v202: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.3 KiB/s wr, 101 op/s 2024-09-21T08:23:49.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:49 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:49.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:49 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3162497760' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-09-21T08:23:49.513 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:49 smithi165 ceph-mon[32893]: pgmap v202: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 3.3 KiB/s wr, 101 op/s 2024-09-21T08:23:49.855 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:23:49.856 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":74,"fsid":"ec8566d6-77f1-11ef-baf6-efdc52797490","created":"2024-09-21T08:18:21.794737+0000","modified":"2024-09-21T08:23:41.463588+0000","last_up_change":"2024-09-21T08:23:19.473450+0000","last_in_change":"2024-09-21T08:22:58.590287+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T08:21:15.037508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T08:23:28.246186+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T08:23:29.209879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T08:23:30.851833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T08:23:32.798630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T08:23:34.858938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b007d1d-6c6b-4a68-9e68-2c5fd7d26977","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":3938795530}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":3938795530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":3938795530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":3938795530}]},"public_addr":"172.21.15.137:6801/3938795530","cluster_addr":"172.21.15.137:6802/3938795530","heartbeat_back_addr":"172.21.15.137:6804/3938795530","heartbeat_front_addr":"172.21.15.137:6803/3938795530","state":["exists","up"]},{"osd":1,"uuid":"8651068e-694f-4bd4-9917-5c55670aca5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":2743268361}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":2743268361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":2743268361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":2743268361}]},"public_addr":"172.21.15.137:6805/2743268361","cluster_addr":"172.21.15.137:6806/2743268361","heartbeat_back_addr":"172.21.15.137:6808/2743268361","heartbeat_front_addr":"172.21.15.137:6807/2743268361","state":["exists","up"]},{"osd":2,"uuid":"2a22f0f7-c610-4995-98a7-d4785efb1e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":1500397557}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":1500397557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":1500397557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":1500397557}]},"public_addr":"172.21.15.137:6809/1500397557","cluster_addr":"172.21.15.137:6810/1500397557","heartbeat_back_addr":"172.21.15.137:6812/1500397557","heartbeat_front_addr":"172.21.15.137:6811/1500397557","state":["exists","up"]},{"osd":3,"uuid":"73f21606-38be-4578-a86b-bb417217c0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":617155021}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":617155021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6816","nonce":617155021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":617155021}]},"public_addr":"172.21.15.137:6813/617155021","cluster_addr":"172.21.15.137:6814/617155021","heartbeat_back_addr":"172.21.15.137:6816/617155021","heartbeat_front_addr":"172.21.15.137:6815/617155021","state":["exists","up"]},{"osd":4,"uuid":"e5f6762b-65f5-4dc7-8e54-9087149ae6ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":70360958}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":70360958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":70360958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":70360958}]},"public_addr":"172.21.15.165:6800/70360958","cluster_addr":"172.21.15.165:6801/70360958","heartbeat_back_addr":"172.21.15.165:6803/70360958","heartbeat_front_addr":"172.21.15.165:6802/70360958","state":["exists","up"]},{"osd":5,"uuid":"f7f8c456-b77f-4435-a860-202e64dda8dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":1835779380}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":1835779380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":1835779380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":1835779380}]},"public_addr":"172.21.15.165:6804/1835779380","cluster_addr":"172.21.15.165:6805/1835779380","heartbeat_back_addr":"172.21.15.165:6807/1835779380","heartbeat_front_addr":"172.21.15.165:6806/1835779380","state":["exists","up"]},{"osd":6,"uuid":"b8b3df16-73fa-41a5-a93f-117c599d488b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":3820354208}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":3820354208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":3820354208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":3820354208}]},"public_addr":"172.21.15.165:6808/3820354208","cluster_addr":"172.21.15.165:6809/3820354208","heartbeat_back_addr":"172.21.15.165:6811/3820354208","heartbeat_front_addr":"172.21.15.165:6810/3820354208","state":["exists","up"]},{"osd":7,"uuid":"ddf234e4-1494-40bd-a31b-333b01717c3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":1933524273}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":1933524273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":1933524273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":1933524273}]},"public_addr":"172.21.15.165:6812/1933524273","cluster_addr":"172.21.15.165:6813/1933524273","heartbeat_back_addr":"172.21.15.165:6815/1933524273","heartbeat_front_addr":"172.21.15.165:6814/1933524273","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:00.155347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:36.284738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:35.160722+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:58.117448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:21.051166+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:25.587597+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:51.433176+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:17.735488+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6800/1900990534":"2024-09-22T08:19:12.726560+0000","172.21.15.137:6800/2580000250":"2024-09-22T08:18:48.673163+0000","172.21.15.137:0/223603733":"2024-09-22T08:19:12.726560+0000","172.21.15.137:0/705906958":"2024-09-22T08:19:12.726560+0000","172.21.15.137:0/700535732":"2024-09-22T08:18:48.673163+0000","172.21.15.137:0/3936900881":"2024-09-22T08:19:12.726560+0000","172.21.15.137:0/2643910317":"2024-09-22T08:18:48.673163+0000","172.21.15.137:0/1738156348":"2024-09-22T08:18:48.673163+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T08:23:50.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:50 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2666136630' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T08:23:50.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:50 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2666136630' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T08:23:50.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:50 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2666136630' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T08:23:50.596 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-09-21T08:23:50.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd dump --format=json 2024-09-21T08:23:50.835 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:51.225 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:51 smithi165 ceph-mon[32893]: pgmap v203: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.2 KiB/s wr, 37 op/s 2024-09-21T08:23:51.264 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:51 smithi137 ceph-mon[29634]: pgmap v203: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.2 KiB/s wr, 37 op/s 2024-09-21T08:23:51.264 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:51 smithi137 ceph-mon[41044]: pgmap v203: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 15 KiB/s rd, 1.2 KiB/s wr, 37 op/s 2024-09-21T08:23:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:23:51.522 INFO:teuthology.orchestra.run.smithi137.stdout:{"epoch":74,"fsid":"ec8566d6-77f1-11ef-baf6-efdc52797490","created":"2024-09-21T08:18:21.794737+0000","modified":"2024-09-21T08:23:41.463588+0000","last_up_change":"2024-09-21T08:23:19.473450+0000","last_in_change":"2024-09-21T08:22:58.590287+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":18,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-21T08:21:15.037508+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":7.8899998664855957,"score_stable":7.8899998664855957,"optimal_score":0.37999999523162842,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"datapool","create_time":"2024-09-21T08:23:28.246186+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":3,"pg_placement_num":3,"pg_placement_num_target":3,"pg_num_target":3,"pg_num_pending":3,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"68","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":68,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.6500000953674316,"score_stable":2.6500000953674316,"optimal_score":0.87999999523162842,"raw_score_acting":2.3299999237060547,"raw_score_stable":2.3299999237060547,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":".rgw.root","create_time":"2024-09-21T08:23:29.209879+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"67","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"default.rgw.log","create_time":"2024-09-21T08:23:30.851833+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"69","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":5,"pool_name":"default.rgw.control","create_time":"2024-09-21T08:23:32.798630+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.25,"score_stable":1.25,"optimal_score":1,"raw_score_acting":1.25,"raw_score_stable":1.25,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":6,"pool_name":"default.rgw.meta","create_time":"2024-09-21T08:23:34.858938+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"73","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4},"application_metadata":{"rgw":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.75,"score_stable":1.75,"optimal_score":1,"raw_score_acting":1.75,"raw_score_stable":1.75,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"1b007d1d-6c6b-4a68-9e68-2c5fd7d26977","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6801","nonce":3938795530}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6802","nonce":3938795530}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6804","nonce":3938795530}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6803","nonce":3938795530}]},"public_addr":"172.21.15.137:6801/3938795530","cluster_addr":"172.21.15.137:6802/3938795530","heartbeat_back_addr":"172.21.15.137:6804/3938795530","heartbeat_front_addr":"172.21.15.137:6803/3938795530","state":["exists","up"]},{"osd":1,"uuid":"8651068e-694f-4bd4-9917-5c55670aca5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6805","nonce":2743268361}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6806","nonce":2743268361}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6808","nonce":2743268361}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6807","nonce":2743268361}]},"public_addr":"172.21.15.137:6805/2743268361","cluster_addr":"172.21.15.137:6806/2743268361","heartbeat_back_addr":"172.21.15.137:6808/2743268361","heartbeat_front_addr":"172.21.15.137:6807/2743268361","state":["exists","up"]},{"osd":2,"uuid":"2a22f0f7-c610-4995-98a7-d4785efb1e5d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6809","nonce":1500397557}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6810","nonce":1500397557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6812","nonce":1500397557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6811","nonce":1500397557}]},"public_addr":"172.21.15.137:6809/1500397557","cluster_addr":"172.21.15.137:6810/1500397557","heartbeat_back_addr":"172.21.15.137:6812/1500397557","heartbeat_front_addr":"172.21.15.137:6811/1500397557","state":["exists","up"]},{"osd":3,"uuid":"73f21606-38be-4578-a86b-bb417217c0e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6813","nonce":617155021}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6814","nonce":617155021}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6816","nonce":617155021}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.137:6815","nonce":617155021}]},"public_addr":"172.21.15.137:6813/617155021","cluster_addr":"172.21.15.137:6814/617155021","heartbeat_back_addr":"172.21.15.137:6816/617155021","heartbeat_front_addr":"172.21.15.137:6815/617155021","state":["exists","up"]},{"osd":4,"uuid":"e5f6762b-65f5-4dc7-8e54-9087149ae6ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":31,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6800","nonce":70360958}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6801","nonce":70360958}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6803","nonce":70360958}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6802","nonce":70360958}]},"public_addr":"172.21.15.165:6800/70360958","cluster_addr":"172.21.15.165:6801/70360958","heartbeat_back_addr":"172.21.15.165:6803/70360958","heartbeat_front_addr":"172.21.15.165:6802/70360958","state":["exists","up"]},{"osd":5,"uuid":"f7f8c456-b77f-4435-a860-202e64dda8dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":41,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6804","nonce":1835779380}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6805","nonce":1835779380}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6807","nonce":1835779380}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6806","nonce":1835779380}]},"public_addr":"172.21.15.165:6804/1835779380","cluster_addr":"172.21.15.165:6805/1835779380","heartbeat_back_addr":"172.21.15.165:6807/1835779380","heartbeat_front_addr":"172.21.15.165:6806/1835779380","state":["exists","up"]},{"osd":6,"uuid":"b8b3df16-73fa-41a5-a93f-117c599d488b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":49,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6808","nonce":3820354208}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6809","nonce":3820354208}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6811","nonce":3820354208}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6810","nonce":3820354208}]},"public_addr":"172.21.15.165:6808/3820354208","cluster_addr":"172.21.15.165:6809/3820354208","heartbeat_back_addr":"172.21.15.165:6811/3820354208","heartbeat_front_addr":"172.21.15.165:6810/3820354208","state":["exists","up"]},{"osd":7,"uuid":"ddf234e4-1494-40bd-a31b-333b01717c3f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":71,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6812","nonce":1933524273}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6813","nonce":1933524273}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6815","nonce":1933524273}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.165:6814","nonce":1933524273}]},"public_addr":"172.21.15.165:6812/1933524273","cluster_addr":"172.21.15.165:6813/1933524273","heartbeat_back_addr":"172.21.15.165:6815/1933524273","heartbeat_front_addr":"172.21.15.165:6814/1933524273","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:00.155347+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:36.284738+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:35.160722+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:58.117448+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:21.051166+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:25.587597+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:22:51.433176+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-21T08:23:17.735488+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.137:6800/1900990534":"2024-09-22T08:19:12.726560+0000","172.21.15.137:6800/2580000250":"2024-09-22T08:18:48.673163+0000","172.21.15.137:0/223603733":"2024-09-22T08:19:12.726560+0000","172.21.15.137:0/705906958":"2024-09-22T08:19:12.726560+0000","172.21.15.137:0/700535732":"2024-09-22T08:18:48.673163+0000","172.21.15.137:0/3936900881":"2024-09-22T08:19:12.726560+0000","172.21.15.137:0/2643910317":"2024-09-22T08:18:48.673163+0000","172.21.15.137:0/1738156348":"2024-09-22T08:18:48.673163+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-21T08:23:52.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:23:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:23:52.235 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:52 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/862332812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T08:23:52.236 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:52 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:23:52.236 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:52 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/862332812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T08:23:52.236 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:52 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:23:52.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph tell osd.0 flush_pg_stats 2024-09-21T08:23:52.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph tell osd.1 flush_pg_stats 2024-09-21T08:23:52.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph tell osd.2 flush_pg_stats 2024-09-21T08:23:52.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph tell osd.3 flush_pg_stats 2024-09-21T08:23:52.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph tell osd.4 flush_pg_stats 2024-09-21T08:23:52.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph tell osd.5 flush_pg_stats 2024-09-21T08:23:52.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph tell osd.6 flush_pg_stats 2024-09-21T08:23:52.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph tell osd.7 flush_pg_stats 2024-09-21T08:23:52.322 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:52 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/862332812' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-09-21T08:23:52.323 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:52 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:23:52.770 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:52.787 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:52.788 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:52.790 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:52.793 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:52.825 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:52.838 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:52.838 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:23:53.013 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:52 smithi165 systemd[1]: Starting Ceph prometheus.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:23:53.256 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:53 smithi137 ceph-mon[29634]: pgmap v204: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 362 B/s wr, 8 op/s 2024-09-21T08:23:53.256 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:53 smithi137 ceph-mon[41044]: pgmap v204: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 362 B/s wr, 8 op/s 2024-09-21T08:23:53.419 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-mon[32893]: pgmap v204: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 3.5 KiB/s rd, 362 B/s wr, 8 op/s 2024-09-21T08:23:53.419 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 podman[62784]: 2024-09-21 08:23:53.036852726 +0000 UTC m=+0.016744044 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T08:23:53.419 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 podman[62784]: 2024-09-21 08:23:53.158546825 +0000 UTC m=+0.138438131 container create ae6d54d2591339eeab1590759b85b64aa6cbaa47f70b4754ac4fc11a30658232 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T08:23:53.419 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 podman[62784]: 2024-09-21 08:23:53.419075443 +0000 UTC m=+0.398966748 container init ae6d54d2591339eeab1590759b85b64aa6cbaa47f70b4754ac4fc11a30658232 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 podman[62784]: 2024-09-21 08:23:53.423810982 +0000 UTC m=+0.403702289 container start ae6d54d2591339eeab1590759b85b64aa6cbaa47f70b4754ac4fc11a30658232 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.448Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.448Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.448Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi165 (none))" 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.448Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.448Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.449Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.450Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.450Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.450Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.453Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.453Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.556µs 2024-09-21T08:23:53.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.453Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T08:23:53.765 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.453Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=0 2024-09-21T08:23:53.765 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.453Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=17.718µs wal_replay_duration=370.379µs wbl_replay_duration=126ns total_replay_duration=405.169µs 2024-09-21T08:23:53.765 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.454Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T08:23:53.765 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.454Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T08:23:53.765 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.454Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T08:23:53.765 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.474Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=20.329029ms db_storage=800ns remote_storage=1.079µs web_handler=309ns query_engine=484ns scrape=396.345µs scrape_sd=89.83µs notify=922ns notify_sd=3.182µs rules=19.568773ms tracing=5.189µs 2024-09-21T08:23:53.765 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.475Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T08:23:53.765 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:23:53.475Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T08:23:53.765 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 bash[62784]: ae6d54d2591339eeab1590759b85b64aa6cbaa47f70b4754ac4fc11a30658232 2024-09-21T08:23:53.765 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:23:53 smithi165 systemd[1]: Started Ceph prometheus.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:23:54.320 INFO:teuthology.orchestra.run.smithi137.stdout:133143986201 2024-09-21T08:23:54.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.4 2024-09-21T08:23:54.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:54.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:54.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:54.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T08:23:54.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:54.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:54.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:54.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T08:23:54.861 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ignoring --setuser ceph since I am not root 2024-09-21T08:23:54.861 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ignoring --setgroup ceph since I am not root 2024-09-21T08:23:54.861 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:54.726+0000 7f53af3b1040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T08:23:54.861 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:54.792+0000 7f53af3b1040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T08:23:54.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:54.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:54.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' 2024-09-21T08:23:54.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:54 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-09-21T08:23:54.937 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:54 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: ignoring --setuser ceph since I am not root 2024-09-21T08:23:54.937 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:54 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: ignoring --setgroup ceph since I am not root 2024-09-21T08:23:54.937 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:54 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:54.727+0000 7fde4d4d9040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-21T08:23:54.937 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:54 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:54.787+0000 7fde4d4d9040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-21T08:23:54.937 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:54 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:54.934+0000 7fde4d4d9040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T08:23:55.181 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:54 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:54.948+0000 7f53af3b1040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-21T08:23:55.181 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:55.028+0000 7f53af3b1040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T08:23:55.181 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:55.179+0000 7f53af3b1040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:23:55.263 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:55 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:55.002+0000 7fde4d4d9040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-21T08:23:55.263 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:55 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:55.147+0000 7fde4d4d9040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:23:55.517 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:55.332+0000 7f53af3b1040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T08:23:55.579 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:55 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:55.289+0000 7fde4d4d9040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-21T08:23:55.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:55 smithi165 ceph-mon[32893]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T08:23:55.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:55 smithi165 ceph-mon[32893]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-21T08:23:55.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:55 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:23:55.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:55 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:23:55.860 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:55 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:55.608+0000 7fde4d4d9040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T08:23:55.860 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:55 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:55.670+0000 7fde4d4d9040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T08:23:55.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-mon[41044]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T08:23:55.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-mon[41044]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-21T08:23:55.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:23:55.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:23:55.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-mon[29634]: from='mgr.14148 v1:172.21.15.137:0/2277979675' entity='mgr.y' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-09-21T08:23:55.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-mon[29634]: mgrmap e17: y(active, since 4m), standbys: x 2024-09-21T08:23:55.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:23:55.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:23:55.925 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:55.662+0000 7f53af3b1040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-21T08:23:55.925 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:55.734+0000 7f53af3b1040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-21T08:23:55.925 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:55.923+0000 7f53af3b1040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T08:23:56.150 INFO:teuthology.orchestra.run.smithi137.stdout:253403070474 2024-09-21T08:23:56.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.7 2024-09-21T08:23:56.263 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:55 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:55.858+0000 7fde4d4d9040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-21T08:23:56.733 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:56 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:23:56.733 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:56 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:23:56.733 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:56 smithi137 ceph-mon[29634]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T08:23:56.733 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:56 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:23:56.734 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:56 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:23:56.734 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:56 smithi137 ceph-mon[41044]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T08:23:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:56 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:23:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:56 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:23:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:56 smithi165 ceph-mon[32893]: osdmap e75: 8 total, 8 up, 8 in 2024-09-21T08:23:57.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:56 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:56.755+0000 7fde4d4d9040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T08:23:57.013 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:56 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:56.824+0000 7fde4d4d9040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T08:23:57.017 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:56 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:56.868+0000 7f53af3b1040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-21T08:23:57.017 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:56 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:56.937+0000 7f53af3b1040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-21T08:23:57.792 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:57 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:57.673+0000 7f53af3b1040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T08:23:57.792 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:57 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:57.790+0000 7f53af3b1040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T08:23:57.812 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:57.550+0000 7fde4d4d9040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-09-21T08:23:57.812 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:57.667+0000 7fde4d4d9040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-21T08:23:57.860 INFO:teuthology.orchestra.run.smithi137.stdout:34359738412 2024-09-21T08:23:57.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.0 2024-09-21T08:23:58.092 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:57.810+0000 7fde4d4d9040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:23:58.092 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:57.873+0000 7fde4d4d9040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T08:23:58.092 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:58 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:58.022+0000 7fde4d4d9040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T08:23:58.093 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:58 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:58.091+0000 7fde4d4d9040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T08:23:58.163 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:57 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:57.943+0000 7f53af3b1040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-21T08:23:58.163 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:58 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:58.008+0000 7f53af3b1040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-21T08:23:58.163 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:58 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:58.161+0000 7f53af3b1040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-09-21T08:23:58.375 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:58 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:58.151+0000 7fde4d4d9040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T08:23:58.376 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:58 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:58.308+0000 7fde4d4d9040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T08:23:58.434 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:58 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:58.230+0000 7f53af3b1040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-21T08:23:58.434 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:58 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:58.291+0000 7f53af3b1040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-21T08:23:58.650 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:58 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:58.374+0000 7fde4d4d9040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T08:23:58.651 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:58 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:58.442+0000 7fde4d4d9040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T08:23:58.651 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:58 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:58.526+0000 7fde4d4d9040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T08:23:58.651 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:58 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:58.588+0000 7fde4d4d9040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T08:23:58.651 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:58 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:58.649+0000 7fde4d4d9040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T08:23:58.732 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:58 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:58.453+0000 7f53af3b1040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-21T08:23:58.732 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:58 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:58.518+0000 7f53af3b1040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-21T08:23:58.733 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:58 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:58.587+0000 7f53af3b1040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-21T08:23:58.733 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:58 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:58.672+0000 7f53af3b1040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-21T08:23:58.733 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:58 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:58.731+0000 7f53af3b1040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-21T08:23:58.774 INFO:teuthology.orchestra.run.smithi137.stdout:176093659155 2024-09-21T08:23:58.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.5 2024-09-21T08:23:58.988 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:58 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:58.815+0000 7f53af3b1040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-21T08:23:59.681 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[29634]: Standby manager daemon x restarted 2024-09-21T08:23:59.681 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[29634]: Standby manager daemon x started 2024-09-21T08:23:59.681 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[29634]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T08:23:59.681 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[29634]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T08:23:59.681 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[29634]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T08:23:59.681 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[29634]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T08:23:59.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[41044]: Standby manager daemon x restarted 2024-09-21T08:23:59.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[41044]: Standby manager daemon x started 2024-09-21T08:23:59.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[41044]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T08:23:59.682 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[41044]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T08:23:59.682 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[41044]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T08:23:59.682 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-mon[41044]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T08:23:59.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-mon[32893]: Standby manager daemon x restarted 2024-09-21T08:23:59.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-mon[32893]: Standby manager daemon x started 2024-09-21T08:23:59.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-mon[32893]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/crt"}]: dispatch 2024-09-21T08:23:59.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-mon[32893]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-09-21T08:23:59.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-mon[32893]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/x/key"}]: dispatch 2024-09-21T08:23:59.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-mon[32893]: from='mgr.? v1:172.21.15.165:0/3142538018' entity='mgr.x' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-09-21T08:23:59.764 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:59.454+0000 7fde4d4d9040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T08:23:59.764 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: [21/Sep/2024:08:23:59] ENGINE Bus STARTING 2024-09-21T08:23:59.764 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: CherryPy Checker: 2024-09-21T08:23:59.764 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: The Application mounted at '' has an empty config. 2024-09-21T08:23:59.764 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: 2024-09-21T08:23:59.764 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: [21/Sep/2024:08:23:59] ENGINE Serving on http://:::9283 2024-09-21T08:23:59.764 INFO:journalctl@ceph.mgr.x.smithi165.stdout:Sep 21 08:23:59 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-x[34878]: [21/Sep/2024:08:23:59] ENGINE Bus STARTED 2024-09-21T08:23:59.965 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:59.679+0000 7f53af3b1040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-21T08:23:59.965 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:23:59] ENGINE Bus STARTING 2024-09-21T08:23:59.965 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: CherryPy Checker: 2024-09-21T08:23:59.965 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: The Application mounted at '' has an empty config. 2024-09-21T08:23:59.965 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:23:59.965 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:23:59] ENGINE Serving on http://:::9283 2024-09-21T08:23:59.965 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:23:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:23:59] ENGINE Bus STARTED 2024-09-21T08:24:00.227 INFO:teuthology.orchestra.run.smithi137.stdout:77309411363 2024-09-21T08:24:00.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.2 2024-09-21T08:24:00.585 INFO:teuthology.orchestra.run.smithi137.stdout:55834574887 2024-09-21T08:24:00.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.1 2024-09-21T08:24:00.873 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:00.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-21T08:24:00.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: Active manager daemon y restarted 2024-09-21T08:24:00.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: Activating manager daemon y 2024-09-21T08:24:00.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T08:24:00.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: mgrmap e19: y(active, starting, since 0.00660691s), standbys: x 2024-09-21T08:24:00.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:24:00.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:24:00.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T08:24:00.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: Manager daemon y is now available 2024-09-21T08:24:00.903 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:24:00.903 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:00.903 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:00.903 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T08:24:00.903 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T08:24:00.903 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T08:24:00.903 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T08:24:00.939 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:24:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:24:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-21T08:24:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: Active manager daemon y restarted 2024-09-21T08:24:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: Activating manager daemon y 2024-09-21T08:24:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: mgrmap e19: y(active, starting, since 0.00660691s), standbys: x 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:24:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: Manager daemon y is now available 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: mgrmap e18: y(active, since 4m), standbys: x 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: Active manager daemon y restarted 2024-09-21T08:24:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: Activating manager daemon y 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: osdmap e76: 8 total, 8 up, 8 in 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: mgrmap e19: y(active, starting, since 0.00660691s), standbys: x 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-09-21T08:24:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: Manager daemon y is now available 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/mirror_snapshot_schedule"}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T08:24:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/y/trash_purge_schedule"}]: dispatch 2024-09-21T08:24:01.241 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:24:00.937+0000 7f5306323640 -1 log_channel(cephadm) log [ERR] : [21/Sep/2024:08:24:00] ENGINE Error in HTTPServer.serve 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Traceback (most recent call last): 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self._connections.run(self.expiration_interval) 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self._run(expiration_interval) 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: s = self.context.wrap_socket( 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T08:24:01.242 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: return self.sslsocket_class._create( 2024-09-21T08:24:01.243 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T08:24:01.243 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self.do_handshake() 2024-09-21T08:24:01.243 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T08:24:01.243 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: self._sslobj.do_handshake() 2024-09-21T08:24:01.243 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T08:24:01.243 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:24:01.710 INFO:teuthology.orchestra.run.smithi137.stdout:107374182429 2024-09-21T08:24:01.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.3 2024-09-21T08:24:01.816 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:24:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:24:01.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:01 smithi165 ceph-mon[32893]: mgrmap e20: y(active, since 1.01201s), standbys: x 2024-09-21T08:24:01.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:01 smithi165 ceph-mon[32893]: [21/Sep/2024:08:24:00] ENGINE Bus STARTING 2024-09-21T08:24:01.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:01 smithi165 ceph-mon[32893]: [21/Sep/2024:08:24:00] ENGINE Serving on http://172.21.15.137:8765 2024-09-21T08:24:01.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:01 smithi165 ceph-mon[32893]: [21/Sep/2024:08:24:00] ENGINE Serving on https://172.21.15.137:7150 2024-09-21T08:24:01.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:01 smithi165 ceph-mon[32893]: [21/Sep/2024:08:24:00] ENGINE Bus STARTED 2024-09-21T08:24:01.816 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:01 smithi165 ceph-mon[32893]: [21/Sep/2024:08:24:00] ENGINE Error in HTTPServer.serve 2024-09-21T08:24:01.816 INFO:journalctl@ceph.mon.b.smithi165.stdout: Traceback (most recent call last): 2024-09-21T08:24:01.816 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._connections.run(self.expiration_interval) 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._run(expiration_interval) 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: s = self.context.wrap_socket( 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: return self.sslsocket_class._create( 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: self.do_handshake() 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T08:24:01.817 INFO:journalctl@ceph.mon.b.smithi165.stdout: self._sslobj.do_handshake() 2024-09-21T08:24:01.818 INFO:journalctl@ceph.mon.b.smithi165.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T08:24:01.818 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:01 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:01.910 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:01.910 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[29634]: mgrmap e20: y(active, since 1.01201s), standbys: x 2024-09-21T08:24:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[29634]: [21/Sep/2024:08:24:00] ENGINE Bus STARTING 2024-09-21T08:24:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[29634]: [21/Sep/2024:08:24:00] ENGINE Serving on http://172.21.15.137:8765 2024-09-21T08:24:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[29634]: [21/Sep/2024:08:24:00] ENGINE Serving on https://172.21.15.137:7150 2024-09-21T08:24:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[29634]: [21/Sep/2024:08:24:00] ENGINE Bus STARTED 2024-09-21T08:24:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[29634]: [21/Sep/2024:08:24:00] ENGINE Error in HTTPServer.serve 2024-09-21T08:24:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout: Traceback (most recent call last): 2024-09-21T08:24:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._run(expiration_interval) 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: s = self.context.wrap_socket( 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: return self.sslsocket_class._create( 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T08:24:02.019 INFO:journalctl@ceph.mon.a.smithi137.stdout: self.do_handshake() 2024-09-21T08:24:02.020 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T08:24:02.020 INFO:journalctl@ceph.mon.a.smithi137.stdout: self._sslobj.do_handshake() 2024-09-21T08:24:02.020 INFO:journalctl@ceph.mon.a.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T08:24:02.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:02.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[41044]: mgrmap e20: y(active, since 1.01201s), standbys: x 2024-09-21T08:24:02.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[41044]: [21/Sep/2024:08:24:00] ENGINE Bus STARTING 2024-09-21T08:24:02.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[41044]: [21/Sep/2024:08:24:00] ENGINE Serving on http://172.21.15.137:8765 2024-09-21T08:24:02.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[41044]: [21/Sep/2024:08:24:00] ENGINE Serving on https://172.21.15.137:7150 2024-09-21T08:24:02.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[41044]: [21/Sep/2024:08:24:00] ENGINE Bus STARTED 2024-09-21T08:24:02.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[41044]: [21/Sep/2024:08:24:00] ENGINE Error in HTTPServer.serve 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: Traceback (most recent call last): 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: self._connections.run(self.expiration_interval) 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: self._run(expiration_interval) 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: s = self.context.wrap_socket( 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: return self.sslsocket_class._create( 2024-09-21T08:24:02.021 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-09-21T08:24:02.022 INFO:journalctl@ceph.mon.c.smithi137.stdout: self.do_handshake() 2024-09-21T08:24:02.022 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-09-21T08:24:02.022 INFO:journalctl@ceph.mon.c.smithi137.stdout: self._sslobj.do_handshake() 2024-09-21T08:24:02.022 INFO:journalctl@ceph.mon.c.smithi137.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1133) 2024-09-21T08:24:02.022 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:01 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:02.076 INFO:teuthology.orchestra.run.smithi137.stdout:133143986202 2024-09-21T08:24:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:02 smithi165 ceph-mon[32893]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-21T08:24:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:02 smithi165 ceph-mon[32893]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:24:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:02 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3293423189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T08:24:02.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:02 smithi137 ceph-mon[29634]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-21T08:24:02.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:02 smithi137 ceph-mon[29634]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:24:02.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:02 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3293423189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T08:24:02.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:02 smithi137 ceph-mon[41044]: mgrmap e21: y(active, since 2s), standbys: x 2024-09-21T08:24:02.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:02 smithi137 ceph-mon[41044]: pgmap v4: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:24:02.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:02 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3293423189' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-09-21T08:24:02.835 INFO:tasks.cephadm.ceph_manager.ceph:need seq 133143986201 got 133143986202 for osd.4 2024-09-21T08:24:02.835 DEBUG:teuthology.parallel:result is None 2024-09-21T08:24:03.089 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:03.202 INFO:teuthology.orchestra.run.smithi137.stdout:253403070475 2024-09-21T08:24:03.995 INFO:teuthology.orchestra.run.smithi137.stdout:34359738413 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1028458991' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: Adjusting osd_memory_target on smithi165 to 3760M 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1028458991' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:03.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: Adjusting osd_memory_target on smithi165 to 3760M 2024-09-21T08:24:03.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:03 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:04.219 INFO:teuthology.orchestra.run.smithi137.stdout:210453397520 2024-09-21T08:24:04.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd last-stat-seq osd.6 2024-09-21T08:24:04.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:04.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:04.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1028458991' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-09-21T08:24:04.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:04.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:04.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:04.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:04.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:04.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:04.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:04.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:04.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:04.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:04.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: Adjusting osd_memory_target on smithi165 to 3760M 2024-09-21T08:24:04.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:03 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:04.909 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738412 got 34359738413 for osd.0 2024-09-21T08:24:04.909 DEBUG:teuthology.parallel:result is None 2024-09-21T08:24:05.131 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:04 smithi137 ceph-mon[29634]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:24:05.131 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:04 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3855660926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T08:24:05.132 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:04 smithi137 ceph-mon[29634]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-21T08:24:05.132 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:04 smithi137 ceph-mon[41044]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:24:05.132 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:04 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3855660926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T08:24:05.132 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:04 smithi137 ceph-mon[41044]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-21T08:24:05.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:04 smithi165 ceph-mon[32893]: pgmap v5: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail 2024-09-21T08:24:05.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:04 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3855660926' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-09-21T08:24:05.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:04 smithi165 ceph-mon[32893]: mgrmap e22: y(active, since 4s), standbys: x 2024-09-21T08:24:05.351 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:05.466 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:05.468 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:05.509 INFO:teuthology.orchestra.run.smithi137.stdout:176093659156 2024-09-21T08:24:05.997 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070474 got 253403070475 for osd.7 2024-09-21T08:24:05.997 DEBUG:teuthology.parallel:result is None 2024-09-21T08:24:06.245 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:05 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1737045795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T08:24:06.246 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:05 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1737045795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T08:24:06.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:05 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1737045795' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-09-21T08:24:06.866 INFO:teuthology.orchestra.run.smithi137.stdout:210453397521 2024-09-21T08:24:07.757 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:07 smithi137 ceph-mon[29634]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-21T08:24:07.757 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:07 smithi137 ceph-mon[29634]: 1.0 deep-scrub starts 2024-09-21T08:24:07.757 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:07 smithi137 ceph-mon[29634]: 1.0 deep-scrub ok 2024-09-21T08:24:07.757 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:07 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1558457751' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T08:24:07.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:07 smithi137 ceph-mon[41044]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-21T08:24:07.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:07 smithi137 ceph-mon[41044]: 1.0 deep-scrub starts 2024-09-21T08:24:07.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:07 smithi137 ceph-mon[41044]: 1.0 deep-scrub ok 2024-09-21T08:24:07.757 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:07 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1558457751' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T08:24:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:07 smithi165 ceph-mon[32893]: pgmap v6: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 42 KiB/s rd, 0 B/s wr, 24 op/s 2024-09-21T08:24:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:07 smithi165 ceph-mon[32893]: 1.0 deep-scrub starts 2024-09-21T08:24:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:07 smithi165 ceph-mon[32893]: 1.0 deep-scrub ok 2024-09-21T08:24:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:07 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1558457751' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-09-21T08:24:07.916 INFO:teuthology.orchestra.run.smithi137.stdout:107374182431 2024-09-21T08:24:08.030 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:08.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:08 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4278442072' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T08:24:08.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:08 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4278442072' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T08:24:08.703 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182429 got 107374182431 for osd.3 2024-09-21T08:24:08.704 DEBUG:teuthology.parallel:result is None 2024-09-21T08:24:08.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:08 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4278442072' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-09-21T08:24:09.059 INFO:tasks.cephadm.ceph_manager.ceph:need seq 210453397520 got 210453397521 for osd.6 2024-09-21T08:24:09.060 DEBUG:teuthology.parallel:result is None 2024-09-21T08:24:09.542 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:09 smithi137 ceph-mon[29634]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T08:24:09.542 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:09 smithi137 ceph-mon[41044]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T08:24:09.551 INFO:teuthology.orchestra.run.smithi137.stdout:55834574889 2024-09-21T08:24:09.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:09 smithi165 ceph-mon[32893]: pgmap v7: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 30 KiB/s rd, 0 B/s wr, 17 op/s 2024-09-21T08:24:10.320 INFO:tasks.cephadm.ceph_manager.ceph:need seq 176093659155 got 176093659156 for osd.5 2024-09-21T08:24:10.321 DEBUG:teuthology.parallel:result is None 2024-09-21T08:24:10.627 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:10 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/197982335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T08:24:10.627 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:10 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/197982335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T08:24:10.678 INFO:teuthology.orchestra.run.smithi137.stdout:77309411365 2024-09-21T08:24:10.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:10 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/197982335' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-09-21T08:24:11.017 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:24:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:24:11.094 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574887 got 55834574889 for osd.1 2024-09-21T08:24:11.094 DEBUG:teuthology.parallel:result is None 2024-09-21T08:24:11.548 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411363 got 77309411365 for osd.2 2024-09-21T08:24:11.548 DEBUG:teuthology.parallel:result is None 2024-09-21T08:24:11.548 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T08:24:11.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:24:11.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:11 smithi165 ceph-mon[32893]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T08:24:11.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:11 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3687877004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T08:24:11.764 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:24:11 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:24:11.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:11 smithi137 ceph-mon[29634]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T08:24:11.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:11 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3687877004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T08:24:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:11 smithi137 ceph-mon[41044]: pgmap v8: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 23 KiB/s rd, 0 B/s wr, 13 op/s 2024-09-21T08:24:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:11 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3687877004' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-09-21T08:24:11.820 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:12.493 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:24:12.495 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:24:12.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:12.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:24:12.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:12 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-09-21T08:24:12.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:12.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:24:12.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:12 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:24:13.251 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":9,"stamp":"2024-09-21T08:24:11.702634+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":969,"num_read_kb":815,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":602,"ondisk_log_size":602,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":231376,"kb_used_data":6844,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749729840,"statfs":{"total":767960285184,"available":767723356160,"internally_reserved":0,"allocated":7008256,"data_stored":3947425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":106,"num_read_kb":216,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.000163"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005780+0000","last_change":"2024-09-21T08:23:30.549696+0000","last_active":"2024-09-21T08:24:00.005780+0000","last_peered":"2024-09-21T08:24:00.005780+0000","last_clean":"2024-09-21T08:24:00.005780+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:24:00.005780+0000","last_undegraded":"2024-09-21T08:24:00.005780+0000","last_fullsized":"2024-09-21T08:24:00.005780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.197261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.687852+0000","last_change":"2024-09-21T08:23:32.556768+0000","last_active":"2024-09-21T08:23:59.687852+0000","last_peered":"2024-09-21T08:23:59.687852+0000","last_clean":"2024-09-21T08:23:59.687852+0000","last_became_active":"2024-09-21T08:23:32.556452+0000","last_became_peered":"2024-09-21T08:23:32.556452+0000","last_unstale":"2024-09-21T08:23:59.687852+0000","last_undegraded":"2024-09-21T08:23:59.687852+0000","last_fullsized":"2024-09-21T08:23:59.687852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.936229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689547+0000","last_change":"2024-09-21T08:23:34.569082+0000","last_active":"2024-09-21T08:23:59.689547+0000","last_peered":"2024-09-21T08:23:59.689547+0000","last_clean":"2024-09-21T08:23:59.689547+0000","last_became_active":"2024-09-21T08:23:34.568762+0000","last_became_peered":"2024-09-21T08:23:34.568762+0000","last_unstale":"2024-09-21T08:23:59.689547+0000","last_undegraded":"2024-09-21T08:23:59.689547+0000","last_fullsized":"2024-09-21T08:23:59.689547+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.577218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688107+0000","last_change":"2024-09-21T08:23:36.575623+0000","last_active":"2024-09-21T08:23:59.688107+0000","last_peered":"2024-09-21T08:23:59.688107+0000","last_clean":"2024-09-21T08:23:59.688107+0000","last_became_active":"2024-09-21T08:23:36.572359+0000","last_became_peered":"2024-09-21T08:23:36.572359+0000","last_unstale":"2024-09-21T08:23:59.688107+0000","last_undegraded":"2024-09-21T08:23:59.688107+0000","last_fullsized":"2024-09-21T08:23:59.688107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.062906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005026+0000","last_change":"2024-09-21T08:23:36.585557+0000","last_active":"2024-09-21T08:24:00.005026+0000","last_peered":"2024-09-21T08:24:00.005026+0000","last_clean":"2024-09-21T08:24:00.005026+0000","last_became_active":"2024-09-21T08:23:36.584820+0000","last_became_peered":"2024-09-21T08:23:36.584820+0000","last_unstale":"2024-09-21T08:24:00.005026+0000","last_undegraded":"2024-09-21T08:24:00.005026+0000","last_fullsized":"2024-09-21T08:24:00.005026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.509136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006185+0000","last_change":"2024-09-21T08:23:30.548436+0000","last_active":"2024-09-21T08:24:00.006185+0000","last_peered":"2024-09-21T08:24:00.006185+0000","last_clean":"2024-09-21T08:24:00.006185+0000","last_became_active":"2024-09-21T08:23:30.548146+0000","last_became_peered":"2024-09-21T08:23:30.548146+0000","last_unstale":"2024-09-21T08:24:00.006185+0000","last_undegraded":"2024-09-21T08:24:00.006185+0000","last_fullsized":"2024-09-21T08:24:00.006185+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.518727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004908+0000","last_change":"2024-09-21T08:23:32.556169+0000","last_active":"2024-09-21T08:24:00.004908+0000","last_peered":"2024-09-21T08:24:00.004908+0000","last_clean":"2024-09-21T08:24:00.004908+0000","last_became_active":"2024-09-21T08:23:32.555789+0000","last_became_peered":"2024-09-21T08:23:32.555789+0000","last_unstale":"2024-09-21T08:24:00.004908+0000","last_undegraded":"2024-09-21T08:24:00.004908+0000","last_fullsized":"2024-09-21T08:24:00.004908+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.359444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.687767+0000","last_change":"2024-09-21T08:23:34.565614+0000","last_active":"2024-09-21T08:23:59.687767+0000","last_peered":"2024-09-21T08:23:59.687767+0000","last_clean":"2024-09-21T08:23:59.687767+0000","last_became_active":"2024-09-21T08:23:34.565276+0000","last_became_peered":"2024-09-21T08:23:34.565276+0000","last_unstale":"2024-09-21T08:23:59.687767+0000","last_undegraded":"2024-09-21T08:23:59.687767+0000","last_fullsized":"2024-09-21T08:23:59.687767+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.926230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773690+0000","last_change":"2024-09-21T08:23:30.543643+0000","last_active":"2024-09-21T08:23:59.773690+0000","last_peered":"2024-09-21T08:23:59.773690+0000","last_clean":"2024-09-21T08:23:59.773690+0000","last_became_active":"2024-09-21T08:23:30.543041+0000","last_became_peered":"2024-09-21T08:23:30.543041+0000","last_unstale":"2024-09-21T08:23:59.773690+0000","last_undegraded":"2024-09-21T08:23:59.773690+0000","last_fullsized":"2024-09-21T08:23:59.773690+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.229786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688319+0000","last_change":"2024-09-21T08:23:32.559542+0000","last_active":"2024-09-21T08:23:59.688319+0000","last_peered":"2024-09-21T08:23:59.688319+0000","last_clean":"2024-09-21T08:23:59.688319+0000","last_became_active":"2024-09-21T08:23:32.559194+0000","last_became_peered":"2024-09-21T08:23:32.559194+0000","last_unstale":"2024-09-21T08:23:59.688319+0000","last_undegraded":"2024-09-21T08:23:59.688319+0000","last_fullsized":"2024-09-21T08:23:59.688319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.251123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773287+0000","last_change":"2024-09-21T08:23:34.567872+0000","last_active":"2024-09-21T08:23:59.773287+0000","last_peered":"2024-09-21T08:23:59.773287+0000","last_clean":"2024-09-21T08:23:59.773287+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:23:59.773287+0000","last_undegraded":"2024-09-21T08:23:59.773287+0000","last_fullsized":"2024-09-21T08:23:59.773287+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.208597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004781+0000","last_change":"2024-09-21T08:23:36.582491+0000","last_active":"2024-09-21T08:24:00.004781+0000","last_peered":"2024-09-21T08:24:00.004781+0000","last_clean":"2024-09-21T08:24:00.004781+0000","last_became_active":"2024-09-21T08:23:36.581720+0000","last_became_peered":"2024-09-21T08:23:36.581720+0000","last_unstale":"2024-09-21T08:24:00.004781+0000","last_undegraded":"2024-09-21T08:24:00.004781+0000","last_fullsized":"2024-09-21T08:24:00.004781+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773331+0000","last_change":"2024-09-21T08:23:30.543477+0000","last_active":"2024-09-21T08:23:59.773331+0000","last_peered":"2024-09-21T08:23:59.773331+0000","last_clean":"2024-09-21T08:23:59.773331+0000","last_became_active":"2024-09-21T08:23:30.543025+0000","last_became_peered":"2024-09-21T08:23:30.543025+0000","last_unstale":"2024-09-21T08:23:59.773331+0000","last_undegraded":"2024-09-21T08:23:59.773331+0000","last_fullsized":"2024-09-21T08:23:59.773331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.290639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":37,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688932+0000","last_change":"2024-09-21T08:23:32.559984+0000","last_active":"2024-09-21T08:23:59.688932+0000","last_peered":"2024-09-21T08:23:59.688932+0000","last_clean":"2024-09-21T08:23:59.688932+0000","last_became_active":"2024-09-21T08:23:32.559544+0000","last_became_peered":"2024-09-21T08:23:32.559544+0000","last_unstale":"2024-09-21T08:23:59.688932+0000","last_undegraded":"2024-09-21T08:23:59.688932+0000","last_fullsized":"2024-09-21T08:23:59.688932+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:31.803588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784212+0000","last_change":"2024-09-21T08:23:34.572687+0000","last_active":"2024-09-21T08:23:59.784212+0000","last_peered":"2024-09-21T08:23:59.784212+0000","last_clean":"2024-09-21T08:23:59.784212+0000","last_became_active":"2024-09-21T08:23:34.572395+0000","last_became_peered":"2024-09-21T08:23:34.572395+0000","last_unstale":"2024-09-21T08:23:59.784212+0000","last_undegraded":"2024-09-21T08:23:59.784212+0000","last_fullsized":"2024-09-21T08:23:59.784212+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.042108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773118+0000","last_change":"2024-09-21T08:23:36.578510+0000","last_active":"2024-09-21T08:23:59.773118+0000","last_peered":"2024-09-21T08:23:59.773118+0000","last_clean":"2024-09-21T08:23:59.773118+0000","last_became_active":"2024-09-21T08:23:36.578227+0000","last_became_peered":"2024-09-21T08:23:36.578227+0000","last_unstale":"2024-09-21T08:23:59.773118+0000","last_undegraded":"2024-09-21T08:23:59.773118+0000","last_fullsized":"2024-09-21T08:23:59.773118+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.087695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688066+0000","last_change":"2024-09-21T08:23:36.573671+0000","last_active":"2024-09-21T08:23:59.688066+0000","last_peered":"2024-09-21T08:23:59.688066+0000","last_clean":"2024-09-21T08:23:59.688066+0000","last_became_active":"2024-09-21T08:23:36.573384+0000","last_became_peered":"2024-09-21T08:23:36.573384+0000","last_unstale":"2024-09-21T08:23:59.688066+0000","last_undegraded":"2024-09-21T08:23:59.688066+0000","last_fullsized":"2024-09-21T08:23:59.688066+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.273680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006256+0000","last_change":"2024-09-21T08:23:30.543356+0000","last_active":"2024-09-21T08:24:00.006256+0000","last_peered":"2024-09-21T08:24:00.006256+0000","last_clean":"2024-09-21T08:24:00.006256+0000","last_became_active":"2024-09-21T08:23:30.542776+0000","last_became_peered":"2024-09-21T08:23:30.542776+0000","last_unstale":"2024-09-21T08:24:00.006256+0000","last_undegraded":"2024-09-21T08:24:00.006256+0000","last_fullsized":"2024-09-21T08:24:00.006256+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.314435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781990+0000","last_change":"2024-09-21T08:23:32.556660+0000","last_active":"2024-09-21T08:23:59.781990+0000","last_peered":"2024-09-21T08:23:59.781990+0000","last_clean":"2024-09-21T08:23:59.781990+0000","last_became_active":"2024-09-21T08:23:32.556378+0000","last_became_peered":"2024-09-21T08:23:32.556378+0000","last_unstale":"2024-09-21T08:23:59.781990+0000","last_undegraded":"2024-09-21T08:23:59.781990+0000","last_fullsized":"2024-09-21T08:23:59.781990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.143529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.690111+0000","last_change":"2024-09-21T08:23:34.569008+0000","last_active":"2024-09-21T08:23:59.690111+0000","last_peered":"2024-09-21T08:23:59.690111+0000","last_clean":"2024-09-21T08:23:59.690111+0000","last_became_active":"2024-09-21T08:23:34.568562+0000","last_became_peered":"2024-09-21T08:23:34.568562+0000","last_unstale":"2024-09-21T08:23:59.690111+0000","last_undegraded":"2024-09-21T08:23:59.690111+0000","last_fullsized":"2024-09-21T08:23:59.690111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.511849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006388+0000","last_change":"2024-09-21T08:23:36.584447+0000","last_active":"2024-09-21T08:24:00.006388+0000","last_peered":"2024-09-21T08:24:00.006388+0000","last_clean":"2024-09-21T08:24:00.006388+0000","last_became_active":"2024-09-21T08:23:36.584154+0000","last_became_peered":"2024-09-21T08:23:36.584154+0000","last_unstale":"2024-09-21T08:24:00.006388+0000","last_undegraded":"2024-09-21T08:24:00.006388+0000","last_fullsized":"2024-09-21T08:24:00.006388+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.079729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688538+0000","last_change":"2024-09-21T08:23:30.547824+0000","last_active":"2024-09-21T08:23:59.688538+0000","last_peered":"2024-09-21T08:23:59.688538+0000","last_clean":"2024-09-21T08:23:59.688538+0000","last_became_active":"2024-09-21T08:23:30.546325+0000","last_became_peered":"2024-09-21T08:23:30.546325+0000","last_unstale":"2024-09-21T08:23:59.688538+0000","last_undegraded":"2024-09-21T08:23:59.688538+0000","last_fullsized":"2024-09-21T08:23:59.688538+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.394437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":50,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005634+0000","last_change":"2024-09-21T08:23:32.563650+0000","last_active":"2024-09-21T08:24:00.005634+0000","last_peered":"2024-09-21T08:24:00.005634+0000","last_clean":"2024-09-21T08:24:00.005634+0000","last_became_active":"2024-09-21T08:23:32.563383+0000","last_became_peered":"2024-09-21T08:23:32.563383+0000","last_unstale":"2024-09-21T08:24:00.005634+0000","last_undegraded":"2024-09-21T08:24:00.005634+0000","last_fullsized":"2024-09-21T08:24:00.005634+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.102032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688485+0000","last_change":"2024-09-21T08:23:34.571589+0000","last_active":"2024-09-21T08:23:59.688485+0000","last_peered":"2024-09-21T08:23:59.688485+0000","last_clean":"2024-09-21T08:23:59.688485+0000","last_became_active":"2024-09-21T08:23:34.571323+0000","last_became_peered":"2024-09-21T08:23:34.571323+0000","last_unstale":"2024-09-21T08:23:59.688485+0000","last_undegraded":"2024-09-21T08:23:59.688485+0000","last_fullsized":"2024-09-21T08:23:59.688485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.905411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784331+0000","last_change":"2024-09-21T08:23:36.584003+0000","last_active":"2024-09-21T08:23:59.784331+0000","last_peered":"2024-09-21T08:23:59.784331+0000","last_clean":"2024-09-21T08:23:59.784331+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:23:59.784331+0000","last_undegraded":"2024-09-21T08:23:59.784331+0000","last_fullsized":"2024-09-21T08:23:59.784331+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.633904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689693+0000","last_change":"2024-09-21T08:23:30.540306+0000","last_active":"2024-09-21T08:23:59.689693+0000","last_peered":"2024-09-21T08:23:59.689693+0000","last_clean":"2024-09-21T08:23:59.689693+0000","last_became_active":"2024-09-21T08:23:30.539931+0000","last_became_peered":"2024-09-21T08:23:30.539931+0000","last_unstale":"2024-09-21T08:23:59.689693+0000","last_undegraded":"2024-09-21T08:23:59.689693+0000","last_fullsized":"2024-09-21T08:23:59.689693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.106584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005002+0000","last_change":"2024-09-21T08:23:32.598036+0000","last_active":"2024-09-21T08:24:00.005002+0000","last_peered":"2024-09-21T08:24:00.005002+0000","last_clean":"2024-09-21T08:24:00.005002+0000","last_became_active":"2024-09-21T08:23:32.597548+0000","last_became_peered":"2024-09-21T08:23:32.597548+0000","last_unstale":"2024-09-21T08:24:00.005002+0000","last_undegraded":"2024-09-21T08:24:00.005002+0000","last_fullsized":"2024-09-21T08:24:00.005002+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.955005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598730+0000","last_change":"2024-09-21T08:23:34.567530+0000","last_active":"2024-09-21T08:24:05.598730+0000","last_peered":"2024-09-21T08:24:05.598730+0000","last_clean":"2024-09-21T08:24:05.598730+0000","last_became_active":"2024-09-21T08:23:34.567257+0000","last_became_peered":"2024-09-21T08:23:34.567257+0000","last_unstale":"2024-09-21T08:24:05.598730+0000","last_undegraded":"2024-09-21T08:24:05.598730+0000","last_fullsized":"2024-09-21T08:24:05.598730+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.508940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.690868+0000","last_change":"2024-09-21T08:23:32.562879+0000","last_active":"2024-09-21T08:23:59.690868+0000","last_peered":"2024-09-21T08:23:59.690868+0000","last_clean":"2024-09-21T08:23:59.690868+0000","last_became_active":"2024-09-21T08:23:32.562594+0000","last_became_peered":"2024-09-21T08:23:32.562594+0000","last_unstale":"2024-09-21T08:23:59.690868+0000","last_undegraded":"2024-09-21T08:23:59.690868+0000","last_fullsized":"2024-09-21T08:23:59.690868+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.780782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004638+0000","last_change":"2024-09-21T08:23:30.546900+0000","last_active":"2024-09-21T08:24:00.004638+0000","last_peered":"2024-09-21T08:24:00.004638+0000","last_clean":"2024-09-21T08:24:00.004638+0000","last_became_active":"2024-09-21T08:23:30.546625+0000","last_became_peered":"2024-09-21T08:23:30.546625+0000","last_unstale":"2024-09-21T08:24:00.004638+0000","last_undegraded":"2024-09-21T08:24:00.004638+0000","last_fullsized":"2024-09-21T08:24:00.004638+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:48.387469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598615+0000","last_change":"2024-09-21T08:23:34.568257+0000","last_active":"2024-09-21T08:24:05.598615+0000","last_peered":"2024-09-21T08:24:05.598615+0000","last_clean":"2024-09-21T08:24:05.598615+0000","last_became_active":"2024-09-21T08:23:34.567963+0000","last_became_peered":"2024-09-21T08:23:34.567963+0000","last_unstale":"2024-09-21T08:24:05.598615+0000","last_undegraded":"2024-09-21T08:24:05.598615+0000","last_fullsized":"2024-09-21T08:24:05.598615+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:43.930223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773366+0000","last_change":"2024-09-21T08:23:36.578919+0000","last_active":"2024-09-21T08:23:59.773366+0000","last_peered":"2024-09-21T08:23:59.773366+0000","last_clean":"2024-09-21T08:23:59.773366+0000","last_became_active":"2024-09-21T08:23:36.578648+0000","last_became_peered":"2024-09-21T08:23:36.578648+0000","last_unstale":"2024-09-21T08:23:59.773366+0000","last_undegraded":"2024-09-21T08:23:59.773366+0000","last_fullsized":"2024-09-21T08:23:59.773366+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.799668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004492+0000","last_change":"2024-09-21T08:23:32.597719+0000","last_active":"2024-09-21T08:24:00.004492+0000","last_peered":"2024-09-21T08:24:00.004492+0000","last_clean":"2024-09-21T08:24:00.004492+0000","last_became_active":"2024-09-21T08:23:32.597431+0000","last_became_peered":"2024-09-21T08:23:32.597431+0000","last_unstale":"2024-09-21T08:24:00.004492+0000","last_undegraded":"2024-09-21T08:24:00.004492+0000","last_fullsized":"2024-09-21T08:24:00.004492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.130776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006792+0000","last_change":"2024-09-21T08:23:30.546233+0000","last_active":"2024-09-21T08:24:00.006792+0000","last_peered":"2024-09-21T08:24:00.006792+0000","last_clean":"2024-09-21T08:24:00.006792+0000","last_became_active":"2024-09-21T08:23:30.545922+0000","last_became_peered":"2024-09-21T08:23:30.545922+0000","last_unstale":"2024-09-21T08:24:00.006792+0000","last_undegraded":"2024-09-21T08:24:00.006792+0000","last_fullsized":"2024-09-21T08:24:00.006792+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.861966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.687952+0000","last_change":"2024-09-21T08:23:34.570926+0000","last_active":"2024-09-21T08:23:59.687952+0000","last_peered":"2024-09-21T08:23:59.687952+0000","last_clean":"2024-09-21T08:23:59.687952+0000","last_became_active":"2024-09-21T08:23:34.570648+0000","last_became_peered":"2024-09-21T08:23:34.570648+0000","last_unstale":"2024-09-21T08:23:59.687952+0000","last_undegraded":"2024-09-21T08:23:59.687952+0000","last_fullsized":"2024-09-21T08:23:59.687952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.476055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688214+0000","last_change":"2024-09-21T08:23:36.582947+0000","last_active":"2024-09-21T08:23:59.688214+0000","last_peered":"2024-09-21T08:23:59.688214+0000","last_clean":"2024-09-21T08:23:59.688214+0000","last_became_active":"2024-09-21T08:23:36.582610+0000","last_became_peered":"2024-09-21T08:23:36.582610+0000","last_unstale":"2024-09-21T08:23:59.688214+0000","last_undegraded":"2024-09-21T08:23:59.688214+0000","last_fullsized":"2024-09-21T08:23:59.688214+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.387653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688032+0000","last_change":"2024-09-21T08:23:32.553440+0000","last_active":"2024-09-21T08:23:59.688032+0000","last_peered":"2024-09-21T08:23:59.688032+0000","last_clean":"2024-09-21T08:23:59.688032+0000","last_became_active":"2024-09-21T08:23:32.552980+0000","last_became_peered":"2024-09-21T08:23:32.552980+0000","last_unstale":"2024-09-21T08:23:59.688032+0000","last_undegraded":"2024-09-21T08:23:59.688032+0000","last_fullsized":"2024-09-21T08:23:59.688032+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.960840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005386+0000","last_change":"2024-09-21T08:23:30.543176+0000","last_active":"2024-09-21T08:24:00.005386+0000","last_peered":"2024-09-21T08:24:00.005386+0000","last_clean":"2024-09-21T08:24:00.005386+0000","last_became_active":"2024-09-21T08:23:30.542888+0000","last_became_peered":"2024-09-21T08:23:30.542888+0000","last_unstale":"2024-09-21T08:24:00.005386+0000","last_undegraded":"2024-09-21T08:24:00.005386+0000","last_fullsized":"2024-09-21T08:24:00.005386+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.545006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598254+0000","last_change":"2024-09-21T08:23:34.570603+0000","last_active":"2024-09-21T08:24:05.598254+0000","last_peered":"2024-09-21T08:24:05.598254+0000","last_clean":"2024-09-21T08:24:05.598254+0000","last_became_active":"2024-09-21T08:23:34.570321+0000","last_became_peered":"2024-09-21T08:23:34.570321+0000","last_unstale":"2024-09-21T08:24:05.598254+0000","last_undegraded":"2024-09-21T08:24:05.598254+0000","last_fullsized":"2024-09-21T08:24:05.598254+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.651545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784289+0000","last_change":"2024-09-21T08:23:36.582496+0000","last_active":"2024-09-21T08:23:59.784289+0000","last_peered":"2024-09-21T08:23:59.784289+0000","last_clean":"2024-09-21T08:23:59.784289+0000","last_became_active":"2024-09-21T08:23:36.582039+0000","last_became_peered":"2024-09-21T08:23:36.582039+0000","last_unstale":"2024-09-21T08:23:59.784289+0000","last_undegraded":"2024-09-21T08:23:59.784289+0000","last_fullsized":"2024-09-21T08:23:59.784289+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.819778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689880+0000","last_change":"2024-09-21T08:23:32.557661+0000","last_active":"2024-09-21T08:23:59.689880+0000","last_peered":"2024-09-21T08:23:59.689880+0000","last_clean":"2024-09-21T08:23:59.689880+0000","last_became_active":"2024-09-21T08:23:32.555080+0000","last_became_peered":"2024-09-21T08:23:32.555080+0000","last_unstale":"2024-09-21T08:23:59.689880+0000","last_undegraded":"2024-09-21T08:23:59.689880+0000","last_fullsized":"2024-09-21T08:23:59.689880+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.633944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":37,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773788+0000","last_change":"2024-09-21T08:23:30.546228+0000","last_active":"2024-09-21T08:23:59.773788+0000","last_peered":"2024-09-21T08:23:59.773788+0000","last_clean":"2024-09-21T08:23:59.773788+0000","last_became_active":"2024-09-21T08:23:30.545920+0000","last_became_peered":"2024-09-21T08:23:30.545920+0000","last_unstale":"2024-09-21T08:23:59.773788+0000","last_undegraded":"2024-09-21T08:23:59.773788+0000","last_fullsized":"2024-09-21T08:23:59.773788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.464911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598279+0000","last_change":"2024-09-21T08:23:34.568126+0000","last_active":"2024-09-21T08:24:05.598279+0000","last_peered":"2024-09-21T08:24:05.598279+0000","last_clean":"2024-09-21T08:24:05.598279+0000","last_became_active":"2024-09-21T08:23:34.563906+0000","last_became_peered":"2024-09-21T08:23:34.563906+0000","last_unstale":"2024-09-21T08:24:05.598279+0000","last_undegraded":"2024-09-21T08:24:05.598279+0000","last_fullsized":"2024-09-21T08:24:05.598279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.761252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004937+0000","last_change":"2024-09-21T08:23:36.588118+0000","last_active":"2024-09-21T08:24:00.004937+0000","last_peered":"2024-09-21T08:24:00.004937+0000","last_clean":"2024-09-21T08:24:00.004937+0000","last_became_active":"2024-09-21T08:23:36.587782+0000","last_became_peered":"2024-09-21T08:23:36.587782+0000","last_unstale":"2024-09-21T08:24:00.004937+0000","last_undegraded":"2024-09-21T08:24:00.004937+0000","last_fullsized":"2024-09-21T08:24:00.004937+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.580040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005953+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:24:00.005953+0000","last_peered":"2024-09-21T08:24:00.005953+0000","last_clean":"2024-09-21T08:24:00.005953+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:24:00.005953+0000","last_undegraded":"2024-09-21T08:24:00.005953+0000","last_fullsized":"2024-09-21T08:24:00.005953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.235370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.690997+0000","last_change":"2024-09-21T08:23:30.547730+0000","last_active":"2024-09-21T08:23:59.690997+0000","last_peered":"2024-09-21T08:23:59.690997+0000","last_clean":"2024-09-21T08:23:59.690997+0000","last_became_active":"2024-09-21T08:23:30.547290+0000","last_became_peered":"2024-09-21T08:23:30.547290+0000","last_unstale":"2024-09-21T08:23:59.690997+0000","last_undegraded":"2024-09-21T08:23:59.690997+0000","last_fullsized":"2024-09-21T08:23:59.690997+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.347687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686264+0000","last_change":"2024-09-21T08:23:34.564467+0000","last_active":"2024-09-21T08:23:59.686264+0000","last_peered":"2024-09-21T08:23:59.686264+0000","last_clean":"2024-09-21T08:23:59.686264+0000","last_became_active":"2024-09-21T08:23:34.564157+0000","last_became_peered":"2024-09-21T08:23:34.564157+0000","last_unstale":"2024-09-21T08:23:59.686264+0000","last_undegraded":"2024-09-21T08:23:59.686264+0000","last_fullsized":"2024-09-21T08:23:59.686264+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.544611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689477+0000","last_change":"2024-09-21T08:23:36.582071+0000","last_active":"2024-09-21T08:23:59.689477+0000","last_peered":"2024-09-21T08:23:59.689477+0000","last_clean":"2024-09-21T08:23:59.689477+0000","last_became_active":"2024-09-21T08:23:36.581757+0000","last_became_peered":"2024-09-21T08:23:36.581757+0000","last_unstale":"2024-09-21T08:23:59.689477+0000","last_undegraded":"2024-09-21T08:23:59.689477+0000","last_fullsized":"2024-09-21T08:23:59.689477+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.873559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":84,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598266+0000","last_change":"2024-09-21T08:23:32.562868+0000","last_active":"2024-09-21T08:24:05.598266+0000","last_peered":"2024-09-21T08:24:05.598266+0000","last_clean":"2024-09-21T08:24:05.598266+0000","last_became_active":"2024-09-21T08:23:32.561965+0000","last_became_peered":"2024-09-21T08:23:32.561965+0000","last_unstale":"2024-09-21T08:24:05.598266+0000","last_undegraded":"2024-09-21T08:24:05.598266+0000","last_fullsized":"2024-09-21T08:24:05.598266+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.844304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689252+0000","last_change":"2024-09-21T08:23:30.541459+0000","last_active":"2024-09-21T08:23:59.689252+0000","last_peered":"2024-09-21T08:23:59.689252+0000","last_clean":"2024-09-21T08:23:59.689252+0000","last_became_active":"2024-09-21T08:23:30.541175+0000","last_became_peered":"2024-09-21T08:23:30.541175+0000","last_unstale":"2024-09-21T08:23:59.689252+0000","last_undegraded":"2024-09-21T08:23:59.689252+0000","last_fullsized":"2024-09-21T08:23:59.689252+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.508394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":41,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773320+0000","last_change":"2024-09-21T08:23:32.546026+0000","last_active":"2024-09-21T08:23:59.773320+0000","last_peered":"2024-09-21T08:23:59.773320+0000","last_clean":"2024-09-21T08:23:59.773320+0000","last_became_active":"2024-09-21T08:23:29.531243+0000","last_became_peered":"2024-09-21T08:23:29.531243+0000","last_unstale":"2024-09-21T08:23:59.773320+0000","last_undegraded":"2024-09-21T08:23:59.773320+0000","last_fullsized":"2024-09-21T08:23:59.773320+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.908720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005536+0000","last_change":"2024-09-21T08:23:34.568259+0000","last_active":"2024-09-21T08:24:00.005536+0000","last_peered":"2024-09-21T08:24:00.005536+0000","last_clean":"2024-09-21T08:24:00.005536+0000","last_became_active":"2024-09-21T08:23:34.567955+0000","last_became_peered":"2024-09-21T08:23:34.567955+0000","last_unstale":"2024-09-21T08:24:00.005536+0000","last_undegraded":"2024-09-21T08:24:00.005536+0000","last_fullsized":"2024-09-21T08:24:00.005536+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.600803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005935+0000","last_change":"2024-09-21T08:23:36.577895+0000","last_active":"2024-09-21T08:24:00.005935+0000","last_peered":"2024-09-21T08:24:00.005935+0000","last_clean":"2024-09-21T08:24:00.005935+0000","last_became_active":"2024-09-21T08:23:36.577593+0000","last_became_peered":"2024-09-21T08:23:36.577593+0000","last_unstale":"2024-09-21T08:24:00.005935+0000","last_undegraded":"2024-09-21T08:24:00.005935+0000","last_fullsized":"2024-09-21T08:24:00.005935+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.663315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.691001+0000","last_change":"2024-09-21T08:23:32.560593+0000","last_active":"2024-09-21T08:23:59.691001+0000","last_peered":"2024-09-21T08:23:59.691001+0000","last_clean":"2024-09-21T08:23:59.691001+0000","last_became_active":"2024-09-21T08:23:32.560323+0000","last_became_peered":"2024-09-21T08:23:32.560323+0000","last_unstale":"2024-09-21T08:23:59.691001+0000","last_undegraded":"2024-09-21T08:23:59.691001+0000","last_fullsized":"2024-09-21T08:23:59.691001+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.516260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689756+0000","last_change":"2024-09-21T08:23:30.548894+0000","last_active":"2024-09-21T08:23:59.689756+0000","last_peered":"2024-09-21T08:23:59.689756+0000","last_clean":"2024-09-21T08:23:59.689756+0000","last_became_active":"2024-09-21T08:23:30.548308+0000","last_became_peered":"2024-09-21T08:23:30.548308+0000","last_unstale":"2024-09-21T08:23:59.689756+0000","last_undegraded":"2024-09-21T08:23:59.689756+0000","last_fullsized":"2024-09-21T08:23:59.689756+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.379264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781354+0000","last_change":"2024-09-21T08:23:32.552207+0000","last_active":"2024-09-21T08:23:59.781354+0000","last_peered":"2024-09-21T08:23:59.781354+0000","last_clean":"2024-09-21T08:23:59.781354+0000","last_became_active":"2024-09-21T08:23:29.533672+0000","last_became_peered":"2024-09-21T08:23:29.533672+0000","last_unstale":"2024-09-21T08:23:59.781354+0000","last_undegraded":"2024-09-21T08:23:59.781354+0000","last_fullsized":"2024-09-21T08:23:59.781354+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.009447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781692+0000","last_change":"2024-09-21T08:23:34.567541+0000","last_active":"2024-09-21T08:23:59.781692+0000","last_peered":"2024-09-21T08:23:59.781692+0000","last_clean":"2024-09-21T08:23:59.781692+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:23:59.781692+0000","last_undegraded":"2024-09-21T08:23:59.781692+0000","last_fullsized":"2024-09-21T08:23:59.781692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:51.623026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784355+0000","last_change":"2024-09-21T08:23:36.585190+0000","last_active":"2024-09-21T08:23:59.784355+0000","last_peered":"2024-09-21T08:23:59.784355+0000","last_clean":"2024-09-21T08:23:59.784355+0000","last_became_active":"2024-09-21T08:23:36.584905+0000","last_became_peered":"2024-09-21T08:23:36.584905+0000","last_unstale":"2024-09-21T08:23:59.784355+0000","last_undegraded":"2024-09-21T08:23:59.784355+0000","last_fullsized":"2024-09-21T08:23:59.784355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.290201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":45,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005132+0000","last_change":"2024-09-21T08:23:32.561943+0000","last_active":"2024-09-21T08:24:00.005132+0000","last_peered":"2024-09-21T08:24:00.005132+0000","last_clean":"2024-09-21T08:24:00.005132+0000","last_became_active":"2024-09-21T08:23:32.561617+0000","last_became_peered":"2024-09-21T08:23:32.561617+0000","last_unstale":"2024-09-21T08:24:00.005132+0000","last_undegraded":"2024-09-21T08:24:00.005132+0000","last_fullsized":"2024-09-21T08:24:00.005132+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.493572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005497+0000","last_change":"2024-09-21T08:23:30.547156+0000","last_active":"2024-09-21T08:24:00.005497+0000","last_peered":"2024-09-21T08:24:00.005497+0000","last_clean":"2024-09-21T08:24:00.005497+0000","last_became_active":"2024-09-21T08:23:30.546898+0000","last_became_peered":"2024-09-21T08:23:30.546898+0000","last_unstale":"2024-09-21T08:24:00.005497+0000","last_undegraded":"2024-09-21T08:24:00.005497+0000","last_fullsized":"2024-09-21T08:24:00.005497+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.718486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":77,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:09.549457+0000","last_change":"2024-09-21T08:23:32.597264+0000","last_active":"2024-09-21T08:24:09.549457+0000","last_peered":"2024-09-21T08:24:09.549457+0000","last_clean":"2024-09-21T08:24:09.549457+0000","last_became_active":"2024-09-21T08:23:29.532520+0000","last_became_peered":"2024-09-21T08:23:29.532520+0000","last_unstale":"2024-09-21T08:24:09.549457+0000","last_undegraded":"2024-09-21T08:24:09.549457+0000","last_fullsized":"2024-09-21T08:24:09.549457+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.455707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773786+0000","last_change":"2024-09-21T08:23:34.570449+0000","last_active":"2024-09-21T08:23:59.773786+0000","last_peered":"2024-09-21T08:23:59.773786+0000","last_clean":"2024-09-21T08:23:59.773786+0000","last_became_active":"2024-09-21T08:23:34.569970+0000","last_became_peered":"2024-09-21T08:23:34.569970+0000","last_unstale":"2024-09-21T08:23:59.773786+0000","last_undegraded":"2024-09-21T08:23:59.773786+0000","last_fullsized":"2024-09-21T08:23:59.773786+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.151707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.690541+0000","last_change":"2024-09-21T08:23:36.585195+0000","last_active":"2024-09-21T08:23:59.690541+0000","last_peered":"2024-09-21T08:23:59.690541+0000","last_clean":"2024-09-21T08:23:59.690541+0000","last_became_active":"2024-09-21T08:23:36.584827+0000","last_became_peered":"2024-09-21T08:23:36.584827+0000","last_unstale":"2024-09-21T08:23:59.690541+0000","last_undegraded":"2024-09-21T08:23:59.690541+0000","last_fullsized":"2024-09-21T08:23:59.690541+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.376554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598687+0000","last_change":"2024-09-21T08:23:32.598546+0000","last_active":"2024-09-21T08:24:05.598687+0000","last_peered":"2024-09-21T08:24:05.598687+0000","last_clean":"2024-09-21T08:24:05.598687+0000","last_became_active":"2024-09-21T08:23:32.596949+0000","last_became_peered":"2024-09-21T08:23:32.596949+0000","last_unstale":"2024-09-21T08:24:05.598687+0000","last_undegraded":"2024-09-21T08:24:05.598687+0000","last_fullsized":"2024-09-21T08:24:05.598687+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.826771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005064+0000","last_change":"2024-09-21T08:23:30.546494+0000","last_active":"2024-09-21T08:24:00.005064+0000","last_peered":"2024-09-21T08:24:00.005064+0000","last_clean":"2024-09-21T08:24:00.005064+0000","last_became_active":"2024-09-21T08:23:30.546218+0000","last_became_peered":"2024-09-21T08:23:30.546218+0000","last_unstale":"2024-09-21T08:24:00.005064+0000","last_undegraded":"2024-09-21T08:24:00.005064+0000","last_fullsized":"2024-09-21T08:24:00.005064+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.498978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":286,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.765884+0000","last_change":"2024-09-21T08:24:05.765943+0000","last_active":"2024-09-21T08:24:05.765884+0000","last_peered":"2024-09-21T08:24:05.765884+0000","last_clean":"2024-09-21T08:24:05.765884+0000","last_became_active":"2024-09-21T08:23:23.509818+0000","last_became_peered":"2024-09-21T08:23:23.509818+0000","last_unstale":"2024-09-21T08:24:05.765884+0000","last_undegraded":"2024-09-21T08:24:05.765884+0000","last_fullsized":"2024-09-21T08:24:05.765884+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:24:05.765804+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:05.827066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784511+0000","last_change":"2024-09-21T08:23:34.568359+0000","last_active":"2024-09-21T08:23:59.784511+0000","last_peered":"2024-09-21T08:23:59.784511+0000","last_clean":"2024-09-21T08:23:59.784511+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:23:59.784511+0000","last_undegraded":"2024-09-21T08:23:59.784511+0000","last_fullsized":"2024-09-21T08:23:59.784511+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.246877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773654+0000","last_change":"2024-09-21T08:23:36.584454+0000","last_active":"2024-09-21T08:23:59.773654+0000","last_peered":"2024-09-21T08:23:59.773654+0000","last_clean":"2024-09-21T08:23:59.773654+0000","last_became_active":"2024-09-21T08:23:36.584095+0000","last_became_peered":"2024-09-21T08:23:36.584095+0000","last_unstale":"2024-09-21T08:23:59.773654+0000","last_undegraded":"2024-09-21T08:23:59.773654+0000","last_fullsized":"2024-09-21T08:23:59.773654+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.123137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688254+0000","last_change":"2024-09-21T08:23:32.557075+0000","last_active":"2024-09-21T08:23:59.688254+0000","last_peered":"2024-09-21T08:23:59.688254+0000","last_clean":"2024-09-21T08:23:59.688254+0000","last_became_active":"2024-09-21T08:23:32.556696+0000","last_became_peered":"2024-09-21T08:23:32.556696+0000","last_unstale":"2024-09-21T08:23:59.688254+0000","last_undegraded":"2024-09-21T08:23:59.688254+0000","last_fullsized":"2024-09-21T08:23:59.688254+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.647238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688373+0000","last_change":"2024-09-21T08:23:30.547989+0000","last_active":"2024-09-21T08:23:59.688373+0000","last_peered":"2024-09-21T08:23:59.688373+0000","last_clean":"2024-09-21T08:23:59.688373+0000","last_became_active":"2024-09-21T08:23:30.547176+0000","last_became_peered":"2024-09-21T08:23:30.547176+0000","last_unstale":"2024-09-21T08:23:59.688373+0000","last_undegraded":"2024-09-21T08:23:59.688373+0000","last_fullsized":"2024-09-21T08:23:59.688373+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.295162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773615+0000","last_change":"2024-09-21T08:23:34.570250+0000","last_active":"2024-09-21T08:23:59.773615+0000","last_peered":"2024-09-21T08:23:59.773615+0000","last_clean":"2024-09-21T08:23:59.773615+0000","last_became_active":"2024-09-21T08:23:34.569571+0000","last_became_peered":"2024-09-21T08:23:34.569571+0000","last_unstale":"2024-09-21T08:23:59.773615+0000","last_undegraded":"2024-09-21T08:23:59.773615+0000","last_fullsized":"2024-09-21T08:23:59.773615+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.053187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006390+0000","last_change":"2024-09-21T08:23:36.587842+0000","last_active":"2024-09-21T08:24:00.006390+0000","last_peered":"2024-09-21T08:24:00.006390+0000","last_clean":"2024-09-21T08:24:00.006390+0000","last_became_active":"2024-09-21T08:23:36.587561+0000","last_became_peered":"2024-09-21T08:23:36.587561+0000","last_unstale":"2024-09-21T08:24:00.006390+0000","last_undegraded":"2024-09-21T08:24:00.006390+0000","last_fullsized":"2024-09-21T08:24:00.006390+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.999379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688962+0000","last_change":"2024-09-21T08:23:32.556741+0000","last_active":"2024-09-21T08:23:59.688962+0000","last_peered":"2024-09-21T08:23:59.688962+0000","last_clean":"2024-09-21T08:23:59.688962+0000","last_became_active":"2024-09-21T08:23:32.556355+0000","last_became_peered":"2024-09-21T08:23:32.556355+0000","last_unstale":"2024-09-21T08:23:59.688962+0000","last_undegraded":"2024-09-21T08:23:59.688962+0000","last_fullsized":"2024-09-21T08:23:59.688962+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686539+0000","last_change":"2024-09-21T08:23:30.537030+0000","last_active":"2024-09-21T08:23:59.686539+0000","last_peered":"2024-09-21T08:23:59.686539+0000","last_clean":"2024-09-21T08:23:59.686539+0000","last_became_active":"2024-09-21T08:23:30.536566+0000","last_became_peered":"2024-09-21T08:23:30.536566+0000","last_unstale":"2024-09-21T08:23:59.686539+0000","last_undegraded":"2024-09-21T08:23:59.686539+0000","last_fullsized":"2024-09-21T08:23:59.686539+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.146930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.690635+0000","last_change":"2024-09-21T08:23:34.569384+0000","last_active":"2024-09-21T08:23:59.690635+0000","last_peered":"2024-09-21T08:23:59.690635+0000","last_clean":"2024-09-21T08:23:59.690635+0000","last_became_active":"2024-09-21T08:23:34.568900+0000","last_became_peered":"2024-09-21T08:23:34.568900+0000","last_unstale":"2024-09-21T08:23:59.690635+0000","last_undegraded":"2024-09-21T08:23:59.690635+0000","last_fullsized":"2024-09-21T08:23:59.690635+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.163373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781851+0000","last_change":"2024-09-21T08:23:36.586002+0000","last_active":"2024-09-21T08:23:59.781851+0000","last_peered":"2024-09-21T08:23:59.781851+0000","last_clean":"2024-09-21T08:23:59.781851+0000","last_became_active":"2024-09-21T08:23:36.585695+0000","last_became_peered":"2024-09-21T08:23:36.585695+0000","last_unstale":"2024-09-21T08:23:59.781851+0000","last_undegraded":"2024-09-21T08:23:59.781851+0000","last_fullsized":"2024-09-21T08:23:59.781851+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.916781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688568+0000","last_change":"2024-09-21T08:23:32.555172+0000","last_active":"2024-09-21T08:23:59.688568+0000","last_peered":"2024-09-21T08:23:59.688568+0000","last_clean":"2024-09-21T08:23:59.688568+0000","last_became_active":"2024-09-21T08:23:32.554845+0000","last_became_peered":"2024-09-21T08:23:32.554845+0000","last_unstale":"2024-09-21T08:23:59.688568+0000","last_undegraded":"2024-09-21T08:23:59.688568+0000","last_fullsized":"2024-09-21T08:23:59.688568+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.339439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005423+0000","last_change":"2024-09-21T08:23:30.545460+0000","last_active":"2024-09-21T08:24:00.005423+0000","last_peered":"2024-09-21T08:24:00.005423+0000","last_clean":"2024-09-21T08:24:00.005423+0000","last_became_active":"2024-09-21T08:23:30.545187+0000","last_became_peered":"2024-09-21T08:23:30.545187+0000","last_unstale":"2024-09-21T08:24:00.005423+0000","last_undegraded":"2024-09-21T08:24:00.005423+0000","last_fullsized":"2024-09-21T08:24:00.005423+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.674857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598259+0000","last_change":"2024-09-21T08:23:34.571530+0000","last_active":"2024-09-21T08:24:05.598259+0000","last_peered":"2024-09-21T08:24:05.598259+0000","last_clean":"2024-09-21T08:24:05.598259+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:24:05.598259+0000","last_undegraded":"2024-09-21T08:24:05.598259+0000","last_fullsized":"2024-09-21T08:24:05.598259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.789578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.687971+0000","last_change":"2024-09-21T08:23:36.582871+0000","last_active":"2024-09-21T08:23:59.687971+0000","last_peered":"2024-09-21T08:23:59.687971+0000","last_clean":"2024-09-21T08:23:59.687971+0000","last_became_active":"2024-09-21T08:23:36.582575+0000","last_became_peered":"2024-09-21T08:23:36.582575+0000","last_unstale":"2024-09-21T08:23:59.687971+0000","last_undegraded":"2024-09-21T08:23:59.687971+0000","last_fullsized":"2024-09-21T08:23:59.687971+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.578837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005715+0000","last_change":"2024-09-21T08:23:32.560978+0000","last_active":"2024-09-21T08:24:00.005715+0000","last_peered":"2024-09-21T08:24:00.005715+0000","last_clean":"2024-09-21T08:24:00.005715+0000","last_became_active":"2024-09-21T08:23:32.560658+0000","last_became_peered":"2024-09-21T08:23:32.560658+0000","last_unstale":"2024-09-21T08:24:00.005715+0000","last_undegraded":"2024-09-21T08:24:00.005715+0000","last_fullsized":"2024-09-21T08:24:00.005715+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.965888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773438+0000","last_change":"2024-09-21T08:23:30.542309+0000","last_active":"2024-09-21T08:23:59.773438+0000","last_peered":"2024-09-21T08:23:59.773438+0000","last_clean":"2024-09-21T08:23:59.773438+0000","last_became_active":"2024-09-21T08:23:30.541992+0000","last_became_peered":"2024-09-21T08:23:30.541992+0000","last_unstale":"2024-09-21T08:23:59.773438+0000","last_undegraded":"2024-09-21T08:23:59.773438+0000","last_fullsized":"2024-09-21T08:23:59.773438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.859320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781309+0000","last_change":"2024-09-21T08:23:34.566539+0000","last_active":"2024-09-21T08:23:59.781309+0000","last_peered":"2024-09-21T08:23:59.781309+0000","last_clean":"2024-09-21T08:23:59.781309+0000","last_became_active":"2024-09-21T08:23:34.566242+0000","last_became_peered":"2024-09-21T08:23:34.566242+0000","last_unstale":"2024-09-21T08:23:59.781309+0000","last_undegraded":"2024-09-21T08:23:59.781309+0000","last_fullsized":"2024-09-21T08:23:59.781309+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.194173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005400+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:24:00.005400+0000","last_peered":"2024-09-21T08:24:00.005400+0000","last_clean":"2024-09-21T08:24:00.005400+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:24:00.005400+0000","last_undegraded":"2024-09-21T08:24:00.005400+0000","last_fullsized":"2024-09-21T08:24:00.005400+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.339671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686332+0000","last_change":"2024-09-21T08:23:32.597880+0000","last_active":"2024-09-21T08:23:59.686332+0000","last_peered":"2024-09-21T08:23:59.686332+0000","last_clean":"2024-09-21T08:23:59.686332+0000","last_became_active":"2024-09-21T08:23:32.597589+0000","last_became_peered":"2024-09-21T08:23:32.597589+0000","last_unstale":"2024-09-21T08:23:59.686332+0000","last_undegraded":"2024-09-21T08:23:59.686332+0000","last_fullsized":"2024-09-21T08:23:59.686332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.870820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784798+0000","last_change":"2024-09-21T08:23:30.545405+0000","last_active":"2024-09-21T08:23:59.784798+0000","last_peered":"2024-09-21T08:23:59.784798+0000","last_clean":"2024-09-21T08:23:59.784798+0000","last_became_active":"2024-09-21T08:23:30.545068+0000","last_became_peered":"2024-09-21T08:23:30.545068+0000","last_unstale":"2024-09-21T08:23:59.784798+0000","last_undegraded":"2024-09-21T08:23:59.784798+0000","last_fullsized":"2024-09-21T08:23:59.784798+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.179419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781437+0000","last_change":"2024-09-21T08:23:34.564923+0000","last_active":"2024-09-21T08:23:59.781437+0000","last_peered":"2024-09-21T08:23:59.781437+0000","last_clean":"2024-09-21T08:23:59.781437+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:23:59.781437+0000","last_undegraded":"2024-09-21T08:23:59.781437+0000","last_fullsized":"2024-09-21T08:23:59.781437+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.183697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784514+0000","last_change":"2024-09-21T08:23:36.582099+0000","last_active":"2024-09-21T08:23:59.784514+0000","last_peered":"2024-09-21T08:23:59.784514+0000","last_clean":"2024-09-21T08:23:59.784514+0000","last_became_active":"2024-09-21T08:23:36.581796+0000","last_became_peered":"2024-09-21T08:23:36.581796+0000","last_unstale":"2024-09-21T08:23:59.784514+0000","last_undegraded":"2024-09-21T08:23:59.784514+0000","last_fullsized":"2024-09-21T08:23:59.784514+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.618273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":50,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688187+0000","last_change":"2024-09-21T08:23:32.557284+0000","last_active":"2024-09-21T08:23:59.688187+0000","last_peered":"2024-09-21T08:23:59.688187+0000","last_clean":"2024-09-21T08:23:59.688187+0000","last_became_active":"2024-09-21T08:23:32.556876+0000","last_became_peered":"2024-09-21T08:23:32.556876+0000","last_unstale":"2024-09-21T08:23:59.688187+0000","last_undegraded":"2024-09-21T08:23:59.688187+0000","last_fullsized":"2024-09-21T08:23:59.688187+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.982500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784789+0000","last_change":"2024-09-21T08:23:30.548671+0000","last_active":"2024-09-21T08:23:59.784789+0000","last_peered":"2024-09-21T08:23:59.784789+0000","last_clean":"2024-09-21T08:23:59.784789+0000","last_became_active":"2024-09-21T08:23:30.548350+0000","last_became_peered":"2024-09-21T08:23:30.548350+0000","last_unstale":"2024-09-21T08:23:59.784789+0000","last_undegraded":"2024-09-21T08:23:59.784789+0000","last_fullsized":"2024-09-21T08:23:59.784789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.489110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781151+0000","last_change":"2024-09-21T08:23:34.566472+0000","last_active":"2024-09-21T08:23:59.781151+0000","last_peered":"2024-09-21T08:23:59.781151+0000","last_clean":"2024-09-21T08:23:59.781151+0000","last_became_active":"2024-09-21T08:23:34.566193+0000","last_became_peered":"2024-09-21T08:23:34.566193+0000","last_unstale":"2024-09-21T08:23:59.781151+0000","last_undegraded":"2024-09-21T08:23:59.781151+0000","last_fullsized":"2024-09-21T08:23:59.781151+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:58.818551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005118+0000","last_change":"2024-09-21T08:23:36.579879+0000","last_active":"2024-09-21T08:24:00.005118+0000","last_peered":"2024-09-21T08:24:00.005118+0000","last_clean":"2024-09-21T08:24:00.005118+0000","last_became_active":"2024-09-21T08:23:36.579603+0000","last_became_peered":"2024-09-21T08:23:36.579603+0000","last_unstale":"2024-09-21T08:24:00.005118+0000","last_undegraded":"2024-09-21T08:24:00.005118+0000","last_fullsized":"2024-09-21T08:24:00.005118+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.717936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773021+0000","last_change":"2024-09-21T08:23:32.598217+0000","last_active":"2024-09-21T08:23:59.773021+0000","last_peered":"2024-09-21T08:23:59.773021+0000","last_clean":"2024-09-21T08:23:59.773021+0000","last_became_active":"2024-09-21T08:23:32.597543+0000","last_became_peered":"2024-09-21T08:23:32.597543+0000","last_unstale":"2024-09-21T08:23:59.773021+0000","last_undegraded":"2024-09-21T08:23:59.773021+0000","last_fullsized":"2024-09-21T08:23:59.773021+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.907666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784890+0000","last_change":"2024-09-21T08:23:30.551520+0000","last_active":"2024-09-21T08:23:59.784890+0000","last_peered":"2024-09-21T08:23:59.784890+0000","last_clean":"2024-09-21T08:23:59.784890+0000","last_became_active":"2024-09-21T08:23:30.551203+0000","last_became_peered":"2024-09-21T08:23:30.551203+0000","last_unstale":"2024-09-21T08:23:59.784890+0000","last_undegraded":"2024-09-21T08:23:59.784890+0000","last_fullsized":"2024-09-21T08:23:59.784890+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:29.708394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598668+0000","last_change":"2024-09-21T08:23:34.569818+0000","last_active":"2024-09-21T08:24:05.598668+0000","last_peered":"2024-09-21T08:24:05.598668+0000","last_clean":"2024-09-21T08:24:05.598668+0000","last_became_active":"2024-09-21T08:23:34.569490+0000","last_became_peered":"2024-09-21T08:23:34.569490+0000","last_unstale":"2024-09-21T08:24:05.598668+0000","last_undegraded":"2024-09-21T08:24:05.598668+0000","last_fullsized":"2024-09-21T08:24:05.598668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.884010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773123+0000","last_change":"2024-09-21T08:23:36.579253+0000","last_active":"2024-09-21T08:23:59.773123+0000","last_peered":"2024-09-21T08:23:59.773123+0000","last_clean":"2024-09-21T08:23:59.773123+0000","last_became_active":"2024-09-21T08:23:36.578952+0000","last_became_peered":"2024-09-21T08:23:36.578952+0000","last_unstale":"2024-09-21T08:23:59.773123+0000","last_undegraded":"2024-09-21T08:23:59.773123+0000","last_fullsized":"2024-09-21T08:23:59.773123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.185192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686641+0000","last_change":"2024-09-21T08:23:30.542360+0000","last_active":"2024-09-21T08:23:59.686641+0000","last_peered":"2024-09-21T08:23:59.686641+0000","last_clean":"2024-09-21T08:23:59.686641+0000","last_became_active":"2024-09-21T08:23:30.542060+0000","last_became_peered":"2024-09-21T08:23:30.542060+0000","last_unstale":"2024-09-21T08:23:59.686641+0000","last_undegraded":"2024-09-21T08:23:59.686641+0000","last_fullsized":"2024-09-21T08:23:59.686641+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:58.968344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006630+0000","last_change":"2024-09-21T08:23:32.598912+0000","last_active":"2024-09-21T08:24:00.006630+0000","last_peered":"2024-09-21T08:24:00.006630+0000","last_clean":"2024-09-21T08:24:00.006630+0000","last_became_active":"2024-09-21T08:23:32.598518+0000","last_became_peered":"2024-09-21T08:23:32.598518+0000","last_unstale":"2024-09-21T08:24:00.006630+0000","last_undegraded":"2024-09-21T08:24:00.006630+0000","last_fullsized":"2024-09-21T08:24:00.006630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:32.806988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773598+0000","last_change":"2024-09-21T08:23:34.566165+0000","last_active":"2024-09-21T08:23:59.773598+0000","last_peered":"2024-09-21T08:23:59.773598+0000","last_clean":"2024-09-21T08:23:59.773598+0000","last_became_active":"2024-09-21T08:23:34.565662+0000","last_became_peered":"2024-09-21T08:23:34.565662+0000","last_unstale":"2024-09-21T08:23:59.773598+0000","last_undegraded":"2024-09-21T08:23:59.773598+0000","last_fullsized":"2024-09-21T08:23:59.773598+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.290518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784363+0000","last_change":"2024-09-21T08:23:36.579682+0000","last_active":"2024-09-21T08:23:59.784363+0000","last_peered":"2024-09-21T08:23:59.784363+0000","last_clean":"2024-09-21T08:23:59.784363+0000","last_became_active":"2024-09-21T08:23:36.579384+0000","last_became_peered":"2024-09-21T08:23:36.579384+0000","last_unstale":"2024-09-21T08:23:59.784363+0000","last_undegraded":"2024-09-21T08:23:59.784363+0000","last_fullsized":"2024-09-21T08:23:59.784363+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.156840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005265+0000","last_change":"2024-09-21T08:23:32.599079+0000","last_active":"2024-09-21T08:24:00.005265+0000","last_peered":"2024-09-21T08:24:00.005265+0000","last_clean":"2024-09-21T08:24:00.005265+0000","last_became_active":"2024-09-21T08:23:32.598778+0000","last_became_peered":"2024-09-21T08:23:32.598778+0000","last_unstale":"2024-09-21T08:24:00.005265+0000","last_undegraded":"2024-09-21T08:24:00.005265+0000","last_fullsized":"2024-09-21T08:24:00.005265+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.131011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784449+0000","last_change":"2024-09-21T08:23:30.548443+0000","last_active":"2024-09-21T08:23:59.784449+0000","last_peered":"2024-09-21T08:23:59.784449+0000","last_clean":"2024-09-21T08:23:59.784449+0000","last_became_active":"2024-09-21T08:23:30.548096+0000","last_became_peered":"2024-09-21T08:23:30.548096+0000","last_unstale":"2024-09-21T08:23:59.784449+0000","last_undegraded":"2024-09-21T08:23:59.784449+0000","last_fullsized":"2024-09-21T08:23:59.784449+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.029746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006478+0000","last_change":"2024-09-21T08:23:34.566473+0000","last_active":"2024-09-21T08:24:00.006478+0000","last_peered":"2024-09-21T08:24:00.006478+0000","last_clean":"2024-09-21T08:24:00.006478+0000","last_became_active":"2024-09-21T08:23:34.566160+0000","last_became_peered":"2024-09-21T08:23:34.566160+0000","last_unstale":"2024-09-21T08:24:00.006478+0000","last_undegraded":"2024-09-21T08:24:00.006478+0000","last_fullsized":"2024-09-21T08:24:00.006478+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.960353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781606+0000","last_change":"2024-09-21T08:23:36.587149+0000","last_active":"2024-09-21T08:23:59.781606+0000","last_peered":"2024-09-21T08:23:59.781606+0000","last_clean":"2024-09-21T08:23:59.781606+0000","last_became_active":"2024-09-21T08:23:36.586865+0000","last_became_peered":"2024-09-21T08:23:36.586865+0000","last_unstale":"2024-09-21T08:23:59.781606+0000","last_undegraded":"2024-09-21T08:23:59.781606+0000","last_fullsized":"2024-09-21T08:23:59.781606+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:42.991875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773650+0000","last_change":"2024-09-21T08:23:32.599185+0000","last_active":"2024-09-21T08:23:59.773650+0000","last_peered":"2024-09-21T08:23:59.773650+0000","last_clean":"2024-09-21T08:23:59.773650+0000","last_became_active":"2024-09-21T08:23:32.598748+0000","last_became_peered":"2024-09-21T08:23:32.598748+0000","last_unstale":"2024-09-21T08:23:59.773650+0000","last_undegraded":"2024-09-21T08:23:59.773650+0000","last_fullsized":"2024-09-21T08:23:59.773650+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.141293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005211+0000","last_change":"2024-09-21T08:23:30.548111+0000","last_active":"2024-09-21T08:24:00.005211+0000","last_peered":"2024-09-21T08:24:00.005211+0000","last_clean":"2024-09-21T08:24:00.005211+0000","last_became_active":"2024-09-21T08:23:30.547824+0000","last_became_peered":"2024-09-21T08:23:30.547824+0000","last_unstale":"2024-09-21T08:24:00.005211+0000","last_undegraded":"2024-09-21T08:24:00.005211+0000","last_fullsized":"2024-09-21T08:24:00.005211+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598279+0000","last_change":"2024-09-21T08:23:34.570551+0000","last_active":"2024-09-21T08:24:05.598279+0000","last_peered":"2024-09-21T08:24:05.598279+0000","last_clean":"2024-09-21T08:24:05.598279+0000","last_became_active":"2024-09-21T08:23:34.570211+0000","last_became_peered":"2024-09-21T08:23:34.570211+0000","last_unstale":"2024-09-21T08:24:05.598279+0000","last_undegraded":"2024-09-21T08:24:05.598279+0000","last_fullsized":"2024-09-21T08:24:05.598279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.547474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689308+0000","last_change":"2024-09-21T08:23:36.582684+0000","last_active":"2024-09-21T08:23:59.689308+0000","last_peered":"2024-09-21T08:23:59.689308+0000","last_clean":"2024-09-21T08:23:59.689308+0000","last_became_active":"2024-09-21T08:23:36.582369+0000","last_became_peered":"2024-09-21T08:23:36.582369+0000","last_unstale":"2024-09-21T08:23:59.689308+0000","last_undegraded":"2024-09-21T08:23:59.689308+0000","last_fullsized":"2024-09-21T08:23:59.689308+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:48.140821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006743+0000","last_change":"2024-09-21T08:23:32.597630+0000","last_active":"2024-09-21T08:24:00.006743+0000","last_peered":"2024-09-21T08:24:00.006743+0000","last_clean":"2024-09-21T08:24:00.006743+0000","last_became_active":"2024-09-21T08:23:32.597296+0000","last_became_peered":"2024-09-21T08:23:32.597296+0000","last_unstale":"2024-09-21T08:24:00.006743+0000","last_undegraded":"2024-09-21T08:24:00.006743+0000","last_fullsized":"2024-09-21T08:24:00.006743+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.289337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":37,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784650+0000","last_change":"2024-09-21T08:23:30.549864+0000","last_active":"2024-09-21T08:23:59.784650+0000","last_peered":"2024-09-21T08:23:59.784650+0000","last_clean":"2024-09-21T08:23:59.784650+0000","last_became_active":"2024-09-21T08:23:30.549562+0000","last_became_peered":"2024-09-21T08:23:30.549562+0000","last_unstale":"2024-09-21T08:23:59.784650+0000","last_undegraded":"2024-09-21T08:23:59.784650+0000","last_fullsized":"2024-09-21T08:23:59.784650+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.957399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598624+0000","last_change":"2024-09-21T08:23:34.567042+0000","last_active":"2024-09-21T08:24:05.598624+0000","last_peered":"2024-09-21T08:24:05.598624+0000","last_clean":"2024-09-21T08:24:05.598624+0000","last_became_active":"2024-09-21T08:23:34.566736+0000","last_became_peered":"2024-09-21T08:23:34.566736+0000","last_unstale":"2024-09-21T08:24:05.598624+0000","last_undegraded":"2024-09-21T08:24:05.598624+0000","last_fullsized":"2024-09-21T08:24:05.598624+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004732+0000","last_change":"2024-09-21T08:23:36.589503+0000","last_active":"2024-09-21T08:24:00.004732+0000","last_peered":"2024-09-21T08:24:00.004732+0000","last_clean":"2024-09-21T08:24:00.004732+0000","last_became_active":"2024-09-21T08:23:36.589224+0000","last_became_peered":"2024-09-21T08:23:36.589224+0000","last_unstale":"2024-09-21T08:24:00.004732+0000","last_undegraded":"2024-09-21T08:24:00.004732+0000","last_fullsized":"2024-09-21T08:24:00.004732+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.636183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689040+0000","last_change":"2024-09-21T08:23:32.559882+0000","last_active":"2024-09-21T08:23:59.689040+0000","last_peered":"2024-09-21T08:23:59.689040+0000","last_clean":"2024-09-21T08:23:59.689040+0000","last_became_active":"2024-09-21T08:23:32.559574+0000","last_became_peered":"2024-09-21T08:23:32.559574+0000","last_unstale":"2024-09-21T08:23:59.689040+0000","last_undegraded":"2024-09-21T08:23:59.689040+0000","last_fullsized":"2024-09-21T08:23:59.689040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.305662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688789+0000","last_change":"2024-09-21T08:23:30.546837+0000","last_active":"2024-09-21T08:23:59.688789+0000","last_peered":"2024-09-21T08:23:59.688789+0000","last_clean":"2024-09-21T08:23:59.688789+0000","last_became_active":"2024-09-21T08:23:30.546542+0000","last_became_peered":"2024-09-21T08:23:30.546542+0000","last_unstale":"2024-09-21T08:23:59.688789+0000","last_undegraded":"2024-09-21T08:23:59.688789+0000","last_fullsized":"2024-09-21T08:23:59.688789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689670+0000","last_change":"2024-09-21T08:23:34.571300+0000","last_active":"2024-09-21T08:23:59.689670+0000","last_peered":"2024-09-21T08:23:59.689670+0000","last_clean":"2024-09-21T08:23:59.689670+0000","last_became_active":"2024-09-21T08:23:34.571034+0000","last_became_peered":"2024-09-21T08:23:34.571034+0000","last_unstale":"2024-09-21T08:23:59.689670+0000","last_undegraded":"2024-09-21T08:23:59.689670+0000","last_fullsized":"2024-09-21T08:23:59.689670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.978453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004703+0000","last_change":"2024-09-21T08:23:36.585748+0000","last_active":"2024-09-21T08:24:00.004703+0000","last_peered":"2024-09-21T08:24:00.004703+0000","last_clean":"2024-09-21T08:24:00.004703+0000","last_became_active":"2024-09-21T08:23:36.585492+0000","last_became_peered":"2024-09-21T08:23:36.585492+0000","last_unstale":"2024-09-21T08:24:00.004703+0000","last_undegraded":"2024-09-21T08:24:00.004703+0000","last_fullsized":"2024-09-21T08:24:00.004703+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.492501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689913+0000","last_change":"2024-09-21T08:23:32.556550+0000","last_active":"2024-09-21T08:23:59.689913+0000","last_peered":"2024-09-21T08:23:59.689913+0000","last_clean":"2024-09-21T08:23:59.689913+0000","last_became_active":"2024-09-21T08:23:32.556189+0000","last_became_peered":"2024-09-21T08:23:32.556189+0000","last_unstale":"2024-09-21T08:23:59.689913+0000","last_undegraded":"2024-09-21T08:23:59.689913+0000","last_fullsized":"2024-09-21T08:23:59.689913+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:51.354291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784649+0000","last_change":"2024-09-21T08:23:30.549643+0000","last_active":"2024-09-21T08:23:59.784649+0000","last_peered":"2024-09-21T08:23:59.784649+0000","last_clean":"2024-09-21T08:23:59.784649+0000","last_became_active":"2024-09-21T08:23:30.549317+0000","last_became_peered":"2024-09-21T08:23:30.549317+0000","last_unstale":"2024-09-21T08:23:59.784649+0000","last_undegraded":"2024-09-21T08:23:59.784649+0000","last_fullsized":"2024-09-21T08:23:59.784649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.895759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005199+0000","last_change":"2024-09-21T08:23:34.565128+0000","last_active":"2024-09-21T08:24:00.005199+0000","last_peered":"2024-09-21T08:24:00.005199+0000","last_clean":"2024-09-21T08:24:00.005199+0000","last_became_active":"2024-09-21T08:23:34.564738+0000","last_became_peered":"2024-09-21T08:23:34.564738+0000","last_unstale":"2024-09-21T08:24:00.005199+0000","last_undegraded":"2024-09-21T08:24:00.005199+0000","last_fullsized":"2024-09-21T08:24:00.005199+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.717622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004625+0000","last_change":"2024-09-21T08:23:36.580460+0000","last_active":"2024-09-21T08:24:00.004625+0000","last_peered":"2024-09-21T08:24:00.004625+0000","last_clean":"2024-09-21T08:24:00.004625+0000","last_became_active":"2024-09-21T08:23:36.579949+0000","last_became_peered":"2024-09-21T08:23:36.579949+0000","last_unstale":"2024-09-21T08:24:00.004625+0000","last_undegraded":"2024-09-21T08:24:00.004625+0000","last_fullsized":"2024-09-21T08:24:00.004625+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.377442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004773+0000","last_change":"2024-09-21T08:23:32.597672+0000","last_active":"2024-09-21T08:24:00.004773+0000","last_peered":"2024-09-21T08:24:00.004773+0000","last_clean":"2024-09-21T08:24:00.004773+0000","last_became_active":"2024-09-21T08:23:32.597281+0000","last_became_peered":"2024-09-21T08:23:32.597281+0000","last_unstale":"2024-09-21T08:24:00.004773+0000","last_undegraded":"2024-09-21T08:24:00.004773+0000","last_fullsized":"2024-09-21T08:24:00.004773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.493132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773659+0000","last_change":"2024-09-21T08:23:30.544860+0000","last_active":"2024-09-21T08:23:59.773659+0000","last_peered":"2024-09-21T08:23:59.773659+0000","last_clean":"2024-09-21T08:23:59.773659+0000","last_became_active":"2024-09-21T08:23:30.544486+0000","last_became_peered":"2024-09-21T08:23:30.544486+0000","last_unstale":"2024-09-21T08:23:59.773659+0000","last_undegraded":"2024-09-21T08:23:59.773659+0000","last_fullsized":"2024-09-21T08:23:59.773659+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.462170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784050+0000","last_change":"2024-09-21T08:23:34.569813+0000","last_active":"2024-09-21T08:23:59.784050+0000","last_peered":"2024-09-21T08:23:59.784050+0000","last_clean":"2024-09-21T08:23:59.784050+0000","last_became_active":"2024-09-21T08:23:34.569485+0000","last_became_peered":"2024-09-21T08:23:34.569485+0000","last_unstale":"2024-09-21T08:23:59.784050+0000","last_undegraded":"2024-09-21T08:23:59.784050+0000","last_fullsized":"2024-09-21T08:23:59.784050+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.575850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005340+0000","last_change":"2024-09-21T08:23:36.586500+0000","last_active":"2024-09-21T08:24:00.005340+0000","last_peered":"2024-09-21T08:24:00.005340+0000","last_clean":"2024-09-21T08:24:00.005340+0000","last_became_active":"2024-09-21T08:23:36.586172+0000","last_became_peered":"2024-09-21T08:23:36.586172+0000","last_unstale":"2024-09-21T08:24:00.005340+0000","last_undegraded":"2024-09-21T08:24:00.005340+0000","last_fullsized":"2024-09-21T08:24:00.005340+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.388100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":33,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004429+0000","last_change":"2024-09-21T08:23:32.563756+0000","last_active":"2024-09-21T08:24:00.004429+0000","last_peered":"2024-09-21T08:24:00.004429+0000","last_clean":"2024-09-21T08:24:00.004429+0000","last_became_active":"2024-09-21T08:23:32.563484+0000","last_became_peered":"2024-09-21T08:23:32.563484+0000","last_unstale":"2024-09-21T08:24:00.004429+0000","last_undegraded":"2024-09-21T08:24:00.004429+0000","last_fullsized":"2024-09-21T08:24:00.004429+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.069344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005806+0000","last_change":"2024-09-21T08:23:30.544403+0000","last_active":"2024-09-21T08:24:00.005806+0000","last_peered":"2024-09-21T08:24:00.005806+0000","last_clean":"2024-09-21T08:24:00.005806+0000","last_became_active":"2024-09-21T08:23:30.544066+0000","last_became_peered":"2024-09-21T08:23:30.544066+0000","last_unstale":"2024-09-21T08:24:00.005806+0000","last_undegraded":"2024-09-21T08:24:00.005806+0000","last_fullsized":"2024-09-21T08:24:00.005806+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.462796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686256+0000","last_change":"2024-09-21T08:23:34.566607+0000","last_active":"2024-09-21T08:23:59.686256+0000","last_peered":"2024-09-21T08:23:59.686256+0000","last_clean":"2024-09-21T08:23:59.686256+0000","last_became_active":"2024-09-21T08:23:34.566337+0000","last_became_peered":"2024-09-21T08:23:34.566337+0000","last_unstale":"2024-09-21T08:23:59.686256+0000","last_undegraded":"2024-09-21T08:23:59.686256+0000","last_fullsized":"2024-09-21T08:23:59.686256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.230638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784194+0000","last_change":"2024-09-21T08:23:36.582773+0000","last_active":"2024-09-21T08:23:59.784194+0000","last_peered":"2024-09-21T08:23:59.784194+0000","last_clean":"2024-09-21T08:23:59.784194+0000","last_became_active":"2024-09-21T08:23:36.582150+0000","last_became_peered":"2024-09-21T08:23:36.582150+0000","last_unstale":"2024-09-21T08:23:59.784194+0000","last_undegraded":"2024-09-21T08:23:59.784194+0000","last_fullsized":"2024-09-21T08:23:59.784194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.263690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689860+0000","last_change":"2024-09-21T08:23:36.576004+0000","last_active":"2024-09-21T08:23:59.689860+0000","last_peered":"2024-09-21T08:23:59.689860+0000","last_clean":"2024-09-21T08:23:59.689860+0000","last_became_active":"2024-09-21T08:23:36.575701+0000","last_became_peered":"2024-09-21T08:23:36.575701+0000","last_unstale":"2024-09-21T08:23:59.689860+0000","last_undegraded":"2024-09-21T08:23:59.689860+0000","last_fullsized":"2024-09-21T08:23:59.689860+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.955927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004607+0000","last_change":"2024-09-21T08:23:30.544201+0000","last_active":"2024-09-21T08:24:00.004607+0000","last_peered":"2024-09-21T08:24:00.004607+0000","last_clean":"2024-09-21T08:24:00.004607+0000","last_became_active":"2024-09-21T08:23:30.543932+0000","last_became_peered":"2024-09-21T08:23:30.543932+0000","last_unstale":"2024-09-21T08:24:00.004607+0000","last_undegraded":"2024-09-21T08:24:00.004607+0000","last_fullsized":"2024-09-21T08:24:00.004607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.933119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686986+0000","last_change":"2024-09-21T08:23:32.549263+0000","last_active":"2024-09-21T08:23:59.686986+0000","last_peered":"2024-09-21T08:23:59.686986+0000","last_clean":"2024-09-21T08:23:59.686986+0000","last_became_active":"2024-09-21T08:23:32.548835+0000","last_became_peered":"2024-09-21T08:23:32.548835+0000","last_unstale":"2024-09-21T08:23:59.686986+0000","last_undegraded":"2024-09-21T08:23:59.686986+0000","last_fullsized":"2024-09-21T08:23:59.686986+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.423215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004910+0000","last_change":"2024-09-21T08:23:34.573167+0000","last_active":"2024-09-21T08:24:00.004910+0000","last_peered":"2024-09-21T08:24:00.004910+0000","last_clean":"2024-09-21T08:24:00.004910+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:24:00.004910+0000","last_undegraded":"2024-09-21T08:24:00.004910+0000","last_fullsized":"2024-09-21T08:24:00.004910+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.897951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070477,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32400,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712752,"statfs":{"total":95995035648,"available":95961858048,"internally_reserved":0,"allocated":1245184,"data_stored":863237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397523,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1220608,"data_stored":861159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659158,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27652,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":638976,"data_stored":271599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713320,"statfs":{"total":95995035648,"available":95962439680,"internally_reserved":0,"allocated":659456,"data_stored":272389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182432,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":272131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717508,"statfs":{"total":95995035648,"available":95966728192,"internally_reserved":0,"allocated":630784,"data_stored":272368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717444,"statfs":{"total":95995035648,"available":95966662656,"internally_reserved":0,"allocated":696320,"data_stored":272589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738415,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1245184,"data_stored":861953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:24:13.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:24:13.516 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:13.752 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T08:24:13.752 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: Adjusting osd_memory_target on smithi137 to 3760M 2024-09-21T08:24:13.752 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-21T08:24:13.752 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-21T08:24:13.752 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: from='client.14760 v1:172.21.15.137:0/218162821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: Adjusting osd_memory_target on smithi137 to 3760M 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-21T08:24:13.753 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: from='client.14760 v1:172.21.15.137:0/218162821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:13.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:24:13.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:24:13.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:13 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: pgmap v9: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 20 KiB/s rd, 0 B/s wr, 11 op/s 2024-09-21T08:24:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: Adjusting osd_memory_target on smithi137 to 3760M 2024-09-21T08:24:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: Updating smithi137:/etc/ceph/ceph.conf 2024-09-21T08:24:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: Updating smithi165:/etc/ceph/ceph.conf 2024-09-21T08:24:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: from='client.14760 v1:172.21.15.137:0/218162821' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:24:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:24:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:13 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:14.175 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:24:14.176 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:24:14.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[41044]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:24:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[41044]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:24:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[41044]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:24:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[41044]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:24:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[41044]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T08:24:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[41044]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T08:24:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[41044]: Deploying daemon alertmanager.a on smithi137 2024-09-21T08:24:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[41044]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T08:24:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[29634]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:24:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[29634]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:24:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[29634]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:24:14.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[29634]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:24:14.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[29634]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T08:24:14.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[29634]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T08:24:14.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[29634]: Deploying daemon alertmanager.a on smithi137 2024-09-21T08:24:14.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:14 smithi137 ceph-mon[29634]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T08:24:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:14 smithi165 ceph-mon[32893]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:24:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:14 smithi165 ceph-mon[32893]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.conf 2024-09-21T08:24:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:14 smithi165 ceph-mon[32893]: Updating smithi137:/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:24:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:14 smithi165 ceph-mon[32893]: Updating smithi165:/etc/ceph/ceph.client.admin.keyring 2024-09-21T08:24:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:14 smithi165 ceph-mon[32893]: Updating smithi137:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T08:24:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:14 smithi165 ceph-mon[32893]: Updating smithi165:/var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/config/ceph.client.admin.keyring 2024-09-21T08:24:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:14 smithi165 ceph-mon[32893]: Deploying daemon alertmanager.a on smithi137 2024-09-21T08:24:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:14 smithi165 ceph-mon[32893]: osdmap e77: 8 total, 8 up, 8 in 2024-09-21T08:24:15.094 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":11,"stamp":"2024-09-21T08:24:13.703523+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":969,"num_read_kb":815,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":602,"ondisk_log_size":602,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":231376,"kb_used_data":6844,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749729840,"statfs":{"total":767960285184,"available":767723356160,"internally_reserved":0,"allocated":7008256,"data_stored":3947425,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":106,"num_read_kb":216,"num_write":20,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002613"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005780+0000","last_change":"2024-09-21T08:23:30.549696+0000","last_active":"2024-09-21T08:24:00.005780+0000","last_peered":"2024-09-21T08:24:00.005780+0000","last_clean":"2024-09-21T08:24:00.005780+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:24:00.005780+0000","last_undegraded":"2024-09-21T08:24:00.005780+0000","last_fullsized":"2024-09-21T08:24:00.005780+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.197261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.687852+0000","last_change":"2024-09-21T08:23:32.556768+0000","last_active":"2024-09-21T08:23:59.687852+0000","last_peered":"2024-09-21T08:23:59.687852+0000","last_clean":"2024-09-21T08:23:59.687852+0000","last_became_active":"2024-09-21T08:23:32.556452+0000","last_became_peered":"2024-09-21T08:23:32.556452+0000","last_unstale":"2024-09-21T08:23:59.687852+0000","last_undegraded":"2024-09-21T08:23:59.687852+0000","last_fullsized":"2024-09-21T08:23:59.687852+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.936229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689547+0000","last_change":"2024-09-21T08:23:34.569082+0000","last_active":"2024-09-21T08:23:59.689547+0000","last_peered":"2024-09-21T08:23:59.689547+0000","last_clean":"2024-09-21T08:23:59.689547+0000","last_became_active":"2024-09-21T08:23:34.568762+0000","last_became_peered":"2024-09-21T08:23:34.568762+0000","last_unstale":"2024-09-21T08:23:59.689547+0000","last_undegraded":"2024-09-21T08:23:59.689547+0000","last_fullsized":"2024-09-21T08:23:59.689547+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.577218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688107+0000","last_change":"2024-09-21T08:23:36.575623+0000","last_active":"2024-09-21T08:23:59.688107+0000","last_peered":"2024-09-21T08:23:59.688107+0000","last_clean":"2024-09-21T08:23:59.688107+0000","last_became_active":"2024-09-21T08:23:36.572359+0000","last_became_peered":"2024-09-21T08:23:36.572359+0000","last_unstale":"2024-09-21T08:23:59.688107+0000","last_undegraded":"2024-09-21T08:23:59.688107+0000","last_fullsized":"2024-09-21T08:23:59.688107+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.062906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005026+0000","last_change":"2024-09-21T08:23:36.585557+0000","last_active":"2024-09-21T08:24:00.005026+0000","last_peered":"2024-09-21T08:24:00.005026+0000","last_clean":"2024-09-21T08:24:00.005026+0000","last_became_active":"2024-09-21T08:23:36.584820+0000","last_became_peered":"2024-09-21T08:23:36.584820+0000","last_unstale":"2024-09-21T08:24:00.005026+0000","last_undegraded":"2024-09-21T08:24:00.005026+0000","last_fullsized":"2024-09-21T08:24:00.005026+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.509136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006185+0000","last_change":"2024-09-21T08:23:30.548436+0000","last_active":"2024-09-21T08:24:00.006185+0000","last_peered":"2024-09-21T08:24:00.006185+0000","last_clean":"2024-09-21T08:24:00.006185+0000","last_became_active":"2024-09-21T08:23:30.548146+0000","last_became_peered":"2024-09-21T08:23:30.548146+0000","last_unstale":"2024-09-21T08:24:00.006185+0000","last_undegraded":"2024-09-21T08:24:00.006185+0000","last_fullsized":"2024-09-21T08:24:00.006185+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.518727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004908+0000","last_change":"2024-09-21T08:23:32.556169+0000","last_active":"2024-09-21T08:24:00.004908+0000","last_peered":"2024-09-21T08:24:00.004908+0000","last_clean":"2024-09-21T08:24:00.004908+0000","last_became_active":"2024-09-21T08:23:32.555789+0000","last_became_peered":"2024-09-21T08:23:32.555789+0000","last_unstale":"2024-09-21T08:24:00.004908+0000","last_undegraded":"2024-09-21T08:24:00.004908+0000","last_fullsized":"2024-09-21T08:24:00.004908+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.359444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.687767+0000","last_change":"2024-09-21T08:23:34.565614+0000","last_active":"2024-09-21T08:23:59.687767+0000","last_peered":"2024-09-21T08:23:59.687767+0000","last_clean":"2024-09-21T08:23:59.687767+0000","last_became_active":"2024-09-21T08:23:34.565276+0000","last_became_peered":"2024-09-21T08:23:34.565276+0000","last_unstale":"2024-09-21T08:23:59.687767+0000","last_undegraded":"2024-09-21T08:23:59.687767+0000","last_fullsized":"2024-09-21T08:23:59.687767+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.926230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773690+0000","last_change":"2024-09-21T08:23:30.543643+0000","last_active":"2024-09-21T08:23:59.773690+0000","last_peered":"2024-09-21T08:23:59.773690+0000","last_clean":"2024-09-21T08:23:59.773690+0000","last_became_active":"2024-09-21T08:23:30.543041+0000","last_became_peered":"2024-09-21T08:23:30.543041+0000","last_unstale":"2024-09-21T08:23:59.773690+0000","last_undegraded":"2024-09-21T08:23:59.773690+0000","last_fullsized":"2024-09-21T08:23:59.773690+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.229786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688319+0000","last_change":"2024-09-21T08:23:32.559542+0000","last_active":"2024-09-21T08:23:59.688319+0000","last_peered":"2024-09-21T08:23:59.688319+0000","last_clean":"2024-09-21T08:23:59.688319+0000","last_became_active":"2024-09-21T08:23:32.559194+0000","last_became_peered":"2024-09-21T08:23:32.559194+0000","last_unstale":"2024-09-21T08:23:59.688319+0000","last_undegraded":"2024-09-21T08:23:59.688319+0000","last_fullsized":"2024-09-21T08:23:59.688319+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.251123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773287+0000","last_change":"2024-09-21T08:23:34.567872+0000","last_active":"2024-09-21T08:23:59.773287+0000","last_peered":"2024-09-21T08:23:59.773287+0000","last_clean":"2024-09-21T08:23:59.773287+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:23:59.773287+0000","last_undegraded":"2024-09-21T08:23:59.773287+0000","last_fullsized":"2024-09-21T08:23:59.773287+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.208597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004781+0000","last_change":"2024-09-21T08:23:36.582491+0000","last_active":"2024-09-21T08:24:00.004781+0000","last_peered":"2024-09-21T08:24:00.004781+0000","last_clean":"2024-09-21T08:24:00.004781+0000","last_became_active":"2024-09-21T08:23:36.581720+0000","last_became_peered":"2024-09-21T08:23:36.581720+0000","last_unstale":"2024-09-21T08:24:00.004781+0000","last_undegraded":"2024-09-21T08:24:00.004781+0000","last_fullsized":"2024-09-21T08:24:00.004781+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773331+0000","last_change":"2024-09-21T08:23:30.543477+0000","last_active":"2024-09-21T08:23:59.773331+0000","last_peered":"2024-09-21T08:23:59.773331+0000","last_clean":"2024-09-21T08:23:59.773331+0000","last_became_active":"2024-09-21T08:23:30.543025+0000","last_became_peered":"2024-09-21T08:23:30.543025+0000","last_unstale":"2024-09-21T08:23:59.773331+0000","last_undegraded":"2024-09-21T08:23:59.773331+0000","last_fullsized":"2024-09-21T08:23:59.773331+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.290639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":37,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688932+0000","last_change":"2024-09-21T08:23:32.559984+0000","last_active":"2024-09-21T08:23:59.688932+0000","last_peered":"2024-09-21T08:23:59.688932+0000","last_clean":"2024-09-21T08:23:59.688932+0000","last_became_active":"2024-09-21T08:23:32.559544+0000","last_became_peered":"2024-09-21T08:23:32.559544+0000","last_unstale":"2024-09-21T08:23:59.688932+0000","last_undegraded":"2024-09-21T08:23:59.688932+0000","last_fullsized":"2024-09-21T08:23:59.688932+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:31.803588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784212+0000","last_change":"2024-09-21T08:23:34.572687+0000","last_active":"2024-09-21T08:23:59.784212+0000","last_peered":"2024-09-21T08:23:59.784212+0000","last_clean":"2024-09-21T08:23:59.784212+0000","last_became_active":"2024-09-21T08:23:34.572395+0000","last_became_peered":"2024-09-21T08:23:34.572395+0000","last_unstale":"2024-09-21T08:23:59.784212+0000","last_undegraded":"2024-09-21T08:23:59.784212+0000","last_fullsized":"2024-09-21T08:23:59.784212+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.042108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773118+0000","last_change":"2024-09-21T08:23:36.578510+0000","last_active":"2024-09-21T08:23:59.773118+0000","last_peered":"2024-09-21T08:23:59.773118+0000","last_clean":"2024-09-21T08:23:59.773118+0000","last_became_active":"2024-09-21T08:23:36.578227+0000","last_became_peered":"2024-09-21T08:23:36.578227+0000","last_unstale":"2024-09-21T08:23:59.773118+0000","last_undegraded":"2024-09-21T08:23:59.773118+0000","last_fullsized":"2024-09-21T08:23:59.773118+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.087695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688066+0000","last_change":"2024-09-21T08:23:36.573671+0000","last_active":"2024-09-21T08:23:59.688066+0000","last_peered":"2024-09-21T08:23:59.688066+0000","last_clean":"2024-09-21T08:23:59.688066+0000","last_became_active":"2024-09-21T08:23:36.573384+0000","last_became_peered":"2024-09-21T08:23:36.573384+0000","last_unstale":"2024-09-21T08:23:59.688066+0000","last_undegraded":"2024-09-21T08:23:59.688066+0000","last_fullsized":"2024-09-21T08:23:59.688066+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.273680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006256+0000","last_change":"2024-09-21T08:23:30.543356+0000","last_active":"2024-09-21T08:24:00.006256+0000","last_peered":"2024-09-21T08:24:00.006256+0000","last_clean":"2024-09-21T08:24:00.006256+0000","last_became_active":"2024-09-21T08:23:30.542776+0000","last_became_peered":"2024-09-21T08:23:30.542776+0000","last_unstale":"2024-09-21T08:24:00.006256+0000","last_undegraded":"2024-09-21T08:24:00.006256+0000","last_fullsized":"2024-09-21T08:24:00.006256+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.314435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781990+0000","last_change":"2024-09-21T08:23:32.556660+0000","last_active":"2024-09-21T08:23:59.781990+0000","last_peered":"2024-09-21T08:23:59.781990+0000","last_clean":"2024-09-21T08:23:59.781990+0000","last_became_active":"2024-09-21T08:23:32.556378+0000","last_became_peered":"2024-09-21T08:23:32.556378+0000","last_unstale":"2024-09-21T08:23:59.781990+0000","last_undegraded":"2024-09-21T08:23:59.781990+0000","last_fullsized":"2024-09-21T08:23:59.781990+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.143529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.690111+0000","last_change":"2024-09-21T08:23:34.569008+0000","last_active":"2024-09-21T08:23:59.690111+0000","last_peered":"2024-09-21T08:23:59.690111+0000","last_clean":"2024-09-21T08:23:59.690111+0000","last_became_active":"2024-09-21T08:23:34.568562+0000","last_became_peered":"2024-09-21T08:23:34.568562+0000","last_unstale":"2024-09-21T08:23:59.690111+0000","last_undegraded":"2024-09-21T08:23:59.690111+0000","last_fullsized":"2024-09-21T08:23:59.690111+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.511849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006388+0000","last_change":"2024-09-21T08:23:36.584447+0000","last_active":"2024-09-21T08:24:00.006388+0000","last_peered":"2024-09-21T08:24:00.006388+0000","last_clean":"2024-09-21T08:24:00.006388+0000","last_became_active":"2024-09-21T08:23:36.584154+0000","last_became_peered":"2024-09-21T08:23:36.584154+0000","last_unstale":"2024-09-21T08:24:00.006388+0000","last_undegraded":"2024-09-21T08:24:00.006388+0000","last_fullsized":"2024-09-21T08:24:00.006388+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.079729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688538+0000","last_change":"2024-09-21T08:23:30.547824+0000","last_active":"2024-09-21T08:23:59.688538+0000","last_peered":"2024-09-21T08:23:59.688538+0000","last_clean":"2024-09-21T08:23:59.688538+0000","last_became_active":"2024-09-21T08:23:30.546325+0000","last_became_peered":"2024-09-21T08:23:30.546325+0000","last_unstale":"2024-09-21T08:23:59.688538+0000","last_undegraded":"2024-09-21T08:23:59.688538+0000","last_fullsized":"2024-09-21T08:23:59.688538+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.394437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":50,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005634+0000","last_change":"2024-09-21T08:23:32.563650+0000","last_active":"2024-09-21T08:24:00.005634+0000","last_peered":"2024-09-21T08:24:00.005634+0000","last_clean":"2024-09-21T08:24:00.005634+0000","last_became_active":"2024-09-21T08:23:32.563383+0000","last_became_peered":"2024-09-21T08:23:32.563383+0000","last_unstale":"2024-09-21T08:24:00.005634+0000","last_undegraded":"2024-09-21T08:24:00.005634+0000","last_fullsized":"2024-09-21T08:24:00.005634+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.102032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688485+0000","last_change":"2024-09-21T08:23:34.571589+0000","last_active":"2024-09-21T08:23:59.688485+0000","last_peered":"2024-09-21T08:23:59.688485+0000","last_clean":"2024-09-21T08:23:59.688485+0000","last_became_active":"2024-09-21T08:23:34.571323+0000","last_became_peered":"2024-09-21T08:23:34.571323+0000","last_unstale":"2024-09-21T08:23:59.688485+0000","last_undegraded":"2024-09-21T08:23:59.688485+0000","last_fullsized":"2024-09-21T08:23:59.688485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.905411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":20,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784331+0000","last_change":"2024-09-21T08:23:36.584003+0000","last_active":"2024-09-21T08:23:59.784331+0000","last_peered":"2024-09-21T08:23:59.784331+0000","last_clean":"2024-09-21T08:23:59.784331+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:23:59.784331+0000","last_undegraded":"2024-09-21T08:23:59.784331+0000","last_fullsized":"2024-09-21T08:23:59.784331+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.633904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689693+0000","last_change":"2024-09-21T08:23:30.540306+0000","last_active":"2024-09-21T08:23:59.689693+0000","last_peered":"2024-09-21T08:23:59.689693+0000","last_clean":"2024-09-21T08:23:59.689693+0000","last_became_active":"2024-09-21T08:23:30.539931+0000","last_became_peered":"2024-09-21T08:23:30.539931+0000","last_unstale":"2024-09-21T08:23:59.689693+0000","last_undegraded":"2024-09-21T08:23:59.689693+0000","last_fullsized":"2024-09-21T08:23:59.689693+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.106584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005002+0000","last_change":"2024-09-21T08:23:32.598036+0000","last_active":"2024-09-21T08:24:00.005002+0000","last_peered":"2024-09-21T08:24:00.005002+0000","last_clean":"2024-09-21T08:24:00.005002+0000","last_became_active":"2024-09-21T08:23:32.597548+0000","last_became_peered":"2024-09-21T08:23:32.597548+0000","last_unstale":"2024-09-21T08:24:00.005002+0000","last_undegraded":"2024-09-21T08:24:00.005002+0000","last_fullsized":"2024-09-21T08:24:00.005002+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.955005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598730+0000","last_change":"2024-09-21T08:23:34.567530+0000","last_active":"2024-09-21T08:24:05.598730+0000","last_peered":"2024-09-21T08:24:05.598730+0000","last_clean":"2024-09-21T08:24:05.598730+0000","last_became_active":"2024-09-21T08:23:34.567257+0000","last_became_peered":"2024-09-21T08:23:34.567257+0000","last_unstale":"2024-09-21T08:24:05.598730+0000","last_undegraded":"2024-09-21T08:24:05.598730+0000","last_fullsized":"2024-09-21T08:24:05.598730+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.508940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.690868+0000","last_change":"2024-09-21T08:23:32.562879+0000","last_active":"2024-09-21T08:23:59.690868+0000","last_peered":"2024-09-21T08:23:59.690868+0000","last_clean":"2024-09-21T08:23:59.690868+0000","last_became_active":"2024-09-21T08:23:32.562594+0000","last_became_peered":"2024-09-21T08:23:32.562594+0000","last_unstale":"2024-09-21T08:23:59.690868+0000","last_undegraded":"2024-09-21T08:23:59.690868+0000","last_fullsized":"2024-09-21T08:23:59.690868+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.780782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004638+0000","last_change":"2024-09-21T08:23:30.546900+0000","last_active":"2024-09-21T08:24:00.004638+0000","last_peered":"2024-09-21T08:24:00.004638+0000","last_clean":"2024-09-21T08:24:00.004638+0000","last_became_active":"2024-09-21T08:23:30.546625+0000","last_became_peered":"2024-09-21T08:23:30.546625+0000","last_unstale":"2024-09-21T08:24:00.004638+0000","last_undegraded":"2024-09-21T08:24:00.004638+0000","last_fullsized":"2024-09-21T08:24:00.004638+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:48.387469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598615+0000","last_change":"2024-09-21T08:23:34.568257+0000","last_active":"2024-09-21T08:24:05.598615+0000","last_peered":"2024-09-21T08:24:05.598615+0000","last_clean":"2024-09-21T08:24:05.598615+0000","last_became_active":"2024-09-21T08:23:34.567963+0000","last_became_peered":"2024-09-21T08:23:34.567963+0000","last_unstale":"2024-09-21T08:24:05.598615+0000","last_undegraded":"2024-09-21T08:24:05.598615+0000","last_fullsized":"2024-09-21T08:24:05.598615+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:43.930223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773366+0000","last_change":"2024-09-21T08:23:36.578919+0000","last_active":"2024-09-21T08:23:59.773366+0000","last_peered":"2024-09-21T08:23:59.773366+0000","last_clean":"2024-09-21T08:23:59.773366+0000","last_became_active":"2024-09-21T08:23:36.578648+0000","last_became_peered":"2024-09-21T08:23:36.578648+0000","last_unstale":"2024-09-21T08:23:59.773366+0000","last_undegraded":"2024-09-21T08:23:59.773366+0000","last_fullsized":"2024-09-21T08:23:59.773366+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.799668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":59,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004492+0000","last_change":"2024-09-21T08:23:32.597719+0000","last_active":"2024-09-21T08:24:00.004492+0000","last_peered":"2024-09-21T08:24:00.004492+0000","last_clean":"2024-09-21T08:24:00.004492+0000","last_became_active":"2024-09-21T08:23:32.597431+0000","last_became_peered":"2024-09-21T08:23:32.597431+0000","last_unstale":"2024-09-21T08:24:00.004492+0000","last_undegraded":"2024-09-21T08:24:00.004492+0000","last_fullsized":"2024-09-21T08:24:00.004492+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.130776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006792+0000","last_change":"2024-09-21T08:23:30.546233+0000","last_active":"2024-09-21T08:24:00.006792+0000","last_peered":"2024-09-21T08:24:00.006792+0000","last_clean":"2024-09-21T08:24:00.006792+0000","last_became_active":"2024-09-21T08:23:30.545922+0000","last_became_peered":"2024-09-21T08:23:30.545922+0000","last_unstale":"2024-09-21T08:24:00.006792+0000","last_undegraded":"2024-09-21T08:24:00.006792+0000","last_fullsized":"2024-09-21T08:24:00.006792+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.861966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.687952+0000","last_change":"2024-09-21T08:23:34.570926+0000","last_active":"2024-09-21T08:23:59.687952+0000","last_peered":"2024-09-21T08:23:59.687952+0000","last_clean":"2024-09-21T08:23:59.687952+0000","last_became_active":"2024-09-21T08:23:34.570648+0000","last_became_peered":"2024-09-21T08:23:34.570648+0000","last_unstale":"2024-09-21T08:23:59.687952+0000","last_undegraded":"2024-09-21T08:23:59.687952+0000","last_fullsized":"2024-09-21T08:23:59.687952+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.476055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688214+0000","last_change":"2024-09-21T08:23:36.582947+0000","last_active":"2024-09-21T08:23:59.688214+0000","last_peered":"2024-09-21T08:23:59.688214+0000","last_clean":"2024-09-21T08:23:59.688214+0000","last_became_active":"2024-09-21T08:23:36.582610+0000","last_became_peered":"2024-09-21T08:23:36.582610+0000","last_unstale":"2024-09-21T08:23:59.688214+0000","last_undegraded":"2024-09-21T08:23:59.688214+0000","last_fullsized":"2024-09-21T08:23:59.688214+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.387653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":48,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688032+0000","last_change":"2024-09-21T08:23:32.553440+0000","last_active":"2024-09-21T08:23:59.688032+0000","last_peered":"2024-09-21T08:23:59.688032+0000","last_clean":"2024-09-21T08:23:59.688032+0000","last_became_active":"2024-09-21T08:23:32.552980+0000","last_became_peered":"2024-09-21T08:23:32.552980+0000","last_unstale":"2024-09-21T08:23:59.688032+0000","last_undegraded":"2024-09-21T08:23:59.688032+0000","last_fullsized":"2024-09-21T08:23:59.688032+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.960840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":32,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005386+0000","last_change":"2024-09-21T08:23:30.543176+0000","last_active":"2024-09-21T08:24:00.005386+0000","last_peered":"2024-09-21T08:24:00.005386+0000","last_clean":"2024-09-21T08:24:00.005386+0000","last_became_active":"2024-09-21T08:23:30.542888+0000","last_became_peered":"2024-09-21T08:23:30.542888+0000","last_unstale":"2024-09-21T08:24:00.005386+0000","last_undegraded":"2024-09-21T08:24:00.005386+0000","last_fullsized":"2024-09-21T08:24:00.005386+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.545006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598254+0000","last_change":"2024-09-21T08:23:34.570603+0000","last_active":"2024-09-21T08:24:05.598254+0000","last_peered":"2024-09-21T08:24:05.598254+0000","last_clean":"2024-09-21T08:24:05.598254+0000","last_became_active":"2024-09-21T08:23:34.570321+0000","last_became_peered":"2024-09-21T08:23:34.570321+0000","last_unstale":"2024-09-21T08:24:05.598254+0000","last_undegraded":"2024-09-21T08:24:05.598254+0000","last_fullsized":"2024-09-21T08:24:05.598254+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.651545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784289+0000","last_change":"2024-09-21T08:23:36.582496+0000","last_active":"2024-09-21T08:23:59.784289+0000","last_peered":"2024-09-21T08:23:59.784289+0000","last_clean":"2024-09-21T08:23:59.784289+0000","last_became_active":"2024-09-21T08:23:36.582039+0000","last_became_peered":"2024-09-21T08:23:36.582039+0000","last_unstale":"2024-09-21T08:23:59.784289+0000","last_undegraded":"2024-09-21T08:23:59.784289+0000","last_fullsized":"2024-09-21T08:23:59.784289+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.819778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689880+0000","last_change":"2024-09-21T08:23:32.557661+0000","last_active":"2024-09-21T08:23:59.689880+0000","last_peered":"2024-09-21T08:23:59.689880+0000","last_clean":"2024-09-21T08:23:59.689880+0000","last_became_active":"2024-09-21T08:23:32.555080+0000","last_became_peered":"2024-09-21T08:23:32.555080+0000","last_unstale":"2024-09-21T08:23:59.689880+0000","last_undegraded":"2024-09-21T08:23:59.689880+0000","last_fullsized":"2024-09-21T08:23:59.689880+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.633944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":37,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773788+0000","last_change":"2024-09-21T08:23:30.546228+0000","last_active":"2024-09-21T08:23:59.773788+0000","last_peered":"2024-09-21T08:23:59.773788+0000","last_clean":"2024-09-21T08:23:59.773788+0000","last_became_active":"2024-09-21T08:23:30.545920+0000","last_became_peered":"2024-09-21T08:23:30.545920+0000","last_unstale":"2024-09-21T08:23:59.773788+0000","last_undegraded":"2024-09-21T08:23:59.773788+0000","last_fullsized":"2024-09-21T08:23:59.773788+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.464911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598279+0000","last_change":"2024-09-21T08:23:34.568126+0000","last_active":"2024-09-21T08:24:05.598279+0000","last_peered":"2024-09-21T08:24:05.598279+0000","last_clean":"2024-09-21T08:24:05.598279+0000","last_became_active":"2024-09-21T08:23:34.563906+0000","last_became_peered":"2024-09-21T08:23:34.563906+0000","last_unstale":"2024-09-21T08:24:05.598279+0000","last_undegraded":"2024-09-21T08:24:05.598279+0000","last_fullsized":"2024-09-21T08:24:05.598279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.761252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004937+0000","last_change":"2024-09-21T08:23:36.588118+0000","last_active":"2024-09-21T08:24:00.004937+0000","last_peered":"2024-09-21T08:24:00.004937+0000","last_clean":"2024-09-21T08:24:00.004937+0000","last_became_active":"2024-09-21T08:23:36.587782+0000","last_became_peered":"2024-09-21T08:23:36.587782+0000","last_unstale":"2024-09-21T08:24:00.004937+0000","last_undegraded":"2024-09-21T08:24:00.004937+0000","last_fullsized":"2024-09-21T08:24:00.004937+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.580040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":63,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005953+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:24:00.005953+0000","last_peered":"2024-09-21T08:24:00.005953+0000","last_clean":"2024-09-21T08:24:00.005953+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:24:00.005953+0000","last_undegraded":"2024-09-21T08:24:00.005953+0000","last_fullsized":"2024-09-21T08:24:00.005953+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.235370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.690997+0000","last_change":"2024-09-21T08:23:30.547730+0000","last_active":"2024-09-21T08:23:59.690997+0000","last_peered":"2024-09-21T08:23:59.690997+0000","last_clean":"2024-09-21T08:23:59.690997+0000","last_became_active":"2024-09-21T08:23:30.547290+0000","last_became_peered":"2024-09-21T08:23:30.547290+0000","last_unstale":"2024-09-21T08:23:59.690997+0000","last_undegraded":"2024-09-21T08:23:59.690997+0000","last_fullsized":"2024-09-21T08:23:59.690997+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.347687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686264+0000","last_change":"2024-09-21T08:23:34.564467+0000","last_active":"2024-09-21T08:23:59.686264+0000","last_peered":"2024-09-21T08:23:59.686264+0000","last_clean":"2024-09-21T08:23:59.686264+0000","last_became_active":"2024-09-21T08:23:34.564157+0000","last_became_peered":"2024-09-21T08:23:34.564157+0000","last_unstale":"2024-09-21T08:23:59.686264+0000","last_undegraded":"2024-09-21T08:23:59.686264+0000","last_fullsized":"2024-09-21T08:23:59.686264+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.544611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689477+0000","last_change":"2024-09-21T08:23:36.582071+0000","last_active":"2024-09-21T08:23:59.689477+0000","last_peered":"2024-09-21T08:23:59.689477+0000","last_clean":"2024-09-21T08:23:59.689477+0000","last_became_active":"2024-09-21T08:23:36.581757+0000","last_became_peered":"2024-09-21T08:23:36.581757+0000","last_unstale":"2024-09-21T08:23:59.689477+0000","last_undegraded":"2024-09-21T08:23:59.689477+0000","last_fullsized":"2024-09-21T08:23:59.689477+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.873559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":84,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598266+0000","last_change":"2024-09-21T08:23:32.562868+0000","last_active":"2024-09-21T08:24:05.598266+0000","last_peered":"2024-09-21T08:24:05.598266+0000","last_clean":"2024-09-21T08:24:05.598266+0000","last_became_active":"2024-09-21T08:23:32.561965+0000","last_became_peered":"2024-09-21T08:23:32.561965+0000","last_unstale":"2024-09-21T08:24:05.598266+0000","last_undegraded":"2024-09-21T08:24:05.598266+0000","last_fullsized":"2024-09-21T08:24:05.598266+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.844304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689252+0000","last_change":"2024-09-21T08:23:30.541459+0000","last_active":"2024-09-21T08:23:59.689252+0000","last_peered":"2024-09-21T08:23:59.689252+0000","last_clean":"2024-09-21T08:23:59.689252+0000","last_became_active":"2024-09-21T08:23:30.541175+0000","last_became_peered":"2024-09-21T08:23:30.541175+0000","last_unstale":"2024-09-21T08:23:59.689252+0000","last_undegraded":"2024-09-21T08:23:59.689252+0000","last_fullsized":"2024-09-21T08:23:59.689252+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.508394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":41,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773320+0000","last_change":"2024-09-21T08:23:32.546026+0000","last_active":"2024-09-21T08:23:59.773320+0000","last_peered":"2024-09-21T08:23:59.773320+0000","last_clean":"2024-09-21T08:23:59.773320+0000","last_became_active":"2024-09-21T08:23:29.531243+0000","last_became_peered":"2024-09-21T08:23:29.531243+0000","last_unstale":"2024-09-21T08:23:59.773320+0000","last_undegraded":"2024-09-21T08:23:59.773320+0000","last_fullsized":"2024-09-21T08:23:59.773320+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.908720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005536+0000","last_change":"2024-09-21T08:23:34.568259+0000","last_active":"2024-09-21T08:24:00.005536+0000","last_peered":"2024-09-21T08:24:00.005536+0000","last_clean":"2024-09-21T08:24:00.005536+0000","last_became_active":"2024-09-21T08:23:34.567955+0000","last_became_peered":"2024-09-21T08:23:34.567955+0000","last_unstale":"2024-09-21T08:24:00.005536+0000","last_undegraded":"2024-09-21T08:24:00.005536+0000","last_fullsized":"2024-09-21T08:24:00.005536+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.600803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005935+0000","last_change":"2024-09-21T08:23:36.577895+0000","last_active":"2024-09-21T08:24:00.005935+0000","last_peered":"2024-09-21T08:24:00.005935+0000","last_clean":"2024-09-21T08:24:00.005935+0000","last_became_active":"2024-09-21T08:23:36.577593+0000","last_became_peered":"2024-09-21T08:23:36.577593+0000","last_unstale":"2024-09-21T08:24:00.005935+0000","last_undegraded":"2024-09-21T08:24:00.005935+0000","last_fullsized":"2024-09-21T08:24:00.005935+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.663315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":54,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.691001+0000","last_change":"2024-09-21T08:23:32.560593+0000","last_active":"2024-09-21T08:23:59.691001+0000","last_peered":"2024-09-21T08:23:59.691001+0000","last_clean":"2024-09-21T08:23:59.691001+0000","last_became_active":"2024-09-21T08:23:32.560323+0000","last_became_peered":"2024-09-21T08:23:32.560323+0000","last_unstale":"2024-09-21T08:23:59.691001+0000","last_undegraded":"2024-09-21T08:23:59.691001+0000","last_fullsized":"2024-09-21T08:23:59.691001+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.516260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689756+0000","last_change":"2024-09-21T08:23:30.548894+0000","last_active":"2024-09-21T08:23:59.689756+0000","last_peered":"2024-09-21T08:23:59.689756+0000","last_clean":"2024-09-21T08:23:59.689756+0000","last_became_active":"2024-09-21T08:23:30.548308+0000","last_became_peered":"2024-09-21T08:23:30.548308+0000","last_unstale":"2024-09-21T08:23:59.689756+0000","last_undegraded":"2024-09-21T08:23:59.689756+0000","last_fullsized":"2024-09-21T08:23:59.689756+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.379264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781354+0000","last_change":"2024-09-21T08:23:32.552207+0000","last_active":"2024-09-21T08:23:59.781354+0000","last_peered":"2024-09-21T08:23:59.781354+0000","last_clean":"2024-09-21T08:23:59.781354+0000","last_became_active":"2024-09-21T08:23:29.533672+0000","last_became_peered":"2024-09-21T08:23:29.533672+0000","last_unstale":"2024-09-21T08:23:59.781354+0000","last_undegraded":"2024-09-21T08:23:59.781354+0000","last_fullsized":"2024-09-21T08:23:59.781354+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.009447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781692+0000","last_change":"2024-09-21T08:23:34.567541+0000","last_active":"2024-09-21T08:23:59.781692+0000","last_peered":"2024-09-21T08:23:59.781692+0000","last_clean":"2024-09-21T08:23:59.781692+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:23:59.781692+0000","last_undegraded":"2024-09-21T08:23:59.781692+0000","last_fullsized":"2024-09-21T08:23:59.781692+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:51.623026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784355+0000","last_change":"2024-09-21T08:23:36.585190+0000","last_active":"2024-09-21T08:23:59.784355+0000","last_peered":"2024-09-21T08:23:59.784355+0000","last_clean":"2024-09-21T08:23:59.784355+0000","last_became_active":"2024-09-21T08:23:36.584905+0000","last_became_peered":"2024-09-21T08:23:36.584905+0000","last_unstale":"2024-09-21T08:23:59.784355+0000","last_undegraded":"2024-09-21T08:23:59.784355+0000","last_fullsized":"2024-09-21T08:23:59.784355+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.290201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":45,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005132+0000","last_change":"2024-09-21T08:23:32.561943+0000","last_active":"2024-09-21T08:24:00.005132+0000","last_peered":"2024-09-21T08:24:00.005132+0000","last_clean":"2024-09-21T08:24:00.005132+0000","last_became_active":"2024-09-21T08:23:32.561617+0000","last_became_peered":"2024-09-21T08:23:32.561617+0000","last_unstale":"2024-09-21T08:24:00.005132+0000","last_undegraded":"2024-09-21T08:24:00.005132+0000","last_fullsized":"2024-09-21T08:24:00.005132+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.493572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005497+0000","last_change":"2024-09-21T08:23:30.547156+0000","last_active":"2024-09-21T08:24:00.005497+0000","last_peered":"2024-09-21T08:24:00.005497+0000","last_clean":"2024-09-21T08:24:00.005497+0000","last_became_active":"2024-09-21T08:23:30.546898+0000","last_became_peered":"2024-09-21T08:23:30.546898+0000","last_unstale":"2024-09-21T08:24:00.005497+0000","last_undegraded":"2024-09-21T08:24:00.005497+0000","last_fullsized":"2024-09-21T08:24:00.005497+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.718486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":77,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:09.549457+0000","last_change":"2024-09-21T08:23:32.597264+0000","last_active":"2024-09-21T08:24:09.549457+0000","last_peered":"2024-09-21T08:24:09.549457+0000","last_clean":"2024-09-21T08:24:09.549457+0000","last_became_active":"2024-09-21T08:23:29.532520+0000","last_became_peered":"2024-09-21T08:23:29.532520+0000","last_unstale":"2024-09-21T08:24:09.549457+0000","last_undegraded":"2024-09-21T08:24:09.549457+0000","last_fullsized":"2024-09-21T08:24:09.549457+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.455707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773786+0000","last_change":"2024-09-21T08:23:34.570449+0000","last_active":"2024-09-21T08:23:59.773786+0000","last_peered":"2024-09-21T08:23:59.773786+0000","last_clean":"2024-09-21T08:23:59.773786+0000","last_became_active":"2024-09-21T08:23:34.569970+0000","last_became_peered":"2024-09-21T08:23:34.569970+0000","last_unstale":"2024-09-21T08:23:59.773786+0000","last_undegraded":"2024-09-21T08:23:59.773786+0000","last_fullsized":"2024-09-21T08:23:59.773786+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.151707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.690541+0000","last_change":"2024-09-21T08:23:36.585195+0000","last_active":"2024-09-21T08:23:59.690541+0000","last_peered":"2024-09-21T08:23:59.690541+0000","last_clean":"2024-09-21T08:23:59.690541+0000","last_became_active":"2024-09-21T08:23:36.584827+0000","last_became_peered":"2024-09-21T08:23:36.584827+0000","last_unstale":"2024-09-21T08:23:59.690541+0000","last_undegraded":"2024-09-21T08:23:59.690541+0000","last_fullsized":"2024-09-21T08:23:59.690541+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.376554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598687+0000","last_change":"2024-09-21T08:23:32.598546+0000","last_active":"2024-09-21T08:24:05.598687+0000","last_peered":"2024-09-21T08:24:05.598687+0000","last_clean":"2024-09-21T08:24:05.598687+0000","last_became_active":"2024-09-21T08:23:32.596949+0000","last_became_peered":"2024-09-21T08:23:32.596949+0000","last_unstale":"2024-09-21T08:24:05.598687+0000","last_undegraded":"2024-09-21T08:24:05.598687+0000","last_fullsized":"2024-09-21T08:24:05.598687+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.826771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005064+0000","last_change":"2024-09-21T08:23:30.546494+0000","last_active":"2024-09-21T08:24:00.005064+0000","last_peered":"2024-09-21T08:24:00.005064+0000","last_clean":"2024-09-21T08:24:00.005064+0000","last_became_active":"2024-09-21T08:23:30.546218+0000","last_became_peered":"2024-09-21T08:23:30.546218+0000","last_unstale":"2024-09-21T08:24:00.005064+0000","last_undegraded":"2024-09-21T08:24:00.005064+0000","last_fullsized":"2024-09-21T08:24:00.005064+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.498978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":286,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.765884+0000","last_change":"2024-09-21T08:24:05.765943+0000","last_active":"2024-09-21T08:24:05.765884+0000","last_peered":"2024-09-21T08:24:05.765884+0000","last_clean":"2024-09-21T08:24:05.765884+0000","last_became_active":"2024-09-21T08:23:23.509818+0000","last_became_peered":"2024-09-21T08:23:23.509818+0000","last_unstale":"2024-09-21T08:24:05.765884+0000","last_undegraded":"2024-09-21T08:24:05.765884+0000","last_fullsized":"2024-09-21T08:24:05.765884+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:24:05.765804+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:05.827066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784511+0000","last_change":"2024-09-21T08:23:34.568359+0000","last_active":"2024-09-21T08:23:59.784511+0000","last_peered":"2024-09-21T08:23:59.784511+0000","last_clean":"2024-09-21T08:23:59.784511+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:23:59.784511+0000","last_undegraded":"2024-09-21T08:23:59.784511+0000","last_fullsized":"2024-09-21T08:23:59.784511+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.246877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773654+0000","last_change":"2024-09-21T08:23:36.584454+0000","last_active":"2024-09-21T08:23:59.773654+0000","last_peered":"2024-09-21T08:23:59.773654+0000","last_clean":"2024-09-21T08:23:59.773654+0000","last_became_active":"2024-09-21T08:23:36.584095+0000","last_became_peered":"2024-09-21T08:23:36.584095+0000","last_unstale":"2024-09-21T08:23:59.773654+0000","last_undegraded":"2024-09-21T08:23:59.773654+0000","last_fullsized":"2024-09-21T08:23:59.773654+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.123137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688254+0000","last_change":"2024-09-21T08:23:32.557075+0000","last_active":"2024-09-21T08:23:59.688254+0000","last_peered":"2024-09-21T08:23:59.688254+0000","last_clean":"2024-09-21T08:23:59.688254+0000","last_became_active":"2024-09-21T08:23:32.556696+0000","last_became_peered":"2024-09-21T08:23:32.556696+0000","last_unstale":"2024-09-21T08:23:59.688254+0000","last_undegraded":"2024-09-21T08:23:59.688254+0000","last_fullsized":"2024-09-21T08:23:59.688254+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.647238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688373+0000","last_change":"2024-09-21T08:23:30.547989+0000","last_active":"2024-09-21T08:23:59.688373+0000","last_peered":"2024-09-21T08:23:59.688373+0000","last_clean":"2024-09-21T08:23:59.688373+0000","last_became_active":"2024-09-21T08:23:30.547176+0000","last_became_peered":"2024-09-21T08:23:30.547176+0000","last_unstale":"2024-09-21T08:23:59.688373+0000","last_undegraded":"2024-09-21T08:23:59.688373+0000","last_fullsized":"2024-09-21T08:23:59.688373+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.295162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773615+0000","last_change":"2024-09-21T08:23:34.570250+0000","last_active":"2024-09-21T08:23:59.773615+0000","last_peered":"2024-09-21T08:23:59.773615+0000","last_clean":"2024-09-21T08:23:59.773615+0000","last_became_active":"2024-09-21T08:23:34.569571+0000","last_became_peered":"2024-09-21T08:23:34.569571+0000","last_unstale":"2024-09-21T08:23:59.773615+0000","last_undegraded":"2024-09-21T08:23:59.773615+0000","last_fullsized":"2024-09-21T08:23:59.773615+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.053187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006390+0000","last_change":"2024-09-21T08:23:36.587842+0000","last_active":"2024-09-21T08:24:00.006390+0000","last_peered":"2024-09-21T08:24:00.006390+0000","last_clean":"2024-09-21T08:24:00.006390+0000","last_became_active":"2024-09-21T08:23:36.587561+0000","last_became_peered":"2024-09-21T08:23:36.587561+0000","last_unstale":"2024-09-21T08:24:00.006390+0000","last_undegraded":"2024-09-21T08:24:00.006390+0000","last_fullsized":"2024-09-21T08:24:00.006390+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.999379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":55,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688962+0000","last_change":"2024-09-21T08:23:32.556741+0000","last_active":"2024-09-21T08:23:59.688962+0000","last_peered":"2024-09-21T08:23:59.688962+0000","last_clean":"2024-09-21T08:23:59.688962+0000","last_became_active":"2024-09-21T08:23:32.556355+0000","last_became_peered":"2024-09-21T08:23:32.556355+0000","last_unstale":"2024-09-21T08:23:59.688962+0000","last_undegraded":"2024-09-21T08:23:59.688962+0000","last_fullsized":"2024-09-21T08:23:59.688962+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686539+0000","last_change":"2024-09-21T08:23:30.537030+0000","last_active":"2024-09-21T08:23:59.686539+0000","last_peered":"2024-09-21T08:23:59.686539+0000","last_clean":"2024-09-21T08:23:59.686539+0000","last_became_active":"2024-09-21T08:23:30.536566+0000","last_became_peered":"2024-09-21T08:23:30.536566+0000","last_unstale":"2024-09-21T08:23:59.686539+0000","last_undegraded":"2024-09-21T08:23:59.686539+0000","last_fullsized":"2024-09-21T08:23:59.686539+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.146930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.690635+0000","last_change":"2024-09-21T08:23:34.569384+0000","last_active":"2024-09-21T08:23:59.690635+0000","last_peered":"2024-09-21T08:23:59.690635+0000","last_clean":"2024-09-21T08:23:59.690635+0000","last_became_active":"2024-09-21T08:23:34.568900+0000","last_became_peered":"2024-09-21T08:23:34.568900+0000","last_unstale":"2024-09-21T08:23:59.690635+0000","last_undegraded":"2024-09-21T08:23:59.690635+0000","last_fullsized":"2024-09-21T08:23:59.690635+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.163373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781851+0000","last_change":"2024-09-21T08:23:36.586002+0000","last_active":"2024-09-21T08:23:59.781851+0000","last_peered":"2024-09-21T08:23:59.781851+0000","last_clean":"2024-09-21T08:23:59.781851+0000","last_became_active":"2024-09-21T08:23:36.585695+0000","last_became_peered":"2024-09-21T08:23:36.585695+0000","last_unstale":"2024-09-21T08:23:59.781851+0000","last_undegraded":"2024-09-21T08:23:59.781851+0000","last_fullsized":"2024-09-21T08:23:59.781851+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.916781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688568+0000","last_change":"2024-09-21T08:23:32.555172+0000","last_active":"2024-09-21T08:23:59.688568+0000","last_peered":"2024-09-21T08:23:59.688568+0000","last_clean":"2024-09-21T08:23:59.688568+0000","last_became_active":"2024-09-21T08:23:32.554845+0000","last_became_peered":"2024-09-21T08:23:32.554845+0000","last_unstale":"2024-09-21T08:23:59.688568+0000","last_undegraded":"2024-09-21T08:23:59.688568+0000","last_fullsized":"2024-09-21T08:23:59.688568+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.339439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005423+0000","last_change":"2024-09-21T08:23:30.545460+0000","last_active":"2024-09-21T08:24:00.005423+0000","last_peered":"2024-09-21T08:24:00.005423+0000","last_clean":"2024-09-21T08:24:00.005423+0000","last_became_active":"2024-09-21T08:23:30.545187+0000","last_became_peered":"2024-09-21T08:23:30.545187+0000","last_unstale":"2024-09-21T08:24:00.005423+0000","last_undegraded":"2024-09-21T08:24:00.005423+0000","last_fullsized":"2024-09-21T08:24:00.005423+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.674857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598259+0000","last_change":"2024-09-21T08:23:34.571530+0000","last_active":"2024-09-21T08:24:05.598259+0000","last_peered":"2024-09-21T08:24:05.598259+0000","last_clean":"2024-09-21T08:24:05.598259+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:24:05.598259+0000","last_undegraded":"2024-09-21T08:24:05.598259+0000","last_fullsized":"2024-09-21T08:24:05.598259+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.789578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.687971+0000","last_change":"2024-09-21T08:23:36.582871+0000","last_active":"2024-09-21T08:23:59.687971+0000","last_peered":"2024-09-21T08:23:59.687971+0000","last_clean":"2024-09-21T08:23:59.687971+0000","last_became_active":"2024-09-21T08:23:36.582575+0000","last_became_peered":"2024-09-21T08:23:36.582575+0000","last_unstale":"2024-09-21T08:23:59.687971+0000","last_undegraded":"2024-09-21T08:23:59.687971+0000","last_fullsized":"2024-09-21T08:23:59.687971+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.578837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005715+0000","last_change":"2024-09-21T08:23:32.560978+0000","last_active":"2024-09-21T08:24:00.005715+0000","last_peered":"2024-09-21T08:24:00.005715+0000","last_clean":"2024-09-21T08:24:00.005715+0000","last_became_active":"2024-09-21T08:23:32.560658+0000","last_became_peered":"2024-09-21T08:23:32.560658+0000","last_unstale":"2024-09-21T08:24:00.005715+0000","last_undegraded":"2024-09-21T08:24:00.005715+0000","last_fullsized":"2024-09-21T08:24:00.005715+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.965888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773438+0000","last_change":"2024-09-21T08:23:30.542309+0000","last_active":"2024-09-21T08:23:59.773438+0000","last_peered":"2024-09-21T08:23:59.773438+0000","last_clean":"2024-09-21T08:23:59.773438+0000","last_became_active":"2024-09-21T08:23:30.541992+0000","last_became_peered":"2024-09-21T08:23:30.541992+0000","last_unstale":"2024-09-21T08:23:59.773438+0000","last_undegraded":"2024-09-21T08:23:59.773438+0000","last_fullsized":"2024-09-21T08:23:59.773438+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.859320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781309+0000","last_change":"2024-09-21T08:23:34.566539+0000","last_active":"2024-09-21T08:23:59.781309+0000","last_peered":"2024-09-21T08:23:59.781309+0000","last_clean":"2024-09-21T08:23:59.781309+0000","last_became_active":"2024-09-21T08:23:34.566242+0000","last_became_peered":"2024-09-21T08:23:34.566242+0000","last_unstale":"2024-09-21T08:23:59.781309+0000","last_undegraded":"2024-09-21T08:23:59.781309+0000","last_fullsized":"2024-09-21T08:23:59.781309+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.194173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005400+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:24:00.005400+0000","last_peered":"2024-09-21T08:24:00.005400+0000","last_clean":"2024-09-21T08:24:00.005400+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:24:00.005400+0000","last_undegraded":"2024-09-21T08:24:00.005400+0000","last_fullsized":"2024-09-21T08:24:00.005400+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.339671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":58,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686332+0000","last_change":"2024-09-21T08:23:32.597880+0000","last_active":"2024-09-21T08:23:59.686332+0000","last_peered":"2024-09-21T08:23:59.686332+0000","last_clean":"2024-09-21T08:23:59.686332+0000","last_became_active":"2024-09-21T08:23:32.597589+0000","last_became_peered":"2024-09-21T08:23:32.597589+0000","last_unstale":"2024-09-21T08:23:59.686332+0000","last_undegraded":"2024-09-21T08:23:59.686332+0000","last_fullsized":"2024-09-21T08:23:59.686332+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.870820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784798+0000","last_change":"2024-09-21T08:23:30.545405+0000","last_active":"2024-09-21T08:23:59.784798+0000","last_peered":"2024-09-21T08:23:59.784798+0000","last_clean":"2024-09-21T08:23:59.784798+0000","last_became_active":"2024-09-21T08:23:30.545068+0000","last_became_peered":"2024-09-21T08:23:30.545068+0000","last_unstale":"2024-09-21T08:23:59.784798+0000","last_undegraded":"2024-09-21T08:23:59.784798+0000","last_fullsized":"2024-09-21T08:23:59.784798+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.179419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781437+0000","last_change":"2024-09-21T08:23:34.564923+0000","last_active":"2024-09-21T08:23:59.781437+0000","last_peered":"2024-09-21T08:23:59.781437+0000","last_clean":"2024-09-21T08:23:59.781437+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:23:59.781437+0000","last_undegraded":"2024-09-21T08:23:59.781437+0000","last_fullsized":"2024-09-21T08:23:59.781437+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.183697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784514+0000","last_change":"2024-09-21T08:23:36.582099+0000","last_active":"2024-09-21T08:23:59.784514+0000","last_peered":"2024-09-21T08:23:59.784514+0000","last_clean":"2024-09-21T08:23:59.784514+0000","last_became_active":"2024-09-21T08:23:36.581796+0000","last_became_peered":"2024-09-21T08:23:36.581796+0000","last_unstale":"2024-09-21T08:23:59.784514+0000","last_undegraded":"2024-09-21T08:23:59.784514+0000","last_fullsized":"2024-09-21T08:23:59.784514+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.618273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":50,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688187+0000","last_change":"2024-09-21T08:23:32.557284+0000","last_active":"2024-09-21T08:23:59.688187+0000","last_peered":"2024-09-21T08:23:59.688187+0000","last_clean":"2024-09-21T08:23:59.688187+0000","last_became_active":"2024-09-21T08:23:32.556876+0000","last_became_peered":"2024-09-21T08:23:32.556876+0000","last_unstale":"2024-09-21T08:23:59.688187+0000","last_undegraded":"2024-09-21T08:23:59.688187+0000","last_fullsized":"2024-09-21T08:23:59.688187+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.982500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784789+0000","last_change":"2024-09-21T08:23:30.548671+0000","last_active":"2024-09-21T08:23:59.784789+0000","last_peered":"2024-09-21T08:23:59.784789+0000","last_clean":"2024-09-21T08:23:59.784789+0000","last_became_active":"2024-09-21T08:23:30.548350+0000","last_became_peered":"2024-09-21T08:23:30.548350+0000","last_unstale":"2024-09-21T08:23:59.784789+0000","last_undegraded":"2024-09-21T08:23:59.784789+0000","last_fullsized":"2024-09-21T08:23:59.784789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.489110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781151+0000","last_change":"2024-09-21T08:23:34.566472+0000","last_active":"2024-09-21T08:23:59.781151+0000","last_peered":"2024-09-21T08:23:59.781151+0000","last_clean":"2024-09-21T08:23:59.781151+0000","last_became_active":"2024-09-21T08:23:34.566193+0000","last_became_peered":"2024-09-21T08:23:34.566193+0000","last_unstale":"2024-09-21T08:23:59.781151+0000","last_undegraded":"2024-09-21T08:23:59.781151+0000","last_fullsized":"2024-09-21T08:23:59.781151+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:58.818551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005118+0000","last_change":"2024-09-21T08:23:36.579879+0000","last_active":"2024-09-21T08:24:00.005118+0000","last_peered":"2024-09-21T08:24:00.005118+0000","last_clean":"2024-09-21T08:24:00.005118+0000","last_became_active":"2024-09-21T08:23:36.579603+0000","last_became_peered":"2024-09-21T08:23:36.579603+0000","last_unstale":"2024-09-21T08:24:00.005118+0000","last_undegraded":"2024-09-21T08:24:00.005118+0000","last_fullsized":"2024-09-21T08:24:00.005118+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.717936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":52,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773021+0000","last_change":"2024-09-21T08:23:32.598217+0000","last_active":"2024-09-21T08:23:59.773021+0000","last_peered":"2024-09-21T08:23:59.773021+0000","last_clean":"2024-09-21T08:23:59.773021+0000","last_became_active":"2024-09-21T08:23:32.597543+0000","last_became_peered":"2024-09-21T08:23:32.597543+0000","last_unstale":"2024-09-21T08:23:59.773021+0000","last_undegraded":"2024-09-21T08:23:59.773021+0000","last_fullsized":"2024-09-21T08:23:59.773021+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.907666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784890+0000","last_change":"2024-09-21T08:23:30.551520+0000","last_active":"2024-09-21T08:23:59.784890+0000","last_peered":"2024-09-21T08:23:59.784890+0000","last_clean":"2024-09-21T08:23:59.784890+0000","last_became_active":"2024-09-21T08:23:30.551203+0000","last_became_peered":"2024-09-21T08:23:30.551203+0000","last_unstale":"2024-09-21T08:23:59.784890+0000","last_undegraded":"2024-09-21T08:23:59.784890+0000","last_fullsized":"2024-09-21T08:23:59.784890+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:29.708394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598668+0000","last_change":"2024-09-21T08:23:34.569818+0000","last_active":"2024-09-21T08:24:05.598668+0000","last_peered":"2024-09-21T08:24:05.598668+0000","last_clean":"2024-09-21T08:24:05.598668+0000","last_became_active":"2024-09-21T08:23:34.569490+0000","last_became_peered":"2024-09-21T08:23:34.569490+0000","last_unstale":"2024-09-21T08:24:05.598668+0000","last_undegraded":"2024-09-21T08:24:05.598668+0000","last_fullsized":"2024-09-21T08:24:05.598668+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.884010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773123+0000","last_change":"2024-09-21T08:23:36.579253+0000","last_active":"2024-09-21T08:23:59.773123+0000","last_peered":"2024-09-21T08:23:59.773123+0000","last_clean":"2024-09-21T08:23:59.773123+0000","last_became_active":"2024-09-21T08:23:36.578952+0000","last_became_peered":"2024-09-21T08:23:36.578952+0000","last_unstale":"2024-09-21T08:23:59.773123+0000","last_undegraded":"2024-09-21T08:23:59.773123+0000","last_fullsized":"2024-09-21T08:23:59.773123+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.185192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686641+0000","last_change":"2024-09-21T08:23:30.542360+0000","last_active":"2024-09-21T08:23:59.686641+0000","last_peered":"2024-09-21T08:23:59.686641+0000","last_clean":"2024-09-21T08:23:59.686641+0000","last_became_active":"2024-09-21T08:23:30.542060+0000","last_became_peered":"2024-09-21T08:23:30.542060+0000","last_unstale":"2024-09-21T08:23:59.686641+0000","last_undegraded":"2024-09-21T08:23:59.686641+0000","last_fullsized":"2024-09-21T08:23:59.686641+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:58.968344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":36,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006630+0000","last_change":"2024-09-21T08:23:32.598912+0000","last_active":"2024-09-21T08:24:00.006630+0000","last_peered":"2024-09-21T08:24:00.006630+0000","last_clean":"2024-09-21T08:24:00.006630+0000","last_became_active":"2024-09-21T08:23:32.598518+0000","last_became_peered":"2024-09-21T08:23:32.598518+0000","last_unstale":"2024-09-21T08:24:00.006630+0000","last_undegraded":"2024-09-21T08:24:00.006630+0000","last_fullsized":"2024-09-21T08:24:00.006630+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:32.806988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773598+0000","last_change":"2024-09-21T08:23:34.566165+0000","last_active":"2024-09-21T08:23:59.773598+0000","last_peered":"2024-09-21T08:23:59.773598+0000","last_clean":"2024-09-21T08:23:59.773598+0000","last_became_active":"2024-09-21T08:23:34.565662+0000","last_became_peered":"2024-09-21T08:23:34.565662+0000","last_unstale":"2024-09-21T08:23:59.773598+0000","last_undegraded":"2024-09-21T08:23:59.773598+0000","last_fullsized":"2024-09-21T08:23:59.773598+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.290518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784363+0000","last_change":"2024-09-21T08:23:36.579682+0000","last_active":"2024-09-21T08:23:59.784363+0000","last_peered":"2024-09-21T08:23:59.784363+0000","last_clean":"2024-09-21T08:23:59.784363+0000","last_became_active":"2024-09-21T08:23:36.579384+0000","last_became_peered":"2024-09-21T08:23:36.579384+0000","last_unstale":"2024-09-21T08:23:59.784363+0000","last_undegraded":"2024-09-21T08:23:59.784363+0000","last_fullsized":"2024-09-21T08:23:59.784363+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.156840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005265+0000","last_change":"2024-09-21T08:23:32.599079+0000","last_active":"2024-09-21T08:24:00.005265+0000","last_peered":"2024-09-21T08:24:00.005265+0000","last_clean":"2024-09-21T08:24:00.005265+0000","last_became_active":"2024-09-21T08:23:32.598778+0000","last_became_peered":"2024-09-21T08:23:32.598778+0000","last_unstale":"2024-09-21T08:24:00.005265+0000","last_undegraded":"2024-09-21T08:24:00.005265+0000","last_fullsized":"2024-09-21T08:24:00.005265+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.131011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784449+0000","last_change":"2024-09-21T08:23:30.548443+0000","last_active":"2024-09-21T08:23:59.784449+0000","last_peered":"2024-09-21T08:23:59.784449+0000","last_clean":"2024-09-21T08:23:59.784449+0000","last_became_active":"2024-09-21T08:23:30.548096+0000","last_became_peered":"2024-09-21T08:23:30.548096+0000","last_unstale":"2024-09-21T08:23:59.784449+0000","last_undegraded":"2024-09-21T08:23:59.784449+0000","last_fullsized":"2024-09-21T08:23:59.784449+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.029746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006478+0000","last_change":"2024-09-21T08:23:34.566473+0000","last_active":"2024-09-21T08:24:00.006478+0000","last_peered":"2024-09-21T08:24:00.006478+0000","last_clean":"2024-09-21T08:24:00.006478+0000","last_became_active":"2024-09-21T08:23:34.566160+0000","last_became_peered":"2024-09-21T08:23:34.566160+0000","last_unstale":"2024-09-21T08:24:00.006478+0000","last_undegraded":"2024-09-21T08:24:00.006478+0000","last_fullsized":"2024-09-21T08:24:00.006478+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.960353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.781606+0000","last_change":"2024-09-21T08:23:36.587149+0000","last_active":"2024-09-21T08:23:59.781606+0000","last_peered":"2024-09-21T08:23:59.781606+0000","last_clean":"2024-09-21T08:23:59.781606+0000","last_became_active":"2024-09-21T08:23:36.586865+0000","last_became_peered":"2024-09-21T08:23:36.586865+0000","last_unstale":"2024-09-21T08:23:59.781606+0000","last_undegraded":"2024-09-21T08:23:59.781606+0000","last_fullsized":"2024-09-21T08:23:59.781606+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:42.991875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773650+0000","last_change":"2024-09-21T08:23:32.599185+0000","last_active":"2024-09-21T08:23:59.773650+0000","last_peered":"2024-09-21T08:23:59.773650+0000","last_clean":"2024-09-21T08:23:59.773650+0000","last_became_active":"2024-09-21T08:23:32.598748+0000","last_became_peered":"2024-09-21T08:23:32.598748+0000","last_unstale":"2024-09-21T08:23:59.773650+0000","last_undegraded":"2024-09-21T08:23:59.773650+0000","last_fullsized":"2024-09-21T08:23:59.773650+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.141293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005211+0000","last_change":"2024-09-21T08:23:30.548111+0000","last_active":"2024-09-21T08:24:00.005211+0000","last_peered":"2024-09-21T08:24:00.005211+0000","last_clean":"2024-09-21T08:24:00.005211+0000","last_became_active":"2024-09-21T08:23:30.547824+0000","last_became_peered":"2024-09-21T08:23:30.547824+0000","last_unstale":"2024-09-21T08:24:00.005211+0000","last_undegraded":"2024-09-21T08:24:00.005211+0000","last_fullsized":"2024-09-21T08:24:00.005211+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598279+0000","last_change":"2024-09-21T08:23:34.570551+0000","last_active":"2024-09-21T08:24:05.598279+0000","last_peered":"2024-09-21T08:24:05.598279+0000","last_clean":"2024-09-21T08:24:05.598279+0000","last_became_active":"2024-09-21T08:23:34.570211+0000","last_became_peered":"2024-09-21T08:23:34.570211+0000","last_unstale":"2024-09-21T08:24:05.598279+0000","last_undegraded":"2024-09-21T08:24:05.598279+0000","last_fullsized":"2024-09-21T08:24:05.598279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.547474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689308+0000","last_change":"2024-09-21T08:23:36.582684+0000","last_active":"2024-09-21T08:23:59.689308+0000","last_peered":"2024-09-21T08:23:59.689308+0000","last_clean":"2024-09-21T08:23:59.689308+0000","last_became_active":"2024-09-21T08:23:36.582369+0000","last_became_peered":"2024-09-21T08:23:36.582369+0000","last_unstale":"2024-09-21T08:23:59.689308+0000","last_undegraded":"2024-09-21T08:23:59.689308+0000","last_fullsized":"2024-09-21T08:23:59.689308+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:48.140821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":42,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.006743+0000","last_change":"2024-09-21T08:23:32.597630+0000","last_active":"2024-09-21T08:24:00.006743+0000","last_peered":"2024-09-21T08:24:00.006743+0000","last_clean":"2024-09-21T08:24:00.006743+0000","last_became_active":"2024-09-21T08:23:32.597296+0000","last_became_peered":"2024-09-21T08:23:32.597296+0000","last_unstale":"2024-09-21T08:24:00.006743+0000","last_undegraded":"2024-09-21T08:24:00.006743+0000","last_fullsized":"2024-09-21T08:24:00.006743+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.289337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":37,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784650+0000","last_change":"2024-09-21T08:23:30.549864+0000","last_active":"2024-09-21T08:23:59.784650+0000","last_peered":"2024-09-21T08:23:59.784650+0000","last_clean":"2024-09-21T08:23:59.784650+0000","last_became_active":"2024-09-21T08:23:30.549562+0000","last_became_peered":"2024-09-21T08:23:30.549562+0000","last_unstale":"2024-09-21T08:23:59.784650+0000","last_undegraded":"2024-09-21T08:23:59.784650+0000","last_fullsized":"2024-09-21T08:23:59.784650+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.957399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":40,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:05.598624+0000","last_change":"2024-09-21T08:23:34.567042+0000","last_active":"2024-09-21T08:24:05.598624+0000","last_peered":"2024-09-21T08:24:05.598624+0000","last_clean":"2024-09-21T08:24:05.598624+0000","last_became_active":"2024-09-21T08:23:34.566736+0000","last_became_peered":"2024-09-21T08:23:34.566736+0000","last_unstale":"2024-09-21T08:24:05.598624+0000","last_undegraded":"2024-09-21T08:24:05.598624+0000","last_fullsized":"2024-09-21T08:24:05.598624+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004732+0000","last_change":"2024-09-21T08:23:36.589503+0000","last_active":"2024-09-21T08:24:00.004732+0000","last_peered":"2024-09-21T08:24:00.004732+0000","last_clean":"2024-09-21T08:24:00.004732+0000","last_became_active":"2024-09-21T08:23:36.589224+0000","last_became_peered":"2024-09-21T08:23:36.589224+0000","last_unstale":"2024-09-21T08:24:00.004732+0000","last_undegraded":"2024-09-21T08:24:00.004732+0000","last_fullsized":"2024-09-21T08:24:00.004732+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.636183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689040+0000","last_change":"2024-09-21T08:23:32.559882+0000","last_active":"2024-09-21T08:23:59.689040+0000","last_peered":"2024-09-21T08:23:59.689040+0000","last_clean":"2024-09-21T08:23:59.689040+0000","last_became_active":"2024-09-21T08:23:32.559574+0000","last_became_peered":"2024-09-21T08:23:32.559574+0000","last_unstale":"2024-09-21T08:23:59.689040+0000","last_undegraded":"2024-09-21T08:23:59.689040+0000","last_fullsized":"2024-09-21T08:23:59.689040+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.305662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.688789+0000","last_change":"2024-09-21T08:23:30.546837+0000","last_active":"2024-09-21T08:23:59.688789+0000","last_peered":"2024-09-21T08:23:59.688789+0000","last_clean":"2024-09-21T08:23:59.688789+0000","last_became_active":"2024-09-21T08:23:30.546542+0000","last_became_peered":"2024-09-21T08:23:30.546542+0000","last_unstale":"2024-09-21T08:23:59.688789+0000","last_undegraded":"2024-09-21T08:23:59.688789+0000","last_fullsized":"2024-09-21T08:23:59.688789+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689670+0000","last_change":"2024-09-21T08:23:34.571300+0000","last_active":"2024-09-21T08:23:59.689670+0000","last_peered":"2024-09-21T08:23:59.689670+0000","last_clean":"2024-09-21T08:23:59.689670+0000","last_became_active":"2024-09-21T08:23:34.571034+0000","last_became_peered":"2024-09-21T08:23:34.571034+0000","last_unstale":"2024-09-21T08:23:59.689670+0000","last_undegraded":"2024-09-21T08:23:59.689670+0000","last_fullsized":"2024-09-21T08:23:59.689670+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.978453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004703+0000","last_change":"2024-09-21T08:23:36.585748+0000","last_active":"2024-09-21T08:24:00.004703+0000","last_peered":"2024-09-21T08:24:00.004703+0000","last_clean":"2024-09-21T08:24:00.004703+0000","last_became_active":"2024-09-21T08:23:36.585492+0000","last_became_peered":"2024-09-21T08:23:36.585492+0000","last_unstale":"2024-09-21T08:24:00.004703+0000","last_undegraded":"2024-09-21T08:24:00.004703+0000","last_fullsized":"2024-09-21T08:24:00.004703+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.492501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":43,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689913+0000","last_change":"2024-09-21T08:23:32.556550+0000","last_active":"2024-09-21T08:23:59.689913+0000","last_peered":"2024-09-21T08:23:59.689913+0000","last_clean":"2024-09-21T08:23:59.689913+0000","last_became_active":"2024-09-21T08:23:32.556189+0000","last_became_peered":"2024-09-21T08:23:32.556189+0000","last_unstale":"2024-09-21T08:23:59.689913+0000","last_undegraded":"2024-09-21T08:23:59.689913+0000","last_fullsized":"2024-09-21T08:23:59.689913+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:51.354291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784649+0000","last_change":"2024-09-21T08:23:30.549643+0000","last_active":"2024-09-21T08:23:59.784649+0000","last_peered":"2024-09-21T08:23:59.784649+0000","last_clean":"2024-09-21T08:23:59.784649+0000","last_became_active":"2024-09-21T08:23:30.549317+0000","last_became_peered":"2024-09-21T08:23:30.549317+0000","last_unstale":"2024-09-21T08:23:59.784649+0000","last_undegraded":"2024-09-21T08:23:59.784649+0000","last_fullsized":"2024-09-21T08:23:59.784649+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.895759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005199+0000","last_change":"2024-09-21T08:23:34.565128+0000","last_active":"2024-09-21T08:24:00.005199+0000","last_peered":"2024-09-21T08:24:00.005199+0000","last_clean":"2024-09-21T08:24:00.005199+0000","last_became_active":"2024-09-21T08:23:34.564738+0000","last_became_peered":"2024-09-21T08:23:34.564738+0000","last_unstale":"2024-09-21T08:24:00.005199+0000","last_undegraded":"2024-09-21T08:24:00.005199+0000","last_fullsized":"2024-09-21T08:24:00.005199+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.717622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004625+0000","last_change":"2024-09-21T08:23:36.580460+0000","last_active":"2024-09-21T08:24:00.004625+0000","last_peered":"2024-09-21T08:24:00.004625+0000","last_clean":"2024-09-21T08:24:00.004625+0000","last_became_active":"2024-09-21T08:23:36.579949+0000","last_became_peered":"2024-09-21T08:23:36.579949+0000","last_unstale":"2024-09-21T08:24:00.004625+0000","last_undegraded":"2024-09-21T08:24:00.004625+0000","last_fullsized":"2024-09-21T08:24:00.004625+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.377442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004773+0000","last_change":"2024-09-21T08:23:32.597672+0000","last_active":"2024-09-21T08:24:00.004773+0000","last_peered":"2024-09-21T08:24:00.004773+0000","last_clean":"2024-09-21T08:24:00.004773+0000","last_became_active":"2024-09-21T08:23:32.597281+0000","last_became_peered":"2024-09-21T08:23:32.597281+0000","last_unstale":"2024-09-21T08:24:00.004773+0000","last_undegraded":"2024-09-21T08:24:00.004773+0000","last_fullsized":"2024-09-21T08:24:00.004773+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.493132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.773659+0000","last_change":"2024-09-21T08:23:30.544860+0000","last_active":"2024-09-21T08:23:59.773659+0000","last_peered":"2024-09-21T08:23:59.773659+0000","last_clean":"2024-09-21T08:23:59.773659+0000","last_became_active":"2024-09-21T08:23:30.544486+0000","last_became_peered":"2024-09-21T08:23:30.544486+0000","last_unstale":"2024-09-21T08:23:59.773659+0000","last_undegraded":"2024-09-21T08:23:59.773659+0000","last_fullsized":"2024-09-21T08:23:59.773659+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.462170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784050+0000","last_change":"2024-09-21T08:23:34.569813+0000","last_active":"2024-09-21T08:23:59.784050+0000","last_peered":"2024-09-21T08:23:59.784050+0000","last_clean":"2024-09-21T08:23:59.784050+0000","last_became_active":"2024-09-21T08:23:34.569485+0000","last_became_peered":"2024-09-21T08:23:34.569485+0000","last_unstale":"2024-09-21T08:23:59.784050+0000","last_undegraded":"2024-09-21T08:23:59.784050+0000","last_fullsized":"2024-09-21T08:23:59.784050+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.575850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005340+0000","last_change":"2024-09-21T08:23:36.586500+0000","last_active":"2024-09-21T08:24:00.005340+0000","last_peered":"2024-09-21T08:24:00.005340+0000","last_clean":"2024-09-21T08:24:00.005340+0000","last_became_active":"2024-09-21T08:23:36.586172+0000","last_became_peered":"2024-09-21T08:23:36.586172+0000","last_unstale":"2024-09-21T08:24:00.005340+0000","last_undegraded":"2024-09-21T08:24:00.005340+0000","last_fullsized":"2024-09-21T08:24:00.005340+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.388100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":33,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004429+0000","last_change":"2024-09-21T08:23:32.563756+0000","last_active":"2024-09-21T08:24:00.004429+0000","last_peered":"2024-09-21T08:24:00.004429+0000","last_clean":"2024-09-21T08:24:00.004429+0000","last_became_active":"2024-09-21T08:23:32.563484+0000","last_became_peered":"2024-09-21T08:23:32.563484+0000","last_unstale":"2024-09-21T08:24:00.004429+0000","last_undegraded":"2024-09-21T08:24:00.004429+0000","last_fullsized":"2024-09-21T08:24:00.004429+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.069344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.005806+0000","last_change":"2024-09-21T08:23:30.544403+0000","last_active":"2024-09-21T08:24:00.005806+0000","last_peered":"2024-09-21T08:24:00.005806+0000","last_clean":"2024-09-21T08:24:00.005806+0000","last_became_active":"2024-09-21T08:23:30.544066+0000","last_became_peered":"2024-09-21T08:23:30.544066+0000","last_unstale":"2024-09-21T08:24:00.005806+0000","last_undegraded":"2024-09-21T08:24:00.005806+0000","last_fullsized":"2024-09-21T08:24:00.005806+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.462796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686256+0000","last_change":"2024-09-21T08:23:34.566607+0000","last_active":"2024-09-21T08:23:59.686256+0000","last_peered":"2024-09-21T08:23:59.686256+0000","last_clean":"2024-09-21T08:23:59.686256+0000","last_became_active":"2024-09-21T08:23:34.566337+0000","last_became_peered":"2024-09-21T08:23:34.566337+0000","last_unstale":"2024-09-21T08:23:59.686256+0000","last_undegraded":"2024-09-21T08:23:59.686256+0000","last_fullsized":"2024-09-21T08:23:59.686256+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.230638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.784194+0000","last_change":"2024-09-21T08:23:36.582773+0000","last_active":"2024-09-21T08:23:59.784194+0000","last_peered":"2024-09-21T08:23:59.784194+0000","last_clean":"2024-09-21T08:23:59.784194+0000","last_became_active":"2024-09-21T08:23:36.582150+0000","last_became_peered":"2024-09-21T08:23:36.582150+0000","last_unstale":"2024-09-21T08:23:59.784194+0000","last_undegraded":"2024-09-21T08:23:59.784194+0000","last_fullsized":"2024-09-21T08:23:59.784194+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.263690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":19,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.689860+0000","last_change":"2024-09-21T08:23:36.576004+0000","last_active":"2024-09-21T08:23:59.689860+0000","last_peered":"2024-09-21T08:23:59.689860+0000","last_clean":"2024-09-21T08:23:59.689860+0000","last_became_active":"2024-09-21T08:23:36.575701+0000","last_became_peered":"2024-09-21T08:23:36.575701+0000","last_unstale":"2024-09-21T08:23:59.689860+0000","last_undegraded":"2024-09-21T08:23:59.689860+0000","last_fullsized":"2024-09-21T08:23:59.689860+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.955927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":31,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004607+0000","last_change":"2024-09-21T08:23:30.544201+0000","last_active":"2024-09-21T08:24:00.004607+0000","last_peered":"2024-09-21T08:24:00.004607+0000","last_clean":"2024-09-21T08:24:00.004607+0000","last_became_active":"2024-09-21T08:23:30.543932+0000","last_became_peered":"2024-09-21T08:23:30.543932+0000","last_unstale":"2024-09-21T08:24:00.004607+0000","last_undegraded":"2024-09-21T08:24:00.004607+0000","last_fullsized":"2024-09-21T08:24:00.004607+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.933119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":46,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:23:59.686986+0000","last_change":"2024-09-21T08:23:32.549263+0000","last_active":"2024-09-21T08:23:59.686986+0000","last_peered":"2024-09-21T08:23:59.686986+0000","last_clean":"2024-09-21T08:23:59.686986+0000","last_became_active":"2024-09-21T08:23:32.548835+0000","last_became_peered":"2024-09-21T08:23:32.548835+0000","last_unstale":"2024-09-21T08:23:59.686986+0000","last_undegraded":"2024-09-21T08:23:59.686986+0000","last_fullsized":"2024-09-21T08:23:59.686986+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.423215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":23,"reported_epoch":76,"state":"active+clean","last_fresh":"2024-09-21T08:24:00.004910+0000","last_change":"2024-09-21T08:23:34.573167+0000","last_active":"2024-09-21T08:24:00.004910+0000","last_peered":"2024-09-21T08:24:00.004910+0000","last_clean":"2024-09-21T08:24:00.004910+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:24:00.004910+0000","last_undegraded":"2024-09-21T08:24:00.004910+0000","last_fullsized":"2024-09-21T08:24:00.004910+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.897951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":35,"num_read_kb":30,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070477,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32400,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712752,"statfs":{"total":95995035648,"available":95961858048,"internally_reserved":0,"allocated":1245184,"data_stored":863237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397523,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28216,"kb_used_data":1192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716936,"statfs":{"total":95995035648,"available":95966142464,"internally_reserved":0,"allocated":1220608,"data_stored":861159,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659158,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27652,"kb_used_data":624,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717500,"statfs":{"total":95995035648,"available":95966720000,"internally_reserved":0,"allocated":638976,"data_stored":271599,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986204,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31832,"kb_used_data":644,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713320,"statfs":{"total":95995035648,"available":95962439680,"internally_reserved":0,"allocated":659456,"data_stored":272389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182432,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27684,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717468,"statfs":{"total":95995035648,"available":95966687232,"internally_reserved":0,"allocated":671744,"data_stored":272131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411366,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27644,"kb_used_data":616,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717508,"statfs":{"total":95995035648,"available":95966728192,"internally_reserved":0,"allocated":630784,"data_stored":272368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574890,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27708,"kb_used_data":680,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717444,"statfs":{"total":95995035648,"available":95966662656,"internally_reserved":0,"allocated":696320,"data_stored":272589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738415,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28240,"kb_used_data":1216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716912,"statfs":{"total":95995035648,"available":95966117888,"internally_reserved":0,"allocated":1245184,"data_stored":861953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:24:15.099 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T08:24:15.099 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-09-21T08:24:15.099 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-09-21T08:24:15.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph health --format=json 2024-09-21T08:24:15.358 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:15.736 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:15 smithi137 ceph-mon[29634]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T08:24:15.736 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:15 smithi137 ceph-mon[29634]: from='client.44657 v1:172.21.15.137:0/4061004404' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:15.736 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:15 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:15.736 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:15 smithi137 ceph-mon[41044]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T08:24:15.736 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:15 smithi137 ceph-mon[41044]: from='client.44657 v1:172.21.15.137:0/4061004404' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:15.736 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:15 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:15 smithi165 ceph-mon[32893]: pgmap v11: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 22 KiB/s rd, 0 B/s wr, 12 op/s 2024-09-21T08:24:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:15 smithi165 ceph-mon[32893]: from='client.44657 v1:172.21.15.137:0/4061004404' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:15 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:16.193 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:24:16.193 INFO:teuthology.orchestra.run.smithi137.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-21T08:24:16.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:16 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3286425905' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T08:24:16.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:16 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3286425905' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T08:24:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:16 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3286425905' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-09-21T08:24:17.365 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-09-21T08:24:17.365 INFO:tasks.cephadm:Setup complete, yielding 2024-09-21T08:24:17.365 INFO:teuthology.run_tasks:Running task thrashosds... 2024-09-21T08:24:17.374 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T08:24:17.374 INFO:tasks.thrashosds:overrides is {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'global': {'ms bind msgr2': False, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon min osdmap epochs': 50, 'mon osdmap full prune interval': 2, 'mon osdmap full prune min': 15, 'mon osdmap full prune txsize': 2, 'paxos service trim min': 10}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd debug reject backfill probability': 0.3, 'osd delete sleep': 1, 'osd max backfills': 3, 'osd scrub max interval': 120, 'osd scrub min interval': 60, 'osd shutdown pgref assert': True, 'osd snap trim sleep': 2}}, 'flavor': 'default', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'CEPHADM_STRAY_DAEMON', 'CEPHADM_FAILED_DAEMON', 'but it is still running', 'objects unfound and apparently lost', '\\(POOL_APP_NOT_ENABLED\\)'], 'log-only-match': ['CEPHADM_'], 'mon_bind_msgr2': False, 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'cephadm': {'cephadm_mode': 'root'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '9e2ac70bb7aea67d34f0d50002344abbe7803736'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'workunit': {'branch': 'squid', 'sha1': '4e45e6b490e6cbe082cdf155fea780438ec53fc7'}} 2024-09-21T08:24:17.375 INFO:tasks.thrashosds:config is {'chance_pgnum_grow': 1, 'chance_pgnum_shrink': 1, 'chance_pgpnum_fix': 1, 'timeout': 1200, 'sighup_delay': 0.1, 'optrack_toggle_delay': 2.0, 'dump_ops_enable': 'true', 'noscrub_toggle_delay': 2.0, 'random_eio': 0.0} 2024-09-21T08:24:17.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph config set mgr mgr_debug_aggressive_pg_num_changes true 2024-09-21T08:24:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:17 smithi137 ceph-mon[29634]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:17 smithi137 ceph-mon[41044]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:17 smithi165 ceph-mon[32893]: pgmap v12: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:18.421 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:19.517 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:19 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:19] ENGINE Bus STOPPING 2024-09-21T08:24:19.787 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:19 smithi137 ceph-mon[29634]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:19.787 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:19 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3062256251' entity='client.admin' 2024-09-21T08:24:19.787 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:19 smithi137 ceph-mon[41044]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:19.787 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:19 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3062256251' entity='client.admin' 2024-09-21T08:24:19.787 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:19 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:19] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T08:24:19.787 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:19 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:19] ENGINE Bus STOPPED 2024-09-21T08:24:19.788 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:19 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:19] ENGINE Bus STARTING 2024-09-21T08:24:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:19 smithi165 ceph-mon[32893]: pgmap v13: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:19 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3062256251' entity='client.admin' 2024-09-21T08:24:20.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:19 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:19] ENGINE Serving on http://:::9283 2024-09-21T08:24:20.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:19 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:19] ENGINE Bus STARTED 2024-09-21T08:24:20.772 INFO:tasks.thrashosds:Beginning thrashosds... 2024-09-21T08:24:20.772 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-09-21T08:24:20.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:24:21.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:20 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:24:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:24:21.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:21 smithi137 ceph-mon[29634]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:21.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:21 smithi137 ceph-mon[41044]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:22.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:21 smithi165 ceph-mon[32893]: pgmap v14: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:22.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:24:21 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:24:22.682 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:22.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:22 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:22.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:22 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:22.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:22 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:23.483 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:24:23.485 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:24:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:23 smithi137 ceph-mon[29634]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:23 smithi137 ceph-mon[41044]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:23 smithi165 ceph-mon[32893]: pgmap v15: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1023 B/s rd, 0 op/s 2024-09-21T08:24:24.795 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:24 smithi137 ceph-mon[29634]: from='client.14784 v1:172.21.15.137:0/2601658319' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:24.795 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:24 smithi137 ceph-mon[41044]: from='client.14784 v1:172.21.15.137:0/2601658319' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:24 smithi165 ceph-mon[32893]: from='client.14784 v1:172.21.15.137:0/2601658319' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:25.440 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:25] ENGINE Bus STOPPING 2024-09-21T08:24:25.440 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:25] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T08:24:25.440 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:25] ENGINE Bus STOPPED 2024-09-21T08:24:25.440 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:25] ENGINE Bus STARTING 2024-09-21T08:24:25.440 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:25] ENGINE Serving on http://:::9283 2024-09-21T08:24:25.441 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:24:25] ENGINE Bus STARTED 2024-09-21T08:24:25.468 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":15,"stamp":"2024-09-21T08:24:21.705336+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":979,"num_read_kb":825,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":602,"ondisk_log_size":602,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":231484,"kb_used_data":6940,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749729732,"statfs":{"total":767960285184,"available":767723245568,"internally_reserved":0,"allocated":7106560,"data_stored":3998801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002702"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508268+0000","last_change":"2024-09-21T08:23:30.549696+0000","last_active":"2024-09-21T08:24:13.508268+0000","last_peered":"2024-09-21T08:24:13.508268+0000","last_clean":"2024-09-21T08:24:13.508268+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:24:13.508268+0000","last_undegraded":"2024-09-21T08:24:13.508268+0000","last_fullsized":"2024-09-21T08:24:13.508268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.197261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.511847+0000","last_change":"2024-09-21T08:23:32.556768+0000","last_active":"2024-09-21T08:24:13.511847+0000","last_peered":"2024-09-21T08:24:13.511847+0000","last_clean":"2024-09-21T08:24:13.511847+0000","last_became_active":"2024-09-21T08:23:32.556452+0000","last_became_peered":"2024-09-21T08:23:32.556452+0000","last_unstale":"2024-09-21T08:24:13.511847+0000","last_undegraded":"2024-09-21T08:24:13.511847+0000","last_fullsized":"2024-09-21T08:24:13.511847+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.936229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879709+0000","last_change":"2024-09-21T08:23:34.569082+0000","last_active":"2024-09-21T08:24:13.879709+0000","last_peered":"2024-09-21T08:24:13.879709+0000","last_clean":"2024-09-21T08:24:13.879709+0000","last_became_active":"2024-09-21T08:23:34.568762+0000","last_became_peered":"2024-09-21T08:23:34.568762+0000","last_unstale":"2024-09-21T08:24:13.879709+0000","last_undegraded":"2024-09-21T08:24:13.879709+0000","last_fullsized":"2024-09-21T08:24:13.879709+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.577218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.511973+0000","last_change":"2024-09-21T08:23:36.575623+0000","last_active":"2024-09-21T08:24:13.511973+0000","last_peered":"2024-09-21T08:24:13.511973+0000","last_clean":"2024-09-21T08:24:13.511973+0000","last_became_active":"2024-09-21T08:23:36.572359+0000","last_became_peered":"2024-09-21T08:23:36.572359+0000","last_unstale":"2024-09-21T08:24:13.511973+0000","last_undegraded":"2024-09-21T08:24:13.511973+0000","last_fullsized":"2024-09-21T08:24:13.511973+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.062906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880039+0000","last_change":"2024-09-21T08:23:36.585557+0000","last_active":"2024-09-21T08:24:13.880039+0000","last_peered":"2024-09-21T08:24:13.880039+0000","last_clean":"2024-09-21T08:24:13.880039+0000","last_became_active":"2024-09-21T08:23:36.584820+0000","last_became_peered":"2024-09-21T08:23:36.584820+0000","last_unstale":"2024-09-21T08:24:13.880039+0000","last_undegraded":"2024-09-21T08:24:13.880039+0000","last_fullsized":"2024-09-21T08:24:13.880039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.509136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881615+0000","last_change":"2024-09-21T08:23:30.548436+0000","last_active":"2024-09-21T08:24:13.881615+0000","last_peered":"2024-09-21T08:24:13.881615+0000","last_clean":"2024-09-21T08:24:13.881615+0000","last_became_active":"2024-09-21T08:23:30.548146+0000","last_became_peered":"2024-09-21T08:23:30.548146+0000","last_unstale":"2024-09-21T08:24:13.881615+0000","last_undegraded":"2024-09-21T08:24:13.881615+0000","last_fullsized":"2024-09-21T08:24:13.881615+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.518727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880248+0000","last_change":"2024-09-21T08:23:32.556169+0000","last_active":"2024-09-21T08:24:13.880248+0000","last_peered":"2024-09-21T08:24:13.880248+0000","last_clean":"2024-09-21T08:24:13.880248+0000","last_became_active":"2024-09-21T08:23:32.555789+0000","last_became_peered":"2024-09-21T08:23:32.555789+0000","last_unstale":"2024-09-21T08:24:13.880248+0000","last_undegraded":"2024-09-21T08:24:13.880248+0000","last_fullsized":"2024-09-21T08:24:13.880248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.359444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.511839+0000","last_change":"2024-09-21T08:23:34.565614+0000","last_active":"2024-09-21T08:24:13.511839+0000","last_peered":"2024-09-21T08:24:13.511839+0000","last_clean":"2024-09-21T08:24:13.511839+0000","last_became_active":"2024-09-21T08:23:34.565276+0000","last_became_peered":"2024-09-21T08:23:34.565276+0000","last_unstale":"2024-09-21T08:24:13.511839+0000","last_undegraded":"2024-09-21T08:24:13.511839+0000","last_fullsized":"2024-09-21T08:24:13.511839+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.926230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880265+0000","last_change":"2024-09-21T08:23:30.543643+0000","last_active":"2024-09-21T08:24:13.880265+0000","last_peered":"2024-09-21T08:24:13.880265+0000","last_clean":"2024-09-21T08:24:13.880265+0000","last_became_active":"2024-09-21T08:23:30.543041+0000","last_became_peered":"2024-09-21T08:23:30.543041+0000","last_unstale":"2024-09-21T08:24:13.880265+0000","last_undegraded":"2024-09-21T08:24:13.880265+0000","last_fullsized":"2024-09-21T08:24:13.880265+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.229786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512124+0000","last_change":"2024-09-21T08:23:32.559542+0000","last_active":"2024-09-21T08:24:13.512124+0000","last_peered":"2024-09-21T08:24:13.512124+0000","last_clean":"2024-09-21T08:24:13.512124+0000","last_became_active":"2024-09-21T08:23:32.559194+0000","last_became_peered":"2024-09-21T08:23:32.559194+0000","last_unstale":"2024-09-21T08:24:13.512124+0000","last_undegraded":"2024-09-21T08:24:13.512124+0000","last_fullsized":"2024-09-21T08:24:13.512124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.251123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879717+0000","last_change":"2024-09-21T08:23:34.567872+0000","last_active":"2024-09-21T08:24:13.879717+0000","last_peered":"2024-09-21T08:24:13.879717+0000","last_clean":"2024-09-21T08:24:13.879717+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:24:13.879717+0000","last_undegraded":"2024-09-21T08:24:13.879717+0000","last_fullsized":"2024-09-21T08:24:13.879717+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.208597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.507725+0000","last_change":"2024-09-21T08:23:36.582491+0000","last_active":"2024-09-21T08:24:13.507725+0000","last_peered":"2024-09-21T08:24:13.507725+0000","last_clean":"2024-09-21T08:24:13.507725+0000","last_became_active":"2024-09-21T08:23:36.581720+0000","last_became_peered":"2024-09-21T08:23:36.581720+0000","last_unstale":"2024-09-21T08:24:13.507725+0000","last_undegraded":"2024-09-21T08:24:13.507725+0000","last_fullsized":"2024-09-21T08:24:13.507725+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879889+0000","last_change":"2024-09-21T08:23:30.543477+0000","last_active":"2024-09-21T08:24:13.879889+0000","last_peered":"2024-09-21T08:24:13.879889+0000","last_clean":"2024-09-21T08:24:13.879889+0000","last_became_active":"2024-09-21T08:23:30.543025+0000","last_became_peered":"2024-09-21T08:23:30.543025+0000","last_unstale":"2024-09-21T08:24:13.879889+0000","last_undegraded":"2024-09-21T08:24:13.879889+0000","last_fullsized":"2024-09-21T08:24:13.879889+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.290639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512026+0000","last_change":"2024-09-21T08:23:32.559984+0000","last_active":"2024-09-21T08:24:13.512026+0000","last_peered":"2024-09-21T08:24:13.512026+0000","last_clean":"2024-09-21T08:24:13.512026+0000","last_became_active":"2024-09-21T08:23:32.559544+0000","last_became_peered":"2024-09-21T08:23:32.559544+0000","last_unstale":"2024-09-21T08:24:13.512026+0000","last_undegraded":"2024-09-21T08:24:13.512026+0000","last_fullsized":"2024-09-21T08:24:13.512026+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:31.803588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880065+0000","last_change":"2024-09-21T08:23:34.572687+0000","last_active":"2024-09-21T08:24:13.880065+0000","last_peered":"2024-09-21T08:24:13.880065+0000","last_clean":"2024-09-21T08:24:13.880065+0000","last_became_active":"2024-09-21T08:23:34.572395+0000","last_became_peered":"2024-09-21T08:23:34.572395+0000","last_unstale":"2024-09-21T08:24:13.880065+0000","last_undegraded":"2024-09-21T08:24:13.880065+0000","last_fullsized":"2024-09-21T08:24:13.880065+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.042108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879431+0000","last_change":"2024-09-21T08:23:36.578510+0000","last_active":"2024-09-21T08:24:13.879431+0000","last_peered":"2024-09-21T08:24:13.879431+0000","last_clean":"2024-09-21T08:24:13.879431+0000","last_became_active":"2024-09-21T08:23:36.578227+0000","last_became_peered":"2024-09-21T08:23:36.578227+0000","last_unstale":"2024-09-21T08:24:13.879431+0000","last_undegraded":"2024-09-21T08:24:13.879431+0000","last_fullsized":"2024-09-21T08:24:13.879431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.087695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512303+0000","last_change":"2024-09-21T08:23:36.573671+0000","last_active":"2024-09-21T08:24:13.512303+0000","last_peered":"2024-09-21T08:24:13.512303+0000","last_clean":"2024-09-21T08:24:13.512303+0000","last_became_active":"2024-09-21T08:23:36.573384+0000","last_became_peered":"2024-09-21T08:23:36.573384+0000","last_unstale":"2024-09-21T08:24:13.512303+0000","last_undegraded":"2024-09-21T08:24:13.512303+0000","last_fullsized":"2024-09-21T08:24:13.512303+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.273680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508422+0000","last_change":"2024-09-21T08:23:30.543356+0000","last_active":"2024-09-21T08:24:13.508422+0000","last_peered":"2024-09-21T08:24:13.508422+0000","last_clean":"2024-09-21T08:24:13.508422+0000","last_became_active":"2024-09-21T08:23:30.542776+0000","last_became_peered":"2024-09-21T08:23:30.542776+0000","last_unstale":"2024-09-21T08:24:13.508422+0000","last_undegraded":"2024-09-21T08:24:13.508422+0000","last_fullsized":"2024-09-21T08:24:13.508422+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.314435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881662+0000","last_change":"2024-09-21T08:23:32.556660+0000","last_active":"2024-09-21T08:24:13.881662+0000","last_peered":"2024-09-21T08:24:13.881662+0000","last_clean":"2024-09-21T08:24:13.881662+0000","last_became_active":"2024-09-21T08:23:32.556378+0000","last_became_peered":"2024-09-21T08:23:32.556378+0000","last_unstale":"2024-09-21T08:24:13.881662+0000","last_undegraded":"2024-09-21T08:24:13.881662+0000","last_fullsized":"2024-09-21T08:24:13.881662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.143529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880911+0000","last_change":"2024-09-21T08:23:34.569008+0000","last_active":"2024-09-21T08:24:13.880911+0000","last_peered":"2024-09-21T08:24:13.880911+0000","last_clean":"2024-09-21T08:24:13.880911+0000","last_became_active":"2024-09-21T08:23:34.568562+0000","last_became_peered":"2024-09-21T08:23:34.568562+0000","last_unstale":"2024-09-21T08:24:13.880911+0000","last_undegraded":"2024-09-21T08:24:13.880911+0000","last_fullsized":"2024-09-21T08:24:13.880911+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.511849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880329+0000","last_change":"2024-09-21T08:23:36.584447+0000","last_active":"2024-09-21T08:24:13.880329+0000","last_peered":"2024-09-21T08:24:13.880329+0000","last_clean":"2024-09-21T08:24:13.880329+0000","last_became_active":"2024-09-21T08:23:36.584154+0000","last_became_peered":"2024-09-21T08:23:36.584154+0000","last_unstale":"2024-09-21T08:24:13.880329+0000","last_undegraded":"2024-09-21T08:24:13.880329+0000","last_fullsized":"2024-09-21T08:24:13.880329+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.079729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512274+0000","last_change":"2024-09-21T08:23:30.547824+0000","last_active":"2024-09-21T08:24:13.512274+0000","last_peered":"2024-09-21T08:24:13.512274+0000","last_clean":"2024-09-21T08:24:13.512274+0000","last_became_active":"2024-09-21T08:23:30.546325+0000","last_became_peered":"2024-09-21T08:23:30.546325+0000","last_unstale":"2024-09-21T08:24:13.512274+0000","last_undegraded":"2024-09-21T08:24:13.512274+0000","last_fullsized":"2024-09-21T08:24:13.512274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.394437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880052+0000","last_change":"2024-09-21T08:23:32.563650+0000","last_active":"2024-09-21T08:24:13.880052+0000","last_peered":"2024-09-21T08:24:13.880052+0000","last_clean":"2024-09-21T08:24:13.880052+0000","last_became_active":"2024-09-21T08:23:32.563383+0000","last_became_peered":"2024-09-21T08:23:32.563383+0000","last_unstale":"2024-09-21T08:24:13.880052+0000","last_undegraded":"2024-09-21T08:24:13.880052+0000","last_fullsized":"2024-09-21T08:24:13.880052+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.102032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512155+0000","last_change":"2024-09-21T08:23:34.571589+0000","last_active":"2024-09-21T08:24:13.512155+0000","last_peered":"2024-09-21T08:24:13.512155+0000","last_clean":"2024-09-21T08:24:13.512155+0000","last_became_active":"2024-09-21T08:23:34.571323+0000","last_became_peered":"2024-09-21T08:23:34.571323+0000","last_unstale":"2024-09-21T08:24:13.512155+0000","last_undegraded":"2024-09-21T08:24:13.512155+0000","last_fullsized":"2024-09-21T08:24:13.512155+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.905411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880474+0000","last_change":"2024-09-21T08:23:36.584003+0000","last_active":"2024-09-21T08:24:13.880474+0000","last_peered":"2024-09-21T08:24:13.880474+0000","last_clean":"2024-09-21T08:24:13.880474+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:24:13.880474+0000","last_undegraded":"2024-09-21T08:24:13.880474+0000","last_fullsized":"2024-09-21T08:24:13.880474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.633904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879860+0000","last_change":"2024-09-21T08:23:30.540306+0000","last_active":"2024-09-21T08:24:13.879860+0000","last_peered":"2024-09-21T08:24:13.879860+0000","last_clean":"2024-09-21T08:24:13.879860+0000","last_became_active":"2024-09-21T08:23:30.539931+0000","last_became_peered":"2024-09-21T08:23:30.539931+0000","last_unstale":"2024-09-21T08:24:13.879860+0000","last_undegraded":"2024-09-21T08:24:13.879860+0000","last_fullsized":"2024-09-21T08:24:13.879860+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.106584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508651+0000","last_change":"2024-09-21T08:23:32.598036+0000","last_active":"2024-09-21T08:24:13.508651+0000","last_peered":"2024-09-21T08:24:13.508651+0000","last_clean":"2024-09-21T08:24:13.508651+0000","last_became_active":"2024-09-21T08:23:32.597548+0000","last_became_peered":"2024-09-21T08:23:32.597548+0000","last_unstale":"2024-09-21T08:24:13.508651+0000","last_undegraded":"2024-09-21T08:24:13.508651+0000","last_fullsized":"2024-09-21T08:24:13.508651+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.955005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":50,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:15.600462+0000","last_change":"2024-09-21T08:23:34.567530+0000","last_active":"2024-09-21T08:24:15.600462+0000","last_peered":"2024-09-21T08:24:15.600462+0000","last_clean":"2024-09-21T08:24:15.600462+0000","last_became_active":"2024-09-21T08:23:34.567257+0000","last_became_peered":"2024-09-21T08:23:34.567257+0000","last_unstale":"2024-09-21T08:24:15.600462+0000","last_undegraded":"2024-09-21T08:24:15.600462+0000","last_fullsized":"2024-09-21T08:24:15.600462+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.508940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881320+0000","last_change":"2024-09-21T08:23:32.562879+0000","last_active":"2024-09-21T08:24:13.881320+0000","last_peered":"2024-09-21T08:24:13.881320+0000","last_clean":"2024-09-21T08:24:13.881320+0000","last_became_active":"2024-09-21T08:23:32.562594+0000","last_became_peered":"2024-09-21T08:23:32.562594+0000","last_unstale":"2024-09-21T08:24:13.881320+0000","last_undegraded":"2024-09-21T08:24:13.881320+0000","last_fullsized":"2024-09-21T08:24:13.881320+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.780782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879940+0000","last_change":"2024-09-21T08:23:30.546900+0000","last_active":"2024-09-21T08:24:13.879940+0000","last_peered":"2024-09-21T08:24:13.879940+0000","last_clean":"2024-09-21T08:24:13.879940+0000","last_became_active":"2024-09-21T08:23:30.546625+0000","last_became_peered":"2024-09-21T08:23:30.546625+0000","last_unstale":"2024-09-21T08:24:13.879940+0000","last_undegraded":"2024-09-21T08:24:13.879940+0000","last_fullsized":"2024-09-21T08:24:13.879940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:48.387469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:15.600442+0000","last_change":"2024-09-21T08:23:34.568257+0000","last_active":"2024-09-21T08:24:15.600442+0000","last_peered":"2024-09-21T08:24:15.600442+0000","last_clean":"2024-09-21T08:24:15.600442+0000","last_became_active":"2024-09-21T08:23:34.567963+0000","last_became_peered":"2024-09-21T08:23:34.567963+0000","last_unstale":"2024-09-21T08:24:15.600442+0000","last_undegraded":"2024-09-21T08:24:15.600442+0000","last_fullsized":"2024-09-21T08:24:15.600442+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:43.930223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879810+0000","last_change":"2024-09-21T08:23:36.578919+0000","last_active":"2024-09-21T08:24:13.879810+0000","last_peered":"2024-09-21T08:24:13.879810+0000","last_clean":"2024-09-21T08:24:13.879810+0000","last_became_active":"2024-09-21T08:23:36.578648+0000","last_became_peered":"2024-09-21T08:23:36.578648+0000","last_unstale":"2024-09-21T08:24:13.879810+0000","last_undegraded":"2024-09-21T08:24:13.879810+0000","last_fullsized":"2024-09-21T08:24:13.879810+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.799668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":61,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879717+0000","last_change":"2024-09-21T08:23:32.597719+0000","last_active":"2024-09-21T08:24:13.879717+0000","last_peered":"2024-09-21T08:24:13.879717+0000","last_clean":"2024-09-21T08:24:13.879717+0000","last_became_active":"2024-09-21T08:23:32.597431+0000","last_became_peered":"2024-09-21T08:23:32.597431+0000","last_unstale":"2024-09-21T08:24:13.879717+0000","last_undegraded":"2024-09-21T08:24:13.879717+0000","last_fullsized":"2024-09-21T08:24:13.879717+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.130776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880512+0000","last_change":"2024-09-21T08:23:30.546233+0000","last_active":"2024-09-21T08:24:13.880512+0000","last_peered":"2024-09-21T08:24:13.880512+0000","last_clean":"2024-09-21T08:24:13.880512+0000","last_became_active":"2024-09-21T08:23:30.545922+0000","last_became_peered":"2024-09-21T08:23:30.545922+0000","last_unstale":"2024-09-21T08:24:13.880512+0000","last_undegraded":"2024-09-21T08:24:13.880512+0000","last_fullsized":"2024-09-21T08:24:13.880512+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.861966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.511851+0000","last_change":"2024-09-21T08:23:34.570926+0000","last_active":"2024-09-21T08:24:13.511851+0000","last_peered":"2024-09-21T08:24:13.511851+0000","last_clean":"2024-09-21T08:24:13.511851+0000","last_became_active":"2024-09-21T08:23:34.570648+0000","last_became_peered":"2024-09-21T08:23:34.570648+0000","last_unstale":"2024-09-21T08:24:13.511851+0000","last_undegraded":"2024-09-21T08:24:13.511851+0000","last_fullsized":"2024-09-21T08:24:13.511851+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.476055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512075+0000","last_change":"2024-09-21T08:23:36.582947+0000","last_active":"2024-09-21T08:24:13.512075+0000","last_peered":"2024-09-21T08:24:13.512075+0000","last_clean":"2024-09-21T08:24:13.512075+0000","last_became_active":"2024-09-21T08:23:36.582610+0000","last_became_peered":"2024-09-21T08:23:36.582610+0000","last_unstale":"2024-09-21T08:24:13.512075+0000","last_undegraded":"2024-09-21T08:24:13.512075+0000","last_fullsized":"2024-09-21T08:24:13.512075+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.387653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":50,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.511887+0000","last_change":"2024-09-21T08:23:32.553440+0000","last_active":"2024-09-21T08:24:13.511887+0000","last_peered":"2024-09-21T08:24:13.511887+0000","last_clean":"2024-09-21T08:24:13.511887+0000","last_became_active":"2024-09-21T08:23:32.552980+0000","last_became_peered":"2024-09-21T08:23:32.552980+0000","last_unstale":"2024-09-21T08:24:13.511887+0000","last_undegraded":"2024-09-21T08:24:13.511887+0000","last_fullsized":"2024-09-21T08:24:13.511887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.960840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508904+0000","last_change":"2024-09-21T08:23:30.543176+0000","last_active":"2024-09-21T08:24:13.508904+0000","last_peered":"2024-09-21T08:24:13.508904+0000","last_clean":"2024-09-21T08:24:13.508904+0000","last_became_active":"2024-09-21T08:23:30.542888+0000","last_became_peered":"2024-09-21T08:23:30.542888+0000","last_unstale":"2024-09-21T08:24:13.508904+0000","last_undegraded":"2024-09-21T08:24:13.508904+0000","last_fullsized":"2024-09-21T08:24:13.508904+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.545006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:15.600279+0000","last_change":"2024-09-21T08:23:34.570603+0000","last_active":"2024-09-21T08:24:15.600279+0000","last_peered":"2024-09-21T08:24:15.600279+0000","last_clean":"2024-09-21T08:24:15.600279+0000","last_became_active":"2024-09-21T08:23:34.570321+0000","last_became_peered":"2024-09-21T08:23:34.570321+0000","last_unstale":"2024-09-21T08:24:15.600279+0000","last_undegraded":"2024-09-21T08:24:15.600279+0000","last_fullsized":"2024-09-21T08:24:15.600279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.651545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880188+0000","last_change":"2024-09-21T08:23:36.582496+0000","last_active":"2024-09-21T08:24:13.880188+0000","last_peered":"2024-09-21T08:24:13.880188+0000","last_clean":"2024-09-21T08:24:13.880188+0000","last_became_active":"2024-09-21T08:23:36.582039+0000","last_became_peered":"2024-09-21T08:23:36.582039+0000","last_unstale":"2024-09-21T08:24:13.880188+0000","last_undegraded":"2024-09-21T08:24:13.880188+0000","last_fullsized":"2024-09-21T08:24:13.880188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.819778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880197+0000","last_change":"2024-09-21T08:23:32.557661+0000","last_active":"2024-09-21T08:24:13.880197+0000","last_peered":"2024-09-21T08:24:13.880197+0000","last_clean":"2024-09-21T08:24:13.880197+0000","last_became_active":"2024-09-21T08:23:32.555080+0000","last_became_peered":"2024-09-21T08:23:32.555080+0000","last_unstale":"2024-09-21T08:24:13.880197+0000","last_undegraded":"2024-09-21T08:24:13.880197+0000","last_fullsized":"2024-09-21T08:24:13.880197+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.633944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880353+0000","last_change":"2024-09-21T08:23:30.546228+0000","last_active":"2024-09-21T08:24:13.880353+0000","last_peered":"2024-09-21T08:24:13.880353+0000","last_clean":"2024-09-21T08:24:13.880353+0000","last_became_active":"2024-09-21T08:23:30.545920+0000","last_became_peered":"2024-09-21T08:23:30.545920+0000","last_unstale":"2024-09-21T08:24:13.880353+0000","last_undegraded":"2024-09-21T08:24:13.880353+0000","last_fullsized":"2024-09-21T08:24:13.880353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.464911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:15.600281+0000","last_change":"2024-09-21T08:23:34.568126+0000","last_active":"2024-09-21T08:24:15.600281+0000","last_peered":"2024-09-21T08:24:15.600281+0000","last_clean":"2024-09-21T08:24:15.600281+0000","last_became_active":"2024-09-21T08:23:34.563906+0000","last_became_peered":"2024-09-21T08:23:34.563906+0000","last_unstale":"2024-09-21T08:24:15.600281+0000","last_undegraded":"2024-09-21T08:24:15.600281+0000","last_fullsized":"2024-09-21T08:24:15.600281+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.761252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.507792+0000","last_change":"2024-09-21T08:23:36.588118+0000","last_active":"2024-09-21T08:24:13.507792+0000","last_peered":"2024-09-21T08:24:13.507792+0000","last_clean":"2024-09-21T08:24:13.507792+0000","last_became_active":"2024-09-21T08:23:36.587782+0000","last_became_peered":"2024-09-21T08:23:36.587782+0000","last_unstale":"2024-09-21T08:24:13.507792+0000","last_undegraded":"2024-09-21T08:24:13.507792+0000","last_fullsized":"2024-09-21T08:24:13.507792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.580040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":65,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508334+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:24:13.508334+0000","last_peered":"2024-09-21T08:24:13.508334+0000","last_clean":"2024-09-21T08:24:13.508334+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:24:13.508334+0000","last_undegraded":"2024-09-21T08:24:13.508334+0000","last_fullsized":"2024-09-21T08:24:13.508334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.235370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881010+0000","last_change":"2024-09-21T08:23:30.547730+0000","last_active":"2024-09-21T08:24:13.881010+0000","last_peered":"2024-09-21T08:24:13.881010+0000","last_clean":"2024-09-21T08:24:13.881010+0000","last_became_active":"2024-09-21T08:23:30.547290+0000","last_became_peered":"2024-09-21T08:23:30.547290+0000","last_unstale":"2024-09-21T08:24:13.881010+0000","last_undegraded":"2024-09-21T08:24:13.881010+0000","last_fullsized":"2024-09-21T08:24:13.881010+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.347687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508691+0000","last_change":"2024-09-21T08:23:34.564467+0000","last_active":"2024-09-21T08:24:13.508691+0000","last_peered":"2024-09-21T08:24:13.508691+0000","last_clean":"2024-09-21T08:24:13.508691+0000","last_became_active":"2024-09-21T08:23:34.564157+0000","last_became_peered":"2024-09-21T08:23:34.564157+0000","last_unstale":"2024-09-21T08:24:13.508691+0000","last_undegraded":"2024-09-21T08:24:13.508691+0000","last_fullsized":"2024-09-21T08:24:13.508691+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.544611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879617+0000","last_change":"2024-09-21T08:23:36.582071+0000","last_active":"2024-09-21T08:24:13.879617+0000","last_peered":"2024-09-21T08:24:13.879617+0000","last_clean":"2024-09-21T08:24:13.879617+0000","last_became_active":"2024-09-21T08:23:36.581757+0000","last_became_peered":"2024-09-21T08:23:36.581757+0000","last_unstale":"2024-09-21T08:24:13.879617+0000","last_undegraded":"2024-09-21T08:24:13.879617+0000","last_fullsized":"2024-09-21T08:24:13.879617+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.873559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":88,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:15.600282+0000","last_change":"2024-09-21T08:23:32.562868+0000","last_active":"2024-09-21T08:24:15.600282+0000","last_peered":"2024-09-21T08:24:15.600282+0000","last_clean":"2024-09-21T08:24:15.600282+0000","last_became_active":"2024-09-21T08:23:32.561965+0000","last_became_peered":"2024-09-21T08:23:32.561965+0000","last_unstale":"2024-09-21T08:24:15.600282+0000","last_undegraded":"2024-09-21T08:24:15.600282+0000","last_fullsized":"2024-09-21T08:24:15.600282+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.844304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512678+0000","last_change":"2024-09-21T08:23:30.541459+0000","last_active":"2024-09-21T08:24:13.512678+0000","last_peered":"2024-09-21T08:24:13.512678+0000","last_clean":"2024-09-21T08:24:13.512678+0000","last_became_active":"2024-09-21T08:23:30.541175+0000","last_became_peered":"2024-09-21T08:23:30.541175+0000","last_unstale":"2024-09-21T08:24:13.512678+0000","last_undegraded":"2024-09-21T08:24:13.512678+0000","last_fullsized":"2024-09-21T08:24:13.512678+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.508394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":43,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879627+0000","last_change":"2024-09-21T08:23:32.546026+0000","last_active":"2024-09-21T08:24:13.879627+0000","last_peered":"2024-09-21T08:24:13.879627+0000","last_clean":"2024-09-21T08:24:13.879627+0000","last_became_active":"2024-09-21T08:23:29.531243+0000","last_became_peered":"2024-09-21T08:23:29.531243+0000","last_unstale":"2024-09-21T08:24:13.879627+0000","last_undegraded":"2024-09-21T08:24:13.879627+0000","last_fullsized":"2024-09-21T08:24:13.879627+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.908720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508023+0000","last_change":"2024-09-21T08:23:34.568259+0000","last_active":"2024-09-21T08:24:13.508023+0000","last_peered":"2024-09-21T08:24:13.508023+0000","last_clean":"2024-09-21T08:24:13.508023+0000","last_became_active":"2024-09-21T08:23:34.567955+0000","last_became_peered":"2024-09-21T08:23:34.567955+0000","last_unstale":"2024-09-21T08:24:13.508023+0000","last_undegraded":"2024-09-21T08:24:13.508023+0000","last_fullsized":"2024-09-21T08:24:13.508023+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.600803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881509+0000","last_change":"2024-09-21T08:23:36.577895+0000","last_active":"2024-09-21T08:24:13.881509+0000","last_peered":"2024-09-21T08:24:13.881509+0000","last_clean":"2024-09-21T08:24:13.881509+0000","last_became_active":"2024-09-21T08:23:36.577593+0000","last_became_peered":"2024-09-21T08:23:36.577593+0000","last_unstale":"2024-09-21T08:24:13.881509+0000","last_undegraded":"2024-09-21T08:24:13.881509+0000","last_fullsized":"2024-09-21T08:24:13.881509+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.663315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881409+0000","last_change":"2024-09-21T08:23:32.560593+0000","last_active":"2024-09-21T08:24:13.881409+0000","last_peered":"2024-09-21T08:24:13.881409+0000","last_clean":"2024-09-21T08:24:13.881409+0000","last_became_active":"2024-09-21T08:23:32.560323+0000","last_became_peered":"2024-09-21T08:23:32.560323+0000","last_unstale":"2024-09-21T08:24:13.881409+0000","last_undegraded":"2024-09-21T08:24:13.881409+0000","last_fullsized":"2024-09-21T08:24:13.881409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.516260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879720+0000","last_change":"2024-09-21T08:23:30.548894+0000","last_active":"2024-09-21T08:24:13.879720+0000","last_peered":"2024-09-21T08:24:13.879720+0000","last_clean":"2024-09-21T08:24:13.879720+0000","last_became_active":"2024-09-21T08:23:30.548308+0000","last_became_peered":"2024-09-21T08:23:30.548308+0000","last_unstale":"2024-09-21T08:24:13.879720+0000","last_undegraded":"2024-09-21T08:24:13.879720+0000","last_fullsized":"2024-09-21T08:24:13.879720+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.379264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879512+0000","last_change":"2024-09-21T08:23:32.552207+0000","last_active":"2024-09-21T08:24:13.879512+0000","last_peered":"2024-09-21T08:24:13.879512+0000","last_clean":"2024-09-21T08:24:13.879512+0000","last_became_active":"2024-09-21T08:23:29.533672+0000","last_became_peered":"2024-09-21T08:23:29.533672+0000","last_unstale":"2024-09-21T08:24:13.879512+0000","last_undegraded":"2024-09-21T08:24:13.879512+0000","last_fullsized":"2024-09-21T08:24:13.879512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.009447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881131+0000","last_change":"2024-09-21T08:23:34.567541+0000","last_active":"2024-09-21T08:24:13.881131+0000","last_peered":"2024-09-21T08:24:13.881131+0000","last_clean":"2024-09-21T08:24:13.881131+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:24:13.881131+0000","last_undegraded":"2024-09-21T08:24:13.881131+0000","last_fullsized":"2024-09-21T08:24:13.881131+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:51.623026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880555+0000","last_change":"2024-09-21T08:23:36.585190+0000","last_active":"2024-09-21T08:24:13.880555+0000","last_peered":"2024-09-21T08:24:13.880555+0000","last_clean":"2024-09-21T08:24:13.880555+0000","last_became_active":"2024-09-21T08:23:36.584905+0000","last_became_peered":"2024-09-21T08:23:36.584905+0000","last_unstale":"2024-09-21T08:24:13.880555+0000","last_undegraded":"2024-09-21T08:24:13.880555+0000","last_fullsized":"2024-09-21T08:24:13.880555+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.290201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":47,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508731+0000","last_change":"2024-09-21T08:23:32.561943+0000","last_active":"2024-09-21T08:24:13.508731+0000","last_peered":"2024-09-21T08:24:13.508731+0000","last_clean":"2024-09-21T08:24:13.508731+0000","last_became_active":"2024-09-21T08:23:32.561617+0000","last_became_peered":"2024-09-21T08:23:32.561617+0000","last_unstale":"2024-09-21T08:24:13.508731+0000","last_undegraded":"2024-09-21T08:24:13.508731+0000","last_fullsized":"2024-09-21T08:24:13.508731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.493572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508060+0000","last_change":"2024-09-21T08:23:30.547156+0000","last_active":"2024-09-21T08:24:13.508060+0000","last_peered":"2024-09-21T08:24:13.508060+0000","last_clean":"2024-09-21T08:24:13.508060+0000","last_became_active":"2024-09-21T08:23:30.546898+0000","last_became_peered":"2024-09-21T08:23:30.546898+0000","last_unstale":"2024-09-21T08:24:13.508060+0000","last_undegraded":"2024-09-21T08:24:13.508060+0000","last_fullsized":"2024-09-21T08:24:13.508060+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.718486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":89,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:19.564162+0000","last_change":"2024-09-21T08:23:32.597264+0000","last_active":"2024-09-21T08:24:19.564162+0000","last_peered":"2024-09-21T08:24:19.564162+0000","last_clean":"2024-09-21T08:24:19.564162+0000","last_became_active":"2024-09-21T08:23:29.532520+0000","last_became_peered":"2024-09-21T08:23:29.532520+0000","last_unstale":"2024-09-21T08:24:19.564162+0000","last_undegraded":"2024-09-21T08:24:19.564162+0000","last_fullsized":"2024-09-21T08:24:19.564162+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.455707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880519+0000","last_change":"2024-09-21T08:23:34.570449+0000","last_active":"2024-09-21T08:24:13.880519+0000","last_peered":"2024-09-21T08:24:13.880519+0000","last_clean":"2024-09-21T08:24:13.880519+0000","last_became_active":"2024-09-21T08:23:34.569970+0000","last_became_peered":"2024-09-21T08:23:34.569970+0000","last_unstale":"2024-09-21T08:24:13.880519+0000","last_undegraded":"2024-09-21T08:24:13.880519+0000","last_fullsized":"2024-09-21T08:24:13.880519+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.151707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880615+0000","last_change":"2024-09-21T08:23:36.585195+0000","last_active":"2024-09-21T08:24:13.880615+0000","last_peered":"2024-09-21T08:24:13.880615+0000","last_clean":"2024-09-21T08:24:13.880615+0000","last_became_active":"2024-09-21T08:23:36.584827+0000","last_became_peered":"2024-09-21T08:23:36.584827+0000","last_unstale":"2024-09-21T08:24:13.880615+0000","last_undegraded":"2024-09-21T08:24:13.880615+0000","last_fullsized":"2024-09-21T08:24:13.880615+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.376554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":62,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:15.600478+0000","last_change":"2024-09-21T08:23:32.598546+0000","last_active":"2024-09-21T08:24:15.600478+0000","last_peered":"2024-09-21T08:24:15.600478+0000","last_clean":"2024-09-21T08:24:15.600478+0000","last_became_active":"2024-09-21T08:23:32.596949+0000","last_became_peered":"2024-09-21T08:23:32.596949+0000","last_unstale":"2024-09-21T08:24:15.600478+0000","last_undegraded":"2024-09-21T08:24:15.600478+0000","last_fullsized":"2024-09-21T08:24:15.600478+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.826771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879980+0000","last_change":"2024-09-21T08:23:30.546494+0000","last_active":"2024-09-21T08:24:13.879980+0000","last_peered":"2024-09-21T08:24:13.879980+0000","last_clean":"2024-09-21T08:24:13.879980+0000","last_became_active":"2024-09-21T08:23:30.546218+0000","last_became_peered":"2024-09-21T08:23:30.546218+0000","last_unstale":"2024-09-21T08:24:13.879980+0000","last_undegraded":"2024-09-21T08:24:13.879980+0000","last_fullsized":"2024-09-21T08:24:13.879980+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.498978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":288,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879849+0000","last_change":"2024-09-21T08:24:05.765943+0000","last_active":"2024-09-21T08:24:13.879849+0000","last_peered":"2024-09-21T08:24:13.879849+0000","last_clean":"2024-09-21T08:24:13.879849+0000","last_became_active":"2024-09-21T08:23:23.509818+0000","last_became_peered":"2024-09-21T08:23:23.509818+0000","last_unstale":"2024-09-21T08:24:13.879849+0000","last_undegraded":"2024-09-21T08:24:13.879849+0000","last_fullsized":"2024-09-21T08:24:13.879849+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:24:05.765804+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:05.827066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880557+0000","last_change":"2024-09-21T08:23:34.568359+0000","last_active":"2024-09-21T08:24:13.880557+0000","last_peered":"2024-09-21T08:24:13.880557+0000","last_clean":"2024-09-21T08:24:13.880557+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:24:13.880557+0000","last_undegraded":"2024-09-21T08:24:13.880557+0000","last_fullsized":"2024-09-21T08:24:13.880557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.246877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880258+0000","last_change":"2024-09-21T08:23:36.584454+0000","last_active":"2024-09-21T08:24:13.880258+0000","last_peered":"2024-09-21T08:24:13.880258+0000","last_clean":"2024-09-21T08:24:13.880258+0000","last_became_active":"2024-09-21T08:23:36.584095+0000","last_became_peered":"2024-09-21T08:23:36.584095+0000","last_unstale":"2024-09-21T08:24:13.880258+0000","last_undegraded":"2024-09-21T08:24:13.880258+0000","last_fullsized":"2024-09-21T08:24:13.880258+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.123137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512430+0000","last_change":"2024-09-21T08:23:32.557075+0000","last_active":"2024-09-21T08:24:13.512430+0000","last_peered":"2024-09-21T08:24:13.512430+0000","last_clean":"2024-09-21T08:24:13.512430+0000","last_became_active":"2024-09-21T08:23:32.556696+0000","last_became_peered":"2024-09-21T08:23:32.556696+0000","last_unstale":"2024-09-21T08:24:13.512430+0000","last_undegraded":"2024-09-21T08:24:13.512430+0000","last_fullsized":"2024-09-21T08:24:13.512430+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.647238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512569+0000","last_change":"2024-09-21T08:23:30.547989+0000","last_active":"2024-09-21T08:24:13.512569+0000","last_peered":"2024-09-21T08:24:13.512569+0000","last_clean":"2024-09-21T08:24:13.512569+0000","last_became_active":"2024-09-21T08:23:30.547176+0000","last_became_peered":"2024-09-21T08:23:30.547176+0000","last_unstale":"2024-09-21T08:24:13.512569+0000","last_undegraded":"2024-09-21T08:24:13.512569+0000","last_fullsized":"2024-09-21T08:24:13.512569+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.295162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880222+0000","last_change":"2024-09-21T08:23:34.570250+0000","last_active":"2024-09-21T08:24:13.880222+0000","last_peered":"2024-09-21T08:24:13.880222+0000","last_clean":"2024-09-21T08:24:13.880222+0000","last_became_active":"2024-09-21T08:23:34.569571+0000","last_became_peered":"2024-09-21T08:23:34.569571+0000","last_unstale":"2024-09-21T08:24:13.880222+0000","last_undegraded":"2024-09-21T08:24:13.880222+0000","last_fullsized":"2024-09-21T08:24:13.880222+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.053187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881229+0000","last_change":"2024-09-21T08:23:36.587842+0000","last_active":"2024-09-21T08:24:13.881229+0000","last_peered":"2024-09-21T08:24:13.881229+0000","last_clean":"2024-09-21T08:24:13.881229+0000","last_became_active":"2024-09-21T08:23:36.587561+0000","last_became_peered":"2024-09-21T08:23:36.587561+0000","last_unstale":"2024-09-21T08:24:13.881229+0000","last_undegraded":"2024-09-21T08:24:13.881229+0000","last_fullsized":"2024-09-21T08:24:13.881229+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.999379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512426+0000","last_change":"2024-09-21T08:23:32.556741+0000","last_active":"2024-09-21T08:24:13.512426+0000","last_peered":"2024-09-21T08:24:13.512426+0000","last_clean":"2024-09-21T08:24:13.512426+0000","last_became_active":"2024-09-21T08:23:32.556355+0000","last_became_peered":"2024-09-21T08:23:32.556355+0000","last_unstale":"2024-09-21T08:24:13.512426+0000","last_undegraded":"2024-09-21T08:24:13.512426+0000","last_fullsized":"2024-09-21T08:24:13.512426+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.509084+0000","last_change":"2024-09-21T08:23:30.537030+0000","last_active":"2024-09-21T08:24:13.509084+0000","last_peered":"2024-09-21T08:24:13.509084+0000","last_clean":"2024-09-21T08:24:13.509084+0000","last_became_active":"2024-09-21T08:23:30.536566+0000","last_became_peered":"2024-09-21T08:23:30.536566+0000","last_unstale":"2024-09-21T08:24:13.509084+0000","last_undegraded":"2024-09-21T08:24:13.509084+0000","last_fullsized":"2024-09-21T08:24:13.509084+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.146930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880704+0000","last_change":"2024-09-21T08:23:34.569384+0000","last_active":"2024-09-21T08:24:13.880704+0000","last_peered":"2024-09-21T08:24:13.880704+0000","last_clean":"2024-09-21T08:24:13.880704+0000","last_became_active":"2024-09-21T08:23:34.568900+0000","last_became_peered":"2024-09-21T08:23:34.568900+0000","last_unstale":"2024-09-21T08:24:13.880704+0000","last_undegraded":"2024-09-21T08:24:13.880704+0000","last_fullsized":"2024-09-21T08:24:13.880704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.163373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881351+0000","last_change":"2024-09-21T08:23:36.586002+0000","last_active":"2024-09-21T08:24:13.881351+0000","last_peered":"2024-09-21T08:24:13.881351+0000","last_clean":"2024-09-21T08:24:13.881351+0000","last_became_active":"2024-09-21T08:23:36.585695+0000","last_became_peered":"2024-09-21T08:23:36.585695+0000","last_unstale":"2024-09-21T08:24:13.881351+0000","last_undegraded":"2024-09-21T08:24:13.881351+0000","last_fullsized":"2024-09-21T08:24:13.881351+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.916781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512467+0000","last_change":"2024-09-21T08:23:32.555172+0000","last_active":"2024-09-21T08:24:13.512467+0000","last_peered":"2024-09-21T08:24:13.512467+0000","last_clean":"2024-09-21T08:24:13.512467+0000","last_became_active":"2024-09-21T08:23:32.554845+0000","last_became_peered":"2024-09-21T08:23:32.554845+0000","last_unstale":"2024-09-21T08:24:13.512467+0000","last_undegraded":"2024-09-21T08:24:13.512467+0000","last_fullsized":"2024-09-21T08:24:13.512467+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.339439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880410+0000","last_change":"2024-09-21T08:23:30.545460+0000","last_active":"2024-09-21T08:24:13.880410+0000","last_peered":"2024-09-21T08:24:13.880410+0000","last_clean":"2024-09-21T08:24:13.880410+0000","last_became_active":"2024-09-21T08:23:30.545187+0000","last_became_peered":"2024-09-21T08:23:30.545187+0000","last_unstale":"2024-09-21T08:24:13.880410+0000","last_undegraded":"2024-09-21T08:24:13.880410+0000","last_fullsized":"2024-09-21T08:24:13.880410+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.674857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:15.600308+0000","last_change":"2024-09-21T08:23:34.571530+0000","last_active":"2024-09-21T08:24:15.600308+0000","last_peered":"2024-09-21T08:24:15.600308+0000","last_clean":"2024-09-21T08:24:15.600308+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:24:15.600308+0000","last_undegraded":"2024-09-21T08:24:15.600308+0000","last_fullsized":"2024-09-21T08:24:15.600308+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.789578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512171+0000","last_change":"2024-09-21T08:23:36.582871+0000","last_active":"2024-09-21T08:24:13.512171+0000","last_peered":"2024-09-21T08:24:13.512171+0000","last_clean":"2024-09-21T08:24:13.512171+0000","last_became_active":"2024-09-21T08:23:36.582575+0000","last_became_peered":"2024-09-21T08:23:36.582575+0000","last_unstale":"2024-09-21T08:24:13.512171+0000","last_undegraded":"2024-09-21T08:24:13.512171+0000","last_fullsized":"2024-09-21T08:24:13.512171+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.578837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508278+0000","last_change":"2024-09-21T08:23:32.560978+0000","last_active":"2024-09-21T08:24:13.508278+0000","last_peered":"2024-09-21T08:24:13.508278+0000","last_clean":"2024-09-21T08:24:13.508278+0000","last_became_active":"2024-09-21T08:23:32.560658+0000","last_became_peered":"2024-09-21T08:23:32.560658+0000","last_unstale":"2024-09-21T08:24:13.508278+0000","last_undegraded":"2024-09-21T08:24:13.508278+0000","last_fullsized":"2024-09-21T08:24:13.508278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.965888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879972+0000","last_change":"2024-09-21T08:23:30.542309+0000","last_active":"2024-09-21T08:24:13.879972+0000","last_peered":"2024-09-21T08:24:13.879972+0000","last_clean":"2024-09-21T08:24:13.879972+0000","last_became_active":"2024-09-21T08:23:30.541992+0000","last_became_peered":"2024-09-21T08:23:30.541992+0000","last_unstale":"2024-09-21T08:24:13.879972+0000","last_undegraded":"2024-09-21T08:24:13.879972+0000","last_fullsized":"2024-09-21T08:24:13.879972+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.859320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880681+0000","last_change":"2024-09-21T08:23:34.566539+0000","last_active":"2024-09-21T08:24:13.880681+0000","last_peered":"2024-09-21T08:24:13.880681+0000","last_clean":"2024-09-21T08:24:13.880681+0000","last_became_active":"2024-09-21T08:23:34.566242+0000","last_became_peered":"2024-09-21T08:23:34.566242+0000","last_unstale":"2024-09-21T08:24:13.880681+0000","last_undegraded":"2024-09-21T08:24:13.880681+0000","last_fullsized":"2024-09-21T08:24:13.880681+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.194173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508008+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:24:13.508008+0000","last_peered":"2024-09-21T08:24:13.508008+0000","last_clean":"2024-09-21T08:24:13.508008+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:24:13.508008+0000","last_undegraded":"2024-09-21T08:24:13.508008+0000","last_fullsized":"2024-09-21T08:24:13.508008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.339671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508748+0000","last_change":"2024-09-21T08:23:32.597880+0000","last_active":"2024-09-21T08:24:13.508748+0000","last_peered":"2024-09-21T08:24:13.508748+0000","last_clean":"2024-09-21T08:24:13.508748+0000","last_became_active":"2024-09-21T08:23:32.597589+0000","last_became_peered":"2024-09-21T08:23:32.597589+0000","last_unstale":"2024-09-21T08:24:13.508748+0000","last_undegraded":"2024-09-21T08:24:13.508748+0000","last_fullsized":"2024-09-21T08:24:13.508748+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.870820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880724+0000","last_change":"2024-09-21T08:23:30.545405+0000","last_active":"2024-09-21T08:24:13.880724+0000","last_peered":"2024-09-21T08:24:13.880724+0000","last_clean":"2024-09-21T08:24:13.880724+0000","last_became_active":"2024-09-21T08:23:30.545068+0000","last_became_peered":"2024-09-21T08:23:30.545068+0000","last_unstale":"2024-09-21T08:24:13.880724+0000","last_undegraded":"2024-09-21T08:24:13.880724+0000","last_fullsized":"2024-09-21T08:24:13.880724+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.179419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880834+0000","last_change":"2024-09-21T08:23:34.564923+0000","last_active":"2024-09-21T08:24:13.880834+0000","last_peered":"2024-09-21T08:24:13.880834+0000","last_clean":"2024-09-21T08:24:13.880834+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:24:13.880834+0000","last_undegraded":"2024-09-21T08:24:13.880834+0000","last_fullsized":"2024-09-21T08:24:13.880834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.183697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879766+0000","last_change":"2024-09-21T08:23:36.582099+0000","last_active":"2024-09-21T08:24:13.879766+0000","last_peered":"2024-09-21T08:24:13.879766+0000","last_clean":"2024-09-21T08:24:13.879766+0000","last_became_active":"2024-09-21T08:23:36.581796+0000","last_became_peered":"2024-09-21T08:23:36.581796+0000","last_unstale":"2024-09-21T08:24:13.879766+0000","last_undegraded":"2024-09-21T08:24:13.879766+0000","last_fullsized":"2024-09-21T08:24:13.879766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.618273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512020+0000","last_change":"2024-09-21T08:23:32.557284+0000","last_active":"2024-09-21T08:24:13.512020+0000","last_peered":"2024-09-21T08:24:13.512020+0000","last_clean":"2024-09-21T08:24:13.512020+0000","last_became_active":"2024-09-21T08:23:32.556876+0000","last_became_peered":"2024-09-21T08:23:32.556876+0000","last_unstale":"2024-09-21T08:24:13.512020+0000","last_undegraded":"2024-09-21T08:24:13.512020+0000","last_fullsized":"2024-09-21T08:24:13.512020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.982500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880726+0000","last_change":"2024-09-21T08:23:30.548671+0000","last_active":"2024-09-21T08:24:13.880726+0000","last_peered":"2024-09-21T08:24:13.880726+0000","last_clean":"2024-09-21T08:24:13.880726+0000","last_became_active":"2024-09-21T08:23:30.548350+0000","last_became_peered":"2024-09-21T08:23:30.548350+0000","last_unstale":"2024-09-21T08:24:13.880726+0000","last_undegraded":"2024-09-21T08:24:13.880726+0000","last_fullsized":"2024-09-21T08:24:13.880726+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.489110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879297+0000","last_change":"2024-09-21T08:23:34.566472+0000","last_active":"2024-09-21T08:24:13.879297+0000","last_peered":"2024-09-21T08:24:13.879297+0000","last_clean":"2024-09-21T08:24:13.879297+0000","last_became_active":"2024-09-21T08:23:34.566193+0000","last_became_peered":"2024-09-21T08:23:34.566193+0000","last_unstale":"2024-09-21T08:24:13.879297+0000","last_undegraded":"2024-09-21T08:24:13.879297+0000","last_fullsized":"2024-09-21T08:24:13.879297+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:58.818551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880129+0000","last_change":"2024-09-21T08:23:36.579879+0000","last_active":"2024-09-21T08:24:13.880129+0000","last_peered":"2024-09-21T08:24:13.880129+0000","last_clean":"2024-09-21T08:24:13.880129+0000","last_became_active":"2024-09-21T08:23:36.579603+0000","last_became_peered":"2024-09-21T08:23:36.579603+0000","last_unstale":"2024-09-21T08:24:13.880129+0000","last_undegraded":"2024-09-21T08:24:13.880129+0000","last_fullsized":"2024-09-21T08:24:13.880129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.717936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879309+0000","last_change":"2024-09-21T08:23:32.598217+0000","last_active":"2024-09-21T08:24:13.879309+0000","last_peered":"2024-09-21T08:24:13.879309+0000","last_clean":"2024-09-21T08:24:13.879309+0000","last_became_active":"2024-09-21T08:23:32.597543+0000","last_became_peered":"2024-09-21T08:23:32.597543+0000","last_unstale":"2024-09-21T08:24:13.879309+0000","last_undegraded":"2024-09-21T08:24:13.879309+0000","last_fullsized":"2024-09-21T08:24:13.879309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.907666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880865+0000","last_change":"2024-09-21T08:23:30.551520+0000","last_active":"2024-09-21T08:24:13.880865+0000","last_peered":"2024-09-21T08:24:13.880865+0000","last_clean":"2024-09-21T08:24:13.880865+0000","last_became_active":"2024-09-21T08:23:30.551203+0000","last_became_peered":"2024-09-21T08:23:30.551203+0000","last_unstale":"2024-09-21T08:24:13.880865+0000","last_undegraded":"2024-09-21T08:24:13.880865+0000","last_fullsized":"2024-09-21T08:24:13.880865+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:29.708394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:15.600365+0000","last_change":"2024-09-21T08:23:34.569818+0000","last_active":"2024-09-21T08:24:15.600365+0000","last_peered":"2024-09-21T08:24:15.600365+0000","last_clean":"2024-09-21T08:24:15.600365+0000","last_became_active":"2024-09-21T08:23:34.569490+0000","last_became_peered":"2024-09-21T08:23:34.569490+0000","last_unstale":"2024-09-21T08:24:15.600365+0000","last_undegraded":"2024-09-21T08:24:15.600365+0000","last_fullsized":"2024-09-21T08:24:15.600365+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.884010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879465+0000","last_change":"2024-09-21T08:23:36.579253+0000","last_active":"2024-09-21T08:24:13.879465+0000","last_peered":"2024-09-21T08:24:13.879465+0000","last_clean":"2024-09-21T08:24:13.879465+0000","last_became_active":"2024-09-21T08:23:36.578952+0000","last_became_peered":"2024-09-21T08:23:36.578952+0000","last_unstale":"2024-09-21T08:24:13.879465+0000","last_undegraded":"2024-09-21T08:24:13.879465+0000","last_fullsized":"2024-09-21T08:24:13.879465+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.185192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508773+0000","last_change":"2024-09-21T08:23:30.542360+0000","last_active":"2024-09-21T08:24:13.508773+0000","last_peered":"2024-09-21T08:24:13.508773+0000","last_clean":"2024-09-21T08:24:13.508773+0000","last_became_active":"2024-09-21T08:23:30.542060+0000","last_became_peered":"2024-09-21T08:23:30.542060+0000","last_unstale":"2024-09-21T08:24:13.508773+0000","last_undegraded":"2024-09-21T08:24:13.508773+0000","last_fullsized":"2024-09-21T08:24:13.508773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:58.968344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":38,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880484+0000","last_change":"2024-09-21T08:23:32.598912+0000","last_active":"2024-09-21T08:24:13.880484+0000","last_peered":"2024-09-21T08:24:13.880484+0000","last_clean":"2024-09-21T08:24:13.880484+0000","last_became_active":"2024-09-21T08:23:32.598518+0000","last_became_peered":"2024-09-21T08:23:32.598518+0000","last_unstale":"2024-09-21T08:24:13.880484+0000","last_undegraded":"2024-09-21T08:24:13.880484+0000","last_fullsized":"2024-09-21T08:24:13.880484+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:32.806988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880017+0000","last_change":"2024-09-21T08:23:34.566165+0000","last_active":"2024-09-21T08:24:13.880017+0000","last_peered":"2024-09-21T08:24:13.880017+0000","last_clean":"2024-09-21T08:24:13.880017+0000","last_became_active":"2024-09-21T08:23:34.565662+0000","last_became_peered":"2024-09-21T08:23:34.565662+0000","last_unstale":"2024-09-21T08:24:13.880017+0000","last_undegraded":"2024-09-21T08:24:13.880017+0000","last_fullsized":"2024-09-21T08:24:13.880017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.290518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880450+0000","last_change":"2024-09-21T08:23:36.579682+0000","last_active":"2024-09-21T08:24:13.880450+0000","last_peered":"2024-09-21T08:24:13.880450+0000","last_clean":"2024-09-21T08:24:13.880450+0000","last_became_active":"2024-09-21T08:23:36.579384+0000","last_became_peered":"2024-09-21T08:23:36.579384+0000","last_unstale":"2024-09-21T08:24:13.880450+0000","last_undegraded":"2024-09-21T08:24:13.880450+0000","last_fullsized":"2024-09-21T08:24:13.880450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.156840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508810+0000","last_change":"2024-09-21T08:23:32.599079+0000","last_active":"2024-09-21T08:24:13.508810+0000","last_peered":"2024-09-21T08:24:13.508810+0000","last_clean":"2024-09-21T08:24:13.508810+0000","last_became_active":"2024-09-21T08:23:32.598778+0000","last_became_peered":"2024-09-21T08:23:32.598778+0000","last_unstale":"2024-09-21T08:24:13.508810+0000","last_undegraded":"2024-09-21T08:24:13.508810+0000","last_fullsized":"2024-09-21T08:24:13.508810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.131011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879936+0000","last_change":"2024-09-21T08:23:30.548443+0000","last_active":"2024-09-21T08:24:13.879936+0000","last_peered":"2024-09-21T08:24:13.879936+0000","last_clean":"2024-09-21T08:24:13.879936+0000","last_became_active":"2024-09-21T08:23:30.548096+0000","last_became_peered":"2024-09-21T08:23:30.548096+0000","last_unstale":"2024-09-21T08:24:13.879936+0000","last_undegraded":"2024-09-21T08:24:13.879936+0000","last_fullsized":"2024-09-21T08:24:13.879936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.029746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880389+0000","last_change":"2024-09-21T08:23:34.566473+0000","last_active":"2024-09-21T08:24:13.880389+0000","last_peered":"2024-09-21T08:24:13.880389+0000","last_clean":"2024-09-21T08:24:13.880389+0000","last_became_active":"2024-09-21T08:23:34.566160+0000","last_became_peered":"2024-09-21T08:23:34.566160+0000","last_unstale":"2024-09-21T08:24:13.880389+0000","last_undegraded":"2024-09-21T08:24:13.880389+0000","last_fullsized":"2024-09-21T08:24:13.880389+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.960353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881308+0000","last_change":"2024-09-21T08:23:36.587149+0000","last_active":"2024-09-21T08:24:13.881308+0000","last_peered":"2024-09-21T08:24:13.881308+0000","last_clean":"2024-09-21T08:24:13.881308+0000","last_became_active":"2024-09-21T08:23:36.586865+0000","last_became_peered":"2024-09-21T08:23:36.586865+0000","last_unstale":"2024-09-21T08:24:13.881308+0000","last_undegraded":"2024-09-21T08:24:13.881308+0000","last_fullsized":"2024-09-21T08:24:13.881308+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:42.991875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880113+0000","last_change":"2024-09-21T08:23:32.599185+0000","last_active":"2024-09-21T08:24:13.880113+0000","last_peered":"2024-09-21T08:24:13.880113+0000","last_clean":"2024-09-21T08:24:13.880113+0000","last_became_active":"2024-09-21T08:23:32.598748+0000","last_became_peered":"2024-09-21T08:23:32.598748+0000","last_unstale":"2024-09-21T08:24:13.880113+0000","last_undegraded":"2024-09-21T08:24:13.880113+0000","last_fullsized":"2024-09-21T08:24:13.880113+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.141293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508197+0000","last_change":"2024-09-21T08:23:30.548111+0000","last_active":"2024-09-21T08:24:13.508197+0000","last_peered":"2024-09-21T08:24:13.508197+0000","last_clean":"2024-09-21T08:24:13.508197+0000","last_became_active":"2024-09-21T08:23:30.547824+0000","last_became_peered":"2024-09-21T08:23:30.547824+0000","last_unstale":"2024-09-21T08:24:13.508197+0000","last_undegraded":"2024-09-21T08:24:13.508197+0000","last_fullsized":"2024-09-21T08:24:13.508197+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:15.600296+0000","last_change":"2024-09-21T08:23:34.570551+0000","last_active":"2024-09-21T08:24:15.600296+0000","last_peered":"2024-09-21T08:24:15.600296+0000","last_clean":"2024-09-21T08:24:15.600296+0000","last_became_active":"2024-09-21T08:23:34.570211+0000","last_became_peered":"2024-09-21T08:23:34.570211+0000","last_unstale":"2024-09-21T08:24:15.600296+0000","last_undegraded":"2024-09-21T08:24:15.600296+0000","last_fullsized":"2024-09-21T08:24:15.600296+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.547474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512312+0000","last_change":"2024-09-21T08:23:36.582684+0000","last_active":"2024-09-21T08:24:13.512312+0000","last_peered":"2024-09-21T08:24:13.512312+0000","last_clean":"2024-09-21T08:24:13.512312+0000","last_became_active":"2024-09-21T08:23:36.582369+0000","last_became_peered":"2024-09-21T08:23:36.582369+0000","last_unstale":"2024-09-21T08:24:13.512312+0000","last_undegraded":"2024-09-21T08:24:13.512312+0000","last_fullsized":"2024-09-21T08:24:13.512312+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:48.140821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881447+0000","last_change":"2024-09-21T08:23:32.597630+0000","last_active":"2024-09-21T08:24:13.881447+0000","last_peered":"2024-09-21T08:24:13.881447+0000","last_clean":"2024-09-21T08:24:13.881447+0000","last_became_active":"2024-09-21T08:23:32.597296+0000","last_became_peered":"2024-09-21T08:23:32.597296+0000","last_unstale":"2024-09-21T08:24:13.881447+0000","last_undegraded":"2024-09-21T08:24:13.881447+0000","last_fullsized":"2024-09-21T08:24:13.881447+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.289337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880378+0000","last_change":"2024-09-21T08:23:30.549864+0000","last_active":"2024-09-21T08:24:13.880378+0000","last_peered":"2024-09-21T08:24:13.880378+0000","last_clean":"2024-09-21T08:24:13.880378+0000","last_became_active":"2024-09-21T08:23:30.549562+0000","last_became_peered":"2024-09-21T08:23:30.549562+0000","last_unstale":"2024-09-21T08:24:13.880378+0000","last_undegraded":"2024-09-21T08:24:13.880378+0000","last_fullsized":"2024-09-21T08:24:13.880378+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.957399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:15.600403+0000","last_change":"2024-09-21T08:23:34.567042+0000","last_active":"2024-09-21T08:24:15.600403+0000","last_peered":"2024-09-21T08:24:15.600403+0000","last_clean":"2024-09-21T08:24:15.600403+0000","last_became_active":"2024-09-21T08:23:34.566736+0000","last_became_peered":"2024-09-21T08:23:34.566736+0000","last_unstale":"2024-09-21T08:24:15.600403+0000","last_undegraded":"2024-09-21T08:24:15.600403+0000","last_fullsized":"2024-09-21T08:24:15.600403+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508446+0000","last_change":"2024-09-21T08:23:36.589503+0000","last_active":"2024-09-21T08:24:13.508446+0000","last_peered":"2024-09-21T08:24:13.508446+0000","last_clean":"2024-09-21T08:24:13.508446+0000","last_became_active":"2024-09-21T08:23:36.589224+0000","last_became_peered":"2024-09-21T08:23:36.589224+0000","last_unstale":"2024-09-21T08:24:13.508446+0000","last_undegraded":"2024-09-21T08:24:13.508446+0000","last_fullsized":"2024-09-21T08:24:13.508446+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.636183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512283+0000","last_change":"2024-09-21T08:23:32.559882+0000","last_active":"2024-09-21T08:24:13.512283+0000","last_peered":"2024-09-21T08:24:13.512283+0000","last_clean":"2024-09-21T08:24:13.512283+0000","last_became_active":"2024-09-21T08:23:32.559574+0000","last_became_peered":"2024-09-21T08:23:32.559574+0000","last_unstale":"2024-09-21T08:24:13.512283+0000","last_undegraded":"2024-09-21T08:24:13.512283+0000","last_fullsized":"2024-09-21T08:24:13.512283+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.305662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512646+0000","last_change":"2024-09-21T08:23:30.546837+0000","last_active":"2024-09-21T08:24:13.512646+0000","last_peered":"2024-09-21T08:24:13.512646+0000","last_clean":"2024-09-21T08:24:13.512646+0000","last_became_active":"2024-09-21T08:23:30.546542+0000","last_became_peered":"2024-09-21T08:23:30.546542+0000","last_unstale":"2024-09-21T08:24:13.512646+0000","last_undegraded":"2024-09-21T08:24:13.512646+0000","last_fullsized":"2024-09-21T08:24:13.512646+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880060+0000","last_change":"2024-09-21T08:23:34.571300+0000","last_active":"2024-09-21T08:24:13.880060+0000","last_peered":"2024-09-21T08:24:13.880060+0000","last_clean":"2024-09-21T08:24:13.880060+0000","last_became_active":"2024-09-21T08:23:34.571034+0000","last_became_peered":"2024-09-21T08:23:34.571034+0000","last_unstale":"2024-09-21T08:24:13.880060+0000","last_undegraded":"2024-09-21T08:24:13.880060+0000","last_fullsized":"2024-09-21T08:24:13.880060+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.978453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880287+0000","last_change":"2024-09-21T08:23:36.585748+0000","last_active":"2024-09-21T08:24:13.880287+0000","last_peered":"2024-09-21T08:24:13.880287+0000","last_clean":"2024-09-21T08:24:13.880287+0000","last_became_active":"2024-09-21T08:23:36.585492+0000","last_became_peered":"2024-09-21T08:23:36.585492+0000","last_unstale":"2024-09-21T08:24:13.880287+0000","last_undegraded":"2024-09-21T08:24:13.880287+0000","last_fullsized":"2024-09-21T08:24:13.880287+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.492501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880287+0000","last_change":"2024-09-21T08:23:32.556550+0000","last_active":"2024-09-21T08:24:13.880287+0000","last_peered":"2024-09-21T08:24:13.880287+0000","last_clean":"2024-09-21T08:24:13.880287+0000","last_became_active":"2024-09-21T08:23:32.556189+0000","last_became_peered":"2024-09-21T08:23:32.556189+0000","last_unstale":"2024-09-21T08:24:13.880287+0000","last_undegraded":"2024-09-21T08:24:13.880287+0000","last_fullsized":"2024-09-21T08:24:13.880287+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:51.354291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880657+0000","last_change":"2024-09-21T08:23:30.549643+0000","last_active":"2024-09-21T08:24:13.880657+0000","last_peered":"2024-09-21T08:24:13.880657+0000","last_clean":"2024-09-21T08:24:13.880657+0000","last_became_active":"2024-09-21T08:23:30.549317+0000","last_became_peered":"2024-09-21T08:23:30.549317+0000","last_unstale":"2024-09-21T08:24:13.880657+0000","last_undegraded":"2024-09-21T08:24:13.880657+0000","last_fullsized":"2024-09-21T08:24:13.880657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.895759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880222+0000","last_change":"2024-09-21T08:23:34.565128+0000","last_active":"2024-09-21T08:24:13.880222+0000","last_peered":"2024-09-21T08:24:13.880222+0000","last_clean":"2024-09-21T08:24:13.880222+0000","last_became_active":"2024-09-21T08:23:34.564738+0000","last_became_peered":"2024-09-21T08:23:34.564738+0000","last_unstale":"2024-09-21T08:24:13.880222+0000","last_undegraded":"2024-09-21T08:24:13.880222+0000","last_fullsized":"2024-09-21T08:24:13.880222+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.717622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.507602+0000","last_change":"2024-09-21T08:23:36.580460+0000","last_active":"2024-09-21T08:24:13.507602+0000","last_peered":"2024-09-21T08:24:13.507602+0000","last_clean":"2024-09-21T08:24:13.507602+0000","last_became_active":"2024-09-21T08:23:36.579949+0000","last_became_peered":"2024-09-21T08:23:36.579949+0000","last_unstale":"2024-09-21T08:24:13.507602+0000","last_undegraded":"2024-09-21T08:24:13.507602+0000","last_fullsized":"2024-09-21T08:24:13.507602+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.377442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879985+0000","last_change":"2024-09-21T08:23:32.597672+0000","last_active":"2024-09-21T08:24:13.879985+0000","last_peered":"2024-09-21T08:24:13.879985+0000","last_clean":"2024-09-21T08:24:13.879985+0000","last_became_active":"2024-09-21T08:23:32.597281+0000","last_became_peered":"2024-09-21T08:23:32.597281+0000","last_unstale":"2024-09-21T08:24:13.879985+0000","last_undegraded":"2024-09-21T08:24:13.879985+0000","last_fullsized":"2024-09-21T08:24:13.879985+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.493132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880120+0000","last_change":"2024-09-21T08:23:30.544860+0000","last_active":"2024-09-21T08:24:13.880120+0000","last_peered":"2024-09-21T08:24:13.880120+0000","last_clean":"2024-09-21T08:24:13.880120+0000","last_became_active":"2024-09-21T08:23:30.544486+0000","last_became_peered":"2024-09-21T08:23:30.544486+0000","last_unstale":"2024-09-21T08:24:13.880120+0000","last_undegraded":"2024-09-21T08:24:13.880120+0000","last_fullsized":"2024-09-21T08:24:13.880120+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.462170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879434+0000","last_change":"2024-09-21T08:23:34.569813+0000","last_active":"2024-09-21T08:24:13.879434+0000","last_peered":"2024-09-21T08:24:13.879434+0000","last_clean":"2024-09-21T08:24:13.879434+0000","last_became_active":"2024-09-21T08:23:34.569485+0000","last_became_peered":"2024-09-21T08:23:34.569485+0000","last_unstale":"2024-09-21T08:24:13.879434+0000","last_undegraded":"2024-09-21T08:24:13.879434+0000","last_fullsized":"2024-09-21T08:24:13.879434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.575850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880334+0000","last_change":"2024-09-21T08:23:36.586500+0000","last_active":"2024-09-21T08:24:13.880334+0000","last_peered":"2024-09-21T08:24:13.880334+0000","last_clean":"2024-09-21T08:24:13.880334+0000","last_became_active":"2024-09-21T08:23:36.586172+0000","last_became_peered":"2024-09-21T08:23:36.586172+0000","last_unstale":"2024-09-21T08:24:13.880334+0000","last_undegraded":"2024-09-21T08:24:13.880334+0000","last_fullsized":"2024-09-21T08:24:13.880334+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.388100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":35,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879627+0000","last_change":"2024-09-21T08:23:32.563756+0000","last_active":"2024-09-21T08:24:13.879627+0000","last_peered":"2024-09-21T08:24:13.879627+0000","last_clean":"2024-09-21T08:24:13.879627+0000","last_became_active":"2024-09-21T08:23:32.563484+0000","last_became_peered":"2024-09-21T08:23:32.563484+0000","last_unstale":"2024-09-21T08:24:13.879627+0000","last_undegraded":"2024-09-21T08:24:13.879627+0000","last_fullsized":"2024-09-21T08:24:13.879627+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.069344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508346+0000","last_change":"2024-09-21T08:23:30.544403+0000","last_active":"2024-09-21T08:24:13.508346+0000","last_peered":"2024-09-21T08:24:13.508346+0000","last_clean":"2024-09-21T08:24:13.508346+0000","last_became_active":"2024-09-21T08:23:30.544066+0000","last_became_peered":"2024-09-21T08:23:30.544066+0000","last_unstale":"2024-09-21T08:24:13.508346+0000","last_undegraded":"2024-09-21T08:24:13.508346+0000","last_fullsized":"2024-09-21T08:24:13.508346+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.462796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508689+0000","last_change":"2024-09-21T08:23:34.566607+0000","last_active":"2024-09-21T08:24:13.508689+0000","last_peered":"2024-09-21T08:24:13.508689+0000","last_clean":"2024-09-21T08:24:13.508689+0000","last_became_active":"2024-09-21T08:23:34.566337+0000","last_became_peered":"2024-09-21T08:23:34.566337+0000","last_unstale":"2024-09-21T08:24:13.508689+0000","last_undegraded":"2024-09-21T08:24:13.508689+0000","last_fullsized":"2024-09-21T08:24:13.508689+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.230638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879522+0000","last_change":"2024-09-21T08:23:36.582773+0000","last_active":"2024-09-21T08:24:13.879522+0000","last_peered":"2024-09-21T08:24:13.879522+0000","last_clean":"2024-09-21T08:24:13.879522+0000","last_became_active":"2024-09-21T08:23:36.582150+0000","last_became_peered":"2024-09-21T08:23:36.582150+0000","last_unstale":"2024-09-21T08:24:13.879522+0000","last_undegraded":"2024-09-21T08:24:13.879522+0000","last_fullsized":"2024-09-21T08:24:13.879522+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.263690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880703+0000","last_change":"2024-09-21T08:23:36.576004+0000","last_active":"2024-09-21T08:24:13.880703+0000","last_peered":"2024-09-21T08:24:13.880703+0000","last_clean":"2024-09-21T08:24:13.880703+0000","last_became_active":"2024-09-21T08:23:36.575701+0000","last_became_peered":"2024-09-21T08:23:36.575701+0000","last_unstale":"2024-09-21T08:24:13.880703+0000","last_undegraded":"2024-09-21T08:24:13.880703+0000","last_fullsized":"2024-09-21T08:24:13.880703+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.955927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879837+0000","last_change":"2024-09-21T08:23:30.544201+0000","last_active":"2024-09-21T08:24:13.879837+0000","last_peered":"2024-09-21T08:24:13.879837+0000","last_clean":"2024-09-21T08:24:13.879837+0000","last_became_active":"2024-09-21T08:23:30.543932+0000","last_became_peered":"2024-09-21T08:23:30.543932+0000","last_unstale":"2024-09-21T08:24:13.879837+0000","last_undegraded":"2024-09-21T08:24:13.879837+0000","last_fullsized":"2024-09-21T08:24:13.879837+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.933119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.509093+0000","last_change":"2024-09-21T08:23:32.549263+0000","last_active":"2024-09-21T08:24:13.509093+0000","last_peered":"2024-09-21T08:24:13.509093+0000","last_clean":"2024-09-21T08:24:13.509093+0000","last_became_active":"2024-09-21T08:23:32.548835+0000","last_became_peered":"2024-09-21T08:23:32.548835+0000","last_unstale":"2024-09-21T08:24:13.509093+0000","last_undegraded":"2024-09-21T08:24:13.509093+0000","last_fullsized":"2024-09-21T08:24:13.509093+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.423215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508578+0000","last_change":"2024-09-21T08:23:34.573167+0000","last_active":"2024-09-21T08:24:13.508578+0000","last_peered":"2024-09-21T08:24:13.508578+0000","last_clean":"2024-09-21T08:24:13.508578+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:24:13.508578+0000","last_undegraded":"2024-09-21T08:24:13.508578+0000","last_fullsized":"2024-09-21T08:24:13.508578+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.897951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":45,"num_read_kb":40,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070479,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32416,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712736,"statfs":{"total":95995035648,"available":95961841664,"internally_reserved":0,"allocated":1257472,"data_stored":869659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397525,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28232,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716920,"statfs":{"total":95995035648,"available":95966126080,"internally_reserved":0,"allocated":1232896,"data_stored":867581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659160,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27664,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":651264,"data_stored":278021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986206,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713308,"statfs":{"total":95995035648,"available":95962427392,"internally_reserved":0,"allocated":671744,"data_stored":278811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182434,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":278553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411368,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":643072,"data_stored":278790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574892,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":279011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738417,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1257472,"data_stored":868375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:24:25.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:24:25.729 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[29634]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-09-21T08:24:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T08:24:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[41044]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-09-21T08:24:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T08:24:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:25 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:25 smithi165 ceph-mon[32893]: pgmap v16: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1004 B/s rd, 0 op/s 2024-09-21T08:24:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:25 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:25 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:25 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:25 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:25 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:26.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:25 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:26.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:25 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T08:24:26.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:25 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:26.474 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:24:26.475 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:24:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:26 smithi137 ceph-mon[29634]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T08:24:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:26 smithi137 ceph-mon[29634]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T08:24:26.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:26 smithi137 ceph-mon[29634]: Deploying daemon grafana.a on smithi165 2024-09-21T08:24:26.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:26 smithi137 ceph-mon[41044]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T08:24:26.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:26 smithi137 ceph-mon[41044]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T08:24:26.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:26 smithi137 ceph-mon[41044]: Deploying daemon grafana.a on smithi165 2024-09-21T08:24:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:26 smithi165 ceph-mon[32893]: Regenerating cephadm self-signed grafana TLS certificates 2024-09-21T08:24:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:26 smithi165 ceph-mon[32893]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-09-21T08:24:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:26 smithi165 ceph-mon[32893]: Deploying daemon grafana.a on smithi165 2024-09-21T08:24:27.073 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:24:26 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[76851]: ts=2024-09-21T08:24:26.818Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000989886s 2024-09-21T08:24:27.102 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":17,"stamp":"2024-09-21T08:24:25.706176+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596507,"num_objects":199,"num_object_clones":0,"num_object_copies":597,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":199,"num_whiteouts":0,"num_read":984,"num_read_kb":830,"num_write":615,"num_write_kb":2461,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":602,"ondisk_log_size":602,"up":396,"acting":396,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":396,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":749961216,"kb_used":231484,"kb_used_data":6940,"kb_used_omap":12,"kb_used_meta":224371,"kb_avail":749729732,"statfs":{"total":767960285184,"available":767723245568,"internally_reserved":0,"allocated":7106560,"data_stored":3998801,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12752,"internal_metadata":229756464},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":15,"num_read_kb":15,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002653"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508268+0000","last_change":"2024-09-21T08:23:30.549696+0000","last_active":"2024-09-21T08:24:13.508268+0000","last_peered":"2024-09-21T08:24:13.508268+0000","last_clean":"2024-09-21T08:24:13.508268+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:24:13.508268+0000","last_undegraded":"2024-09-21T08:24:13.508268+0000","last_fullsized":"2024-09-21T08:24:13.508268+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.197261+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.511847+0000","last_change":"2024-09-21T08:23:32.556768+0000","last_active":"2024-09-21T08:24:13.511847+0000","last_peered":"2024-09-21T08:24:13.511847+0000","last_clean":"2024-09-21T08:24:13.511847+0000","last_became_active":"2024-09-21T08:23:32.556452+0000","last_became_peered":"2024-09-21T08:23:32.556452+0000","last_unstale":"2024-09-21T08:24:13.511847+0000","last_undegraded":"2024-09-21T08:24:13.511847+0000","last_fullsized":"2024-09-21T08:24:13.511847+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.936229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879709+0000","last_change":"2024-09-21T08:23:34.569082+0000","last_active":"2024-09-21T08:24:13.879709+0000","last_peered":"2024-09-21T08:24:13.879709+0000","last_clean":"2024-09-21T08:24:13.879709+0000","last_became_active":"2024-09-21T08:23:34.568762+0000","last_became_peered":"2024-09-21T08:23:34.568762+0000","last_unstale":"2024-09-21T08:24:13.879709+0000","last_undegraded":"2024-09-21T08:24:13.879709+0000","last_fullsized":"2024-09-21T08:24:13.879709+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.577218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.511973+0000","last_change":"2024-09-21T08:23:36.575623+0000","last_active":"2024-09-21T08:24:13.511973+0000","last_peered":"2024-09-21T08:24:13.511973+0000","last_clean":"2024-09-21T08:24:13.511973+0000","last_became_active":"2024-09-21T08:23:36.572359+0000","last_became_peered":"2024-09-21T08:23:36.572359+0000","last_unstale":"2024-09-21T08:24:13.511973+0000","last_undegraded":"2024-09-21T08:24:13.511973+0000","last_fullsized":"2024-09-21T08:24:13.511973+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.062906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880039+0000","last_change":"2024-09-21T08:23:36.585557+0000","last_active":"2024-09-21T08:24:13.880039+0000","last_peered":"2024-09-21T08:24:13.880039+0000","last_clean":"2024-09-21T08:24:13.880039+0000","last_became_active":"2024-09-21T08:23:36.584820+0000","last_became_peered":"2024-09-21T08:23:36.584820+0000","last_unstale":"2024-09-21T08:24:13.880039+0000","last_undegraded":"2024-09-21T08:24:13.880039+0000","last_fullsized":"2024-09-21T08:24:13.880039+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.509136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881615+0000","last_change":"2024-09-21T08:23:30.548436+0000","last_active":"2024-09-21T08:24:13.881615+0000","last_peered":"2024-09-21T08:24:13.881615+0000","last_clean":"2024-09-21T08:24:13.881615+0000","last_became_active":"2024-09-21T08:23:30.548146+0000","last_became_peered":"2024-09-21T08:23:30.548146+0000","last_unstale":"2024-09-21T08:24:13.881615+0000","last_undegraded":"2024-09-21T08:24:13.881615+0000","last_fullsized":"2024-09-21T08:24:13.881615+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.518727+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,2],"acting":[3,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880248+0000","last_change":"2024-09-21T08:23:32.556169+0000","last_active":"2024-09-21T08:24:13.880248+0000","last_peered":"2024-09-21T08:24:13.880248+0000","last_clean":"2024-09-21T08:24:13.880248+0000","last_became_active":"2024-09-21T08:23:32.555789+0000","last_became_peered":"2024-09-21T08:23:32.555789+0000","last_unstale":"2024-09-21T08:24:13.880248+0000","last_undegraded":"2024-09-21T08:24:13.880248+0000","last_fullsized":"2024-09-21T08:24:13.880248+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.359444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.511839+0000","last_change":"2024-09-21T08:23:34.565614+0000","last_active":"2024-09-21T08:24:13.511839+0000","last_peered":"2024-09-21T08:24:13.511839+0000","last_clean":"2024-09-21T08:24:13.511839+0000","last_became_active":"2024-09-21T08:23:34.565276+0000","last_became_peered":"2024-09-21T08:23:34.565276+0000","last_unstale":"2024-09-21T08:24:13.511839+0000","last_undegraded":"2024-09-21T08:24:13.511839+0000","last_fullsized":"2024-09-21T08:24:13.511839+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.926230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880265+0000","last_change":"2024-09-21T08:23:30.543643+0000","last_active":"2024-09-21T08:24:13.880265+0000","last_peered":"2024-09-21T08:24:13.880265+0000","last_clean":"2024-09-21T08:24:13.880265+0000","last_became_active":"2024-09-21T08:23:30.543041+0000","last_became_peered":"2024-09-21T08:23:30.543041+0000","last_unstale":"2024-09-21T08:24:13.880265+0000","last_undegraded":"2024-09-21T08:24:13.880265+0000","last_fullsized":"2024-09-21T08:24:13.880265+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.229786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512124+0000","last_change":"2024-09-21T08:23:32.559542+0000","last_active":"2024-09-21T08:24:13.512124+0000","last_peered":"2024-09-21T08:24:13.512124+0000","last_clean":"2024-09-21T08:24:13.512124+0000","last_became_active":"2024-09-21T08:23:32.559194+0000","last_became_peered":"2024-09-21T08:23:32.559194+0000","last_unstale":"2024-09-21T08:24:13.512124+0000","last_undegraded":"2024-09-21T08:24:13.512124+0000","last_fullsized":"2024-09-21T08:24:13.512124+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.251123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,0],"acting":[4,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879717+0000","last_change":"2024-09-21T08:23:34.567872+0000","last_active":"2024-09-21T08:24:13.879717+0000","last_peered":"2024-09-21T08:24:13.879717+0000","last_clean":"2024-09-21T08:24:13.879717+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:24:13.879717+0000","last_undegraded":"2024-09-21T08:24:13.879717+0000","last_fullsized":"2024-09-21T08:24:13.879717+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.208597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.507725+0000","last_change":"2024-09-21T08:23:36.582491+0000","last_active":"2024-09-21T08:24:13.507725+0000","last_peered":"2024-09-21T08:24:13.507725+0000","last_clean":"2024-09-21T08:24:13.507725+0000","last_became_active":"2024-09-21T08:23:36.581720+0000","last_became_peered":"2024-09-21T08:23:36.581720+0000","last_unstale":"2024-09-21T08:24:13.507725+0000","last_undegraded":"2024-09-21T08:24:13.507725+0000","last_fullsized":"2024-09-21T08:24:13.507725+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.837636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879889+0000","last_change":"2024-09-21T08:23:30.543477+0000","last_active":"2024-09-21T08:24:13.879889+0000","last_peered":"2024-09-21T08:24:13.879889+0000","last_clean":"2024-09-21T08:24:13.879889+0000","last_became_active":"2024-09-21T08:23:30.543025+0000","last_became_peered":"2024-09-21T08:23:30.543025+0000","last_unstale":"2024-09-21T08:24:13.879889+0000","last_undegraded":"2024-09-21T08:24:13.879889+0000","last_fullsized":"2024-09-21T08:24:13.879889+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.290639+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512026+0000","last_change":"2024-09-21T08:23:32.559984+0000","last_active":"2024-09-21T08:24:13.512026+0000","last_peered":"2024-09-21T08:24:13.512026+0000","last_clean":"2024-09-21T08:24:13.512026+0000","last_became_active":"2024-09-21T08:23:32.559544+0000","last_became_peered":"2024-09-21T08:23:32.559544+0000","last_unstale":"2024-09-21T08:24:13.512026+0000","last_undegraded":"2024-09-21T08:24:13.512026+0000","last_fullsized":"2024-09-21T08:24:13.512026+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:31.803588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":7,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,1],"acting":[4,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880065+0000","last_change":"2024-09-21T08:23:34.572687+0000","last_active":"2024-09-21T08:24:13.880065+0000","last_peered":"2024-09-21T08:24:13.880065+0000","last_clean":"2024-09-21T08:24:13.880065+0000","last_became_active":"2024-09-21T08:23:34.572395+0000","last_became_peered":"2024-09-21T08:23:34.572395+0000","last_unstale":"2024-09-21T08:24:13.880065+0000","last_undegraded":"2024-09-21T08:24:13.880065+0000","last_fullsized":"2024-09-21T08:24:13.880065+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.042108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879431+0000","last_change":"2024-09-21T08:23:36.578510+0000","last_active":"2024-09-21T08:24:13.879431+0000","last_peered":"2024-09-21T08:24:13.879431+0000","last_clean":"2024-09-21T08:24:13.879431+0000","last_became_active":"2024-09-21T08:23:36.578227+0000","last_became_peered":"2024-09-21T08:23:36.578227+0000","last_unstale":"2024-09-21T08:24:13.879431+0000","last_undegraded":"2024-09-21T08:24:13.879431+0000","last_fullsized":"2024-09-21T08:24:13.879431+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.087695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,3],"acting":[5,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512303+0000","last_change":"2024-09-21T08:23:36.573671+0000","last_active":"2024-09-21T08:24:13.512303+0000","last_peered":"2024-09-21T08:24:13.512303+0000","last_clean":"2024-09-21T08:24:13.512303+0000","last_became_active":"2024-09-21T08:23:36.573384+0000","last_became_peered":"2024-09-21T08:23:36.573384+0000","last_unstale":"2024-09-21T08:24:13.512303+0000","last_undegraded":"2024-09-21T08:24:13.512303+0000","last_fullsized":"2024-09-21T08:24:13.512303+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.273680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5],"acting":[4,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508422+0000","last_change":"2024-09-21T08:23:30.543356+0000","last_active":"2024-09-21T08:24:13.508422+0000","last_peered":"2024-09-21T08:24:13.508422+0000","last_clean":"2024-09-21T08:24:13.508422+0000","last_became_active":"2024-09-21T08:23:30.542776+0000","last_became_peered":"2024-09-21T08:23:30.542776+0000","last_unstale":"2024-09-21T08:24:13.508422+0000","last_undegraded":"2024-09-21T08:24:13.508422+0000","last_fullsized":"2024-09-21T08:24:13.508422+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.314435+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,7],"acting":[0,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881662+0000","last_change":"2024-09-21T08:23:32.556660+0000","last_active":"2024-09-21T08:24:13.881662+0000","last_peered":"2024-09-21T08:24:13.881662+0000","last_clean":"2024-09-21T08:24:13.881662+0000","last_became_active":"2024-09-21T08:23:32.556378+0000","last_became_peered":"2024-09-21T08:23:32.556378+0000","last_unstale":"2024-09-21T08:24:13.881662+0000","last_undegraded":"2024-09-21T08:24:13.881662+0000","last_fullsized":"2024-09-21T08:24:13.881662+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.143529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,3],"acting":[2,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880911+0000","last_change":"2024-09-21T08:23:34.569008+0000","last_active":"2024-09-21T08:24:13.880911+0000","last_peered":"2024-09-21T08:24:13.880911+0000","last_clean":"2024-09-21T08:24:13.880911+0000","last_became_active":"2024-09-21T08:23:34.568562+0000","last_became_peered":"2024-09-21T08:23:34.568562+0000","last_unstale":"2024-09-21T08:24:13.880911+0000","last_undegraded":"2024-09-21T08:24:13.880911+0000","last_fullsized":"2024-09-21T08:24:13.880911+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.511849+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880329+0000","last_change":"2024-09-21T08:23:36.584447+0000","last_active":"2024-09-21T08:24:13.880329+0000","last_peered":"2024-09-21T08:24:13.880329+0000","last_clean":"2024-09-21T08:24:13.880329+0000","last_became_active":"2024-09-21T08:23:36.584154+0000","last_became_peered":"2024-09-21T08:23:36.584154+0000","last_unstale":"2024-09-21T08:24:13.880329+0000","last_undegraded":"2024-09-21T08:24:13.880329+0000","last_fullsized":"2024-09-21T08:24:13.880329+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.079729+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512274+0000","last_change":"2024-09-21T08:23:30.547824+0000","last_active":"2024-09-21T08:24:13.512274+0000","last_peered":"2024-09-21T08:24:13.512274+0000","last_clean":"2024-09-21T08:24:13.512274+0000","last_became_active":"2024-09-21T08:23:30.546325+0000","last_became_peered":"2024-09-21T08:23:30.546325+0000","last_unstale":"2024-09-21T08:24:13.512274+0000","last_undegraded":"2024-09-21T08:24:13.512274+0000","last_fullsized":"2024-09-21T08:24:13.512274+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.394437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880052+0000","last_change":"2024-09-21T08:23:32.563650+0000","last_active":"2024-09-21T08:24:13.880052+0000","last_peered":"2024-09-21T08:24:13.880052+0000","last_clean":"2024-09-21T08:24:13.880052+0000","last_became_active":"2024-09-21T08:23:32.563383+0000","last_became_peered":"2024-09-21T08:23:32.563383+0000","last_unstale":"2024-09-21T08:24:13.880052+0000","last_undegraded":"2024-09-21T08:24:13.880052+0000","last_fullsized":"2024-09-21T08:24:13.880052+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.102032+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512155+0000","last_change":"2024-09-21T08:23:34.571589+0000","last_active":"2024-09-21T08:24:13.512155+0000","last_peered":"2024-09-21T08:24:13.512155+0000","last_clean":"2024-09-21T08:24:13.512155+0000","last_became_active":"2024-09-21T08:23:34.571323+0000","last_became_peered":"2024-09-21T08:23:34.571323+0000","last_unstale":"2024-09-21T08:24:13.512155+0000","last_undegraded":"2024-09-21T08:24:13.512155+0000","last_fullsized":"2024-09-21T08:24:13.512155+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.905411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":22,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880474+0000","last_change":"2024-09-21T08:23:36.584003+0000","last_active":"2024-09-21T08:24:13.880474+0000","last_peered":"2024-09-21T08:24:13.880474+0000","last_clean":"2024-09-21T08:24:13.880474+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:24:13.880474+0000","last_undegraded":"2024-09-21T08:24:13.880474+0000","last_fullsized":"2024-09-21T08:24:13.880474+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.633904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879860+0000","last_change":"2024-09-21T08:23:30.540306+0000","last_active":"2024-09-21T08:24:13.879860+0000","last_peered":"2024-09-21T08:24:13.879860+0000","last_clean":"2024-09-21T08:24:13.879860+0000","last_became_active":"2024-09-21T08:23:30.539931+0000","last_became_peered":"2024-09-21T08:23:30.539931+0000","last_unstale":"2024-09-21T08:24:13.879860+0000","last_undegraded":"2024-09-21T08:24:13.879860+0000","last_fullsized":"2024-09-21T08:24:13.879860+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.106584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508651+0000","last_change":"2024-09-21T08:23:32.598036+0000","last_active":"2024-09-21T08:24:13.508651+0000","last_peered":"2024-09-21T08:24:13.508651+0000","last_clean":"2024-09-21T08:24:13.508651+0000","last_became_active":"2024-09-21T08:23:32.597548+0000","last_became_peered":"2024-09-21T08:23:32.597548+0000","last_unstale":"2024-09-21T08:24:13.508651+0000","last_undegraded":"2024-09-21T08:24:13.508651+0000","last_fullsized":"2024-09-21T08:24:13.508651+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.955005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"73'8","reported_seq":51,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:20.601392+0000","last_change":"2024-09-21T08:23:34.567530+0000","last_active":"2024-09-21T08:24:20.601392+0000","last_peered":"2024-09-21T08:24:20.601392+0000","last_clean":"2024-09-21T08:24:20.601392+0000","last_became_active":"2024-09-21T08:23:34.567257+0000","last_became_peered":"2024-09-21T08:23:34.567257+0000","last_unstale":"2024-09-21T08:24:20.601392+0000","last_undegraded":"2024-09-21T08:24:20.601392+0000","last_fullsized":"2024-09-21T08:24:20.601392+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.508940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.f","version":"73'15","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881320+0000","last_change":"2024-09-21T08:23:32.562879+0000","last_active":"2024-09-21T08:24:13.881320+0000","last_peered":"2024-09-21T08:24:13.881320+0000","last_clean":"2024-09-21T08:24:13.881320+0000","last_became_active":"2024-09-21T08:23:32.562594+0000","last_became_peered":"2024-09-21T08:23:32.562594+0000","last_unstale":"2024-09-21T08:24:13.881320+0000","last_undegraded":"2024-09-21T08:24:13.881320+0000","last_fullsized":"2024-09-21T08:24:13.881320+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.780782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3,4],"acting":[1,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879940+0000","last_change":"2024-09-21T08:23:30.546900+0000","last_active":"2024-09-21T08:24:13.879940+0000","last_peered":"2024-09-21T08:24:13.879940+0000","last_clean":"2024-09-21T08:24:13.879940+0000","last_became_active":"2024-09-21T08:23:30.546625+0000","last_became_peered":"2024-09-21T08:23:30.546625+0000","last_unstale":"2024-09-21T08:24:13.879940+0000","last_undegraded":"2024-09-21T08:24:13.879940+0000","last_fullsized":"2024-09-21T08:24:13.879940+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:48.387469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.e","version":"73'8","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:20.601373+0000","last_change":"2024-09-21T08:23:34.568257+0000","last_active":"2024-09-21T08:24:20.601373+0000","last_peered":"2024-09-21T08:24:20.601373+0000","last_clean":"2024-09-21T08:24:20.601373+0000","last_became_active":"2024-09-21T08:23:34.567963+0000","last_became_peered":"2024-09-21T08:23:34.567963+0000","last_unstale":"2024-09-21T08:24:20.601373+0000","last_undegraded":"2024-09-21T08:24:20.601373+0000","last_fullsized":"2024-09-21T08:24:20.601373+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:43.930223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,0],"acting":[4,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879810+0000","last_change":"2024-09-21T08:23:36.578919+0000","last_active":"2024-09-21T08:24:13.879810+0000","last_peered":"2024-09-21T08:24:13.879810+0000","last_clean":"2024-09-21T08:24:13.879810+0000","last_became_active":"2024-09-21T08:23:36.578648+0000","last_became_peered":"2024-09-21T08:23:36.578648+0000","last_unstale":"2024-09-21T08:24:13.879810+0000","last_undegraded":"2024-09-21T08:24:13.879810+0000","last_fullsized":"2024-09-21T08:24:13.879810+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.799668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.0","version":"73'18","reported_seq":61,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879717+0000","last_change":"2024-09-21T08:23:32.597719+0000","last_active":"2024-09-21T08:24:13.879717+0000","last_peered":"2024-09-21T08:24:13.879717+0000","last_clean":"2024-09-21T08:24:13.879717+0000","last_became_active":"2024-09-21T08:23:32.597431+0000","last_became_peered":"2024-09-21T08:23:32.597431+0000","last_unstale":"2024-09-21T08:24:13.879717+0000","last_undegraded":"2024-09-21T08:24:13.879717+0000","last_fullsized":"2024-09-21T08:24:13.879717+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.130776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":34,"num_read_kb":22,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880512+0000","last_change":"2024-09-21T08:23:30.546233+0000","last_active":"2024-09-21T08:24:13.880512+0000","last_peered":"2024-09-21T08:24:13.880512+0000","last_clean":"2024-09-21T08:24:13.880512+0000","last_became_active":"2024-09-21T08:23:30.545922+0000","last_became_peered":"2024-09-21T08:23:30.545922+0000","last_unstale":"2024-09-21T08:24:13.880512+0000","last_undegraded":"2024-09-21T08:24:13.880512+0000","last_fullsized":"2024-09-21T08:24:13.880512+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.861966+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.511851+0000","last_change":"2024-09-21T08:23:34.570926+0000","last_active":"2024-09-21T08:24:13.511851+0000","last_peered":"2024-09-21T08:24:13.511851+0000","last_clean":"2024-09-21T08:24:13.511851+0000","last_became_active":"2024-09-21T08:23:34.570648+0000","last_became_peered":"2024-09-21T08:23:34.570648+0000","last_unstale":"2024-09-21T08:24:13.511851+0000","last_undegraded":"2024-09-21T08:24:13.511851+0000","last_fullsized":"2024-09-21T08:24:13.511851+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.476055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,7],"acting":[4,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512075+0000","last_change":"2024-09-21T08:23:36.582947+0000","last_active":"2024-09-21T08:24:13.512075+0000","last_peered":"2024-09-21T08:24:13.512075+0000","last_clean":"2024-09-21T08:24:13.512075+0000","last_became_active":"2024-09-21T08:23:36.582610+0000","last_became_peered":"2024-09-21T08:23:36.582610+0000","last_unstale":"2024-09-21T08:24:13.512075+0000","last_undegraded":"2024-09-21T08:24:13.512075+0000","last_fullsized":"2024-09-21T08:24:13.512075+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.387653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,2],"acting":[4,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1","version":"73'14","reported_seq":50,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.511887+0000","last_change":"2024-09-21T08:23:32.553440+0000","last_active":"2024-09-21T08:24:13.511887+0000","last_peered":"2024-09-21T08:24:13.511887+0000","last_clean":"2024-09-21T08:24:13.511887+0000","last_became_active":"2024-09-21T08:23:32.552980+0000","last_became_peered":"2024-09-21T08:23:32.552980+0000","last_unstale":"2024-09-21T08:24:13.511887+0000","last_undegraded":"2024-09-21T08:24:13.511887+0000","last_fullsized":"2024-09-21T08:24:13.511887+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.960840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":14,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,6],"acting":[4,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":34,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508904+0000","last_change":"2024-09-21T08:23:30.543176+0000","last_active":"2024-09-21T08:24:13.508904+0000","last_peered":"2024-09-21T08:24:13.508904+0000","last_clean":"2024-09-21T08:24:13.508904+0000","last_became_active":"2024-09-21T08:23:30.542888+0000","last_became_peered":"2024-09-21T08:23:30.542888+0000","last_unstale":"2024-09-21T08:24:13.508904+0000","last_undegraded":"2024-09-21T08:24:13.508904+0000","last_fullsized":"2024-09-21T08:24:13.508904+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.545006+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"73'8","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:20.601278+0000","last_change":"2024-09-21T08:23:34.570603+0000","last_active":"2024-09-21T08:24:20.601278+0000","last_peered":"2024-09-21T08:24:20.601278+0000","last_clean":"2024-09-21T08:24:20.601278+0000","last_became_active":"2024-09-21T08:23:34.570321+0000","last_became_peered":"2024-09-21T08:23:34.570321+0000","last_unstale":"2024-09-21T08:24:20.601278+0000","last_undegraded":"2024-09-21T08:24:20.601278+0000","last_fullsized":"2024-09-21T08:24:20.601278+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.651545+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,4],"acting":[3,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880188+0000","last_change":"2024-09-21T08:23:36.582496+0000","last_active":"2024-09-21T08:24:13.880188+0000","last_peered":"2024-09-21T08:24:13.880188+0000","last_clean":"2024-09-21T08:24:13.880188+0000","last_became_active":"2024-09-21T08:23:36.582039+0000","last_became_peered":"2024-09-21T08:23:36.582039+0000","last_unstale":"2024-09-21T08:24:13.880188+0000","last_undegraded":"2024-09-21T08:24:13.880188+0000","last_fullsized":"2024-09-21T08:24:13.880188+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.819778+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,2],"acting":[7,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.2","version":"73'10","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880197+0000","last_change":"2024-09-21T08:23:32.557661+0000","last_active":"2024-09-21T08:24:13.880197+0000","last_peered":"2024-09-21T08:24:13.880197+0000","last_clean":"2024-09-21T08:24:13.880197+0000","last_became_active":"2024-09-21T08:23:32.555080+0000","last_became_peered":"2024-09-21T08:23:32.555080+0000","last_unstale":"2024-09-21T08:24:13.880197+0000","last_undegraded":"2024-09-21T08:24:13.880197+0000","last_fullsized":"2024-09-21T08:24:13.880197+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.633944+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880353+0000","last_change":"2024-09-21T08:23:30.546228+0000","last_active":"2024-09-21T08:24:13.880353+0000","last_peered":"2024-09-21T08:24:13.880353+0000","last_clean":"2024-09-21T08:24:13.880353+0000","last_became_active":"2024-09-21T08:23:30.545920+0000","last_became_peered":"2024-09-21T08:23:30.545920+0000","last_unstale":"2024-09-21T08:24:13.880353+0000","last_undegraded":"2024-09-21T08:24:13.880353+0000","last_fullsized":"2024-09-21T08:24:13.880353+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.464911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"73'8","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:20.601318+0000","last_change":"2024-09-21T08:23:34.568126+0000","last_active":"2024-09-21T08:24:20.601318+0000","last_peered":"2024-09-21T08:24:20.601318+0000","last_clean":"2024-09-21T08:24:20.601318+0000","last_became_active":"2024-09-21T08:23:34.563906+0000","last_became_peered":"2024-09-21T08:23:34.563906+0000","last_unstale":"2024-09-21T08:24:20.601318+0000","last_undegraded":"2024-09-21T08:24:20.601318+0000","last_fullsized":"2024-09-21T08:24:20.601318+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.761252+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,5],"acting":[0,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.507792+0000","last_change":"2024-09-21T08:23:36.588118+0000","last_active":"2024-09-21T08:24:13.507792+0000","last_peered":"2024-09-21T08:24:13.507792+0000","last_clean":"2024-09-21T08:24:13.507792+0000","last_became_active":"2024-09-21T08:23:36.587782+0000","last_became_peered":"2024-09-21T08:23:36.587782+0000","last_unstale":"2024-09-21T08:24:13.507792+0000","last_undegraded":"2024-09-21T08:24:13.507792+0000","last_fullsized":"2024-09-21T08:24:13.507792+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:44.580040+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.3","version":"73'19","reported_seq":65,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508334+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:24:13.508334+0000","last_peered":"2024-09-21T08:24:13.508334+0000","last_clean":"2024-09-21T08:24:13.508334+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:24:13.508334+0000","last_undegraded":"2024-09-21T08:24:13.508334+0000","last_fullsized":"2024-09-21T08:24:13.508334+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.235370+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881010+0000","last_change":"2024-09-21T08:23:30.547730+0000","last_active":"2024-09-21T08:24:13.881010+0000","last_peered":"2024-09-21T08:24:13.881010+0000","last_clean":"2024-09-21T08:24:13.881010+0000","last_became_active":"2024-09-21T08:23:30.547290+0000","last_became_peered":"2024-09-21T08:23:30.547290+0000","last_unstale":"2024-09-21T08:24:13.881010+0000","last_undegraded":"2024-09-21T08:24:13.881010+0000","last_fullsized":"2024-09-21T08:24:13.881010+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.347687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508691+0000","last_change":"2024-09-21T08:23:34.564467+0000","last_active":"2024-09-21T08:24:13.508691+0000","last_peered":"2024-09-21T08:24:13.508691+0000","last_clean":"2024-09-21T08:24:13.508691+0000","last_became_active":"2024-09-21T08:23:34.564157+0000","last_became_peered":"2024-09-21T08:23:34.564157+0000","last_unstale":"2024-09-21T08:24:13.508691+0000","last_undegraded":"2024-09-21T08:24:13.508691+0000","last_fullsized":"2024-09-21T08:24:13.508691+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.544611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879617+0000","last_change":"2024-09-21T08:23:36.582071+0000","last_active":"2024-09-21T08:24:13.879617+0000","last_peered":"2024-09-21T08:24:13.879617+0000","last_clean":"2024-09-21T08:24:13.879617+0000","last_became_active":"2024-09-21T08:23:36.581757+0000","last_became_peered":"2024-09-21T08:23:36.581757+0000","last_unstale":"2024-09-21T08:24:13.879617+0000","last_undegraded":"2024-09-21T08:24:13.879617+0000","last_fullsized":"2024-09-21T08:24:13.879617+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.873559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"73'28","reported_seq":89,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:20.601296+0000","last_change":"2024-09-21T08:23:32.562868+0000","last_active":"2024-09-21T08:24:20.601296+0000","last_peered":"2024-09-21T08:24:20.601296+0000","last_clean":"2024-09-21T08:24:20.601296+0000","last_became_active":"2024-09-21T08:23:32.561965+0000","last_became_peered":"2024-09-21T08:23:32.561965+0000","last_unstale":"2024-09-21T08:24:20.601296+0000","last_undegraded":"2024-09-21T08:24:20.601296+0000","last_fullsized":"2024-09-21T08:24:20.601296+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.844304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,3],"acting":[1,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512678+0000","last_change":"2024-09-21T08:23:30.541459+0000","last_active":"2024-09-21T08:24:13.512678+0000","last_peered":"2024-09-21T08:24:13.512678+0000","last_clean":"2024-09-21T08:24:13.512678+0000","last_became_active":"2024-09-21T08:23:30.541175+0000","last_became_peered":"2024-09-21T08:23:30.541175+0000","last_unstale":"2024-09-21T08:24:13.512678+0000","last_undegraded":"2024-09-21T08:24:13.512678+0000","last_fullsized":"2024-09-21T08:24:13.512678+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.508394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0,6],"acting":[4,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":43,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879627+0000","last_change":"2024-09-21T08:23:32.546026+0000","last_active":"2024-09-21T08:24:13.879627+0000","last_peered":"2024-09-21T08:24:13.879627+0000","last_clean":"2024-09-21T08:24:13.879627+0000","last_became_active":"2024-09-21T08:23:29.531243+0000","last_became_peered":"2024-09-21T08:23:29.531243+0000","last_unstale":"2024-09-21T08:24:13.879627+0000","last_undegraded":"2024-09-21T08:24:13.879627+0000","last_fullsized":"2024-09-21T08:24:13.879627+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:34.908720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,6],"acting":[5,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508023+0000","last_change":"2024-09-21T08:23:34.568259+0000","last_active":"2024-09-21T08:24:13.508023+0000","last_peered":"2024-09-21T08:24:13.508023+0000","last_clean":"2024-09-21T08:24:13.508023+0000","last_became_active":"2024-09-21T08:23:34.567955+0000","last_became_peered":"2024-09-21T08:23:34.567955+0000","last_unstale":"2024-09-21T08:24:13.508023+0000","last_undegraded":"2024-09-21T08:24:13.508023+0000","last_fullsized":"2024-09-21T08:24:13.508023+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.600803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881509+0000","last_change":"2024-09-21T08:23:36.577895+0000","last_active":"2024-09-21T08:24:13.881509+0000","last_peered":"2024-09-21T08:24:13.881509+0000","last_clean":"2024-09-21T08:24:13.881509+0000","last_became_active":"2024-09-21T08:23:36.577593+0000","last_became_peered":"2024-09-21T08:23:36.577593+0000","last_unstale":"2024-09-21T08:24:13.881509+0000","last_undegraded":"2024-09-21T08:24:13.881509+0000","last_fullsized":"2024-09-21T08:24:13.881509+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.663315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,7],"acting":[3,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":56,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881409+0000","last_change":"2024-09-21T08:23:32.560593+0000","last_active":"2024-09-21T08:24:13.881409+0000","last_peered":"2024-09-21T08:24:13.881409+0000","last_clean":"2024-09-21T08:24:13.881409+0000","last_became_active":"2024-09-21T08:23:32.560323+0000","last_became_peered":"2024-09-21T08:23:32.560323+0000","last_unstale":"2024-09-21T08:24:13.881409+0000","last_undegraded":"2024-09-21T08:24:13.881409+0000","last_fullsized":"2024-09-21T08:24:13.881409+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.516260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879720+0000","last_change":"2024-09-21T08:23:30.548894+0000","last_active":"2024-09-21T08:24:13.879720+0000","last_peered":"2024-09-21T08:24:13.879720+0000","last_clean":"2024-09-21T08:24:13.879720+0000","last_became_active":"2024-09-21T08:23:30.548308+0000","last_became_peered":"2024-09-21T08:23:30.548308+0000","last_unstale":"2024-09-21T08:24:13.879720+0000","last_undegraded":"2024-09-21T08:24:13.879720+0000","last_fullsized":"2024-09-21T08:24:13.879720+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.379264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,6],"acting":[1,2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":42,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879512+0000","last_change":"2024-09-21T08:23:32.552207+0000","last_active":"2024-09-21T08:24:13.879512+0000","last_peered":"2024-09-21T08:24:13.879512+0000","last_clean":"2024-09-21T08:24:13.879512+0000","last_became_active":"2024-09-21T08:23:29.533672+0000","last_became_peered":"2024-09-21T08:23:29.533672+0000","last_unstale":"2024-09-21T08:24:13.879512+0000","last_undegraded":"2024-09-21T08:24:13.879512+0000","last_fullsized":"2024-09-21T08:24:13.879512+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.009447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881131+0000","last_change":"2024-09-21T08:23:34.567541+0000","last_active":"2024-09-21T08:24:13.881131+0000","last_peered":"2024-09-21T08:24:13.881131+0000","last_clean":"2024-09-21T08:24:13.881131+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:24:13.881131+0000","last_undegraded":"2024-09-21T08:24:13.881131+0000","last_fullsized":"2024-09-21T08:24:13.881131+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:51.623026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880555+0000","last_change":"2024-09-21T08:23:36.585190+0000","last_active":"2024-09-21T08:24:13.880555+0000","last_peered":"2024-09-21T08:24:13.880555+0000","last_clean":"2024-09-21T08:24:13.880555+0000","last_became_active":"2024-09-21T08:23:36.584905+0000","last_became_peered":"2024-09-21T08:23:36.584905+0000","last_unstale":"2024-09-21T08:24:13.880555+0000","last_undegraded":"2024-09-21T08:24:13.880555+0000","last_fullsized":"2024-09-21T08:24:13.880555+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.290201+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,3],"acting":[7,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"73'12","reported_seq":47,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508731+0000","last_change":"2024-09-21T08:23:32.561943+0000","last_active":"2024-09-21T08:24:13.508731+0000","last_peered":"2024-09-21T08:24:13.508731+0000","last_clean":"2024-09-21T08:24:13.508731+0000","last_became_active":"2024-09-21T08:23:32.561617+0000","last_became_peered":"2024-09-21T08:23:32.561617+0000","last_unstale":"2024-09-21T08:24:13.508731+0000","last_undegraded":"2024-09-21T08:24:13.508731+0000","last_fullsized":"2024-09-21T08:24:13.508731+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.493572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508060+0000","last_change":"2024-09-21T08:23:30.547156+0000","last_active":"2024-09-21T08:24:13.508060+0000","last_peered":"2024-09-21T08:24:13.508060+0000","last_clean":"2024-09-21T08:24:13.508060+0000","last_became_active":"2024-09-21T08:23:30.546898+0000","last_became_peered":"2024-09-21T08:23:30.546898+0000","last_unstale":"2024-09-21T08:24:13.508060+0000","last_undegraded":"2024-09-21T08:24:13.508060+0000","last_fullsized":"2024-09-21T08:24:13.508060+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:54.718486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,3],"acting":[0,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":94,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:24.570724+0000","last_change":"2024-09-21T08:23:32.597264+0000","last_active":"2024-09-21T08:24:24.570724+0000","last_peered":"2024-09-21T08:24:24.570724+0000","last_clean":"2024-09-21T08:24:24.570724+0000","last_became_active":"2024-09-21T08:23:29.532520+0000","last_became_peered":"2024-09-21T08:23:29.532520+0000","last_unstale":"2024-09-21T08:24:24.570724+0000","last_undegraded":"2024-09-21T08:24:24.570724+0000","last_fullsized":"2024-09-21T08:24:24.570724+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.455707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880519+0000","last_change":"2024-09-21T08:23:34.570449+0000","last_active":"2024-09-21T08:24:13.880519+0000","last_peered":"2024-09-21T08:24:13.880519+0000","last_clean":"2024-09-21T08:24:13.880519+0000","last_became_active":"2024-09-21T08:23:34.569970+0000","last_became_peered":"2024-09-21T08:23:34.569970+0000","last_unstale":"2024-09-21T08:24:13.880519+0000","last_undegraded":"2024-09-21T08:24:13.880519+0000","last_fullsized":"2024-09-21T08:24:13.880519+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:40.151707+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880615+0000","last_change":"2024-09-21T08:23:36.585195+0000","last_active":"2024-09-21T08:24:13.880615+0000","last_peered":"2024-09-21T08:24:13.880615+0000","last_clean":"2024-09-21T08:24:13.880615+0000","last_became_active":"2024-09-21T08:23:36.584827+0000","last_became_peered":"2024-09-21T08:23:36.584827+0000","last_unstale":"2024-09-21T08:24:13.880615+0000","last_undegraded":"2024-09-21T08:24:13.880615+0000","last_fullsized":"2024-09-21T08:24:13.880615+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.376554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.5","version":"73'16","reported_seq":63,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:20.601427+0000","last_change":"2024-09-21T08:23:32.598546+0000","last_active":"2024-09-21T08:24:20.601427+0000","last_peered":"2024-09-21T08:24:20.601427+0000","last_clean":"2024-09-21T08:24:20.601427+0000","last_became_active":"2024-09-21T08:23:32.596949+0000","last_became_peered":"2024-09-21T08:23:32.596949+0000","last_unstale":"2024-09-21T08:24:20.601427+0000","last_undegraded":"2024-09-21T08:24:20.601427+0000","last_fullsized":"2024-09-21T08:24:20.601427+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.826771+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":25,"num_read_kb":15,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879980+0000","last_change":"2024-09-21T08:23:30.546494+0000","last_active":"2024-09-21T08:24:13.879980+0000","last_peered":"2024-09-21T08:24:13.879980+0000","last_clean":"2024-09-21T08:24:13.879980+0000","last_became_active":"2024-09-21T08:23:30.546218+0000","last_became_peered":"2024-09-21T08:23:30.546218+0000","last_unstale":"2024-09-21T08:24:13.879980+0000","last_undegraded":"2024-09-21T08:24:13.879980+0000","last_fullsized":"2024-09-21T08:24:13.879980+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.498978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,6],"acting":[3,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":288,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879849+0000","last_change":"2024-09-21T08:24:05.765943+0000","last_active":"2024-09-21T08:24:13.879849+0000","last_peered":"2024-09-21T08:24:13.879849+0000","last_clean":"2024-09-21T08:24:13.879849+0000","last_became_active":"2024-09-21T08:23:23.509818+0000","last_became_peered":"2024-09-21T08:23:23.509818+0000","last_unstale":"2024-09-21T08:24:13.879849+0000","last_undegraded":"2024-09-21T08:24:13.879849+0000","last_fullsized":"2024-09-21T08:24:13.879849+0000","mapping_epoch":62,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":63,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:24:05.765804+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:05.827066+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,6],"acting":[7,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880557+0000","last_change":"2024-09-21T08:23:34.568359+0000","last_active":"2024-09-21T08:24:13.880557+0000","last_peered":"2024-09-21T08:24:13.880557+0000","last_clean":"2024-09-21T08:24:13.880557+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:24:13.880557+0000","last_undegraded":"2024-09-21T08:24:13.880557+0000","last_fullsized":"2024-09-21T08:24:13.880557+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.246877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880258+0000","last_change":"2024-09-21T08:23:36.584454+0000","last_active":"2024-09-21T08:24:13.880258+0000","last_peered":"2024-09-21T08:24:13.880258+0000","last_clean":"2024-09-21T08:24:13.880258+0000","last_became_active":"2024-09-21T08:23:36.584095+0000","last_became_peered":"2024-09-21T08:23:36.584095+0000","last_unstale":"2024-09-21T08:24:13.880258+0000","last_undegraded":"2024-09-21T08:24:13.880258+0000","last_fullsized":"2024-09-21T08:24:13.880258+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.123137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,4],"acting":[5,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512430+0000","last_change":"2024-09-21T08:23:32.557075+0000","last_active":"2024-09-21T08:24:13.512430+0000","last_peered":"2024-09-21T08:24:13.512430+0000","last_clean":"2024-09-21T08:24:13.512430+0000","last_became_active":"2024-09-21T08:23:32.556696+0000","last_became_peered":"2024-09-21T08:23:32.556696+0000","last_unstale":"2024-09-21T08:24:13.512430+0000","last_undegraded":"2024-09-21T08:24:13.512430+0000","last_fullsized":"2024-09-21T08:24:13.512430+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.647238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512569+0000","last_change":"2024-09-21T08:23:30.547989+0000","last_active":"2024-09-21T08:24:13.512569+0000","last_peered":"2024-09-21T08:24:13.512569+0000","last_clean":"2024-09-21T08:24:13.512569+0000","last_became_active":"2024-09-21T08:23:30.547176+0000","last_became_peered":"2024-09-21T08:23:30.547176+0000","last_unstale":"2024-09-21T08:24:13.512569+0000","last_undegraded":"2024-09-21T08:24:13.512569+0000","last_fullsized":"2024-09-21T08:24:13.512569+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.295162+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880222+0000","last_change":"2024-09-21T08:23:34.570250+0000","last_active":"2024-09-21T08:24:13.880222+0000","last_peered":"2024-09-21T08:24:13.880222+0000","last_clean":"2024-09-21T08:24:13.880222+0000","last_became_active":"2024-09-21T08:23:34.569571+0000","last_became_peered":"2024-09-21T08:23:34.569571+0000","last_unstale":"2024-09-21T08:24:13.880222+0000","last_undegraded":"2024-09-21T08:24:13.880222+0000","last_fullsized":"2024-09-21T08:24:13.880222+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.053187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,6],"acting":[5,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881229+0000","last_change":"2024-09-21T08:23:36.587842+0000","last_active":"2024-09-21T08:24:13.881229+0000","last_peered":"2024-09-21T08:24:13.881229+0000","last_clean":"2024-09-21T08:24:13.881229+0000","last_became_active":"2024-09-21T08:23:36.587561+0000","last_became_peered":"2024-09-21T08:23:36.587561+0000","last_unstale":"2024-09-21T08:24:13.881229+0000","last_undegraded":"2024-09-21T08:24:13.881229+0000","last_fullsized":"2024-09-21T08:24:13.881229+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:03.999379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6,5],"acting":[3,6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"73'17","reported_seq":57,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512426+0000","last_change":"2024-09-21T08:23:32.556741+0000","last_active":"2024-09-21T08:24:13.512426+0000","last_peered":"2024-09-21T08:24:13.512426+0000","last_clean":"2024-09-21T08:24:13.512426+0000","last_became_active":"2024-09-21T08:23:32.556355+0000","last_became_peered":"2024-09-21T08:23:32.556355+0000","last_unstale":"2024-09-21T08:24:13.512426+0000","last_undegraded":"2024-09-21T08:24:13.512426+0000","last_fullsized":"2024-09-21T08:24:13.512426+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.708055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":19,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.509084+0000","last_change":"2024-09-21T08:23:30.537030+0000","last_active":"2024-09-21T08:24:13.509084+0000","last_peered":"2024-09-21T08:24:13.509084+0000","last_clean":"2024-09-21T08:24:13.509084+0000","last_became_active":"2024-09-21T08:23:30.536566+0000","last_became_peered":"2024-09-21T08:23:30.536566+0000","last_unstale":"2024-09-21T08:24:13.509084+0000","last_undegraded":"2024-09-21T08:24:13.509084+0000","last_fullsized":"2024-09-21T08:24:13.509084+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.146930+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,1],"acting":[6,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880704+0000","last_change":"2024-09-21T08:23:34.569384+0000","last_active":"2024-09-21T08:24:13.880704+0000","last_peered":"2024-09-21T08:24:13.880704+0000","last_clean":"2024-09-21T08:24:13.880704+0000","last_became_active":"2024-09-21T08:23:34.568900+0000","last_became_peered":"2024-09-21T08:23:34.568900+0000","last_unstale":"2024-09-21T08:24:13.880704+0000","last_undegraded":"2024-09-21T08:24:13.880704+0000","last_fullsized":"2024-09-21T08:24:13.880704+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.163373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,0],"acting":[1,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881351+0000","last_change":"2024-09-21T08:23:36.586002+0000","last_active":"2024-09-21T08:24:13.881351+0000","last_peered":"2024-09-21T08:24:13.881351+0000","last_clean":"2024-09-21T08:24:13.881351+0000","last_became_active":"2024-09-21T08:23:36.585695+0000","last_became_peered":"2024-09-21T08:23:36.585695+0000","last_unstale":"2024-09-21T08:24:13.881351+0000","last_undegraded":"2024-09-21T08:24:13.881351+0000","last_fullsized":"2024-09-21T08:24:13.881351+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.916781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,4],"acting":[2,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.c","version":"73'10","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512467+0000","last_change":"2024-09-21T08:23:32.555172+0000","last_active":"2024-09-21T08:24:13.512467+0000","last_peered":"2024-09-21T08:24:13.512467+0000","last_clean":"2024-09-21T08:24:13.512467+0000","last_became_active":"2024-09-21T08:23:32.554845+0000","last_became_peered":"2024-09-21T08:23:32.554845+0000","last_unstale":"2024-09-21T08:24:13.512467+0000","last_undegraded":"2024-09-21T08:24:13.512467+0000","last_fullsized":"2024-09-21T08:24:13.512467+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:50.339439+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,6],"acting":[4,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880410+0000","last_change":"2024-09-21T08:23:30.545460+0000","last_active":"2024-09-21T08:24:13.880410+0000","last_peered":"2024-09-21T08:24:13.880410+0000","last_clean":"2024-09-21T08:24:13.880410+0000","last_became_active":"2024-09-21T08:23:30.545187+0000","last_became_peered":"2024-09-21T08:23:30.545187+0000","last_unstale":"2024-09-21T08:24:13.880410+0000","last_undegraded":"2024-09-21T08:24:13.880410+0000","last_fullsized":"2024-09-21T08:24:13.880410+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:52.674857+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,4],"acting":[3,0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:20.601279+0000","last_change":"2024-09-21T08:23:34.571530+0000","last_active":"2024-09-21T08:24:20.601279+0000","last_peered":"2024-09-21T08:24:20.601279+0000","last_clean":"2024-09-21T08:24:20.601279+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:24:20.601279+0000","last_undegraded":"2024-09-21T08:24:20.601279+0000","last_fullsized":"2024-09-21T08:24:20.601279+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.789578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512171+0000","last_change":"2024-09-21T08:23:36.582871+0000","last_active":"2024-09-21T08:24:13.512171+0000","last_peered":"2024-09-21T08:24:13.512171+0000","last_clean":"2024-09-21T08:24:13.512171+0000","last_became_active":"2024-09-21T08:23:36.582575+0000","last_became_peered":"2024-09-21T08:23:36.582575+0000","last_unstale":"2024-09-21T08:24:13.512171+0000","last_undegraded":"2024-09-21T08:24:13.512171+0000","last_fullsized":"2024-09-21T08:24:13.512171+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.578837+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.b","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508278+0000","last_change":"2024-09-21T08:23:32.560978+0000","last_active":"2024-09-21T08:24:13.508278+0000","last_peered":"2024-09-21T08:24:13.508278+0000","last_clean":"2024-09-21T08:24:13.508278+0000","last_became_active":"2024-09-21T08:23:32.560658+0000","last_became_peered":"2024-09-21T08:23:32.560658+0000","last_unstale":"2024-09-21T08:24:13.508278+0000","last_undegraded":"2024-09-21T08:24:13.508278+0000","last_fullsized":"2024-09-21T08:24:13.508278+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.965888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,4],"acting":[0,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879972+0000","last_change":"2024-09-21T08:23:30.542309+0000","last_active":"2024-09-21T08:24:13.879972+0000","last_peered":"2024-09-21T08:24:13.879972+0000","last_clean":"2024-09-21T08:24:13.879972+0000","last_became_active":"2024-09-21T08:23:30.541992+0000","last_became_peered":"2024-09-21T08:23:30.541992+0000","last_unstale":"2024-09-21T08:24:13.879972+0000","last_undegraded":"2024-09-21T08:24:13.879972+0000","last_fullsized":"2024-09-21T08:24:13.879972+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.859320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,6],"acting":[5,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880681+0000","last_change":"2024-09-21T08:23:34.566539+0000","last_active":"2024-09-21T08:24:13.880681+0000","last_peered":"2024-09-21T08:24:13.880681+0000","last_clean":"2024-09-21T08:24:13.880681+0000","last_became_active":"2024-09-21T08:23:34.566242+0000","last_became_peered":"2024-09-21T08:23:34.566242+0000","last_unstale":"2024-09-21T08:24:13.880681+0000","last_undegraded":"2024-09-21T08:24:13.880681+0000","last_fullsized":"2024-09-21T08:24:13.880681+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.194173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,3],"acting":[2,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508008+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:24:13.508008+0000","last_peered":"2024-09-21T08:24:13.508008+0000","last_clean":"2024-09-21T08:24:13.508008+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:24:13.508008+0000","last_undegraded":"2024-09-21T08:24:13.508008+0000","last_fullsized":"2024-09-21T08:24:13.508008+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:53.339671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":60,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508748+0000","last_change":"2024-09-21T08:23:32.597880+0000","last_active":"2024-09-21T08:24:13.508748+0000","last_peered":"2024-09-21T08:24:13.508748+0000","last_clean":"2024-09-21T08:24:13.508748+0000","last_became_active":"2024-09-21T08:23:32.597589+0000","last_became_peered":"2024-09-21T08:23:32.597589+0000","last_unstale":"2024-09-21T08:24:13.508748+0000","last_undegraded":"2024-09-21T08:24:13.508748+0000","last_fullsized":"2024-09-21T08:24:13.508748+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.870820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":32,"num_read_kb":21,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,1,7],"acting":[6,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880724+0000","last_change":"2024-09-21T08:23:30.545405+0000","last_active":"2024-09-21T08:24:13.880724+0000","last_peered":"2024-09-21T08:24:13.880724+0000","last_clean":"2024-09-21T08:24:13.880724+0000","last_became_active":"2024-09-21T08:23:30.545068+0000","last_became_peered":"2024-09-21T08:23:30.545068+0000","last_unstale":"2024-09-21T08:24:13.880724+0000","last_undegraded":"2024-09-21T08:24:13.880724+0000","last_fullsized":"2024-09-21T08:24:13.880724+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.179419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,6],"acting":[7,5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880834+0000","last_change":"2024-09-21T08:23:34.564923+0000","last_active":"2024-09-21T08:24:13.880834+0000","last_peered":"2024-09-21T08:24:13.880834+0000","last_clean":"2024-09-21T08:24:13.880834+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:24:13.880834+0000","last_undegraded":"2024-09-21T08:24:13.880834+0000","last_fullsized":"2024-09-21T08:24:13.880834+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.183697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879766+0000","last_change":"2024-09-21T08:23:36.582099+0000","last_active":"2024-09-21T08:24:13.879766+0000","last_peered":"2024-09-21T08:24:13.879766+0000","last_clean":"2024-09-21T08:24:13.879766+0000","last_became_active":"2024-09-21T08:23:36.581796+0000","last_became_peered":"2024-09-21T08:23:36.581796+0000","last_unstale":"2024-09-21T08:24:13.879766+0000","last_undegraded":"2024-09-21T08:24:13.879766+0000","last_fullsized":"2024-09-21T08:24:13.879766+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.618273+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.9","version":"73'12","reported_seq":52,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512020+0000","last_change":"2024-09-21T08:23:32.557284+0000","last_active":"2024-09-21T08:24:13.512020+0000","last_peered":"2024-09-21T08:24:13.512020+0000","last_clean":"2024-09-21T08:24:13.512020+0000","last_became_active":"2024-09-21T08:23:32.556876+0000","last_became_peered":"2024-09-21T08:23:32.556876+0000","last_unstale":"2024-09-21T08:24:13.512020+0000","last_undegraded":"2024-09-21T08:24:13.512020+0000","last_fullsized":"2024-09-21T08:24:13.512020+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.982500+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,3],"acting":[4,1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880726+0000","last_change":"2024-09-21T08:23:30.548671+0000","last_active":"2024-09-21T08:24:13.880726+0000","last_peered":"2024-09-21T08:24:13.880726+0000","last_clean":"2024-09-21T08:24:13.880726+0000","last_became_active":"2024-09-21T08:23:30.548350+0000","last_became_peered":"2024-09-21T08:23:30.548350+0000","last_unstale":"2024-09-21T08:24:13.880726+0000","last_undegraded":"2024-09-21T08:24:13.880726+0000","last_fullsized":"2024-09-21T08:24:13.880726+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:56.489110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879297+0000","last_change":"2024-09-21T08:23:34.566472+0000","last_active":"2024-09-21T08:24:13.879297+0000","last_peered":"2024-09-21T08:24:13.879297+0000","last_clean":"2024-09-21T08:24:13.879297+0000","last_became_active":"2024-09-21T08:23:34.566193+0000","last_became_peered":"2024-09-21T08:23:34.566193+0000","last_unstale":"2024-09-21T08:24:13.879297+0000","last_undegraded":"2024-09-21T08:24:13.879297+0000","last_fullsized":"2024-09-21T08:24:13.879297+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:58.818551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880129+0000","last_change":"2024-09-21T08:23:36.579879+0000","last_active":"2024-09-21T08:24:13.880129+0000","last_peered":"2024-09-21T08:24:13.880129+0000","last_clean":"2024-09-21T08:24:13.880129+0000","last_became_active":"2024-09-21T08:23:36.579603+0000","last_became_peered":"2024-09-21T08:23:36.579603+0000","last_unstale":"2024-09-21T08:24:13.880129+0000","last_undegraded":"2024-09-21T08:24:13.880129+0000","last_fullsized":"2024-09-21T08:24:13.880129+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.717936+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.8","version":"73'15","reported_seq":54,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879309+0000","last_change":"2024-09-21T08:23:32.598217+0000","last_active":"2024-09-21T08:24:13.879309+0000","last_peered":"2024-09-21T08:24:13.879309+0000","last_clean":"2024-09-21T08:24:13.879309+0000","last_became_active":"2024-09-21T08:23:32.597543+0000","last_became_peered":"2024-09-21T08:23:32.597543+0000","last_unstale":"2024-09-21T08:24:13.879309+0000","last_undegraded":"2024-09-21T08:24:13.879309+0000","last_fullsized":"2024-09-21T08:24:13.879309+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.907666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,6],"acting":[5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880865+0000","last_change":"2024-09-21T08:23:30.551520+0000","last_active":"2024-09-21T08:24:13.880865+0000","last_peered":"2024-09-21T08:24:13.880865+0000","last_clean":"2024-09-21T08:24:13.880865+0000","last_became_active":"2024-09-21T08:23:30.551203+0000","last_became_peered":"2024-09-21T08:23:30.551203+0000","last_unstale":"2024-09-21T08:24:13.880865+0000","last_undegraded":"2024-09-21T08:24:13.880865+0000","last_fullsized":"2024-09-21T08:24:13.880865+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:29.708394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,0],"acting":[7,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"73'8","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:20.601374+0000","last_change":"2024-09-21T08:23:34.569818+0000","last_active":"2024-09-21T08:24:20.601374+0000","last_peered":"2024-09-21T08:24:20.601374+0000","last_clean":"2024-09-21T08:24:20.601374+0000","last_became_active":"2024-09-21T08:23:34.569490+0000","last_became_peered":"2024-09-21T08:23:34.569490+0000","last_unstale":"2024-09-21T08:24:20.601374+0000","last_undegraded":"2024-09-21T08:24:20.601374+0000","last_fullsized":"2024-09-21T08:24:20.601374+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.884010+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879465+0000","last_change":"2024-09-21T08:23:36.579253+0000","last_active":"2024-09-21T08:24:13.879465+0000","last_peered":"2024-09-21T08:24:13.879465+0000","last_clean":"2024-09-21T08:24:13.879465+0000","last_became_active":"2024-09-21T08:23:36.578952+0000","last_became_peered":"2024-09-21T08:23:36.578952+0000","last_unstale":"2024-09-21T08:24:13.879465+0000","last_undegraded":"2024-09-21T08:24:13.879465+0000","last_fullsized":"2024-09-21T08:24:13.879465+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.185192+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,0],"acting":[5,6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508773+0000","last_change":"2024-09-21T08:23:30.542360+0000","last_active":"2024-09-21T08:24:13.508773+0000","last_peered":"2024-09-21T08:24:13.508773+0000","last_clean":"2024-09-21T08:24:13.508773+0000","last_became_active":"2024-09-21T08:23:30.542060+0000","last_became_peered":"2024-09-21T08:23:30.542060+0000","last_unstale":"2024-09-21T08:24:13.508773+0000","last_undegraded":"2024-09-21T08:24:13.508773+0000","last_fullsized":"2024-09-21T08:24:13.508773+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:58.968344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,5],"acting":[6,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":38,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880484+0000","last_change":"2024-09-21T08:23:32.598912+0000","last_active":"2024-09-21T08:24:13.880484+0000","last_peered":"2024-09-21T08:24:13.880484+0000","last_clean":"2024-09-21T08:24:13.880484+0000","last_became_active":"2024-09-21T08:23:32.598518+0000","last_became_peered":"2024-09-21T08:23:32.598518+0000","last_unstale":"2024-09-21T08:24:13.880484+0000","last_undegraded":"2024-09-21T08:24:13.880484+0000","last_fullsized":"2024-09-21T08:24:13.880484+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:32.806988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":9,"num_read_kb":6,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1],"acting":[3,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880017+0000","last_change":"2024-09-21T08:23:34.566165+0000","last_active":"2024-09-21T08:24:13.880017+0000","last_peered":"2024-09-21T08:24:13.880017+0000","last_clean":"2024-09-21T08:24:13.880017+0000","last_became_active":"2024-09-21T08:23:34.565662+0000","last_became_peered":"2024-09-21T08:23:34.565662+0000","last_unstale":"2024-09-21T08:24:13.880017+0000","last_undegraded":"2024-09-21T08:24:13.880017+0000","last_fullsized":"2024-09-21T08:24:13.880017+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:01.290518+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,1],"acting":[5,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880450+0000","last_change":"2024-09-21T08:23:36.579682+0000","last_active":"2024-09-21T08:24:13.880450+0000","last_peered":"2024-09-21T08:24:13.880450+0000","last_clean":"2024-09-21T08:24:13.880450+0000","last_became_active":"2024-09-21T08:23:36.579384+0000","last_became_peered":"2024-09-21T08:23:36.579384+0000","last_unstale":"2024-09-21T08:24:13.880450+0000","last_undegraded":"2024-09-21T08:24:13.880450+0000","last_fullsized":"2024-09-21T08:24:13.880450+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.156840+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,4],"acting":[7,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508810+0000","last_change":"2024-09-21T08:23:32.599079+0000","last_active":"2024-09-21T08:24:13.508810+0000","last_peered":"2024-09-21T08:24:13.508810+0000","last_clean":"2024-09-21T08:24:13.508810+0000","last_became_active":"2024-09-21T08:23:32.598778+0000","last_became_peered":"2024-09-21T08:23:32.598778+0000","last_unstale":"2024-09-21T08:24:13.508810+0000","last_undegraded":"2024-09-21T08:24:13.508810+0000","last_fullsized":"2024-09-21T08:24:13.508810+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.131011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879936+0000","last_change":"2024-09-21T08:23:30.548443+0000","last_active":"2024-09-21T08:24:13.879936+0000","last_peered":"2024-09-21T08:24:13.879936+0000","last_clean":"2024-09-21T08:24:13.879936+0000","last_became_active":"2024-09-21T08:23:30.548096+0000","last_became_peered":"2024-09-21T08:23:30.548096+0000","last_unstale":"2024-09-21T08:24:13.879936+0000","last_undegraded":"2024-09-21T08:24:13.879936+0000","last_fullsized":"2024-09-21T08:24:13.879936+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:45.029746+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880389+0000","last_change":"2024-09-21T08:23:34.566473+0000","last_active":"2024-09-21T08:24:13.880389+0000","last_peered":"2024-09-21T08:24:13.880389+0000","last_clean":"2024-09-21T08:24:13.880389+0000","last_became_active":"2024-09-21T08:23:34.566160+0000","last_became_peered":"2024-09-21T08:23:34.566160+0000","last_unstale":"2024-09-21T08:24:13.880389+0000","last_undegraded":"2024-09-21T08:24:13.880389+0000","last_fullsized":"2024-09-21T08:24:13.880389+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.960353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881308+0000","last_change":"2024-09-21T08:23:36.587149+0000","last_active":"2024-09-21T08:24:13.881308+0000","last_peered":"2024-09-21T08:24:13.881308+0000","last_clean":"2024-09-21T08:24:13.881308+0000","last_became_active":"2024-09-21T08:23:36.586865+0000","last_became_peered":"2024-09-21T08:23:36.586865+0000","last_unstale":"2024-09-21T08:24:13.881308+0000","last_undegraded":"2024-09-21T08:24:13.881308+0000","last_fullsized":"2024-09-21T08:24:13.881308+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:42.991875+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880113+0000","last_change":"2024-09-21T08:23:32.599185+0000","last_active":"2024-09-21T08:24:13.880113+0000","last_peered":"2024-09-21T08:24:13.880113+0000","last_clean":"2024-09-21T08:24:13.880113+0000","last_became_active":"2024-09-21T08:23:32.598748+0000","last_became_peered":"2024-09-21T08:23:32.598748+0000","last_unstale":"2024-09-21T08:24:13.880113+0000","last_undegraded":"2024-09-21T08:24:13.880113+0000","last_fullsized":"2024-09-21T08:24:13.880113+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.141293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508197+0000","last_change":"2024-09-21T08:23:30.548111+0000","last_active":"2024-09-21T08:24:13.508197+0000","last_peered":"2024-09-21T08:24:13.508197+0000","last_clean":"2024-09-21T08:24:13.508197+0000","last_became_active":"2024-09-21T08:23:30.547824+0000","last_became_peered":"2024-09-21T08:23:30.547824+0000","last_unstale":"2024-09-21T08:24:13.508197+0000","last_undegraded":"2024-09-21T08:24:13.508197+0000","last_fullsized":"2024-09-21T08:24:13.508197+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:55.463183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"73'8","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:20.601323+0000","last_change":"2024-09-21T08:23:34.570551+0000","last_active":"2024-09-21T08:24:20.601323+0000","last_peered":"2024-09-21T08:24:20.601323+0000","last_clean":"2024-09-21T08:24:20.601323+0000","last_became_active":"2024-09-21T08:23:34.570211+0000","last_became_peered":"2024-09-21T08:23:34.570211+0000","last_unstale":"2024-09-21T08:24:20.601323+0000","last_undegraded":"2024-09-21T08:24:20.601323+0000","last_fullsized":"2024-09-21T08:24:20.601323+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:57.547474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512312+0000","last_change":"2024-09-21T08:23:36.582684+0000","last_active":"2024-09-21T08:24:13.512312+0000","last_peered":"2024-09-21T08:24:13.512312+0000","last_clean":"2024-09-21T08:24:13.512312+0000","last_became_active":"2024-09-21T08:23:36.582369+0000","last_became_peered":"2024-09-21T08:23:36.582369+0000","last_unstale":"2024-09-21T08:24:13.512312+0000","last_undegraded":"2024-09-21T08:24:13.512312+0000","last_fullsized":"2024-09-21T08:24:13.512312+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:48.140821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.14","version":"73'10","reported_seq":44,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.881447+0000","last_change":"2024-09-21T08:23:32.597630+0000","last_active":"2024-09-21T08:24:13.881447+0000","last_peered":"2024-09-21T08:24:13.881447+0000","last_clean":"2024-09-21T08:24:13.881447+0000","last_became_active":"2024-09-21T08:23:32.597296+0000","last_became_peered":"2024-09-21T08:23:32.597296+0000","last_unstale":"2024-09-21T08:24:13.881447+0000","last_undegraded":"2024-09-21T08:24:13.881447+0000","last_fullsized":"2024-09-21T08:24:13.881447+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:46.289337+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,7],"acting":[3,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":39,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880378+0000","last_change":"2024-09-21T08:23:30.549864+0000","last_active":"2024-09-21T08:24:13.880378+0000","last_peered":"2024-09-21T08:24:13.880378+0000","last_clean":"2024-09-21T08:24:13.880378+0000","last_became_active":"2024-09-21T08:23:30.549562+0000","last_became_peered":"2024-09-21T08:23:30.549562+0000","last_unstale":"2024-09-21T08:24:13.880378+0000","last_undegraded":"2024-09-21T08:24:13.880378+0000","last_fullsized":"2024-09-21T08:24:13.880378+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.957399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:20.601315+0000","last_change":"2024-09-21T08:23:34.567042+0000","last_active":"2024-09-21T08:24:20.601315+0000","last_peered":"2024-09-21T08:24:20.601315+0000","last_clean":"2024-09-21T08:24:20.601315+0000","last_became_active":"2024-09-21T08:23:34.566736+0000","last_became_peered":"2024-09-21T08:23:34.566736+0000","last_unstale":"2024-09-21T08:24:20.601315+0000","last_undegraded":"2024-09-21T08:24:20.601315+0000","last_fullsized":"2024-09-21T08:24:20.601315+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:33.785039+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508446+0000","last_change":"2024-09-21T08:23:36.589503+0000","last_active":"2024-09-21T08:24:13.508446+0000","last_peered":"2024-09-21T08:24:13.508446+0000","last_clean":"2024-09-21T08:24:13.508446+0000","last_became_active":"2024-09-21T08:23:36.589224+0000","last_became_peered":"2024-09-21T08:23:36.589224+0000","last_unstale":"2024-09-21T08:24:13.508446+0000","last_undegraded":"2024-09-21T08:24:13.508446+0000","last_fullsized":"2024-09-21T08:24:13.508446+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.636183+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512283+0000","last_change":"2024-09-21T08:23:32.559882+0000","last_active":"2024-09-21T08:24:13.512283+0000","last_peered":"2024-09-21T08:24:13.512283+0000","last_clean":"2024-09-21T08:24:13.512283+0000","last_became_active":"2024-09-21T08:23:32.559574+0000","last_became_peered":"2024-09-21T08:23:32.559574+0000","last_unstale":"2024-09-21T08:24:13.512283+0000","last_undegraded":"2024-09-21T08:24:13.512283+0000","last_fullsized":"2024-09-21T08:24:13.512283+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.305662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,1],"acting":[4,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.512646+0000","last_change":"2024-09-21T08:23:30.546837+0000","last_active":"2024-09-21T08:24:13.512646+0000","last_peered":"2024-09-21T08:24:13.512646+0000","last_clean":"2024-09-21T08:24:13.512646+0000","last_became_active":"2024-09-21T08:23:30.546542+0000","last_became_peered":"2024-09-21T08:23:30.546542+0000","last_unstale":"2024-09-21T08:24:13.512646+0000","last_undegraded":"2024-09-21T08:24:13.512646+0000","last_fullsized":"2024-09-21T08:24:13.512646+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.479344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,6],"acting":[4,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880060+0000","last_change":"2024-09-21T08:23:34.571300+0000","last_active":"2024-09-21T08:24:13.880060+0000","last_peered":"2024-09-21T08:24:13.880060+0000","last_clean":"2024-09-21T08:24:13.880060+0000","last_became_active":"2024-09-21T08:23:34.571034+0000","last_became_peered":"2024-09-21T08:23:34.571034+0000","last_unstale":"2024-09-21T08:24:13.880060+0000","last_undegraded":"2024-09-21T08:24:13.880060+0000","last_fullsized":"2024-09-21T08:24:13.880060+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.978453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,3],"acting":[1,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880287+0000","last_change":"2024-09-21T08:23:36.585748+0000","last_active":"2024-09-21T08:24:13.880287+0000","last_peered":"2024-09-21T08:24:13.880287+0000","last_clean":"2024-09-21T08:24:13.880287+0000","last_became_active":"2024-09-21T08:23:36.585492+0000","last_became_peered":"2024-09-21T08:23:36.585492+0000","last_unstale":"2024-09-21T08:24:13.880287+0000","last_undegraded":"2024-09-21T08:24:13.880287+0000","last_fullsized":"2024-09-21T08:24:13.880287+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:36.492501+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":45,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880287+0000","last_change":"2024-09-21T08:23:32.556550+0000","last_active":"2024-09-21T08:24:13.880287+0000","last_peered":"2024-09-21T08:24:13.880287+0000","last_clean":"2024-09-21T08:24:13.880287+0000","last_became_active":"2024-09-21T08:23:32.556189+0000","last_became_peered":"2024-09-21T08:23:32.556189+0000","last_unstale":"2024-09-21T08:24:13.880287+0000","last_undegraded":"2024-09-21T08:24:13.880287+0000","last_fullsized":"2024-09-21T08:24:13.880287+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:51.354291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6,2],"acting":[1,6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880657+0000","last_change":"2024-09-21T08:23:30.549643+0000","last_active":"2024-09-21T08:24:13.880657+0000","last_peered":"2024-09-21T08:24:13.880657+0000","last_clean":"2024-09-21T08:24:13.880657+0000","last_became_active":"2024-09-21T08:23:30.549317+0000","last_became_peered":"2024-09-21T08:23:30.549317+0000","last_unstale":"2024-09-21T08:24:13.880657+0000","last_undegraded":"2024-09-21T08:24:13.880657+0000","last_fullsized":"2024-09-21T08:24:13.880657+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.895759+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,4],"acting":[7,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880222+0000","last_change":"2024-09-21T08:23:34.565128+0000","last_active":"2024-09-21T08:24:13.880222+0000","last_peered":"2024-09-21T08:24:13.880222+0000","last_clean":"2024-09-21T08:24:13.880222+0000","last_became_active":"2024-09-21T08:23:34.564738+0000","last_became_peered":"2024-09-21T08:23:34.564738+0000","last_unstale":"2024-09-21T08:24:13.880222+0000","last_undegraded":"2024-09-21T08:24:13.880222+0000","last_fullsized":"2024-09-21T08:24:13.880222+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.717622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.507602+0000","last_change":"2024-09-21T08:23:36.580460+0000","last_active":"2024-09-21T08:24:13.507602+0000","last_peered":"2024-09-21T08:24:13.507602+0000","last_clean":"2024-09-21T08:24:13.507602+0000","last_became_active":"2024-09-21T08:23:36.579949+0000","last_became_peered":"2024-09-21T08:23:36.579949+0000","last_unstale":"2024-09-21T08:24:13.507602+0000","last_undegraded":"2024-09-21T08:24:13.507602+0000","last_fullsized":"2024-09-21T08:24:13.507602+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:49.377442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879985+0000","last_change":"2024-09-21T08:23:32.597672+0000","last_active":"2024-09-21T08:24:13.879985+0000","last_peered":"2024-09-21T08:24:13.879985+0000","last_clean":"2024-09-21T08:24:13.879985+0000","last_became_active":"2024-09-21T08:23:32.597281+0000","last_became_peered":"2024-09-21T08:23:32.597281+0000","last_unstale":"2024-09-21T08:24:13.879985+0000","last_undegraded":"2024-09-21T08:24:13.879985+0000","last_fullsized":"2024-09-21T08:24:13.879985+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:00.493132+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,6],"acting":[3,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880120+0000","last_change":"2024-09-21T08:23:30.544860+0000","last_active":"2024-09-21T08:24:13.880120+0000","last_peered":"2024-09-21T08:24:13.880120+0000","last_clean":"2024-09-21T08:24:13.880120+0000","last_became_active":"2024-09-21T08:23:30.544486+0000","last_became_peered":"2024-09-21T08:23:30.544486+0000","last_unstale":"2024-09-21T08:24:13.880120+0000","last_undegraded":"2024-09-21T08:24:13.880120+0000","last_fullsized":"2024-09-21T08:24:13.880120+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.462170+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879434+0000","last_change":"2024-09-21T08:23:34.569813+0000","last_active":"2024-09-21T08:24:13.879434+0000","last_peered":"2024-09-21T08:24:13.879434+0000","last_clean":"2024-09-21T08:24:13.879434+0000","last_became_active":"2024-09-21T08:23:34.569485+0000","last_became_peered":"2024-09-21T08:23:34.569485+0000","last_unstale":"2024-09-21T08:24:13.879434+0000","last_undegraded":"2024-09-21T08:24:13.879434+0000","last_fullsized":"2024-09-21T08:24:13.879434+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:59.575850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880334+0000","last_change":"2024-09-21T08:23:36.586500+0000","last_active":"2024-09-21T08:24:13.880334+0000","last_peered":"2024-09-21T08:24:13.880334+0000","last_clean":"2024-09-21T08:24:13.880334+0000","last_became_active":"2024-09-21T08:23:36.586172+0000","last_became_peered":"2024-09-21T08:23:36.586172+0000","last_unstale":"2024-09-21T08:24:13.880334+0000","last_undegraded":"2024-09-21T08:24:13.880334+0000","last_fullsized":"2024-09-21T08:24:13.880334+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:02.388100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,6],"acting":[3,0,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":35,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879627+0000","last_change":"2024-09-21T08:23:32.563756+0000","last_active":"2024-09-21T08:24:13.879627+0000","last_peered":"2024-09-21T08:24:13.879627+0000","last_clean":"2024-09-21T08:24:13.879627+0000","last_became_active":"2024-09-21T08:23:32.563484+0000","last_became_peered":"2024-09-21T08:23:32.563484+0000","last_unstale":"2024-09-21T08:24:13.879627+0000","last_undegraded":"2024-09-21T08:24:13.879627+0000","last_fullsized":"2024-09-21T08:24:13.879627+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.069344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1,6],"acting":[3,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508346+0000","last_change":"2024-09-21T08:23:30.544403+0000","last_active":"2024-09-21T08:24:13.508346+0000","last_peered":"2024-09-21T08:24:13.508346+0000","last_clean":"2024-09-21T08:24:13.508346+0000","last_became_active":"2024-09-21T08:23:30.544066+0000","last_became_peered":"2024-09-21T08:23:30.544066+0000","last_unstale":"2024-09-21T08:24:13.508346+0000","last_undegraded":"2024-09-21T08:24:13.508346+0000","last_fullsized":"2024-09-21T08:24:13.508346+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.462796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508689+0000","last_change":"2024-09-21T08:23:34.566607+0000","last_active":"2024-09-21T08:24:13.508689+0000","last_peered":"2024-09-21T08:24:13.508689+0000","last_clean":"2024-09-21T08:24:13.508689+0000","last_became_active":"2024-09-21T08:23:34.566337+0000","last_became_peered":"2024-09-21T08:23:34.566337+0000","last_unstale":"2024-09-21T08:24:13.508689+0000","last_undegraded":"2024-09-21T08:24:13.508689+0000","last_fullsized":"2024-09-21T08:24:13.508689+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:39.230638+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,7],"acting":[6,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879522+0000","last_change":"2024-09-21T08:23:36.582773+0000","last_active":"2024-09-21T08:24:13.879522+0000","last_peered":"2024-09-21T08:24:13.879522+0000","last_clean":"2024-09-21T08:24:13.879522+0000","last_became_active":"2024-09-21T08:23:36.582150+0000","last_became_peered":"2024-09-21T08:23:36.582150+0000","last_unstale":"2024-09-21T08:24:13.879522+0000","last_undegraded":"2024-09-21T08:24:13.879522+0000","last_fullsized":"2024-09-21T08:24:13.879522+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:47.263690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,4],"acting":[7,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":21,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.880703+0000","last_change":"2024-09-21T08:23:36.576004+0000","last_active":"2024-09-21T08:24:13.880703+0000","last_peered":"2024-09-21T08:24:13.880703+0000","last_clean":"2024-09-21T08:24:13.880703+0000","last_became_active":"2024-09-21T08:23:36.575701+0000","last_became_peered":"2024-09-21T08:23:36.575701+0000","last_unstale":"2024-09-21T08:24:13.880703+0000","last_undegraded":"2024-09-21T08:24:13.880703+0000","last_fullsized":"2024-09-21T08:24:13.880703+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:38.955927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.879837+0000","last_change":"2024-09-21T08:23:30.544201+0000","last_active":"2024-09-21T08:24:13.879837+0000","last_peered":"2024-09-21T08:24:13.879837+0000","last_clean":"2024-09-21T08:24:13.879837+0000","last_became_active":"2024-09-21T08:23:30.543932+0000","last_became_peered":"2024-09-21T08:23:30.543932+0000","last_unstale":"2024-09-21T08:24:13.879837+0000","last_undegraded":"2024-09-21T08:24:13.879837+0000","last_fullsized":"2024-09-21T08:24:13.879837+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:37.933119+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":48,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.509093+0000","last_change":"2024-09-21T08:23:32.549263+0000","last_active":"2024-09-21T08:24:13.509093+0000","last_peered":"2024-09-21T08:24:13.509093+0000","last_clean":"2024-09-21T08:24:13.509093+0000","last_became_active":"2024-09-21T08:23:32.548835+0000","last_became_peered":"2024-09-21T08:23:32.548835+0000","last_unstale":"2024-09-21T08:24:13.509093+0000","last_undegraded":"2024-09-21T08:24:13.509093+0000","last_fullsized":"2024-09-21T08:24:13.509093+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:41.423215+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,1],"acting":[6,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":25,"reported_epoch":77,"state":"active+clean","last_fresh":"2024-09-21T08:24:13.508578+0000","last_change":"2024-09-21T08:23:34.573167+0000","last_active":"2024-09-21T08:24:13.508578+0000","last_peered":"2024-09-21T08:24:13.508578+0000","last_clean":"2024-09-21T08:24:13.508578+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:24:13.508578+0000","last_undegraded":"2024-09-21T08:24:13.508578+0000","last_fullsized":"2024-09-21T08:24:13.508578+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:24:35.897951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":64,"ondisk_log_size":64,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":698,"num_read_kb":455,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":11106,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":393,"ondisk_log_size":393,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":4839,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":45,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1771104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1781760,"data_stored":1771104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":3}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070480,"num_pgs":53,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32416,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712736,"statfs":{"total":95995035648,"available":95961841664,"internally_reserved":0,"allocated":1257472,"data_stored":869659,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":49,"seq":210453397526,"num_pgs":43,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28232,"kb_used_data":1204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716920,"statfs":{"total":95995035648,"available":95966126080,"internally_reserved":0,"allocated":1232896,"data_stored":867581,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659161,"num_pgs":47,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27664,"kb_used_data":636,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717488,"statfs":{"total":95995035648,"available":95966707712,"internally_reserved":0,"allocated":651264,"data_stored":278021,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986207,"num_pgs":58,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":31844,"kb_used_data":656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713308,"statfs":{"total":95995035648,"available":95962427392,"internally_reserved":0,"allocated":671744,"data_stored":278811,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182435,"num_pgs":56,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27696,"kb_used_data":668,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717456,"statfs":{"total":95995035648,"available":95966674944,"internally_reserved":0,"allocated":684032,"data_stored":278553,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411369,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27656,"kb_used_data":628,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717496,"statfs":{"total":95995035648,"available":95966715904,"internally_reserved":0,"allocated":643072,"data_stored":278790,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574893,"num_pgs":57,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27720,"kb_used_data":692,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717432,"statfs":{"total":95995035648,"available":95966650368,"internally_reserved":0,"allocated":708608,"data_stored":279011,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738418,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28256,"kb_used_data":1228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716896,"statfs":{"total":95995035648,"available":95966101504,"internally_reserved":0,"allocated":1257472,"data_stored":868375,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":92,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":1429,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1320,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":990,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1034,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":1254,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:24:27.107 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-09-21T08:24:27.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd dump 2024-09-21T08:24:27.354 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:27.743 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:27 smithi137 ceph-mon[29634]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:27.743 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:27 smithi137 ceph-mon[29634]: from='client.44681 v1:172.21.15.137:0/584312318' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:27.744 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:27 smithi137 ceph-mon[41044]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:27.744 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:27 smithi137 ceph-mon[41044]: from='client.44681 v1:172.21.15.137:0/584312318' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:27.999 INFO:teuthology.orchestra.run.smithi137.stdout:epoch 77 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:fsid ec8566d6-77f1-11ef-baf6-efdc52797490 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:created 2024-09-21T08:18:21.794737+0000 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:modified 2024-09-21T08:24:13.497405+0000 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:crush_version 18 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:full_ratio 0.95 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:backfillfull_ratio 0.9 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:nearfull_ratio 0.85 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:require_min_compat_client luminous 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:min_compat_client jewel 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:require_osd_release squid 2024-09-21T08:24:28.000 INFO:teuthology.orchestra.run.smithi137.stdout:stretch_mode_enabled false 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 7.89 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:pool 2 'datapool' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 3 pgp_num 3 autoscale_mode off last_change 68 flags hashpspool,selfmanaged_snaps stripe_width 0 application rbd read_balance_score 2.65 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:pool 3 '.rgw.root' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 67 flags hashpspool stripe_width 0 application rgw read_balance_score 1.50 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:pool 4 'default.rgw.log' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 69 flags hashpspool stripe_width 0 application rgw read_balance_score 2.25 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:pool 5 'default.rgw.control' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 71 flags hashpspool stripe_width 0 application rgw read_balance_score 1.25 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:pool 6 'default.rgw.meta' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 73 flags hashpspool stripe_width 0 pg_autoscale_bias 4 application rgw read_balance_score 1.75 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:max_osd 8 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0 up in weight 1 up_from 8 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6801/3938795530 v1:172.21.15.137:6802/3938795530 exists,up 1b007d1d-6c6b-4a68-9e68-2c5fd7d26977 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1 up in weight 1 up_from 13 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6805/2743268361 v1:172.21.15.137:6806/2743268361 exists,up 8651068e-694f-4bd4-9917-5c55670aca5d 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2 up in weight 1 up_from 18 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6809/1500397557 v1:172.21.15.137:6810/1500397557 exists,up 2a22f0f7-c610-4995-98a7-d4785efb1e5d 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3 up in weight 1 up_from 25 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6813/617155021 v1:172.21.15.137:6814/617155021 exists,up 73f21606-38be-4578-a86b-bb417217c0e0 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4 up in weight 1 up_from 31 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6800/70360958 v1:172.21.15.165:6801/70360958 exists,up e5f6762b-65f5-4dc7-8e54-9087149ae6ae 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5 up in weight 1 up_from 41 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6804/1835779380 v1:172.21.15.165:6805/1835779380 exists,up f7f8c456-b77f-4435-a860-202e64dda8dd 2024-09-21T08:24:28.001 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6 up in weight 1 up_from 49 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6808/3820354208 v1:172.21.15.165:6809/3820354208 exists,up b8b3df16-73fa-41a5-a93f-117c599d488b 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7 up in weight 1 up_from 59 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6812/1933524273 v1:172.21.15.165:6813/1933524273 exists,up ddf234e4-1494-40bd-a31b-333b01717c3f 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/1478868457 expires 2024-09-22T08:23:59.682278+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/1738156348 expires 2024-09-22T08:18:48.673163+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/2643910317 expires 2024-09-22T08:18:48.673163+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/705906958 expires 2024-09-22T08:19:12.726560+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/3936900881 expires 2024-09-22T08:19:12.726560+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/223603733 expires 2024-09-22T08:19:12.726560+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/1718301516 expires 2024-09-22T08:23:59.682278+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:6800/2580000250 expires 2024-09-22T08:18:48.673163+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/3166961425 expires 2024-09-22T08:23:59.682278+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/700535732 expires 2024-09-22T08:18:48.673163+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:6800/1900990534 expires 2024-09-22T08:19:12.726560+0000 2024-09-21T08:24:28.002 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:6800/1825289826 expires 2024-09-22T08:23:59.682278+0000 2024-09-21T08:24:28.003 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/3010106803 expires 2024-09-22T08:23:59.682278+0000 2024-09-21T08:24:28.003 INFO:teuthology.orchestra.run.smithi137.stdout:blocklist 172.21.15.137:0/3442439903 expires 2024-09-22T08:23:59.682278+0000 2024-09-21T08:24:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:27 smithi165 ceph-mon[32893]: pgmap v17: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:27 smithi165 ceph-mon[32893]: from='client.44681 v1:172.21.15.137:0/584312318' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:24:28.764 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:28 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/692004227' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-21T08:24:28.765 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:28 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/692004227' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-21T08:24:28.793 INFO:tasks.cephadm.ceph_manager.ceph:['osd.0 up in weight 1 up_from 8 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6801/3938795530 v1:172.21.15.137:6802/3938795530 exists,up 1b007d1d-6c6b-4a68-9e68-2c5fd7d26977', 'osd.1 up in weight 1 up_from 13 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6805/2743268361 v1:172.21.15.137:6806/2743268361 exists,up 8651068e-694f-4bd4-9917-5c55670aca5d', 'osd.2 up in weight 1 up_from 18 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6809/1500397557 v1:172.21.15.137:6810/1500397557 exists,up 2a22f0f7-c610-4995-98a7-d4785efb1e5d', 'osd.3 up in weight 1 up_from 25 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.137:6813/617155021 v1:172.21.15.137:6814/617155021 exists,up 73f21606-38be-4578-a86b-bb417217c0e0', 'osd.4 up in weight 1 up_from 31 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6800/70360958 v1:172.21.15.165:6801/70360958 exists,up e5f6762b-65f5-4dc7-8e54-9087149ae6ae', 'osd.5 up in weight 1 up_from 41 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6804/1835779380 v1:172.21.15.165:6805/1835779380 exists,up f7f8c456-b77f-4435-a860-202e64dda8dd', 'osd.6 up in weight 1 up_from 49 up_thru 69 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6808/3820354208 v1:172.21.15.165:6809/3820354208 exists,up b8b3df16-73fa-41a5-a93f-117c599d488b', 'osd.7 up in weight 1 up_from 59 up_thru 71 down_at 0 last_clean_interval [0,0) v1:172.21.15.165:6812/1933524273 v1:172.21.15.165:6813/1933524273 exists,up ddf234e4-1494-40bd-a31b-333b01717c3f'] 2024-09-21T08:24:28.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon mon.a config show 2024-09-21T08:24:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:28 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/692004227' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-09-21T08:24:29.036 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:29.504 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:24:29.504 INFO:teuthology.orchestra.run.smithi137.stdout: "name": "mon.a", 2024-09-21T08:24:29.504 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster": "ceph", 2024-09-21T08:24:29.504 INFO:teuthology.orchestra.run.smithi137.stdout: "admin_socket": "/var/run/ceph/ceph-mon.a.asok", 2024-09-21T08:24:29.504 INFO:teuthology.orchestra.run.smithi137.stdout: "admin_socket_mode": "", 2024-09-21T08:24:29.504 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_allow_insecure_global_id_reclaim": "false", 2024-09-21T08:24:29.504 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_client_required": "cephx, none", 2024-09-21T08:24:29.504 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_cluster_required": "cephx", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_debug": "true", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_expose_insecure_global_id_reclaim": "true", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_mon_ticket_ttl": "660.000000", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_service_required": "cephx", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_service_ticket_ttl": "240.000000", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "auth_supported": "", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_aio": "true", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_aio_max_queue_depth": "1024", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_aio_poll_ms": "250", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_aio_reap_max": "16", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_async_discard": "false", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_block_size": "4096", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_debug_aio": "false", 2024-09-21T08:24:29.505 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_debug_aio_log_age": "5.000000", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_debug_aio_suicide_timeout": "60.000000", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_debug_inflight_ios": "false", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_enable_discard": "false", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_flock_retry": "3", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_flock_retry_interval": "0.100000", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_inject_crash": "0", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_inject_crash_flush_delay": "2", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_ioring": "false", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_ioring_hipri": "false", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_ioring_sqthread_poll": "false", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_nvme_unbind_from_kernel": "false", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_read_buffer_alignment": "4096", 2024-09-21T08:24:29.506 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_read_preallocated_huge_buffers": "", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_stalled_read_warn_lifetime": "86400", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_stalled_read_warn_threshold": "1", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bdev_type": "", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_alloc_size": "1048576", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_allocator": "hybrid", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_buffered_io": "true", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_check_for_zeros": "false", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_check_volume_selector_often": "false", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_check_volume_selector_on_umount": "false", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_compact_log_sync": "false", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_failed_shared_alloc_cooldown": "600.000000", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_log_compact_min_ratio": "5.000000", 2024-09-21T08:24:29.507 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_log_compact_min_size": "16777216", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_log_replay_check_allocations": "true", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_max_log_runway": "4194304", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_max_prefetch": "1048576", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_min_flush_size": "524288", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_min_log_runway": "1048576", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_replay_recovery": "false", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_replay_recovery_disable_compact": "false", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_shared_alloc_size": "65536", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluefs_sync_write": "false", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_2q_cache_kin_ratio": "0.500000", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_2q_cache_kout_ratio": "0.500000", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_alloc_stats_dump_interval": "86400.000000", 2024-09-21T08:24:29.508 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_allocation_from_file": "true", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_allocator": "hybrid", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_avl_alloc_bf_free_pct": "4", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_avl_alloc_bf_threshold": "131072", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_avl_alloc_ff_max_search_bytes": "16777216", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_avl_alloc_ff_max_search_count": "100", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bdev_label_multi": "true", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bdev_label_multi_upgrade": "false", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bdev_label_require_all": "true", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bitmapallocator_blocks_per_zone": "1024", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bitmapallocator_span_size": "1024", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_blobid_prealloc": "10240", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_create": "true", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_db_create": "false", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_db_path": "", 2024-09-21T08:24:29.509 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_db_size": "0", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_path": "", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_preallocate_file": "false", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_size": "107374182400", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_wal_create": "false", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_wal_path": "", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_block_wal_size": "100663296", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bluefs": "true", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bluefs_alloc_failure_dump_interval": "0.000000", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bluefs_env_mirror": "false", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_bluefs_max_free": "10737418240", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_age_bin_interval": "1.000000", 2024-09-21T08:24:29.510 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_age_bins_data": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_age_bins_kv": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_age_bins_kv_onode": "0 0 0 0 0 0 0 0 0 720", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_age_bins_meta": "1 2 6 24 120 720 0 0 0 0", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_autotune": "true", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_autotune_interval": "5.000000", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_kv_onode_ratio": "0.040000", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_kv_ratio": "0.450000", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_meta_ratio": "0.450000", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_size": "0", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_size_hdd": "1073741824", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_size_ssd": "3221225472", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_trim_interval": "0.050000", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_trim_max_skip_pinned": "1000", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cache_type": "2q", 2024-09-21T08:24:29.511 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_cleaner_sleep_interval": "5.000000", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_clone_cow": "true", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_algorithm": "snappy", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_max_blob_size": "0", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_max_blob_size_hdd": "65536", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_max_blob_size_ssd": "65536", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_min_blob_size": "0", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_min_blob_size_hdd": "8192", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_min_blob_size_ssd": "65536", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_mode": "none", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_compression_required_ratio": "0.875000", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_csum_type": "crc32c", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_enforce_settings": "default", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_freelist": "false", 2024-09-21T08:24:29.512 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_fsck_abort": "false", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_inject_allocation_from_file_failure": "0.000000", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_inject_csum_err_probability": "0.000000", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_inject_read_err": "false", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_legacy_omap": "false", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_no_reuse_blocks": "false", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_omit_block_device_write": "false", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_omit_kv_commit": "false", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_permit_any_bdev_label": "false", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_prefragment_max": "1048576", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_random_read_err": "0.000000", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_randomize_serial_transaction": "0", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_small_allocations": "0", 2024-09-21T08:24:29.513 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_debug_too_many_blobs_threshold": "24576", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_default_buffered_read": "true", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_default_buffered_write": "false", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_deferred_batch_ops": "0", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_deferred_batch_ops_hdd": "64", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_deferred_batch_ops_ssd": "16", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_elastic_shared_blobs": "true", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_extent_map_inline_shard_prealloc_size": "256", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_extent_map_shard_max_size": "1200", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_extent_map_shard_min_size": "150", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_extent_map_shard_target_size": "500", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_extent_map_shard_target_size_slop": "0.200000", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fail_eio": "false", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_freelist_blocks_per_key": "128", 2024-09-21T08:24:29.514 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_error_on_no_per_pg_omap": "false", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_error_on_no_per_pool_omap": "false", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_error_on_no_per_pool_stats": "false", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_mkfs": "true", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_mkfs_deep": "false", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_mount": "false", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_mount_deep": "false", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_umount": "false", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_on_umount_deep": "false", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_quick_fix_on_mount": "false", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_quick_fix_threads": "2", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_read_bytes_cap": "67108864", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_fsck_shared_blob_tracker_size": "0.031250", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_gc_enable_blob_threshold": "0", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_gc_enable_total_threshold": "0", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_hybrid_alloc_mem_cap": "67108864", 2024-09-21T08:24:29.515 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_ignore_data_csum": "false", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_kv_sync_util_logging_s": "10.000000", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_kvbackend": "rocksdb", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_log_collection_list_age": "60.000000", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_log_omap_iterator_age": "5.000000", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_log_op_age": "5.000000", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_alloc_size": "0", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_blob_size": "0", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_blob_size_hdd": "65536", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_blob_size_ssd": "65536", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_defer_interval": "3.000000", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_max_deferred_txc": "32", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_min_alloc_size": "0", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_min_alloc_size_hdd": "4096", 2024-09-21T08:24:29.516 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_min_alloc_size_ssd": "4096", 2024-09-21T08:24:29.517 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_nid_prealloc": "1024", 2024-09-21T08:24:29.517 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_prefer_deferred_size": "0", 2024-09-21T08:24:29.517 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_prefer_deferred_size_hdd": "65536", 2024-09-21T08:24:29.517 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_prefer_deferred_size_ssd": "0", 2024-09-21T08:24:29.517 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_qfsck_on_mount": "true", 2024-09-21T08:24:29.517 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_retry_disk_reads": "3", 2024-09-21T08:24:29.517 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_rocksdb_cf": "true", 2024-09-21T08:24:29.518 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_rocksdb_cfs": "m(3) p(3,0-12) O(3,0-13)=block_cache={type=binned_lru} L=min_write_buffer_number_to_merge=32 P=min_write_buffer_number_to_merge=32", 2024-09-21T08:24:29.518 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_rocksdb_options": "compression=kLZ4Compression,max_write_buffer_number=64,min_write_buffer_number_to_merge=6,compaction_style=kCompactionStyleLevel,write_buffer_size=16777216,max_background_jobs=4,level0_file_num_compaction_trigger=8,max_bytes_for_level_base=1073741824,max_bytes_for_level_multiplier=8,compaction_readahead_size=2MB,max_total_wal_size=1073741824,writable_file_max_buffer_size=0", 2024-09-21T08:24:29.518 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_rocksdb_options_annex": "", 2024-09-21T08:24:29.518 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_slow_ops_warn_lifetime": "86400", 2024-09-21T08:24:29.518 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_slow_ops_warn_threshold": "1", 2024-09-21T08:24:29.518 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_spdk_coremask": "0x1", 2024-09-21T08:24:29.518 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_spdk_io_sleep": "5", 2024-09-21T08:24:29.518 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_spdk_max_io_completion": "0", 2024-09-21T08:24:29.518 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_spdk_mem": "512", 2024-09-21T08:24:29.518 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_sync_submit_transaction": "false", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_bytes": "67108864", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_cost_per_io": "0", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_cost_per_io_hdd": "670000", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_cost_per_io_ssd": "4000", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_deferred_bytes": "134217728", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_throttle_trace_rate": "0.000000", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_tracing": "false", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_use_optimal_io_size_for_min_alloc_size": "false", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_volume_selection_policy": "use_some_extra", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_volume_selection_reserved": "0", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_volume_selection_reserved_factor": "2.000000", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_warn_on_bluefs_spillover": "true", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_warn_on_legacy_statfs": "true", 2024-09-21T08:24:29.519 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_warn_on_no_per_pg_omap": "false", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_warn_on_no_per_pool_omap": "true", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_warn_on_spurious_read_errors": "true", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "bluestore_zero_block_detection": "false", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephadm_path": "/usr/sbin/cephadm", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_action_update_interval": "2", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_directory_scan_interval": "10", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_max_concurrent_directory_syncs": "3", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_max_consecutive_failures_per_directory": "10", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_max_snapshot_sync_per_cycle": "3", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_mount_timeout": "10", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_perf_stats_prio": "5", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_restart_mirror_on_blocklist_interval": "30", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_restart_mirror_on_failure_interval": "20", 2024-09-21T08:24:29.520 INFO:teuthology.orchestra.run.smithi137.stdout: "cephfs_mirror_retry_failed_directories_interval": "60", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "cephsqlite_blocklist_dead_locker": "true", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "cephsqlite_lock_renewal_interval": "2000", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "cephsqlite_lock_renewal_timeout": "30000", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_cluster_require_signatures": "false", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_cluster_require_version": "2", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_require_signatures": "false", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_require_version": "2", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_service_require_signatures": "false", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_service_require_version": "2", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "cephx_sign_messages": "true", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "chdir": "", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "client_acl_type": "", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "client_asio_thread_count": "2", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "client_cache_mid": "0.750000", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "client_cache_size": "16384", 2024-09-21T08:24:29.521 INFO:teuthology.orchestra.run.smithi137.stdout: "client_caps_release_delay": "5", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_check_pool_perm": "true", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_collect_and_send_global_metrics": "false", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_debug_force_sync_read": "false", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_debug_getattr_caps": "false", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_debug_inject_features": "", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_debug_inject_tick_delay": "0", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_die_on_failed_dentry_invalidate": "true", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_die_on_failed_remount": "false", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_dirsize_rbytes": "true", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_force_lazyio": "false", 2024-09-21T08:24:29.522 INFO:teuthology.orchestra.run.smithi137.stdout: "client_fs": "", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_inject_fixed_oldest_tid": "false", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_inject_release_failure": "false", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_max_inline_size": "4096", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_max_retries_on_remount_failure": "5", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_mds_namespace": "", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_metadata": "", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_mount_gid": "-1", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_mount_timeout": "300", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_mount_uid": "-1", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_mountpoint": "/", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_notify_timeout": "10", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc": "true", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc_max_dirty": "104857600", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc_max_dirty_age": "5.000000", 2024-09-21T08:24:29.523 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc_max_objects": "1000", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc_size": "209715200", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_oc_target_dirty": "8388608", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_permissions": "true", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_quota": "true", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_quota_df": "true", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_readahead_max_bytes": "0", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_readahead_max_periods": "4", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_readahead_min": "131072", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_reconnect_stale": "false", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_shutdown_timeout": "30", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_snapdir": ".snap", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_tick_interval": "1", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_trace": "", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_try_dentry_invalidate": "false", 2024-09-21T08:24:29.524 INFO:teuthology.orchestra.run.smithi137.stdout: "client_use_faked_inos": "false", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "client_use_random_mds": "false", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_graylog": "false", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_graylog_host": "127.0.0.1", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_graylog_port": "12201", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_monitors": "default=true", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_syslog": "false", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_syslog_facility": "default=daemon audit=local0", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "clog_to_syslog_level": "info", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "-", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_network": "", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_network_interface": "", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "compressor_zlib_isal": "false", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "compressor_zlib_level": "5", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "compressor_zlib_winsize": "-15", 2024-09-21T08:24:29.525 INFO:teuthology.orchestra.run.smithi137.stdout: "compressor_zstd_level": "1", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "container_image": "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crash_dir": "/var/lib/ceph/crash", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_alien_op_num_threads": "6", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_alien_thread_cpu_cores": "", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_osd_obc_lru_size": "512", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_osd_scheduler_concurrency": "0", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_osd_stat_interval": "0", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_seastar_cpu_cores": "", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crimson_seastar_num_threads": "0", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_location": "", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_location_hook": "", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_location_hook_timeout": "10", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "daemonize": "false", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "daos_pool": "tank", 2024-09-21T08:24:29.526 INFO:teuthology.orchestra.run.smithi137.stdout: "dbstore_config_uri": "file:/var/lib/ceph/radosgw/dbstore-config.db", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "dbstore_db_dir": "/var/lib/ceph/radosgw", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "dbstore_db_name_prefix": "dbstore", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_alienstore": "0/5", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_allow_any_pool_priority": "false", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_asok": "1/5", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_asok_assert_abort": "false", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_asserts_on_shutdown": "true", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_auth": "1/5", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_bdev": "1/3", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_bluefs": "1/5", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_bluestore": "1/5", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_buffer": "0/1", 2024-09-21T08:24:29.527 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_ceph_exporter": "1/5", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_cephfs_mirror": "0/5", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_cephsqlite": "0/5", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_client": "0/5", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_compressor": "1/5", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_context": "0/1", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_crimson_interrupt": "0/5", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_crush": "1/1", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_crypto": "1/5", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_cyanstore": "0/5", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_deliberately_leak_memory": "false", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_disable_randomized_ping": "false", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_dpdk": "1/5", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_eventtrace": "1/5", 2024-09-21T08:24:29.528 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_filer": "0/1", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_filestore": "1/3", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_finisher": "1/1", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_fuse": "1/5", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_heartbeat_testing_span": "0", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_heartbeatmap": "1/5", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_immutable_obj_cache": "0/5", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_javaclient": "1/5", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_journal": "1/3", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_journaler": "0/5", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_kstore": "1/5", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_lockdep": "0/1", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mclock": "1/5", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds": "1/5", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_balancer": "1/5", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_locker": "1/5", 2024-09-21T08:24:29.529 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_log": "1/5", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_log_expire": "1/5", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_migrator": "1/5", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mds_quiesce": "3/5", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_memstore": "1/5", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mgr": "2/5", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mgrc": "1/5", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_mon": "20/20", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_monc": "0/10", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_ms": "1/1", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_none": "0/5", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_objclass": "0/5", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_objectcacher": "0/5", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_objecter": "0/1", 2024-09-21T08:24:29.530 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_optracker": "0/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_osd": "1/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_paxos": "20/20", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_perfcounter": "1/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_prioritycache": "1/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rados": "0/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rbd": "0/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rbd_mirror": "0/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rbd_pwl": "0/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rbd_replay": "0/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_refs": "0/0", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_reserver": "1/1", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw": "1/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_access": "1/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_datacache": "1/5", 2024-09-21T08:24:29.531 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_dbstore": "1/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_flight": "1/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_lifecycle": "1/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rgw_sync": "1/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_rocksdb": "4/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore": "0/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_backref": "0/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_cache": "0/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_cleaner": "0/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_device": "0/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_epm": "0/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_fixedkv_tree": "0/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_journal": "0/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_lba": "0/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_odata": "0/5", 2024-09-21T08:24:29.532 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_omap": "0/5", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_onode": "0/5", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_t": "0/5", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_seastore_tm": "0/5", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_striper": "0/1", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_test": "0/5", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_throttle": "1/1", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_timer": "0/1", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_tp": "0/5", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "debug_trace": "1/5", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "defer_client_eviction_on_laggy_osds": "false", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "device_failure_prediction_mode": "none", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "enable_experimental_unrecoverable_data_corrupting_features": "", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_dir": "/usr/lib64/ceph/erasure-code", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "err_to_graylog": "false", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "err_to_journald": "false", 2024-09-21T08:24:29.533 INFO:teuthology.orchestra.run.smithi137.stdout: "err_to_stderr": "true", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "err_to_syslog": "false", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event_tracing": "false", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_addr": "0.0.0.0", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_http_port": "9926", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_prio_limit": "5", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_sock_dir": "/var/run/ceph/", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_sort_metrics": "true", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "exporter_stats_period": "5", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "fake_statfs_for_testing": "0", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "fatal_signal_handlers": "true", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "filer_max_purge_ops": "10", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "filer_max_truncate_ops": "128", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_apply_finisher_threads": "1", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_blackhole": "false", 2024-09-21T08:24:29.534 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_btrfs_clone_range": "true", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_btrfs_snap": "true", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_caller_concurrency": "10", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_collect_device_partition_information": "true", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_commit_timeout": "600.000000", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_debug_inject_read_err": "false", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_debug_omap_check": "false", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_debug_random_read_err": "0.000000", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_debug_verify_split": "false", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_dump_file": "", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_expected_throughput_bytes": "209715200.000000", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_expected_throughput_ops": "200.000000", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fadvise": "true", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fail_eio": "true", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fd_cache_shards": "16", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fd_cache_size": "128", 2024-09-21T08:24:29.535 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fiemap": "false", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fiemap_threshold": "4096", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_fsync_flushes_journal_data": "false", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_index_retry_probability": "0.000000", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_inject_stall": "0", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_journal_parallel": "false", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_journal_trailing": "false", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_journal_writeahead": "false", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_kill_at": "0", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_alloc_hint_size": "1048576", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattr_size": "0", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattr_size_btrfs": "2048", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattr_size_other": "512", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattr_size_xfs": "65536", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattrs": "0", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattrs_btrfs": "10", 2024-09-21T08:24:29.536 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattrs_other": "2", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_inline_xattrs_xfs": "10", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_sync_interval": "5.000000", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_xattr_value_size": "0", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_xattr_value_size_btrfs": "65536", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_xattr_value_size_other": "1024", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_max_xattr_value_size_xfs": "65536", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_merge_threshold": "-10", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_min_sync_interval": "0.010000", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_odsync_write": "false", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_omap_backend": "rocksdb", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_omap_backend_path": "", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_omap_header_cache_size": "1024", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_ondisk_finisher_threads": "1", 2024-09-21T08:24:29.537 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_op_thread_suicide_timeout": "180", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_op_thread_timeout": "60", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_op_threads": "2", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_punch_hole": "false", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_high_delay_multiple": "0.000000", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_high_delay_multiple_bytes": "0.000000", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_high_delay_multiple_ops": "0.000000", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_high_threshhold": "0.900000", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_low_threshhold": "0.300000", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_max_bytes": "104857600", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_max_delay_multiple": "0.000000", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_max_delay_multiple_bytes": "0.000000", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_max_delay_multiple_ops": "0.000000", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_queue_max_ops": "50", 2024-09-21T08:24:29.538 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_rocksdb_options": "max_background_jobs=10,compaction_readahead_size=2097152,compression=kNoCompression", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_seek_data_hole": "false", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_sloppy_crc": "false", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_sloppy_crc_block_size": "65536", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_splice": "false", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_split_multiple": "2", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_split_rand_factor": "20", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_update_to": "1000", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_bytes_hard_limit": "419430400", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_bytes_start_flusher": "41943040", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_inodes_hard_limit": "5000", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_inodes_start_flusher": "500", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_ios_hard_limit": "5000", 2024-09-21T08:24:29.539 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_btrfs_ios_start_flusher": "500", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_enable": "true", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_bytes_hard_limit": "419430400", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_bytes_start_flusher": "41943040", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_inodes_hard_limit": "5000", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_inodes_start_flusher": "500", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_ios_hard_limit": "5000", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_wbthrottle_xfs_ios_start_flusher": "500", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_xfs_extsize": "false", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "filestore_zfs_snap": "false", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "fio_dir": "/tmp/fio", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "ec8566d6-77f1-11ef-baf6-efdc52797490", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_allow_other": "true", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_atomic_o_trunc": "true", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_big_writes": "true", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_debug": "false", 2024-09-21T08:24:29.540 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_default_permissions": "false", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_disable_pagecache": "false", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_max_write": "0", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_multithreaded": "true", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_require_active_mds": "true", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_set_user_groups": "true", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_splice_move": "true", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_splice_read": "true", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_splice_write": "true", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_syncfs_on_mksnap": "true", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "fuse_use_invalidate_cb": "true", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "gss_ktab_client_file": "/var/lib/ceph/mon.a/gss_client_mon.a.ktab", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "gss_target_name": "ceph", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_file": "", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_inject_failure": "0", 2024-09-21T08:24:29.541 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_interval": "5", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "host": "smithi137", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_client_dedicated_thread_num": "2", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_max_inflight_ops": "128", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_max_size": "1073741824", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_path": "/tmp/ceph_immutable_object_cache", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_bps_burst": "0", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_bps_burst_seconds": "1", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_bps_limit": "0", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_iops_burst": "0", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_iops_burst_seconds": "1", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_iops_limit": "0", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_qos_schedule_tick_min": "50", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_sock": "/var/run/ceph/immutable_object_cache_sock", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "immutable_object_cache_watermark": "0.900000", 2024-09-21T08:24:29.542 INFO:teuthology.orchestra.run.smithi137.stdout: "inject_early_sigterm": "false", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "jaeger_agent_port": "6799", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "jaeger_tracing_enable": "false", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_aio": "true", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_align_min_size": "65536", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_block_align": "true", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_block_size": "4096", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_dio": "true", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_discard": "false", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_force_aio": "false", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_ignore_corruption": "false", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_max_write_bytes": "10485760", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_max_write_entries": "100", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_replay_from": "0", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_throttle_high_multiple": "0.000000", 2024-09-21T08:24:29.543 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_throttle_high_threshhold": "0.900000", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_throttle_low_threshhold": "0.600000", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_throttle_max_multiple": "0.000000", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_write_header_frequency": "0", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "journal_zero_on_create": "false", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "journaler_prefetch_periods": "10", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "journaler_prezero_periods": "5", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "journaler_write_head_interval": "15", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "key": "", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "keyfile": "", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "keyring": "/var/lib/ceph/mon/ceph-a/keyring", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_backend": "rocksdb", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_default_stripe_size": "65536", 2024-09-21T08:24:29.544 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_fsck_on_mount": "false", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_fsck_on_mount_deep": "true", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_max_bytes": "67108864", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_max_ops": "512", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_nid_prealloc": "1024", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_onode_map_size": "1024", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_rocksdb_options": "compression=kNoCompression", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_sync_submit_transaction": "false", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "kstore_sync_transaction": "false", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "librados_thread_count": "2", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "lockdep": "false", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "lockdep_force_backtrace": "false", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "log_coarse_timestamps": "true", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "log_file": "/var/log/ceph/ceph-mon.a.log", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "log_flush_on_exit": "false", 2024-09-21T08:24:29.545 INFO:teuthology.orchestra.run.smithi137.stdout: "log_graylog_host": "127.0.0.1", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "log_graylog_port": "12201", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "log_max_new": "1000", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "log_max_recent": "10000", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "log_stderr_prefix": "", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "log_stop_at_utilization": "0.970000", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "log_to_file": "true", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "log_to_graylog": "false", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "log_to_journald": "false", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "log_to_stderr": "false", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "log_to_syslog": "false", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "max_rotating_auth_attempts": "10", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_abort_on_newly_corrupt_dentry": "true", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_action_on_write_error": "1", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_alternate_name_max": "8192", 2024-09-21T08:24:29.546 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_asio_thread_count": "2", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_export_pin": "true", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_fragment_dirs": "true", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_fragment_fast_factor": "1.500000", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_fragment_interval": "5", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_fragment_size_max": "100000", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_idle_threshold": "0.000000", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_interval": "10", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_max": "-1", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_max_until": "-1", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_merge_size": "50", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_midchunk": "0.300000", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_min_rebalance": "0.100000", 2024-09-21T08:24:29.547 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_min_start": "0.200000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_minchunk": "0.001000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_mode": "0", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_need_max": "1.200000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_need_min": "0.800000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_overload_epochs": "2", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_replicate_threshold": "8000.000000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_sample_interval": "3.000000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_split_bits": "3", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_split_rd": "25000.000000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_split_size": "10000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_split_wr": "10000.000000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_target_decay": "10.000000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_bal_unreplicate_threshold": "0.000000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_beacon_grace": "15.000000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_beacon_interval": "4.000000", 2024-09-21T08:24:29.548 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_beacon_mon_down_grace": "60", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_memory_limit": "4294967296", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_mid": "0.700000", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_quiesce_decay_rate": "1.000000", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_quiesce_delay": "0", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_quiesce_sleep": "200", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_quiesce_splitauth": "true", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_quiesce_threshold": "524288", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_release_free_interval": "10", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_reservation": "0.050000", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_trim_decay_rate": "1.000000", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_trim_interval": "1", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cache_trim_threshold": "262144", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cap_acquisition_throttle_retry_request_timeout": "0.500000", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_cap_revoke_eviction_timeout": "0.000000", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_client_delegate_inos_pct": "50", 2024-09-21T08:24:29.549 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_client_prealloc_inos": "1000", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_client_writeable_range_max_inc_objs": "1024", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_connect_bootstrapping": "false", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_damage_table_max_entries": "10000", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_data": "/var/lib/ceph/mds/ceph-a", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_debug_auth_pins": "false", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_debug_frag": "false", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_debug_scatterstat": "false", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_debug_subtrees": "false", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_decay_halflife": "5.000000", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_default_dir_hash": "2", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_defer_session_stale": "true", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_deny_all_reconnect": "false", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dir_keys_per_op": "16384", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dir_max_commit_size": "10", 2024-09-21T08:24:29.550 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dir_max_entries": "0", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dir_prefetch": "true", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dirstat_min_interval": "1.000000", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dump_cache_after_rejoin": "false", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dump_cache_on_map": "false", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dump_cache_threshold_file": "0", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_dump_cache_threshold_formatter": "1073741824", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_early_reply": "true", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_enable_op_tracker": "true", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_enforce_unique_name": "true", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_export_ephemeral_distributed": "true", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_export_ephemeral_distributed_factor": "2.000000", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_export_ephemeral_random": "true", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_export_ephemeral_random_max": "0.010000", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_extraordinary_events_dump_interval": "0", 2024-09-21T08:24:29.551 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_forward_all_requests_to_auth": "false", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_freeze_tree_timeout": "30.000000", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_fscrypt_last_block_max_size": "4096", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_go_bad_corrupt_dentry": "true", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_hack_allow_loading_invalid_metadata": "false", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_health_cache_threshold": "1.500000", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_health_summarize_threshold": "10", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_heartbeat_grace": "15.000000", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_heartbeat_reset_grace": "1000", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_health_dummy": "false", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_journal_corrupt_dentry_first": "0.000000", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_migrator_session_race": "false", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_rename_corrupt_dentry_first": "0.000000", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_skip_replaying_inotable": "false", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_inject_traceless_reply_probability": "0.000000", 2024-09-21T08:24:29.552 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_join_fs": "", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_journal_format": "1", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_after_journal_logs_flushed": "false", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_create_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_export_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_import_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_journal_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_journal_expire_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_journal_replay_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_link_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_mdstable_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_openc_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_rename_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_kill_shutdown_at": "0", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_event_large_threshold": "524288", 2024-09-21T08:24:29.553 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_events_per_segment": "1024", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_major_segment_event_ratio": "12", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_max_events": "-1", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_max_segments": "128", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_pause": "false", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_segment_size": "0", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_skip_corrupt_events": "false", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_skip_unbounded_events": "false", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_trim_decay_rate": "1.000000", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_trim_threshold": "128", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_trim_upkeep_interval": "1000", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_log_warn_factor": "2.000000", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_caps_per_client": "1048576", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_completed_flushes": "100000", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_completed_requests": "100000", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_export_size": "20971520", 2024-09-21T08:24:29.554 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_file_recover": "32", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_purge_files": "64", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_purge_ops": "8192", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_purge_ops_per_pg": "0.500000", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_scrub_ops_in_progress": "5", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_max_snaps_per_dir": "100", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_metrics_update_interval": "2", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_min_caps_per_client": "100", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_min_caps_working_set": "10000", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_mon_shutdown_timeout": "5.000000", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_numa_node": "-1", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_oft_prefetch_dirfrags": "false", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_complaint_time": "30.000000", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_history_duration": "600", 2024-09-21T08:24:29.555 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_history_size": "20", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_history_slow_op_size": "20", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_history_slow_op_threshold": "10", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_op_log_threshold": "5", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_ping_grace": "15", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_ping_interval": "5", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_purge_queue_busy_flush_period": "1.000000", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_global_max_decay_threshold": "131072", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_max_caps": "30000", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_max_decay_rate": "1.500000", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_max_decay_threshold": "131072", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_warning_decay_rate": "60.000000", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_recall_warning_threshold": "262144", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_reconnect_timeout": "45.000000", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_replay_interval": "1.000000", 2024-09-21T08:24:29.556 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_replay_unsafe_with_closed_session": "false", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_request_load_average_decay_rate": "60.000000", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_root_ino_gid": "0", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_root_ino_uid": "0", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_scatter_nudge_interval": "5.000000", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_blocklist_on_evict": "true", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_blocklist_on_timeout": "true", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_cache_liveness_decay_rate": "300.000000", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_cache_liveness_magnitude": "10", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_cap_acquisition_decay_rate": "30.000000", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_cap_acquisition_throttle": "100000", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_max_caps_throttle_ratio": "1.100000", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_session_metadata_threshold": "16777216", 2024-09-21T08:24:29.557 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_sessionmap_keys_per_op": "1024", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_shutdown_check": "0", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_skip_ino": "0", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_sleep_rank_change": "0.000000", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_snap_max_uid": "4294967294", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_snap_min_uid": "0", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_snap_rstat": "false", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_standby_replay_damaged": "false", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_symlink_recovery": "true", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_task_status_update_interval": "2.000000", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_thrash_exports": "0", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_thrash_fragments": "0", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_tick_interval": "5.000000", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_valgrind_exit": "false", 2024-09-21T08:24:29.558 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_verify_backtrace": "1", 2024-09-21T08:24:29.559 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_verify_scatter": "false", 2024-09-21T08:24:29.559 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_wipe_ino_prealloc": "false", 2024-09-21T08:24:29.559 INFO:teuthology.orchestra.run.smithi137.stdout: "mds_wipe_sessions": "false", 2024-09-21T08:24:29.559 INFO:teuthology.orchestra.run.smithi137.stdout: "mempool_debug": "false", 2024-09-21T08:24:29.559 INFO:teuthology.orchestra.run.smithi137.stdout: "memstore_debug_omit_block_device_write": "false", 2024-09-21T08:24:29.560 INFO:teuthology.orchestra.run.smithi137.stdout: "memstore_device_bytes": "1073741824", 2024-09-21T08:24:29.560 INFO:teuthology.orchestra.run.smithi137.stdout: "memstore_page_set": "false", 2024-09-21T08:24:29.560 INFO:teuthology.orchestra.run.smithi137.stdout: "memstore_page_size": "65536", 2024-09-21T08:24:29.560 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_client_bytes": "134217728", 2024-09-21T08:24:29.560 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_client_messages": "512", 2024-09-21T08:24:29.560 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_client_service_daemon_unregister_timeout": "1.000000", 2024-09-21T08:24:29.560 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_connect_retry_interval": "1.000000", 2024-09-21T08:24:29.560 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_data": "/var/lib/ceph/mgr/ceph-a", 2024-09-21T08:24:29.560 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_debug_aggressive_pg_num_changes": "false", 2024-09-21T08:24:29.560 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_disabled_modules": "", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_enable_op_tracker": "true", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_initial_modules": "restful iostat nfs", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_max_pg_num_change": "128", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_mds_bytes": "134217728", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_mds_messages": "128", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_module_path": "/usr/share/ceph/mgr", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_mon_bytes": "134217728", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_mon_messages": "128", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_num_op_tracker_shard": "32", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_complaint_time": "30.000000", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_history_duration": "600", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_history_size": "20", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_history_slow_op_size": "20", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_history_slow_op_threshold": "10.000000", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_op_log_threshold": "5", 2024-09-21T08:24:29.561 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_osd_bytes": "536870912", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_osd_messages": "8192", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_pool": "true", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_service_beacon_grace": "60.000000", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_standby_modules": "true", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_stats_period": "5", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_stats_threshold": "5", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_tick_period": "2", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr_ttl_cache_expire_seconds": "0", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_accept_timeout_factor": "2.000000", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_allow_pool_delete": "true", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_allow_pool_size_one": "false", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_auth_validate_all_caps": "true", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cache_target_full_warn_ratio": "0.660000", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_clean_pg_upmaps_per_chunk": "256", 2024-09-21T08:24:29.562 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_bytes": "104857600", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_directed_command_retry": "2", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_hunt_interval": "3.000000", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_hunt_interval_backoff": "1.500000", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_hunt_interval_max_multiple": "10.000000", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_hunt_interval_min_multiple": "1.000000", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_hunt_parallel": "3", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_log_interval": "1.000000", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_max_log_entries_per_message": "1000", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_ping_interval": "10.000000", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_ping_timeout": "30.000000", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_client_target_rank": "-1", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_clock_drift_allowed": "1.000000", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_clock_drift_warn_backoff": "5.000000", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_file": "default=/var/log/ceph/ceph.$channel.log cluster=/var/log/ceph/ceph.log", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_file_level": "debug", 2024-09-21T08:24:29.563 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_file": "true", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_graylog": "false", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_graylog_host": "127.0.0.1", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_graylog_port": "12201", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_journald": "true", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_stderr": "false", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_syslog": "default=false", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_syslog_facility": "daemon", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cluster_log_to_syslog_level": "info", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_compact_on_bootstrap": "false", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_compact_on_start": "false", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_compact_on_trim": "true", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_con_tracker_persist_interval": "10", 2024-09-21T08:24:29.564 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_con_tracker_score_halflife": "43200", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_config_key_max_entry_size": "65536", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_cpu_threads": "4", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_crush_min_required_version": "hammer", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_daemon_bytes": "419430400", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_data": "/var/lib/ceph/mon/ceph-a", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_data_avail_crit": "5", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_data_avail_warn": "5", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_data_size_warn": "16106127360", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_block_osdmap_trim": "false", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_deprecated_as_obsolete": "false", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_dump_json": "false", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_dump_location": "/var/log/ceph/ceph-mon.a.tdump", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_dump_transactions": "false", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_extra_checks": "false", 2024-09-21T08:24:29.565 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_no_initial_persistent_features": "false", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_no_require_bluestore_for_ec_overwrites": "false", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_no_require_reef": "false", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_no_require_squid": "false", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_debug_unsafe_allow_tier_with_nonempty_snaps": "false", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_delta_reset_interval": "10.000000", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_dns_srv_name": "ceph-mon", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_down_mkfs_grace": "60", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_down_uptime_grace": "60", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_election_default_strategy": "1", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_election_timeout": "5.000000", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_elector_ignore_propose_margin": "0.000500", 2024-09-21T08:24:29.566 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_elector_ping_divisor": "2", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_elector_ping_timeout": "2.000000", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_enable_op_tracker": "true", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_fake_pool_delete": "false", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_force_quorum_join": "false", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_fsmap_prune_threshold": "300", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_globalid_prealloc": "10000", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_detail_to_clog": "true", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_log_update_period": "5", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_max_detail": "50", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_to_clog": "true", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_to_clog_interval": "600", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_health_to_clog_tick_interval": "60.000000", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_host": "172.21.15.137:6789/0", 2024-09-21T08:24:29.567 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_host_override": "", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_initial_members": "", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_inject_pg_merge_bounce_probability": "0.000000", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_inject_sync_get_chunk_delay": "0.000000", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_inject_transaction_delay_max": "10.000000", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_inject_transaction_delay_probability": "0.000000", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_keyvaluedb": "rocksdb", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_lease": "5.000000", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_lease_ack_timeout_factor": "2.000000", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_lease_renew_interval_factor": "0.600000", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_log_full_interval": "50", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_log_max": "10000", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_log_max_summary": "50", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_log_entries_per_event": "4096", 2024-09-21T08:24:29.568 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_log_epochs": "500", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_mdsmap_epochs": "500", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_mgrmap_epochs": "500", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_osd": "10000", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_pg_per_osd": "10000", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_pool_pg_num": "65536", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_max_snap_prune_per_epoch": "100", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mds_blocklist_interval": "86400.000000", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mds_force_trim_to": "0", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mds_skip_sanity": "false", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_memory_autotune": "true", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_memory_target": "2147483648", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_beacon_grace": "30", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_blocklist_interval": "86400.000000", 2024-09-21T08:24:29.569 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_digest_period": "5", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_inactive_grace": "60", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_mkfs_grace": "240", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_mgr_proxy_client_bytes_ratio": "0.300000", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_min_osdmap_epochs": "50", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_complaint_time": "30", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_history_duration": "600", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_history_size": "20", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_history_slow_op_size": "20", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_history_slow_op_threshold": "10", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_op_log_threshold": "5", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_adjust_down_out_interval": "true", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_adjust_heartbeat_grace": "true", 2024-09-21T08:24:29.570 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_auto_mark_auto_out_in": "true", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_auto_mark_in": "false", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_auto_mark_new_in": "true", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_backfillfull_ratio": "0.900000", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_blocklist_default_expire": "3600.000000", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_cache_size": "500", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_cache_size_min": "134217728", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_crush_smoke_test": "true", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_destroyed_out_interval": "600", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_down_out_interval": "600", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_down_out_subtree_limit": "rack", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_err_op_age_ratio": "128.000000", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_force_trim_to": "0", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_full_ratio": "0.950000", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_initial_require_min_compat_client": "luminous", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_laggy_halflife": "3600", 2024-09-21T08:24:29.571 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_laggy_max_interval": "300", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_laggy_weight": "0.300000", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_mapping_pgs_per_chunk": "4096", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_max_creating_pgs": "1024", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_max_initial_pgs": "1024", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_min_down_reporters": "2", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_min_in_ratio": "0.750000", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_min_up_ratio": "0.300000", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_nearfull_ratio": "0.850000", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_prime_pg_temp": "true", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_prime_pg_temp_max_estimate": "0.250000", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_prime_pg_temp_max_time": "0.500000", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_report_timeout": "900", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_reporter_subtree_level": "osd", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_snap_trim_queue_warn_on": "32768", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_warn_num_repaired": "10", 2024-09-21T08:24:29.572 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osd_warn_op_age": "32.000000", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osdmap_full_prune_enabled": "true", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osdmap_full_prune_interval": "2", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osdmap_full_prune_min": "15", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_osdmap_full_prune_txsize": "2", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_check_down_all_threshold": "0.500000", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_stuck_threshold": "60", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_warn_max_object_skew": "0.000000", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_warn_min_objects": "10000", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_warn_min_per_osd": "0", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pg_warn_min_pool_objects": "1000", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pool_quota_crit_threshold": "0", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_pool_quota_warn_threshold": "0", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_probe_timeout": "2.000000", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_reweight_max_change": "0.050000", 2024-09-21T08:24:29.573 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_reweight_max_osds": "4", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_reweight_min_bytes_per_osd": "10", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_reweight_min_pgs_per_osd": "4", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_rocksdb_options": "write_buffer_size=33554432,compression=kNoCompression,level_compaction_dynamic_level_bytes=true", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_scrub_inject_crc_mismatch": "0.000000", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_scrub_inject_missing_keys": "0.000000", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_scrub_interval": "86400", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_scrub_max_keys": "100", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_scrub_timeout": "300", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_session_timeout": "300", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_smart_report_timeout": "5", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_stat_smooth_intervals": "6", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_stretch_cluster_recovery_ratio": "0.600000", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_stretch_pool_min_size": "2", 2024-09-21T08:24:29.574 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_stretch_pool_size": "4", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_stretch_recovery_min_wait": "15.000000", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_subscribe_interval": "86400.000000", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_debug": "false", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_max_payload_keys": "2000", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_max_payload_size": "1048576", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_provider_kill_at": "0", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_requester_kill_at": "0", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_sync_timeout": "60.000000", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_target_pg_per_osd": "100", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_tick_interval": "5", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_timecheck_interval": "300.000000", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_timecheck_skew_interval": "30.000000", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_use_min_delay_socket": "false", 2024-09-21T08:24:29.575 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_older_version_delay": "604800", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_cache_pools_without_hit_sets": "true", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_crush_straw_calc_version_zero": "false", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_degraded_stretch_mode": "true", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_filestore_osds": "true", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_insecure_global_id_reclaim": "false", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_insecure_global_id_reclaim_allowed": "false", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_legacy_crush_tunables": "false", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_misplaced": "false", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_msgr2_not_enabled": "true", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_older_version": "true", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_osd_down_out_interval_zero": "false", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_pool_no_app": "true", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_pool_no_redundancy": "true", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_pool_pg_num_not_power_of_two": "false", 2024-09-21T08:24:29.576 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_slow_ping_ratio": "0.050000", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_slow_ping_time": "0.000000", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_on_too_few_osds": "false", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_pg_not_deep_scrubbed_ratio": "0.750000", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "mon_warn_pg_not_scrubbed_ratio": "0.500000", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "monmap": "", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_admin_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_admin_fid": "0x7200000000000001:0x0", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_ha_endpoint": "192.168.180.182@tcp:12345:1:1", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_my_endpoint": "192.168.180.182@tcp:12345:4:1", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_my_fid": "0x7200000000000001:0x0", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_profile_fid": "0x7000000000000001:0x0", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "motr_tracing_enabled": "false", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_op_threads": "3", 2024-09-21T08:24:29.577 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_buffer_size": "131072", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_cm": "false", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_device_name": "", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_dscp": "96", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_enable_hugepage": "false", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_gid_idx": "0", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_local_gid": "", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_polling_us": "1000", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_port_num": "1", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_receive_buffers": "32768", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_receive_queue_len": "4096", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_roce_ver": "1", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_send_buffers": "1024", 2024-09-21T08:24:29.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_sl": "3", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_support_srq": "true", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_rdma_type": "ib", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_async_reap_threshold": "5", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_before_connect": "false", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_ipv4": "true", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_ipv6": "false", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_msgr1": "true", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_msgr2": "false", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_port_max": "7568", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_port_min": "6800", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_prefer_ipv4": "false", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_retry_count": "3", 2024-09-21T08:24:29.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_bind_retry_delay": "5", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_blackhole_client": "false", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_blackhole_mds": "false", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_blackhole_mgr": "false", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_blackhole_mon": "false", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_blackhole_osd": "false", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_client_mode": "crc secure", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_cluster_mode": "crc secure", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_cluster_type": "", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_compress_secure": "false", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_connection_idle_timeout": "900", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_connection_ready_timeout": "10", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_crc_data": "true", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_crc_header": "true", 2024-09-21T08:24:29.580 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_die_on_bad_msg": "false", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_die_on_bug": "true", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_die_on_old_message": "true", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_die_on_skipped_message": "false", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_die_on_unhandled_msg": "false", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dispatch_throttle_bytes": "104857600", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_coremask": "0xF", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_debug_allow_loopback": "false", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_devs_allowlist": "", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_enable_tso": "true", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_gateway_ipv4_addr": "", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_host_ipv4_addr": "", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_hugepages": "", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_hw_flow_control": "true", 2024-09-21T08:24:29.581 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_hw_queue_weight": "1.000000", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_lro": "true", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_memory_channel": "4", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_netmask_ipv4_addr": "", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_pmd": "", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_port_id": "0", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dpdk_rx_buffer_count_per_core": "8192", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dump_corrupt_message_level": "1", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_dump_on_send": "false", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_initial_backoff": "0.200000", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_delay_max": "1.000000", 2024-09-21T08:24:29.582 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_delay_probability": "0.000000", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_delay_type": "", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_internal_delays": "0.000000", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_network_congestion": "0", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_inject_socket_failures": "0", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_learn_addr_from_peer": "true", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_max_accept_failures": "4", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_max_backoff": "15.000000", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_mon_client_mode": "secure crc", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_mon_cluster_mode": "secure crc", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_mon_service_mode": "secure crc", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_osd_compress_min_size": "1024", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_osd_compress_mode": "none", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_osd_compression_algorithm": "snappy", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_pq_max_tokens_per_priority": "16777216", 2024-09-21T08:24:29.583 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_pq_min_cost": "65536", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_public_type": "", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_service_mode": "crc secure", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_tcp_listen_backlog": "512", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_tcp_nodelay": "true", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_tcp_prefetch_max_size": "4096", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_tcp_rcvbuf": "0", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "ms_type": "async+posix", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "no_config_file": "false", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_completion_locks_per_session": "32", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_debug_inject_relock_delay": "false", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_inflight_op_bytes": "104857600", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_inflight_ops": "1024", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_inject_no_watch_ping": "false", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_retry_writes_after_first_reply": "false", 2024-09-21T08:24:29.584 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_tick_interval": "5.000000", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "objecter_timeout": "10.000000", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "objectstore_blackhole": "false", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "objectstore_debug_throw_on_failed_txc": "false", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "openssl_engine_opts": "", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_delay_time": "5.000000", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_hist_halflife": "1000", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_max_low_ops": "2", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_max_ops": "4", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_min_evict_effort": "0.100000", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_quantize_effort": "0.100000", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_agent_slop": "0.020000", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_aggregated_slow_ops_logging": "true", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_allow_recovery_below_min_size": "true", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_asio_thread_count": "2", 2024-09-21T08:24:29.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_async_recovery_min_cost": "100", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_auto_mark_unfound_lost": "false", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backfill_retry_interval": "30.000000", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backfill_scan_max": "512", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backfill_scan_min": "64", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backoff_on_degraded": "false", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backoff_on_peering": "false", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_backoff_on_unfound": "true", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_beacon_report_interval": "300", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_bench_duration": "30", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_bench_large_size_max_throughput": "104857600", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_bench_max_block_size": "67108864", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_bench_small_size_max_iops": "100", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_blkin_trace_all": "false", 2024-09-21T08:24:29.586 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_blocked_scrub_grace_period": "120", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_calc_pg_upmaps_aggressively": "true", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_calc_pg_upmaps_aggressively_fast": "true", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_calc_pg_upmaps_local_fallback_retries": "100", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_check_for_log_corruption": "false", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_check_max_object_name_len_on_startup": "true", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_class_default_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_class_dir": "/usr/lib64/rados-classes", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_class_load_list": "cephfs hello journal lock log numops otp rbd refcount rgw rgw_gc timeindex user version cas cmpomap queue 2pc_queue fifo", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_class_update_on_start": "true", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_client_message_cap": "256", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_client_message_size_cap": "524288000", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_client_op_priority": "63", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_client_watch_timeout": "30", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_command_max_records": "256", 2024-09-21T08:24:29.587 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_compact_on_start": "false", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_copyfrom_max_chunk": "8388608", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_crush_chooseleaf_type": "1", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_crush_initial_weight": "-1.000000", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_crush_update_on_start": "true", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_crush_update_weight_set": "true", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_data": "/var/lib/ceph/osd/ceph-a", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_crash_on_ignored_backoff": "false", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_deep_scrub_sleep": "0.000000", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_drop_ping_duration": "0", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_drop_ping_probability": "0.000000", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_feed_pullee": "-1", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_inject_copyfrom_error": "false", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_inject_dispatch_delay_duration": "0.100000", 2024-09-21T08:24:29.588 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_inject_dispatch_delay_probability": "0.000000", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_misdirected_ops": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_no_acting_change": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_no_purge_strays": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_op_order": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_pg_log_writeout": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_pretend_recovery_active": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_random_push_read_error": "0.000000", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_reject_backfill_probability": "0.000000", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_shutdown": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_skip_full_check_in_backfill_reservation": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_skip_full_check_in_recovery": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_trim_objects": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_verify_cached_snaps": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_verify_missing_on_start": "false", 2024-09-21T08:24:29.589 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_verify_snaps": "false", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_debug_verify_stray_on_activate": "false", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_interval": "604800.000000", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_keys": "1024", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_large_omap_object_key_threshold": "200000", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_large_omap_object_value_sum_threshold": "1073741824", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_randomize_ratio": "0.150000", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_stride": "524288", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_deep_scrub_update_digest_min_age": "7200", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_default_data_pool_replay_window": "45", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_default_notify_timeout": "30", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_delete_sleep": "0.000000", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_delete_sleep_hdd": "5.000000", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_delete_sleep_hybrid": "1.000000", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_delete_sleep_ssd": "1.000000", 2024-09-21T08:24:29.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_discard_disconnected_ops": "true", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_enable_op_tracker": "true", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_erasure_code_plugins": "jerasure lrc", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_extblkdev_plugins": "vdo", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_failsafe_full_ratio": "0.970000", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_fast_fail_on_connection_refused": "true", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_fast_info": "true", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_fast_shutdown": "true", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_fast_shutdown_notify_mon": "true", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_fast_shutdown_timeout": "15", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_find_best_info_ignore_history_les": "false", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_force_auth_primary_missing_objects": "100", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_force_recovery_pg_log_entries_factor": "1.300000", 2024-09-21T08:24:29.591 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_function_tracing": "false", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_grace": "20", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_interval": "6", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_min_healthy_ratio": "0.330000", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_min_peers": "10", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_min_size": "2000", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_stale": "600", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_heartbeat_use_min_delay_socket": "false", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_hit_set_max_size": "100000", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_hit_set_min_size": "1000", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_hit_set_namespace": ".ceph-internal", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_ignore_stale_divergent_priors": "false", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_inject_bad_map_crc_probability": "0.000000", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_inject_failure_on_pg_removal": "false", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_journal": "/var/lib/ceph/osd/ceph-a/journal", 2024-09-21T08:24:29.592 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_journal_flush_on_shutdown": "true", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_journal_size": "5120", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_kill_backfill_at": "0", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_loop_before_reset_tphandle": "64", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_map_cache_size": "50", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_map_dedup": "true", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_map_message_max": "40", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_map_message_max_bytes": "10485760", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_map_share_max_epochs": "40", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_attr_name_len": "100", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_attr_size": "0", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_backfills": "1", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_markdown_count": "5", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_markdown_period": "600", 2024-09-21T08:24:29.593 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_object_name_len": "2048", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_object_namespace_len": "256", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_object_size": "134217728", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_omap_bytes_per_request": "1073741824", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_omap_entries_per_request": "1024", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_pg_blocked_by": "16", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_pg_log_entries": "10000", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_pg_per_osd_hard_ratio": "3.000000", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_pgls": "1024", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_push_cost": "8388608", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_push_objects": "10", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_scrubs": "3", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_snap_prune_intervals_per_epoch": "512", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_trimming_pgs": "2", 2024-09-21T08:24:29.594 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_write_op_reply_len": "64", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_max_write_size": "90", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_force_run_benchmark_on_init": "false", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_iops_capacity_threshold_hdd": "500.000000", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_iops_capacity_threshold_ssd": "80000.000000", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_max_capacity_iops_hdd": "315.000000", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_max_capacity_iops_ssd": "21500.000000", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_max_sequential_bandwidth_hdd": "157286400", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_max_sequential_bandwidth_ssd": "1258291200", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_override_recovery_settings": "false", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_profile": "balanced", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_anticipation_timeout": "0.000000", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_best_effort_lim": "0.000000", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_best_effort_res": "0.000000", 2024-09-21T08:24:29.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_best_effort_wgt": "1", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_recovery_lim": "0.000000", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_recovery_res": "0.000000", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_background_recovery_wgt": "1", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_client_lim": "0.000000", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_client_res": "0.000000", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_scheduler_client_wgt": "1", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mclock_skip_benchmark": "false", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_base": "805306368", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_cache_min": "134217728", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_cache_resize_interval": "1.000000", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_expected_fragmentation": "0.150000", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_target": "4294967296", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_target_autotune": "false", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_memory_target_cgroup_limit_ratio": "0.800000", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_min_pg_log_entries": "250", 2024-09-21T08:24:29.596 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_min_recovery_priority": "0", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mon_heartbeat_interval": "30", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mon_heartbeat_stat_stale": "3600", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mon_report_interval": "5", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mon_report_max_in_flight": "2", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_mon_shutdown_timeout": "5.000000", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_num_cache_shards": "32", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_num_op_tracker_shard": "32", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_numa_auto_affinity": "true", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_numa_node": "-1", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_numa_prefer_iface": "true", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_object_clean_region_max_num_intervals": "10", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_objecter_finishers": "1", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_objectstore": "bluestore", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_objectstore_fuse": "false", 2024-09-21T08:24:29.597 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_objectstore_tracing": "false", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_complaint_time": "30.000000", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_history_duration": "600", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_history_size": "20", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_history_slow_op_size": "20", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_history_slow_op_threshold": "10.000000", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_log_threshold": "5", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_shards": "0", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_shards_hdd": "5", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_shards_ssd": "8", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_threads_per_shard": "0", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_threads_per_shard_hdd": "1", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_num_threads_per_shard_ssd": "2", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_pq_max_tokens_per_priority": "4194304", 2024-09-21T08:24:29.598 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_pq_min_cost": "65536", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_queue": "mclock_scheduler", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_queue_cut_off": "high", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_thread_suicide_timeout": "150", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_op_thread_timeout": "15", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_open_classes_on_start": "true", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_os_flags": "0", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_peering_op_priority": "255", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_delete_cost": "1048576", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_delete_priority": "5", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_epoch_max_lag_factor": "2.000000", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_epoch_persisted_max_stale": "40", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_log_dups_tracked": "3000", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_log_trim_max": "10000", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_log_trim_min": "100", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_max_concurrent_snap_trims": "2", 2024-09-21T08:24:29.599 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_object_context_cache_count": "64", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_stat_report_interval_max_epochs": "500", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pg_stat_report_interval_max_seconds": "5", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_max_evict_check_size": "10", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_min_evict_age": "0", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_min_flush_age": "0", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_target_dirty_high_ratio": "0.600000", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_target_dirty_ratio": "0.400000", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_cache_target_full_ratio": "0.800000", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_crimson": "false", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_crush_rule": "-1", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_ec_fast_read": "false", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_erasure_code_profile": "plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flag_bulk": "false", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flag_hashpspool": "true", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flag_nodelete": "false", 2024-09-21T08:24:29.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flag_nopgchange": "false", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flag_nosizechange": "false", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_flags": "0", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_hit_set_bloom_fpp": "0.050000", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_min_size": "0", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_pg_autoscale_mode": "off", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_pg_num": "32", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_pgp_num": "0", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_read_lease_ratio": "0.800000", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_read_ratio": "70", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_size": "3", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_default_type": "replicated", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_erasure_code_stripe_unit": "4096", 2024-09-21T08:24:29.601 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_pool_use_gmt_hitset": "true", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_push_per_object_cost": "1000", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_read_ec_check_for_errors": "false", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recover_clone_overlap": "true", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recover_clone_overlap_limit": "10", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_cost": "20971520", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_delay_start": "0.000000", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_active": "0", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_active_hdd": "3", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_active_ssd": "10", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_chunk": "8388608", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_omap_entries_per_chunk": "8096", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_max_single_start": "1", 2024-09-21T08:24:29.602 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_op_priority": "3", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_op_warn_multiple": "16", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_priority": "5", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_retry_interval": "30.000000", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_sleep": "0.000000", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_sleep_hdd": "0.100000", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_sleep_hybrid": "0.025000", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_recovery_sleep_ssd": "0.000000", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_repair_during_recovery": "false", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_requested_scrub_priority": "120", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_rocksdb_iterator_bounds_enabled": "true", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_rollback_to_cluster_snap": "", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_auto_repair": "false", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_auto_repair_num_errors": "5", 2024-09-21T08:24:29.603 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_backoff_ratio": "0.660000", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_begin_hour": "0", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_begin_week_day": "0", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_chunk_max": "25", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_chunk_min": "5", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_cost": "52428800", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_disable_reservation_queuing": "false", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_during_recovery": "false", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_end_hour": "0", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_end_week_day": "0", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_event_cost": "4096", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_extended_sleep": "0.000000", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_interval_randomize_ratio": "0.500000", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_invalid_stats": "true", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_load_threshold": "0.500000", 2024-09-21T08:24:29.604 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_max_interval": "604800.000000", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_max_preemptions": "5", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_min_interval": "86400.000000", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_priority": "5", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_scrub_sleep": "0.000000", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_shallow_scrub_chunk_max": "100", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_shallow_scrub_chunk_min": "50", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_shutdown_pgref_assert": "false", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_skip_data_digest": "false", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_smart_report_timeout": "5", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_cost": "1048576", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_priority": "5", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_sleep": "0.000000", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_sleep_hdd": "5.000000", 2024-09-21T08:24:29.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_sleep_hybrid": "2.000000", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_snap_trim_sleep_ssd": "0.000000", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_stats_update_period_not_scrubbing": "120", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_stats_update_period_scrubbing": "15", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_target_pg_log_entries_per_osd": "300000", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_target_transaction_size": "30", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_hit_set_count": "4", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_hit_set_grade_decay_rate": "20", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_hit_set_period": "1200", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_hit_set_search_last_n": "1", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_hit_set_type": "bloom", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_min_read_recency_for_promote": "1", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_min_write_recency_for_promote": "1", 2024-09-21T08:24:29.606 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_default_cache_mode": "writeback", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_promote_max_bytes_sec": "5242880", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tier_promote_max_objects_sec": "25", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_tracing": "false", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_use_stale_snap": "false", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_uuid": "00000000-0000-0000-0000-000000000000", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "osdc_blkin_trace_all": "false", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_kill_at": "0", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_max_join_drift": "10", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_min": "500", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_min_wait": "0.050000", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_propose_interval": "1.000000", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_service_trim_max": "500", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_service_trim_max_multiplier": "20", 2024-09-21T08:24:29.607 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_service_trim_min": "10", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_stash_full_interval": "25", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_trim_max": "500", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "paxos_trim_min": "250", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "perf": "true", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "pid_file": "", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin_crypto_accelerator": "crypto_isal", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin_dir": "/usr/lib64/ceph", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "-", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrv": "", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "public_bind_addr": "-", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "public_network": "", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "public_network_interface": "", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "qat_compressor_busy_polling": "false", 2024-09-21T08:24:29.608 INFO:teuthology.orchestra.run.smithi137.stdout: "qat_compressor_enabled": "false", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "qat_compressor_session_max_number": "256", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rados_mon_op_timeout": "0", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rados_osd_op_timeout": "0", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rados_tracing": "false", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_atime_update_interval": "60", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_auto_exclusive_lock_until_manual_request": "true", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_balance_parent_reads": "false", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_balance_snap_reads": "false", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_blkin_trace_all": "false", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_blocklist_expire_seconds": "0", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_blocklist_on_break_lock": "true", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache": "true", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_block_writes_upfront": "false", 2024-09-21T08:24:29.609 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_max_dirty": "25165824", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_max_dirty_age": "1.000000", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_max_dirty_object": "0", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_policy": "writearound", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_size": "33554432", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_target_dirty": "16777216", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_cache_writethrough_until_flush": "true", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_clone_copy_on_read": "false", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_compression_hint": "none", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_concurrent_management_ops": "10", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_config_pool_override_update_timestamp": "0", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_clone_format": "auto", 2024-09-21T08:24:29.610 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_data_pool": "", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_features": "61", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_format": "2", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_map_options": "", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_order": "22", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_pool": "rbd", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_snapshot_quiesce_mode": "required", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_stripe_count": "0", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_default_stripe_unit": "0", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_disable_zero_copy_writes": "true", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_discard_granularity_bytes": "65536", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_discard_on_zeroed_write_same": "true", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_enable_alloc_hint": "true", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_invalidate_object_map_on_timeout": "true", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_io_scheduler": "simple", 2024-09-21T08:24:29.611 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_io_scheduler_simple_max_delay": "0", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_commit_age": "5.000000", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_max_concurrent_object_sets": "0", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_max_payload_bytes": "16384", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_object_flush_age": "0.000000", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_object_flush_bytes": "1048576", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_object_flush_interval": "0", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_object_max_in_flight_appends": "0", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_object_writethrough_until_flush": "true", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_order": "24", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_pool": "", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_journal_splay_width": "4", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_localize_parent_reads": "false", 2024-09-21T08:24:29.612 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_localize_snap_reads": "false", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_concurrent_image_deletions": "1", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_concurrent_image_syncs": "5", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_delete_retry_interval": "30.000000", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_perf_stats_prio": "5", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_policy_migration_throttle": "300", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_policy_rebalance_timeout": "0.000000", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_policy_type": "simple", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_policy_update_throttle_interval": "1.000000", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_image_state_check_interval": "30", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_journal_commit_age": "5.000000", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_journal_poll_age": "5.000000", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_leader_heartbeat_interval": "5", 2024-09-21T08:24:29.613 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_leader_max_acquire_attempts_before_break": "3", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_leader_max_missed_heartbeats": "2", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_autotune": "true", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_base": "805306368", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_cache_autotune_interval": "30.000000", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_cache_min": "134217728", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_cache_resize_interval": "5.000000", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_expected_fragmentation": "0.150000", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_memory_target": "4294967296", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_perf_stats_prio": "5", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_pool_replayers_refresh_interval": "30", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirror_sync_point_update_age": "30.000000", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirroring_delete_delay": "0", 2024-09-21T08:24:29.614 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirroring_max_mirroring_snapshots": "5", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirroring_replay_delay": "0", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mirroring_resync_after_disconnect": "false", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_move_parent_to_trash_on_remove": "false", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_move_to_trash_on_remove": "false", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_move_to_trash_on_remove_expire_seconds": "0", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_mtime_update_interval": "60", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_non_blocking_aio": "true", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_op_thread_timeout": "60", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_op_threads": "1", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_parent_cache_enabled": "false", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_persistent_cache_mode": "disabled", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_persistent_cache_path": "/tmp", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_persistent_cache_size": "1073741824", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_plugins": "", 2024-09-21T08:24:29.615 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_bps_burst": "0", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_bps_burst_seconds": "1", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_bps_limit": "0", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_exclude_ops": "0", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_iops_burst": "0", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_iops_burst_seconds": "1", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_iops_limit": "0", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_bps_burst": "0", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_bps_burst_seconds": "1", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_bps_limit": "0", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_iops_burst": "0", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_iops_burst_seconds": "1", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_read_iops_limit": "0", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_schedule_tick_min": "50", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_bps_burst": "0", 2024-09-21T08:24:29.616 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_bps_burst_seconds": "1", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_bps_limit": "0", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_iops_burst": "0", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_iops_burst_seconds": "1", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_qos_write_iops_limit": "0", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_quiesce_notification_attempts": "10", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_read_from_replica_policy": "default", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_readahead_disable_after_bytes": "52428800", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_readahead_max_bytes": "524288", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_readahead_trigger_requests": "10", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_request_timed_out_seconds": "30", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_skip_partial_discard": "true", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_sparse_read_threshold_bytes": "65536", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_tracing": "false", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_validate_names": "true", 2024-09-21T08:24:29.617 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd_validate_pool": "true", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "restapi_base_url": "", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "restapi_log_level": "", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_account_default_quota_max_objects": "-1", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_account_default_quota_max_size": "-1", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_acl_grants_max_num": "100", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_admin_entry": "admin", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_allow_notification_secrets_in_cleartext": "false", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_backend_store": "rados", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_barbican_url": "", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_beast_enable_async": "true", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_counters_cache": "false", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_counters_cache_size": "10000", 2024-09-21T08:24:29.618 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_default_quota_max_objects": "-1", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_default_quota_max_size": "-1", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_index_max_aio": "128", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_index_transaction_instrumentation": "false", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_quota_cache_size": "10000", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_quota_ttl": "600", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_bucket_sync_spawn_window": "20", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_cache_enabled": "true", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_cache_expiry_interval": "900", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_cache_lru_size": "25000", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_config_store": "rados", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_content_length_compat": "false", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_copy_obj_progress": "true", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_copy_obj_progress_every_bytes": "1048576", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_cors_rules_max_num": "100", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_cross_domain_policy": "", 2024-09-21T08:24:29.619 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_default_encryption_key": "", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_addr": "", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_ca_path": "", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_client_cert": "", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_client_key": "", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_kms_key_template": "", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_password": "", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_s3_key_template": "$keyid", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_kmip_username": "", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_require_ssl": "true", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_s3_kms_backend": "barbican", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_s3_kms_encryption_keys": "", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_backend": "vault", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_key_template": "%bucket_id", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_addr": "", 2024-09-21T08:24:29.620 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_auth": "token", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_namespace": "", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_prefix": "", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_secret_engine": "transit", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_ssl_cacert": "", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_ssl_clientcert": "", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_ssl_clientkey": "", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_token_file": "", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_sse_s3_vault_verify_ssl": "true", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_suppress_logs": "true", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_addr": "", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_auth": "token", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_namespace": "", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_prefix": "", 2024-09-21T08:24:29.621 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_secret_engine": "transit", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_ssl_cacert": "", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_ssl_clientcert": "", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_ssl_clientkey": "", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_token_file": "", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_crypt_vault_verify_ssl": "true", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_curl_buffersize": "524288", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_curl_low_speed_limit": "1024", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_curl_low_speed_time": "300", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_curl_tcp_keepalive": "0", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_curl_wait_timeout_ms": "1000", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_datacache_persistent_path": "/tmp/rgw_datacache/", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_datacache_size": "1073741824", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_evict_cache_on_start": "true", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_eviction_policy": "lru", 2024-09-21T08:24:29.622 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_fadvise": "4", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_l1_local_datacache_enabled": "false", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_libaio_aio_num": "64", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d3n_libaio_aio_threads": "20", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d4n_host": "127.0.0.1", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_d4n_port": "6379", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data": "/var/lib/ceph/radosgw/ceph-a", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_log_changes_size": "1000", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_log_num_shards": "128", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_log_obj_prefix": "data_log", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_log_window": "30", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_notify_interval_msec": "0", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_sync_poll_interval": "20", 2024-09-21T08:24:29.623 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_data_sync_spawn_window": "20", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_debug_inject_latency_bi_unlink": "0", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_debug_inject_olh_cancel_modification_err": "false", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_debug_inject_set_olh_err": "0", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_default_data_log_backing": "fifo", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_default_realm_info_oid": "default.realm", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_default_region_info_oid": "default.region", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_default_zone_info_oid": "default.zone", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_default_zonegroup_info_oid": "default.zonegroup", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_defer_to_bucket_acls": "", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_delete_multi_obj_max_num": "1000", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_admin_lim": "0.000000", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_admin_res": "100.000000", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_admin_wgt": "100.000000", 2024-09-21T08:24:29.624 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_auth_lim": "0.000000", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_auth_res": "200.000000", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_auth_wgt": "100.000000", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_data_lim": "0.000000", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_data_res": "500.000000", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_data_wgt": "500.000000", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_metadata_lim": "0.000000", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_metadata_res": "500.000000", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dmclock_metadata_wgt": "500.000000", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dns_name": "", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dns_s3website_name": "", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_dynamic_resharding": "true", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_apis": "s3, s3website, swift, swift_auth, admin, sts, iam, notifications", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_gc_threads": "true", 2024-09-21T08:24:29.625 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_lc_threads": "true", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_ops_log": "false", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_quota_threads": "true", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_static_website": "false", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enable_usage_log": "false", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_enforce_swift_acls": "true", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_exit_timeout_secs": "120", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_expose_bucket": "false", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_extended_http_attrs": "", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_filter": "none", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_frontend_defaults": "beast ssl_certificate=config://rgw/cert/$realm/$zone.crt ssl_private_key=config://rgw/cert/$realm/$zone.key", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_frontends": "beast port=7480", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_concurrent_io": "10", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_deferred": "50", 2024-09-21T08:24:29.626 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_deferred_entries_size": "3072", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_objs": "32", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_queue_size": "134213632", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_max_trim_chunk": "16", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_obj_min_wait": "7200", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_processor_max_time": "3600", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_gc_processor_period": "3600", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_get_obj_max_req_size": "4194304", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_get_obj_window_size": "16777216", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_healthcheck_disabling_path": "", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ignore_get_invalid_range": "false", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_init_timeout": "300", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_inject_notify_timeout_probability": "0.000000", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_json_config": "/var/lib/ceph/radosgw/config.json", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_kafka_connection_idle": "300", 2024-09-21T08:24:29.627 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_kafka_message_timeout": "5000", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_kafka_sleep_timeout": "10", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_accepted_admin_roles": "", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_accepted_reader_roles": "", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_accepted_roles": "Member, admin", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_domain": "", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_password": "", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_password_path": "", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_project": "", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_tenant": "", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_token": "", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_token_path": "", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_admin_user": "", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_api_version": "2", 2024-09-21T08:24:29.628 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_barbican_domain": "", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_barbican_password": "", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_barbican_project": "", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_barbican_tenant": "", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_barbican_user": "", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_expired_token_cache_expiration": "3600", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_implicit_tenants": "false", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_service_token_accepted_roles": "admin", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_service_token_enabled": "false", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_token_cache_size": "10000", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_url": "", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_keystone_verify_ssl": "true", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_debug_interval": "-1", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_lock_max_time": "90", 2024-09-21T08:24:29.629 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_max_objs": "32", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_max_rules": "1000", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_max_worker": "3", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_max_wp_worker": "3", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lc_thread_delay": "0", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_binddn": "uid=admin,cn=users,dc=example,dc=com", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_dnattr": "uid", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_searchdn": "cn=users,cn=accounts,dc=example,dc=com", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_searchfilter": "", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_secret": "/etc/openldap/secret", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ldap_uri": "", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lifecycle_work_time": "00:00-06:00", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_list_bucket_min_readahead": "1000", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_list_buckets_max_chunk": "1000", 2024-09-21T08:24:29.630 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_log_http_headers": "", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_log_nonexistent_bucket": "false", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_log_object_name": "%Y-%m-%d-%H-%i-%n", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_log_object_name_utc": "false", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_lua_max_memory_per_state": "128000", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_luarocks_location": "/tmp/rgw_luarocks/mon.a", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_attr_name_len": "0", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_attr_size": "0", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_attrs_num_in_req": "0", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_chunk_size": "4194304", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_concurrent_requests": "1024", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_copy_obj_concurrent_io": "10", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_dynamic_shards": "1999", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_listing_results": "1000", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_notify_retries": "10", 2024-09-21T08:24:29.631 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_objs_per_shard": "100000", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_put_param_size": "1048576", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_put_size": "5368709120", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_max_slo_entries": "1000", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_md_log_max_shards": "64", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_md_notify_interval_msec": "200", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_meta_sync_poll_interval": "20", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_meta_sync_spawn_window": "20", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_mime_types_file": "/etc/mime.types", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_mp_lock_max_time": "600", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_multi_obj_del_max_aio": "16", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_multipart_min_part_size": "5242880", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_multipart_part_upload_limit": "10000", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_fhcache_partitions": "3", 2024-09-21T08:24:29.632 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_fhcache_size": "2017", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_frontends": "rgw-nfs", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_lru_lane_hiwat": "911", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_lru_lanes": "5", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_max_gc": "300", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_namespace_expire_secs": "300", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_run_gc_threads": "false", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_run_lc_threads": "false", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_run_quota_threads": "false", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_run_sync_thread": "false", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_s3_fast_attrs": "false", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_nfs_write_completion_interval_s": "10", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_num_async_rados_threads": "32", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_num_control_oids": "8", 2024-09-21T08:24:29.633 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_numa_node": "-1", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_obj_stripe_size": "4194304", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_obj_tombstone_cache_size": "1000", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_objexp_chunk_size": "100", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_objexp_gc_interval": "600", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_objexp_hints_num_shards": "127", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_olh_pending_timeout_sec": "3600", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_op_thread_suicide_timeout": "0", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_op_thread_timeout": "600", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_op_tracing": "false", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_opa_token": "", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_opa_url": "", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_opa_verify_ssl": "true", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ops_log_data_backlog": "5242880", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ops_log_file_path": "/var/log/ceph/ops-log-ceph-mon.a.log", 2024-09-21T08:24:29.634 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ops_log_rados": "false", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_ops_log_socket_path": "", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_override_bucket_index_max_shards": "0", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_pending_bucket_index_op_expiration": "120", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_period_latest_epoch_info_oid": ".latest_epoch", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_period_push_interval": "2.000000", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_period_push_interval_max": "30.000000", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_period_root_pool": ".rgw.root", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_policy_reject_invalid_principals": "true", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_base_path": "/tmp/rgw_posix_driver", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_cache_lanes": "3", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_cache_lmdb_count": "3", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_cache_max_buckets": "100", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_cache_partitions": "3", 2024-09-21T08:24:29.635 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_posix_database_root": "/var/lib/ceph/radosgw", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_print_continue": "true", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_print_prohibited_content_length": "false", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_put_obj_max_window_size": "67108864", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_put_obj_min_window_size": "16777216", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_rados_pool_autoscale_bias": "4.000000", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_rados_pool_recovery_priority": "5", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_rados_tracing": "false", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_realm": "", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_realm_id": "", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_realm_root_pool": ".rgw.root", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_region": "", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_region_root_pool": ".rgw.root", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_relaxed_region_enforcement": "false", 2024-09-21T08:24:29.636 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_relaxed_s3_bucket_names": "false", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_remote_addr_param": "REMOTE_ADDR", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_request_uri": "", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_batch_size": "64", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_bucket_lock_duration": "360", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_max_aio": "128", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_num_logs": "16", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_reshard_thread_interval": "600", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_resolve_cname": "false", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_rest_getusage_op_compat": "false", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_run_sync_thread": "true", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_disable_signature_url": "false", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_order": "sts, external, local", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_use_keystone": "false", 2024-09-21T08:24:29.637 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_use_ldap": "false", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_use_rados": "true", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_auth_use_sts": "false", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_client_max_sig_ver": "-1", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_s3_success_create_obj_status": "0", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_safe_max_objects_per_shard": "102400", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_scheduler_type": "throttler", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_script_uri": "", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_service_provider_name": "", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_shard_warning_threshold": "90.000000", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_client_id": "", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_client_secret": "", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_entry": "sts", 2024-09-21T08:24:29.638 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_key": "sts", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_max_session_duration": "43200", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_min_session_duration": "900", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sts_token_introspection_url": "", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_account_in_url": "false", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_auth_entry": "auth", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_auth_url": "", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_custom_header": "", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_enforce_content_length": "false", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_need_stats": "true", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_tenant_name": "", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_token_expiration": "86400", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_url": "", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_url_prefix": "swift", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_swift_versioning_enabled": "false", 2024-09-21T08:24:29.639 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_data_full_inject_err_probability": "0.000000", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_data_inject_err_probability": "0.000000", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_lease_period": "120", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_log_trim_concurrent_buckets": "4", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_log_trim_interval": "1200", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_log_trim_max_buckets": "16", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_log_trim_min_cold_buckets": "4", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_meta_inject_err_probability": "0.000000", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_obj_etag_verify": "false", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_trace_history_size": "4096", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_trace_per_node_log_size": "32", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_sync_trace_servicemap_update_interval": "10", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_thread_pool_size": "512", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_topic_persistency_max_retries": "0", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_topic_persistency_sleep_duration": "0", 2024-09-21T08:24:29.640 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_topic_persistency_time_to_live": "0", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_topic_require_publish_policy": "false", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_comment": "", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_createby": "", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_encoding": "", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_flag": "false", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_max_size": "5368709120", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_origin": "", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_sha_unit": "524288", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_torrent_tracker": "", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_trust_forwarded_https": "false", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_usage_log_flush_threshold": "1024", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_usage_log_tick_interval": "30", 2024-09-21T08:24:29.641 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_usage_max_shards": "32", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_usage_max_user_shards": "1", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_use_opa_authz": "false", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_counters_cache": "false", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_counters_cache_size": "10000", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_default_quota_max_objects": "-1", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_default_quota_max_size": "-1", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_max_buckets": "1000", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_policies_max_num": "100", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_quota_bucket_sync_interval": "180", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_quota_sync_idle_users": "false", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_quota_sync_interval": "86400", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_quota_sync_wait_time": "86400", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_user_unique_email": "true", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_verify_ssl": "true", 2024-09-21T08:24:29.642 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_website_routing_rules_max_num": "50", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zone": "", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zone_id": "", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zone_root_pool": ".rgw.root", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zonegroup": "", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zonegroup_id": "", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw_zonegroup_root_pool": ".rgw.root", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rgwlc_auto_session_clear": "true", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rgwlc_skip_bucket_step": "false", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_block_size": "4096", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_bloom_bits_per_key": "20", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_index_and_filter_blocks": "true", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_index_and_filter_blocks_with_high_priority": "false", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_row_ratio": "0.000000", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_shard_bits": "4", 2024-09-21T08:24:29.643 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_size": "536870912", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cache_type": "binned_lru", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cf_compact_on_deletion": "true", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cf_compact_on_deletion_sliding_window": "32768", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_cf_compact_on_deletion_trigger": "16384", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_collect_compaction_stats": "false", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_collect_extended_stats": "false", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_collect_memory_stats": "false", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_delete_range_threshold": "1048576", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_index_type": "binary_search", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_log_to_ceph_log": "true", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_metadata_block_size": "4096", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_partition_filters": "false", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_perf": "false", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rocksdb_pin_l0_filter_and_index_blocks_in_cache": "false", 2024-09-21T08:24:29.644 INFO:teuthology.orchestra.run.smithi137.stdout: "rotating_keys_bootstrap_timeout": "30", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "rotating_keys_renewal_timeout": "10", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "run_dir": "/var/run/ceph", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_block_create": "true", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_cache_lru_size": "67108864", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_cbjournal_size": "5368709120", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_data_delta_based_overwrite": "0", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_default_max_object_size": "16777216", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_default_object_metadata_reservation": "16777216", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_device_size": "53687091200", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_disable_end_to_end_data_protection": "true", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_full_integrity_check": "false", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_journal_batch_capacity": "16", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_journal_batch_flush_size": "16777216", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_journal_batch_preferred_fullness": "0.950000", 2024-09-21T08:24:29.645 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_journal_iodepth_limit": "5", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_main_device_type": "SSD", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_max_concurrent_transactions": "8", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_max_data_allocation_size": "32768", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_multiple_tiers_default_evict_ratio": "0.600000", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_multiple_tiers_fast_evict_ratio": "0.700000", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_multiple_tiers_stop_evict_ratio": "0.500000", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_obj_data_write_amplification": "1.250000", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "seastore_segment_size": "67108864", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "set_keepcaps": "false", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "setgroup": "ceph", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "setuser": "ceph", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "setuser_match_path": "", 2024-09-21T08:24:29.646 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_misplaced_ratio": "0.050000", 2024-09-21T08:24:29.647 INFO:teuthology.orchestra.run.smithi137.stdout: "thp": "false", 2024-09-21T08:24:29.647 INFO:teuthology.orchestra.run.smithi137.stdout: "threadpool_default_timeout": "60", 2024-09-21T08:24:29.647 INFO:teuthology.orchestra.run.smithi137.stdout: "threadpool_empty_queue_max_wait": "2", 2024-09-21T08:24:29.647 INFO:teuthology.orchestra.run.smithi137.stdout: "throttler_perf_counter": "true" 2024-09-21T08:24:29.647 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:24:29.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:29 smithi137 ceph-mon[29634]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T08:24:29.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:29 smithi137 ceph-mon[41044]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T08:24:30.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:29 smithi165 ceph-mon[32893]: pgmap v18: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T08:24:30.364 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'mon.*' injectargs --mon_osd_down_out_interval=0 2024-09-21T08:24:30.611 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:30.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-mon[29634]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T08:24:30.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:30.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:30.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:24:30.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:24:30.998 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:24:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:24:30.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-mon[41044]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T08:24:30.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:30.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:30.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:24:30.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:30 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:24:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:30 smithi165 ceph-mon[32893]: pgmap v19: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T08:24:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:30 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:24:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:30 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:24:31.089 INFO:teuthology.orchestra.run.smithi137.stdout:mon.a: {} 2024-09-21T08:24:31.089 INFO:teuthology.orchestra.run.smithi137.stderr:mon.a: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T08:24:31.098 INFO:teuthology.orchestra.run.smithi137.stdout:mon.b: {} 2024-09-21T08:24:31.098 INFO:teuthology.orchestra.run.smithi137.stderr:mon.b: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T08:24:31.106 INFO:teuthology.orchestra.run.smithi137.stdout:mon.c: {} 2024-09-21T08:24:31.106 INFO:teuthology.orchestra.run.smithi137.stderr:mon.c: mon_osd_down_out_interval = '' (not observed, change may require restart) 2024-09-21T08:24:31.848 INFO:teuthology.run_tasks:Running task rados... 2024-09-21T08:24:31.864 INFO:tasks.rados:Beginning rados... 2024-09-21T08:24:31.864 DEBUG:teuthology.run_tasks:Unwinding manager rados 2024-09-21T08:24:31.891 INFO:tasks.rados:joining rados 2024-09-21T08:24:31.891 INFO:tasks.thrashosds.thrasher:starting do_thrash 2024-09-21T08:24:31.891 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 4, 5, 6, 7] out_osds: [] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:24:31.891 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:24:31.891 INFO:tasks.thrashosds.thrasher:Removing osd 4, in_osds are: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:24:31.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd out 4 2024-09-21T08:24:31.892 INFO:tasks.thrashosds.thrasher:starting do_sighup with a delay of 0.1 2024-09-21T08:24:31.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:31.893 INFO:tasks.thrashosds.thrasher:starting do_optrack_toggle with a delay of 2.0 2024-09-21T08:24:31.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:24:31.893 INFO:tasks.thrashosds.thrasher:starting do_dump_ops 2024-09-21T08:24:31.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T08:24:31.895 INFO:tasks.thrashosds.thrasher:starting do_noscrub_toggle with a delay of 2.0 2024-09-21T08:24:31.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:24:31.896 INFO:tasks.rados:clients are ['client.0', 'client.1'] 2024-09-21T08:24:31.896 INFO:tasks.rados:starting run 0 out of 1 2024-09-21T08:24:31.896 INFO:tasks.cephadm.ceph_manager.ceph:creating pool_name unique_pool_0 2024-09-21T08:24:31.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pool create unique_pool_0 16 2024-09-21T08:24:31.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:31.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:31.928 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:31 smithi165 ceph-mon[32893]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T08:24:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:31 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/331664558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:24:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:31 smithi165 ceph-mon[32893]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T08:24:32.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:31 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:32.014 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:24:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:24:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:31 smithi137 ceph-mon[29634]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T08:24:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:31 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/331664558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:24:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:31 smithi137 ceph-mon[29634]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T08:24:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:31 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:31 smithi137 ceph-mon[41044]: osdmap e78: 8 total, 8 up, 8 in 2024-09-21T08:24:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:31 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/331664558' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-09-21T08:24:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:31 smithi137 ceph-mon[41044]: osdmap e79: 8 total, 8 up, 8 in 2024-09-21T08:24:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:31 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:32.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:32.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:32.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:32.062 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:32.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:32.182 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:32.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:32.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:32.195 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:32.201 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:32.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:32.313 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:32.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:32.314 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:32.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:32.441 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:32.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:32.442 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:32.542 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:32.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:32.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:32.568 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:32.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:32.693 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:32.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:32.694 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:32.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:32.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:32.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:32.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:32.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:32.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:32.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:32.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:33.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:33.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:33.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:33.067 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:33.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:33.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:33.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:33.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:33.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:33.315 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:33.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:33.317 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:33.333 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:24:33.333 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:24:33.333 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:24:33.333 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:24:33.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:33.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:33.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:33.439 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:33.492 INFO:teuthology.orchestra.run.smithi137.stderr:pool 'unique_pool_0' created 2024-09-21T08:24:33.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:33.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:33.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:33.561 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:33.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:33.686 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:33.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:33.687 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:33.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:33 smithi165 ceph-mon[32893]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:33.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:33 smithi165 ceph-mon[32893]: 3.f scrub starts 2024-09-21T08:24:33.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:33 smithi165 ceph-mon[32893]: 3.f scrub ok 2024-09-21T08:24:33.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:33 smithi165 ceph-mon[32893]: 4.1b scrub starts 2024-09-21T08:24:33.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:33 smithi165 ceph-mon[32893]: 4.1b scrub ok 2024-09-21T08:24:33.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:33 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3700954400' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-21T08:24:33.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[29634]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[29634]: 3.f scrub starts 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[29634]: 3.f scrub ok 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[29634]: 4.1b scrub starts 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[29634]: 4.1b scrub ok 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3700954400' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[41044]: pgmap v22: 132 pgs: 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[41044]: 3.f scrub starts 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[41044]: 3.f scrub ok 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[41044]: 4.1b scrub starts 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[41044]: 4.1b scrub ok 2024-09-21T08:24:33.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:33 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3700954400' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]: dispatch 2024-09-21T08:24:33.788 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:33.811 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:33.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:33.812 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:33.913 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:33.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:33.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:33.935 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:34.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:34.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:34.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:34.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:34.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:34.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:34.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:34.188 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:34.222 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T08:24:34.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:34.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:34.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:34.311 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:34.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:34.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:34.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:34.433 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:34.501 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:24:34.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:34.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:34.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:34.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:34.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:34.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:34.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:34.680 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3700954400' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T08:24:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:34 smithi165 ceph-mon[32893]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T08:24:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4154408293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:24:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:34 smithi165 ceph-mon[32893]: from='client.44711 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:24:34.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3700954400' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T08:24:34.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:34 smithi137 ceph-mon[29634]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T08:24:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4154408293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:24:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:34 smithi137 ceph-mon[29634]: from='client.44711 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:24:34.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3700954400' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "unique_pool_0", "pg_num": 16}]': finished 2024-09-21T08:24:34.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:34 smithi137 ceph-mon[41044]: osdmap e80: 8 total, 8 up, 8 in 2024-09-21T08:24:34.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4154408293' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:24:34.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:34 smithi137 ceph-mon[41044]: from='client.44711 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:24:34.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:34.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:34.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:34.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:34.860 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pool application enable unique_pool_0 rados --yes-i-really-mean-it || true 2024-09-21T08:24:34.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:34.927 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:34.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:34.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:35.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:35.050 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:35.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:35.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:35.082 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:35.082 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:35.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:35.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:35.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:35.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:35.207 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:24:34 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[76851]: ts=2024-09-21T08:24:34.821Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.003868494s 2024-09-21T08:24:35.277 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:35.299 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:35.300 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:35.401 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:35.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:35.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:35.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:35.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:35.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:35.549 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:35.555 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:35.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:35.673 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:35.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:35.674 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:35 smithi165 ceph-mon[32893]: pgmap v24: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T08:24:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:35 smithi165 ceph-mon[32893]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T08:24:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:35 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:24:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:35 smithi165 ceph-mon[32893]: from='client.44711 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:24:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:35 smithi165 ceph-mon[32893]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T08:24:35.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-mon[29634]: pgmap v24: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T08:24:35.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-mon[29634]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T08:24:35.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:24:35.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-mon[29634]: from='client.44711 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:24:35.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-mon[29634]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T08:24:35.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-mon[41044]: pgmap v24: 148 pgs: 16 unknown, 132 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 853 B/s rd, 0 op/s 2024-09-21T08:24:35.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-mon[41044]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-09-21T08:24:35.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:24:35.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-mon[41044]: from='client.44711 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:24:35.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-mon[41044]: osdmap e81: 8 total, 8 up, 8 in 2024-09-21T08:24:35.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:35.789 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:35.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:35.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:35.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:35.923 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:35.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:35.924 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:36.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:36.045 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:24:35.899+0000 7f534001e640 -1 log_channel(cephadm) log [ERR] : Failed while placing grafana.a on smithi165: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a 2024-09-21T08:24:36.045 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a 2024-09-21T08:24:36.045 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana.a 2024-09-21T08:24:36.045 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana.a 2024-09-21T08:24:36.045 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Deploy daemon grafana.a ... 2024-09-21T08:24:36.045 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:9.4.12 -e NODE_NAME=smithi165 quay.io/ceph/grafana:9.4.12 -c %u %g /var/lib/grafana 2024-09-21T08:24:36.045 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: stat: stderr Trying to pull quay.io/ceph/grafana:9.4.12... 2024-09-21T08:24:36.045 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: stat: stderr Error: initializing source docker://quay.io/ceph/grafana:9.4.12: reading manifest 9.4.12 in quay.io/ceph/grafana: received unexpected HTTP status: 504 Gateway Time-out 2024-09-21T08:24:36.045 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ERROR: Failed to extract uid/gid for path /var/lib/grafana: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:9.4.12 -e NODE_NAME=smithi165 quay.io/ceph/grafana:9.4.12 -c %u %g /var/lib/grafana: Trying to pull quay.io/ceph/grafana:9.4.12... 2024-09-21T08:24:36.045 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:35 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Error: initializing source docker://quay.io/ceph/grafana:9.4.12: reading manifest 9.4.12 in quay.io/ceph/grafana: received unexpected HTTP status: 504 Gateway Time-out 2024-09-21T08:24:36.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:36.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:36.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:36.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:36.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:36.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:36.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:36.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:36.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:36.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:36.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:36.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:36.420 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:36.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:36.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:36.544 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:36.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:36.544 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:36.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:36 smithi137 ceph-mon[29634]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T08:24:36.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:36 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:36.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:36 smithi137 ceph-mon[29634]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T08:24:36.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:36 smithi137 ceph-mon[41044]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T08:24:36.636 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:36 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:36.636 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:36 smithi137 ceph-mon[41044]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T08:24:36.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:36.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:36.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:36.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:36 smithi165 ceph-mon[32893]: osdmap e82: 8 total, 8 up, 8 in 2024-09-21T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:36 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:36.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:36 smithi165 ceph-mon[32893]: osdmap e83: 8 total, 8 up, 8 in 2024-09-21T08:24:36.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:36.790 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:36.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:36.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:36.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:36.912 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:36.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:36.913 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:37.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:37.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:37.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:37.035 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:37.063 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:24:37.063 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:24:37.063 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:24:37.063 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:24:37.063 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:24:37.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:37.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:37.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:37.159 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:37.259 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:37.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29611]: 2024-09-21T08:24:37.115+0000 7f1058bb7640 -1 mon.a@0(leader).osd e83 definitely_dead 0 2024-09-21T08:24:37.280 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:37.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:37.281 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:37.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:37.403 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:37.403 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:37.404 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:37.505 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:37.517 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.4. 2024-09-21T08:24:37.517 INFO:teuthology.orchestra.run.smithi137.stderr:enabled application 'rados' on pool 'unique_pool_0' 2024-09-21T08:24:37.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:37.539 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:37.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:24:37.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[29634]: pgmap v27: 148 pgs: 148 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[29634]: Failed while placing grafana.a on smithi165: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana.a 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana.a 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout: Deploy daemon grafana.a ... 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout: Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:9.4.12 -e NODE_NAME=smithi165 quay.io/ceph/grafana:9.4.12 -c %u %g /var/lib/grafana 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout: stat: stderr Trying to pull quay.io/ceph/grafana:9.4.12... 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout: stat: stderr Error: initializing source docker://quay.io/ceph/grafana:9.4.12: reading manifest 9.4.12 in quay.io/ceph/grafana: received unexpected HTTP status: 504 Gateway Time-out 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout: ERROR: Failed to extract uid/gid for path /var/lib/grafana: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:9.4.12 -e NODE_NAME=smithi165 quay.io/ceph/grafana:9.4.12 -c %u %g /var/lib/grafana: Trying to pull quay.io/ceph/grafana:9.4.12... 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout: Error: initializing source docker://quay.io/ceph/grafana:9.4.12: reading manifest 9.4.12 in quay.io/ceph/grafana: received unexpected HTTP status: 504 Gateway Time-out 2024-09-21T08:24:37.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[29634]: pgmap v28: 148 pgs: 148 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[29634]: Deploying daemon node-exporter.a on smithi137 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[29634]: Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4259691001' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4007791077' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[29634]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[41044]: pgmap v27: 148 pgs: 148 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[41044]: Failed while placing grafana.a on smithi165: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.c.smithi137.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.c.smithi137.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana.a 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.c.smithi137.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana.a 2024-09-21T08:24:37.661 INFO:journalctl@ceph.mon.c.smithi137.stdout: Deploy daemon grafana.a ... 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:9.4.12 -e NODE_NAME=smithi165 quay.io/ceph/grafana:9.4.12 -c %u %g /var/lib/grafana 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: stat: stderr Trying to pull quay.io/ceph/grafana:9.4.12... 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: stat: stderr Error: initializing source docker://quay.io/ceph/grafana:9.4.12: reading manifest 9.4.12 in quay.io/ceph/grafana: received unexpected HTTP status: 504 Gateway Time-out 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: ERROR: Failed to extract uid/gid for path /var/lib/grafana: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:9.4.12 -e NODE_NAME=smithi165 quay.io/ceph/grafana:9.4.12 -c %u %g /var/lib/grafana: Trying to pull quay.io/ceph/grafana:9.4.12... 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout: Error: initializing source docker://quay.io/ceph/grafana:9.4.12: reading manifest 9.4.12 in quay.io/ceph/grafana: received unexpected HTTP status: 504 Gateway Time-out 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[41044]: pgmap v28: 148 pgs: 148 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[41044]: Deploying daemon node-exporter.a on smithi137 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[41044]: Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4259691001' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4007791077' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-21T08:24:37.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:37 smithi137 ceph-mon[41044]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T08:24:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:37.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:37.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:37.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T08:24:37.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:37.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:37.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:37.791 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:37.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:37.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:37.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:37.915 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:37 smithi165 ceph-mon[32893]: pgmap v27: 148 pgs: 148 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:37 smithi165 ceph-mon[32893]: Failed while placing grafana.a on smithi165: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a 2024-09-21T08:24:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a 2024-09-21T08:24:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana.a 2024-09-21T08:24:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana.a 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: Deploy daemon grafana.a ... 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: Non-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:9.4.12 -e NODE_NAME=smithi165 quay.io/ceph/grafana:9.4.12 -c %u %g /var/lib/grafana 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: stat: stderr Trying to pull quay.io/ceph/grafana:9.4.12... 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: stat: stderr Error: initializing source docker://quay.io/ceph/grafana:9.4.12: reading manifest 9.4.12 in quay.io/ceph/grafana: received unexpected HTTP status: 504 Gateway Time-out 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: ERROR: Failed to extract uid/gid for path /var/lib/grafana: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:9.4.12 -e NODE_NAME=smithi165 quay.io/ceph/grafana:9.4.12 -c %u %g /var/lib/grafana: Trying to pull quay.io/ceph/grafana:9.4.12... 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: Error: initializing source docker://quay.io/ceph/grafana:9.4.12: reading manifest 9.4.12 in quay.io/ceph/grafana: received unexpected HTTP status: 504 Gateway Time-out 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:37 smithi165 ceph-mon[32893]: pgmap v28: 148 pgs: 148 active+clean; 583 KiB data, 226 MiB used, 715 GiB / 715 GiB avail; 1.2 KiB/s rd, 1 op/s 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:37 smithi165 ceph-mon[32893]: Deploying daemon node-exporter.a on smithi137 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:37 smithi165 ceph-mon[32893]: Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL) 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4259691001' entity='client.admin' cmd=[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]: dispatch 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4007791077' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["4"]}]: dispatch 2024-09-21T08:24:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:37 smithi165 ceph-mon[32893]: Client client.admin marked osd.4 out, while it was still marked up 2024-09-21T08:24:38.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:38.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:38.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:38.042 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:38.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:38.168 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:38.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:38.169 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:38.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:38.291 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:38.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:38.292 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:38.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:38.413 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:38.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:38.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:38.515 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:38.551 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:38.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:38.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:38.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:38.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:38.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:38.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:38.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:38.798 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:38.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:38.799 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:38.876 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:38 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4259691001' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T08:24:38.876 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:38 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4007791077' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T08:24:38.876 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:38 smithi137 ceph-mon[41044]: osdmap e84: 8 total, 8 up, 7 in 2024-09-21T08:24:38.876 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:38 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:24:38.876 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:38 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:24:38.876 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:38 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4259691001' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T08:24:38.876 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:38 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4007791077' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T08:24:38.876 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:38 smithi137 ceph-mon[29634]: osdmap e84: 8 total, 8 up, 7 in 2024-09-21T08:24:38.876 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:38 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:24:38.876 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:38 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:24:38.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:38.920 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:38.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:38.921 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:38 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4259691001' entity='client.admin' cmd='[{"prefix": "osd pool application enable", "pool": "unique_pool_0", "app": "rados", "yes_i_really_mean_it": true}]': finished 2024-09-21T08:24:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:38 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4007791077' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["4"]}]': finished 2024-09-21T08:24:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:38 smithi165 ceph-mon[32893]: osdmap e84: 8 total, 8 up, 7 in 2024-09-21T08:24:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:38 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:24:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:38 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:24:39.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:39.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:39.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:39.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:39.068 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:24:39.068 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:24:39.083 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:24:39.084 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:24:39.100 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:24:39.101 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:24:39.120 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:24:39.120 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:24:39.141 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:24:39.141 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:24:39.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:39.160 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:39.160 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:39.161 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:24:39.161 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:24:39.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:39.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:39.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:39.191 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:24:39.192 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:24:39.209 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:24:39.209 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:24:39.233 DEBUG:teuthology.orchestra.run.smithi137:> CEPH_CLIENT_ID=0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph_test_rados --max-ops 4000 --objects 50 --max-in-flight 16 --size 4000000 --min-stride-size 400000 --max-stride-size 800000 --max-seconds 0 --op read 100 --op write 50 --op delete 50 --op snap_create 50 --op snap_remove 50 --op rollback 50 --op copy_from 50 --op write_excl 50 --pool unique_pool_0 2024-09-21T08:24:39.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:39.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:39.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:39.295 INFO:tasks.rados.rados.0.smithi137.stdout:adding op weight read -> 100 2024-09-21T08:24:39.295 INFO:tasks.rados.rados.0.smithi137.stdout:adding op weight write -> 50 2024-09-21T08:24:39.295 INFO:tasks.rados.rados.0.smithi137.stdout:adding op weight delete -> 50 2024-09-21T08:24:39.295 INFO:tasks.rados.rados.0.smithi137.stdout:adding op weight snap_create -> 50 2024-09-21T08:24:39.295 INFO:tasks.rados.rados.0.smithi137.stdout:adding op weight snap_remove -> 50 2024-09-21T08:24:39.295 INFO:tasks.rados.rados.0.smithi137.stdout:adding op weight rollback -> 50 2024-09-21T08:24:39.295 INFO:tasks.rados.rados.0.smithi137.stdout:adding op weight copy_from -> 50 2024-09-21T08:24:39.296 INFO:tasks.rados.rados.0.smithi137.stdout:adding op weight write_excl -> 50 2024-09-21T08:24:39.296 INFO:tasks.rados.rados.0.smithi137.stdout:ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable) 2024-09-21T08:24:39.296 INFO:tasks.rados.rados.0.smithi137.stdout:Configuration: 2024-09-21T08:24:39.296 INFO:tasks.rados.rados.0.smithi137.stdout: Number of operations: 4000 2024-09-21T08:24:39.296 INFO:tasks.rados.rados.0.smithi137.stdout: Number of objects: 50 2024-09-21T08:24:39.296 INFO:tasks.rados.rados.0.smithi137.stdout: Max in flight operations: 16 2024-09-21T08:24:39.296 INFO:tasks.rados.rados.0.smithi137.stdout: Object size (in bytes): 4000000 2024-09-21T08:24:39.296 INFO:tasks.rados.rados.0.smithi137.stdout: Write stride min: 400000 2024-09-21T08:24:39.296 INFO:tasks.rados.rados.0.smithi137.stdout: Write stride max: 800000 2024-09-21T08:24:39.392 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:39.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:39.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:39.419 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:39.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:39.529 INFO:tasks.rados.rados.0.smithi137.stdout:1: write initial oid 1 2024-09-21T08:24:39.529 INFO:tasks.rados.rados.0.smithi137.stdout:1: seq_num 0 ranges {416807=475249,1542129=474678} 2024-09-21T08:24:39.535 INFO:tasks.rados.rados.0.smithi137.stdout:1: writing smithi13779221-1 from 416807 to 892056 tid 1 2024-09-21T08:24:39.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:39.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:39.547 INFO:tasks.rados.rados.0.smithi137.stdout:1: writing smithi13779221-1 from 1542129 to 2016807 tid 2 2024-09-21T08:24:39.547 INFO:tasks.rados.rados.0.smithi137.stdout:2: write initial oid 2 2024-09-21T08:24:39.547 INFO:tasks.rados.rados.0.smithi137.stdout:2: seq_num 1 ranges {416807=475249,1542129=474678} 2024-09-21T08:24:39.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:39.553 INFO:tasks.rados.rados.0.smithi137.stdout:2: writing smithi13779221-2 from 416807 to 892056 tid 1 2024-09-21T08:24:39.561 INFO:tasks.rados.rados.0.smithi137.stdout:2: writing smithi13779221-2 from 1542129 to 2016807 tid 2 2024-09-21T08:24:39.561 INFO:tasks.rados.rados.0.smithi137.stdout:3: write initial oid 3 2024-09-21T08:24:39.561 INFO:tasks.rados.rados.0.smithi137.stdout:3: seq_num 2 ranges {433614=550498,1600611=433003} 2024-09-21T08:24:39.566 INFO:tasks.rados.rados.0.smithi137.stdout:3: writing smithi13779221-3 from 433614 to 984112 tid 1 2024-09-21T08:24:39.572 INFO:tasks.rados.rados.0.smithi137.stdout:3: writing smithi13779221-3 from 1600611 to 2033614 tid 2 2024-09-21T08:24:39.572 INFO:tasks.rados.rados.0.smithi137.stdout:4: write initial oid 4 2024-09-21T08:24:39.572 INFO:tasks.rados.rados.0.smithi137.stdout:4: seq_num 3 ranges {450421=625747,1659093=391328} 2024-09-21T08:24:39.578 INFO:tasks.rados.rados.0.smithi137.stdout:4: writing smithi13779221-4 from 450421 to 1076168 tid 1 2024-09-21T08:24:39.584 INFO:tasks.rados.rados.0.smithi137.stdout:4: writing smithi13779221-4 from 1659093 to 2050421 tid 2 2024-09-21T08:24:39.584 INFO:tasks.rados.rados.0.smithi137.stdout:5: write initial oid 5 2024-09-21T08:24:39.584 INFO:tasks.rados.rados.0.smithi137.stdout:5: seq_num 4 ranges {467228=700996,1717575=349653} 2024-09-21T08:24:39.590 INFO:tasks.rados.rados.0.smithi137.stdout:5: writing smithi13779221-5 from 467228 to 1168224 tid 1 2024-09-21T08:24:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:5: writing smithi13779221-5 from 1717575 to 2067228 tid 2 2024-09-21T08:24:39.596 INFO:tasks.rados.rados.0.smithi137.stdout:6: write initial oid 6 2024-09-21T08:24:39.596 INFO:tasks.rados.rados.0.smithi137.stdout:6: seq_num 5 ranges {484035=776245,2059704=24331} 2024-09-21T08:24:39.602 INFO:tasks.rados.rados.0.smithi137.stdout:6: writing smithi13779221-6 from 484035 to 1260280 tid 1 2024-09-21T08:24:39.607 INFO:tasks.rados.rados.0.smithi137.stdout:6: writing smithi13779221-6 from 2059704 to 2084035 tid 2 2024-09-21T08:24:39.607 INFO:tasks.rados.rados.0.smithi137.stdout:7: write initial oid 7 2024-09-21T08:24:39.607 INFO:tasks.rados.rados.0.smithi137.stdout:7: seq_num 6 ranges {500842=451494,1718186=382656} 2024-09-21T08:24:39.612 INFO:tasks.rados.rados.0.smithi137.stdout:7: writing smithi13779221-7 from 500842 to 952336 tid 1 2024-09-21T08:24:39.618 INFO:tasks.rados.rados.0.smithi137.stdout:7: writing smithi13779221-7 from 1718186 to 2100842 tid 2 2024-09-21T08:24:39.618 INFO:tasks.rados.rados.0.smithi137.stdout:8: write initial oid 8 2024-09-21T08:24:39.619 INFO:tasks.rados.rados.0.smithi137.stdout:8: seq_num 7 ranges {517649=526743,1776668=340981} 2024-09-21T08:24:39.624 INFO:tasks.rados.rados.0.smithi137.stdout:8: writing smithi13779221-8 from 517649 to 1044392 tid 1 2024-09-21T08:24:39.631 INFO:tasks.rados.rados.0.smithi137.stdout:8: writing smithi13779221-8 from 1776668 to 2117649 tid 2 2024-09-21T08:24:39.631 INFO:tasks.rados.rados.0.smithi137.stdout:9: write initial oid 9 2024-09-21T08:24:39.631 INFO:tasks.rados.rados.0.smithi137.stdout:9: seq_num 8 ranges {534456=718345,1951503=182953} 2024-09-21T08:24:39.637 INFO:tasks.rados.rados.0.smithi137.stdout:9: writing smithi13779221-9 from 534456 to 1252801 tid 1 2024-09-21T08:24:39.642 INFO:tasks.rados.rados.0.smithi137.stdout:9: writing smithi13779221-9 from 1951503 to 2134456 tid 2 2024-09-21T08:24:39.642 INFO:tasks.rados.rados.0.smithi137.stdout:10: write initial oid 10 2024-09-21T08:24:39.642 INFO:tasks.rados.rados.0.smithi137.stdout:10: seq_num 9 ranges {551263=793594,1893632=257631} 2024-09-21T08:24:39.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:39.650 INFO:tasks.rados.rados.0.smithi137.stdout:10: writing smithi13779221-10 from 551263 to 1344857 tid 1 2024-09-21T08:24:39.654 INFO:tasks.rados.rados.0.smithi137.stdout:10: writing smithi13779221-10 from 1893632 to 2151263 tid 2 2024-09-21T08:24:39.654 INFO:tasks.rados.rados.0.smithi137.stdout:11: write initial oid 11 2024-09-21T08:24:39.654 INFO:tasks.rados.rados.0.smithi137.stdout:11: seq_num 10 ranges {568070=468843,1552114=615956} 2024-09-21T08:24:39.659 INFO:tasks.rados.rados.0.smithi137.stdout:11: writing smithi13779221-11 from 568070 to 1036913 tid 1 2024-09-21T08:24:39.666 INFO:tasks.rados.rados.0.smithi137.stdout:11: writing smithi13779221-11 from 1552114 to 2168070 tid 2 2024-09-21T08:24:39.666 INFO:tasks.rados.rados.0.smithi137.stdout:12: write initial oid 12 2024-09-21T08:24:39.666 INFO:tasks.rados.rados.0.smithi137.stdout:12: seq_num 11 ranges {584877=544092,1610596=562003} 2024-09-21T08:24:39.672 INFO:tasks.rados.rados.0.smithi137.stdout:12: writing smithi13779221-12 from 584877 to 1128969 tid 1 2024-09-21T08:24:39.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:39.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:39.675 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:39.679 INFO:tasks.rados.rados.0.smithi137.stdout:12: writing smithi13779221-12 from 1610596 to 2172599 tid 2 2024-09-21T08:24:39.679 INFO:tasks.rados.rados.0.smithi137.stdout:13: write initial oid 13 2024-09-21T08:24:39.679 INFO:tasks.rados.rados.0.smithi137.stdout:13: seq_num 12 ranges {601684=619341,1669078=532606} 2024-09-21T08:24:39.685 INFO:tasks.rados.rados.0.smithi137.stdout:13: writing smithi13779221-13 from 601684 to 1221025 tid 1 2024-09-21T08:24:39.691 INFO:tasks.rados.rados.0.smithi137.stdout:13: writing smithi13779221-13 from 1669078 to 2201684 tid 2 2024-09-21T08:24:39.691 INFO:tasks.rados.rados.0.smithi137.stdout:14: write initial oid 14 2024-09-21T08:24:39.691 INFO:tasks.rados.rados.0.smithi137.stdout:14: seq_num 13 ranges {618491=694590,2011207=207284} 2024-09-21T08:24:39.698 INFO:tasks.rados.rados.0.smithi137.stdout:14: writing smithi13779221-14 from 618491 to 1313081 tid 1 2024-09-21T08:24:39.703 INFO:tasks.rados.rados.0.smithi137.stdout:14: writing smithi13779221-14 from 2011207 to 2218491 tid 2 2024-09-21T08:24:39.703 INFO:tasks.rados.rados.0.smithi137.stdout:15: write initial oid 15 2024-09-21T08:24:39.704 INFO:tasks.rados.rados.0.smithi137.stdout:15: seq_num 14 ranges {635298=769839,2069689=165609} 2024-09-21T08:24:39.711 INFO:tasks.rados.rados.0.smithi137.stdout:15: writing smithi13779221-15 from 635298 to 1405137 tid 1 2024-09-21T08:24:39.716 INFO:tasks.rados.rados.0.smithi137.stdout:15: writing smithi13779221-15 from 2069689 to 2235298 tid 2 2024-09-21T08:24:39.716 INFO:tasks.rados.rados.0.smithi137.stdout:16: write initial oid 16 2024-09-21T08:24:39.716 INFO:tasks.rados.rados.0.smithi137.stdout:16: seq_num 15 ranges {652105=445088,1728171=452988} 2024-09-21T08:24:39.722 INFO:tasks.rados.rados.0.smithi137.stdout:16: writing smithi13779221-16 from 652105 to 1097193 tid 1 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:16: writing smithi13779221-16 from 1728171 to 2181159 tid 2 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:1: finishing write tid 1 to smithi13779221-1 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:1: finishing write tid 2 to smithi13779221-1 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:1: finishing write tid 3 to smithi13779221-1 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:1: finishing write tid 4 to smithi13779221-1 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:1: oid 1 updating version 0 to 1 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:1: oid 1 updating version 1 to 2 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:1: oid 1 updating version 2 to 3 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:1: left oid 1 (ObjNum 0 snap 0 seq_num 0) 2024-09-21T08:24:39.729 INFO:tasks.rados.rados.0.smithi137.stdout:2: finishing write tid 1 to smithi13779221-2 2024-09-21T08:24:39.730 INFO:tasks.rados.rados.0.smithi137.stdout:2: finishing write tid 2 to smithi13779221-2 2024-09-21T08:24:39.730 INFO:tasks.rados.rados.0.smithi137.stdout:1: done (15 left) 2024-09-21T08:24:39.730 INFO:tasks.rados.rados.0.smithi137.stdout:17: write initial oid 17 2024-09-21T08:24:39.730 INFO:tasks.rados.rados.0.smithi137.stdout:17: seq_num 16 ranges {668912=636690,1903006=365906} 2024-09-21T08:24:39.730 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:39 smithi137 ceph-mon[29634]: pgmap v31: 148 pgs: 148 active+clean; 583 KiB data, 195 MiB used, 626 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T08:24:39.730 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:39 smithi137 ceph-mon[29634]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T08:24:39.730 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:39 smithi137 ceph-mon[29634]: osdmap e85: 8 total, 8 up, 7 in 2024-09-21T08:24:39.730 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:39 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1574150494' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:39.730 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:39 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3100918956' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-21T08:24:39.731 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:39 smithi137 ceph-mon[41044]: pgmap v31: 148 pgs: 148 active+clean; 583 KiB data, 195 MiB used, 626 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T08:24:39.731 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:39 smithi137 ceph-mon[41044]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T08:24:39.731 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:39 smithi137 ceph-mon[41044]: osdmap e85: 8 total, 8 up, 7 in 2024-09-21T08:24:39.731 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:39 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1574150494' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:39.731 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:39 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3100918956' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-21T08:24:39.736 INFO:tasks.rados.rados.0.smithi137.stdout:17: writing smithi13779221-17 from 668912 to 1305602 tid 1 2024-09-21T08:24:39.742 INFO:tasks.rados.rados.0.smithi137.stdout:17: writing smithi13779221-17 from 1903006 to 2268912 tid 2 2024-09-21T08:24:39.745 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:39.745 INFO:tasks.rados.rados.0.smithi137.stdout:2: finishing write tid 3 to smithi13779221-2 2024-09-21T08:24:39.745 INFO:tasks.rados.rados.0.smithi137.stdout:2: finishing write tid 4 to smithi13779221-2 2024-09-21T08:24:39.745 INFO:tasks.rados.rados.0.smithi137.stdout:2: oid 2 updating version 0 to 2 2024-09-21T08:24:39.745 INFO:tasks.rados.rados.0.smithi137.stdout:2: oid 2 version 2 is already newer than 1 2024-09-21T08:24:39.745 INFO:tasks.rados.rados.0.smithi137.stdout:2: oid 2 updating version 2 to 3 2024-09-21T08:24:39.745 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 3 (ObjNum 1 snap 0 seq_num 1) dirty exists 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:2: left oid 2 (ObjNum 1 snap 0 seq_num 1) 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:3: finishing write tid 1 to smithi13779221-3 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:3: finishing write tid 2 to smithi13779221-3 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:3: finishing write tid 3 to smithi13779221-3 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:3: finishing write tid 4 to smithi13779221-3 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:3: oid 3 updating version 0 to 2 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:3: oid 3 updating version 2 to 3 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:3: oid 3 version 3 is already newer than 1 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 3 (ObjNum 2 snap 0 seq_num 2) dirty exists 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:3: left oid 3 (ObjNum 2 snap 0 seq_num 2) 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:4: finishing write tid 1 to smithi13779221-4 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:4: finishing write tid 2 to smithi13779221-4 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:4: finishing write tid 3 to smithi13779221-4 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:4: finishing write tid 4 to smithi13779221-4 2024-09-21T08:24:39.746 INFO:tasks.rados.rados.0.smithi137.stdout:4: oid 4 updating version 0 to 2 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:4: oid 4 version 2 is already newer than 1 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:4: oid 4 updating version 2 to 3 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 3 (ObjNum 3 snap 0 seq_num 3) dirty exists 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:4: left oid 4 (ObjNum 3 snap 0 seq_num 3) 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:5: finishing write tid 1 to smithi13779221-5 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:5: finishing write tid 2 to smithi13779221-5 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:5: finishing write tid 3 to smithi13779221-5 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:5: finishing write tid 4 to smithi13779221-5 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:5: oid 5 updating version 0 to 3 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:5: oid 5 version 3 is already newer than 2 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:5: oid 5 version 3 is already newer than 1 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:5: left oid 5 (ObjNum 4 snap 0 seq_num 4) 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:6: finishing write tid 1 to smithi13779221-6 2024-09-21T08:24:39.747 INFO:tasks.rados.rados.0.smithi137.stdout:6: finishing write tid 2 to smithi13779221-6 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:6: finishing write tid 3 to smithi13779221-6 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:6: finishing write tid 4 to smithi13779221-6 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:6: oid 6 updating version 0 to 1 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:6: oid 6 updating version 1 to 2 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:6: oid 6 updating version 2 to 3 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 3 (ObjNum 5 snap 0 seq_num 5) dirty exists 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:6: left oid 6 (ObjNum 5 snap 0 seq_num 5) 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:7: finishing write tid 1 to smithi13779221-7 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:7: finishing write tid 2 to smithi13779221-7 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:7: finishing write tid 3 to smithi13779221-7 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:7: finishing write tid 4 to smithi13779221-7 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:7: oid 7 updating version 0 to 3 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:7: oid 7 version 3 is already newer than 1 2024-09-21T08:24:39.748 INFO:tasks.rados.rados.0.smithi137.stdout:7: oid 7 version 3 is already newer than 2 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 3 (ObjNum 6 snap 0 seq_num 6) dirty exists 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:7: left oid 7 (ObjNum 6 snap 0 seq_num 6) 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:8: finishing write tid 1 to smithi13779221-8 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:8: finishing write tid 2 to smithi13779221-8 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:8: finishing write tid 3 to smithi13779221-8 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:8: finishing write tid 4 to smithi13779221-8 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:8: oid 8 updating version 0 to 4 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:8: oid 8 updating version 4 to 6 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:8: oid 8 version 6 is already newer than 5 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 6 (ObjNum 7 snap 0 seq_num 7) dirty exists 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:8: left oid 8 (ObjNum 7 snap 0 seq_num 7) 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:9: finishing write tid 1 to smithi13779221-9 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:9: finishing write tid 2 to smithi13779221-9 2024-09-21T08:24:39.749 INFO:tasks.rados.rados.0.smithi137.stdout:9: finishing write tid 3 to smithi13779221-9 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:9: finishing write tid 4 to smithi13779221-9 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:9: oid 9 updating version 0 to 5 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:9: oid 9 updating version 5 to 6 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:9: oid 9 version 6 is already newer than 4 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 9 v 6 (ObjNum 8 snap 0 seq_num 8) dirty exists 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:9: left oid 9 (ObjNum 8 snap 0 seq_num 8) 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:10: finishing write tid 1 to smithi13779221-10 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:10: finishing write tid 2 to smithi13779221-10 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:10: finishing write tid 3 to smithi13779221-10 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:10: finishing write tid 4 to smithi13779221-10 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:10: oid 10 updating version 0 to 9 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:10: oid 10 version 9 is already newer than 8 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:10: oid 10 version 9 is already newer than 7 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 9 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-21T08:24:39.750 INFO:tasks.rados.rados.0.smithi137.stdout:10: left oid 10 (ObjNum 9 snap 0 seq_num 9) 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:11: finishing write tid 1 to smithi13779221-11 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:11: finishing write tid 2 to smithi13779221-11 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:11: finishing write tid 3 to smithi13779221-11 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:11: finishing write tid 4 to smithi13779221-11 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:11: oid 11 updating version 0 to 1 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:11: oid 11 updating version 1 to 3 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:11: oid 11 version 3 is already newer than 2 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:11: left oid 11 (ObjNum 10 snap 0 seq_num 10) 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:12: finishing write tid 1 to smithi13779221-12 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:12: finishing write tid 2 to smithi13779221-12 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:12: finishing write tid 3 to smithi13779221-12 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:12: finishing write tid 4 to smithi13779221-12 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:12: oid 12 updating version 0 to 12 2024-09-21T08:24:39.751 INFO:tasks.rados.rados.0.smithi137.stdout:12: oid 12 version 12 is already newer than 10 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:12: oid 12 version 12 is already newer than 11 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 12 (ObjNum 11 snap 0 seq_num 11) dirty exists 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:12: left oid 12 (ObjNum 11 snap 0 seq_num 11) 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:13: finishing write tid 1 to smithi13779221-13 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:13: finishing write tid 2 to smithi13779221-13 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:13: finishing write tid 3 to smithi13779221-13 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:13: finishing write tid 4 to smithi13779221-13 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:13: oid 13 updating version 0 to 6 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:13: oid 13 version 6 is already newer than 4 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:13: oid 13 version 6 is already newer than 5 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 6 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:13: left oid 13 (ObjNum 12 snap 0 seq_num 12) 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:14: finishing write tid 1 to smithi13779221-14 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:14: finishing write tid 2 to smithi13779221-14 2024-09-21T08:24:39.752 INFO:tasks.rados.rados.0.smithi137.stdout:14: finishing write tid 3 to smithi13779221-14 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:14: finishing write tid 4 to smithi13779221-14 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:14: oid 14 updating version 0 to 5 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:14: oid 14 updating version 5 to 6 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:14: oid 14 version 6 is already newer than 4 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:14: left oid 14 (ObjNum 13 snap 0 seq_num 13) 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:15: finishing write tid 1 to smithi13779221-15 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:15: finishing write tid 2 to smithi13779221-15 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:15: finishing write tid 3 to smithi13779221-15 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:15: finishing write tid 4 to smithi13779221-15 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:15: oid 15 updating version 0 to 1 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:15: oid 15 updating version 1 to 3 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:15: oid 15 version 3 is already newer than 2 2024-09-21T08:24:39.753 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 3 (ObjNum 14 snap 0 seq_num 14) dirty exists 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:15: left oid 15 (ObjNum 14 snap 0 seq_num 14) 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:16: finishing write tid 1 to smithi13779221-16 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:16: finishing write tid 2 to smithi13779221-16 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:16: finishing write tid 3 to smithi13779221-16 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:16: finishing write tid 4 to smithi13779221-16 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:16: oid 16 updating version 0 to 6 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:16: oid 16 version 6 is already newer than 4 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:16: oid 16 version 6 is already newer than 5 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 6 (ObjNum 15 snap 0 seq_num 15) dirty exists 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:16: left oid 16 (ObjNum 15 snap 0 seq_num 15) 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:17: finishing write tid 1 to smithi13779221-17 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:2: done (15 left) 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:3: done (14 left) 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:4: done (13 left) 2024-09-21T08:24:39.754 INFO:tasks.rados.rados.0.smithi137.stdout:5: done (12 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:6: done (11 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:7: done (10 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:8: done (9 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:9: done (8 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:10: done (7 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:11: done (6 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:12: done (5 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:13: done (4 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:14: done (3 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:15: done (2 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:16: done (1 left) 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:18: write initial oid 18 2024-09-21T08:24:39.755 INFO:tasks.rados.rados.0.smithi137.stdout:18: seq_num 17 ranges {685719=711939,1845135=440584} 2024-09-21T08:24:39.756 INFO:tasks.rados.rados.0.smithi137.stdout:18: writing smithi13779221-18 from 685719 to 1397658 tid 1 2024-09-21T08:24:39.756 INFO:tasks.rados.rados.0.smithi137.stdout:18: writing smithi13779221-18 from 1845135 to 2285719 tid 2 2024-09-21T08:24:39.756 INFO:tasks.rados.rados.0.smithi137.stdout:19: write initial oid 19 2024-09-21T08:24:39.756 INFO:tasks.rados.rados.0.smithi137.stdout:19: seq_num 18 ranges {702526=787188,1903617=398909} 2024-09-21T08:24:39.765 INFO:tasks.rados.rados.0.smithi137.stdout:19: writing smithi13779221-19 from 702526 to 1489714 tid 1 2024-09-21T08:24:39.770 INFO:tasks.rados.rados.0.smithi137.stdout:19: writing smithi13779221-19 from 1903617 to 2302526 tid 2 2024-09-21T08:24:39.770 INFO:tasks.rados.rados.0.smithi137.stdout:20: write initial oid 20 2024-09-21T08:24:39.770 INFO:tasks.rados.rados.0.smithi137.stdout:20: seq_num 19 ranges {719333=462437,1962099=357234} 2024-09-21T08:24:39.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:39.776 INFO:tasks.rados.rados.0.smithi137.stdout:20: writing smithi13779221-20 from 719333 to 1181770 tid 1 2024-09-21T08:24:39.783 INFO:tasks.rados.rados.0.smithi137.stdout:20: writing smithi13779221-20 from 1962099 to 2319333 tid 2 2024-09-21T08:24:39.783 INFO:tasks.rados.rados.0.smithi137.stdout:21: write initial oid 21 2024-09-21T08:24:39.783 INFO:tasks.rados.rados.0.smithi137.stdout:21: seq_num 20 ranges {736140=537686,2020581=315559} 2024-09-21T08:24:39.789 INFO:tasks.rados.rados.0.smithi137.stdout:21: writing smithi13779221-21 from 736140 to 1273826 tid 1 2024-09-21T08:24:39.795 INFO:tasks.rados.rados.0.smithi137.stdout:21: writing smithi13779221-21 from 2020581 to 2336140 tid 2 2024-09-21T08:24:39.795 INFO:tasks.rados.rados.0.smithi137.stdout:22: write initial oid 22 2024-09-21T08:24:39.795 INFO:tasks.rados.rados.0.smithi137.stdout:22: seq_num 21 ranges {752947=612935,1962710=390237} 2024-09-21T08:24:39.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:39.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:39.799 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:39.802 INFO:tasks.rados.rados.0.smithi137.stdout:22: writing smithi13779221-22 from 752947 to 1365882 tid 1 2024-09-21T08:24:39.808 INFO:tasks.rados.rados.0.smithi137.stdout:22: writing smithi13779221-22 from 1962710 to 2352947 tid 2 2024-09-21T08:24:39.808 INFO:tasks.rados.rados.0.smithi137.stdout:23: write initial oid 23 2024-09-21T08:24:39.808 INFO:tasks.rados.rados.0.smithi137.stdout:23: seq_num 22 ranges {769754=688184,2021192=348562} 2024-09-21T08:24:39.816 INFO:tasks.rados.rados.0.smithi137.stdout:23: writing smithi13779221-23 from 769754 to 1457938 tid 1 2024-09-21T08:24:39.821 INFO:tasks.rados.rados.0.smithi137.stdout:23: writing smithi13779221-23 from 2021192 to 2369754 tid 2 2024-09-21T08:24:39.821 INFO:tasks.rados.rados.0.smithi137.stdout:24: write initial oid 24 2024-09-21T08:24:39.821 INFO:tasks.rados.rados.0.smithi137.stdout:24: seq_num 23 ranges {786561=479786,1796027=467664} 2024-09-21T08:24:39.828 INFO:tasks.rados.rados.0.smithi137.stdout:24: writing smithi13779221-24 from 786561 to 1266347 tid 1 2024-09-21T08:24:39.833 INFO:tasks.rados.rados.0.smithi137.stdout:24: writing smithi13779221-24 from 1796027 to 2263691 tid 2 2024-09-21T08:24:39.833 INFO:tasks.rados.rados.0.smithi137.stdout:17: finishing write tid 2 to smithi13779221-17 2024-09-21T08:24:39.833 INFO:tasks.rados.rados.0.smithi137.stdout:25: write initial oid 25 2024-09-21T08:24:39.834 INFO:tasks.rados.rados.0.smithi137.stdout:25: seq_num 24 ranges {403368=555035,1454509=727675} 2024-09-21T08:24:39.838 INFO:tasks.rados.rados.0.smithi137.stdout:25: writing smithi13779221-25 from 403368 to 958403 tid 1 2024-09-21T08:24:39.845 INFO:tasks.rados.rados.0.smithi137.stdout:25: writing smithi13779221-25 from 1454509 to 2182184 tid 2 2024-09-21T08:24:39.845 INFO:tasks.rados.rados.0.smithi137.stdout:26: write initial oid 26 2024-09-21T08:24:39.846 INFO:tasks.rados.rados.0.smithi137.stdout:26: seq_num 25 ranges {420175=630284,1796638=471333} 2024-09-21T08:24:39.851 INFO:tasks.rados.rados.0.smithi137.stdout:26: writing smithi13779221-26 from 420175 to 1050459 tid 1 2024-09-21T08:24:39.858 INFO:tasks.rados.rados.0.smithi137.stdout:26: writing smithi13779221-26 from 1796638 to 2267971 tid 2 2024-09-21T08:24:39.858 INFO:tasks.rados.rados.0.smithi137.stdout:27: write initial oid 27 2024-09-21T08:24:39.858 INFO:tasks.rados.rados.0.smithi137.stdout:27: seq_num 26 ranges {436982=705533,1855120=581862} 2024-09-21T08:24:39.864 INFO:tasks.rados.rados.0.smithi137.stdout:27: writing smithi13779221-27 from 436982 to 1142515 tid 1 2024-09-21T08:24:39.872 INFO:tasks.rados.rados.0.smithi137.stdout:27: writing smithi13779221-27 from 1855120 to 2436982 tid 2 2024-09-21T08:24:39.872 INFO:tasks.rados.rados.0.smithi137.stdout:28: write initial oid 28 2024-09-21T08:24:39.872 INFO:tasks.rados.rados.0.smithi137.stdout:28: seq_num 27 ranges {453789=780782,1913602=475002} 2024-09-21T08:24:39.879 INFO:tasks.rados.rados.0.smithi137.stdout:28: writing smithi13779221-28 from 453789 to 1234571 tid 1 2024-09-21T08:24:39.885 INFO:tasks.rados.rados.0.smithi137.stdout:28: writing smithi13779221-28 from 1913602 to 2388604 tid 2 2024-09-21T08:24:39.885 INFO:tasks.rados.rados.0.smithi137.stdout:29: write initial oid 29 2024-09-21T08:24:39.885 INFO:tasks.rados.rados.0.smithi137.stdout:29: seq_num 28 ranges {470596=456031,1572084=618660} 2024-09-21T08:24:39.890 INFO:tasks.rados.rados.0.smithi137.stdout:29: writing smithi13779221-29 from 470596 to 926627 tid 1 2024-09-21T08:24:39.897 INFO:tasks.rados.rados.0.smithi137.stdout:29: writing smithi13779221-29 from 1572084 to 2190744 tid 2 2024-09-21T08:24:39.897 INFO:tasks.rados.rados.0.smithi137.stdout:30: write initial oid 30 2024-09-21T08:24:39.897 INFO:tasks.rados.rados.0.smithi137.stdout:30: seq_num 29 ranges {487403=531280,1514213=478671} 2024-09-21T08:24:39.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:39.903 INFO:tasks.rados.rados.0.smithi137.stdout:30: writing smithi13779221-30 from 487403 to 1018683 tid 1 2024-09-21T08:24:39.908 INFO:tasks.rados.rados.0.smithi137.stdout:30: writing smithi13779221-30 from 1514213 to 1992884 tid 2 2024-09-21T08:24:39.908 INFO:tasks.rados.rados.0.smithi137.stdout:31: write initial oid 31 2024-09-21T08:24:39.908 INFO:tasks.rados.rados.0.smithi137.stdout:31: seq_num 30 ranges {504210=606529,1572695=622329} 2024-09-21T08:24:39.914 INFO:tasks.rados.rados.0.smithi137.stdout:31: writing smithi13779221-31 from 504210 to 1110739 tid 1 2024-09-21T08:24:39.920 INFO:tasks.rados.rados.0.smithi137.stdout:31: writing smithi13779221-31 from 1572695 to 2195024 tid 2 2024-09-21T08:24:39.920 INFO:tasks.rados.rados.0.smithi137.stdout:32: write initial oid 32 2024-09-21T08:24:39.920 INFO:tasks.rados.rados.0.smithi137.stdout:32: seq_num 31 ranges {521017=798131,1747530=482340} 2024-09-21T08:24:39.921 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:39.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:39.922 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:39.927 INFO:tasks.rados.rados.0.smithi137.stdout:32: writing smithi13779221-32 from 521017 to 1319148 tid 1 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout:32: writing smithi13779221-32 from 1747530 to 2229870 tid 2 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout:17: finishing write tid 3 to smithi13779221-17 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout:17: finishing write tid 4 to smithi13779221-17 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout:17: oid 17 updating version 0 to 2 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout:17: oid 17 version 2 is already newer than 1 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout:17: oid 17 updating version 2 to 3 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 3 (ObjNum 16 snap 0 seq_num 16) dirty exists 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout:17: left oid 17 (ObjNum 16 snap 0 seq_num 16) 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout:18: finishing write tid 1 to smithi13779221-18 2024-09-21T08:24:39.933 INFO:tasks.rados.rados.0.smithi137.stdout:18: finishing write tid 2 to smithi13779221-18 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:18: finishing write tid 3 to smithi13779221-18 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:18: finishing write tid 4 to smithi13779221-18 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:18: oid 18 updating version 0 to 8 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:18: oid 18 version 8 is already newer than 7 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:18: oid 18 updating version 8 to 9 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 9 (ObjNum 17 snap 0 seq_num 17) dirty exists 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:18: left oid 18 (ObjNum 17 snap 0 seq_num 17) 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:19: finishing write tid 1 to smithi13779221-19 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:19: finishing write tid 2 to smithi13779221-19 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:17: done (15 left) 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:18: done (14 left) 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:33: write initial oid 33 2024-09-21T08:24:39.934 INFO:tasks.rados.rados.0.smithi137.stdout:33: seq_num 32 ranges {537824=473380,1806012=625998} 2024-09-21T08:24:39.938 INFO:tasks.rados.rados.0.smithi137.stdout:33: writing smithi13779221-33 from 537824 to 1011204 tid 1 2024-09-21T08:24:39.946 INFO:tasks.rados.rados.0.smithi137.stdout:33: writing smithi13779221-33 from 1806012 to 2432010 tid 2 2024-09-21T08:24:39.946 INFO:tasks.rados.rados.0.smithi137.stdout:34: write initial oid 34 2024-09-21T08:24:39.946 INFO:tasks.rados.rados.0.smithi137.stdout:34: seq_num 33 ranges {554631=548629,1748141=486009} 2024-09-21T08:24:39.952 INFO:tasks.rados.rados.0.smithi137.stdout:34: writing smithi13779221-34 from 554631 to 1103260 tid 1 2024-09-21T08:24:39.958 INFO:tasks.rados.rados.0.smithi137.stdout:34: writing smithi13779221-34 from 1748141 to 2234150 tid 2 2024-09-21T08:24:39.958 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:39.958 INFO:tasks.rados.rados.0.smithi137.stdout:19: finishing write tid 3 to smithi13779221-19 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:19: finishing write tid 4 to smithi13779221-19 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:19: oid 19 updating version 0 to 1 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:19: oid 19 updating version 1 to 2 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:19: oid 19 updating version 2 to 3 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 3 (ObjNum 18 snap 0 seq_num 18) dirty exists 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:19: left oid 19 (ObjNum 18 snap 0 seq_num 18) 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:20: finishing write tid 1 to smithi13779221-20 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:20: finishing write tid 2 to smithi13779221-20 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:19: done (15 left) 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:35: write initial oid 35 2024-09-21T08:24:39.959 INFO:tasks.rados.rados.0.smithi137.stdout:35: seq_num 34 ranges {571438=623878,1806623=629667} 2024-09-21T08:24:39.965 INFO:tasks.rados.rados.0.smithi137.stdout:35: writing smithi13779221-35 from 571438 to 1195316 tid 1 2024-09-21T08:24:39.972 INFO:tasks.rados.rados.0.smithi137.stdout:35: writing smithi13779221-35 from 1806623 to 2436290 tid 2 2024-09-21T08:24:39.972 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:39.972 INFO:tasks.rados.rados.0.smithi137.stdout:20: finishing write tid 3 to smithi13779221-20 2024-09-21T08:24:39.972 INFO:tasks.rados.rados.0.smithi137.stdout:20: finishing write tid 4 to smithi13779221-20 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:20: oid 20 updating version 0 to 3 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:20: oid 20 version 3 is already newer than 1 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:20: oid 20 version 3 is already newer than 2 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 3 (ObjNum 19 snap 0 seq_num 19) dirty exists 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:20: left oid 20 (ObjNum 19 snap 0 seq_num 19) 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:21: finishing write tid 1 to smithi13779221-21 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:21: finishing write tid 2 to smithi13779221-21 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:21: finishing write tid 3 to smithi13779221-21 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:21: finishing write tid 4 to smithi13779221-21 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:21: oid 21 updating version 0 to 5 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:21: oid 21 updating version 5 to 6 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:21: oid 21 version 6 is already newer than 4 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 6 (ObjNum 20 snap 0 seq_num 20) dirty exists 2024-09-21T08:24:39.973 INFO:tasks.rados.rados.0.smithi137.stdout:21: left oid 21 (ObjNum 20 snap 0 seq_num 20) 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:22: finishing write tid 1 to smithi13779221-22 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:22: finishing write tid 2 to smithi13779221-22 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:22: finishing write tid 3 to smithi13779221-22 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:22: finishing write tid 4 to smithi13779221-22 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:22: oid 22 updating version 0 to 8 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:22: oid 22 updating version 8 to 9 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:22: oid 22 version 9 is already newer than 7 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 9 (ObjNum 21 snap 0 seq_num 21) dirty exists 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:22: left oid 22 (ObjNum 21 snap 0 seq_num 21) 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:23: finishing write tid 1 to smithi13779221-23 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:23: finishing write tid 2 to smithi13779221-23 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:23: finishing write tid 3 to smithi13779221-23 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:23: finishing write tid 4 to smithi13779221-23 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:23: oid 23 updating version 0 to 5 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:23: oid 23 updating version 5 to 6 2024-09-21T08:24:39.974 INFO:tasks.rados.rados.0.smithi137.stdout:23: oid 23 version 6 is already newer than 4 2024-09-21T08:24:39.975 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-21T08:24:39.975 INFO:tasks.rados.rados.0.smithi137.stdout:23: left oid 23 (ObjNum 22 snap 0 seq_num 22) 2024-09-21T08:24:39.975 INFO:tasks.rados.rados.0.smithi137.stdout:24: finishing write tid 1 to smithi13779221-24 2024-09-21T08:24:39.975 INFO:tasks.rados.rados.0.smithi137.stdout:20: done (15 left) 2024-09-21T08:24:39.975 INFO:tasks.rados.rados.0.smithi137.stdout:21: done (14 left) 2024-09-21T08:24:39.975 INFO:tasks.rados.rados.0.smithi137.stdout:22: done (13 left) 2024-09-21T08:24:39.975 INFO:tasks.rados.rados.0.smithi137.stdout:23: done (12 left) 2024-09-21T08:24:39.975 INFO:tasks.rados.rados.0.smithi137.stdout:36: write initial oid 36 2024-09-21T08:24:39.975 INFO:tasks.rados.rados.0.smithi137.stdout:36: seq_num 35 ranges {588245=699127,1865105=489678} 2024-09-21T08:24:39.979 INFO:tasks.rados.rados.0.smithi137.stdout:36: writing smithi13779221-36 from 588245 to 1287372 tid 1 2024-09-21T08:24:39.985 INFO:tasks.rados.rados.0.smithi137.stdout:36: writing smithi13779221-36 from 1865105 to 2354783 tid 2 2024-09-21T08:24:39.985 INFO:tasks.rados.rados.0.smithi137.stdout:37: write initial oid 37 2024-09-21T08:24:39.985 INFO:tasks.rados.rados.0.smithi137.stdout:37: seq_num 36 ranges {605052=774376,1923587=633336} 2024-09-21T08:24:39.993 INFO:tasks.rados.rados.0.smithi137.stdout:37: writing smithi13779221-37 from 605052 to 1379428 tid 1 2024-09-21T08:24:39.993 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:24:39.993 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:24:39.999 INFO:tasks.rados.rados.0.smithi137.stdout:37: writing smithi13779221-37 from 1923587 to 2556923 tid 2 2024-09-21T08:24:40.000 INFO:tasks.rados.rados.0.smithi137.stdout:38: write initial oid 38 2024-09-21T08:24:40.000 INFO:tasks.rados.rados.0.smithi137.stdout:38: seq_num 37 ranges {621859=449625,1865716=756143} 2024-09-21T08:24:40.005 INFO:tasks.rados.rados.0.smithi137.stdout:38: writing smithi13779221-38 from 621859 to 1071484 tid 1 2024-09-21T08:24:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:39 smithi165 ceph-mon[32893]: pgmap v31: 148 pgs: 148 active+clean; 583 KiB data, 195 MiB used, 626 GiB / 626 GiB avail; 1.5 KiB/s rd, 1 op/s 2024-09-21T08:24:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:39 smithi165 ceph-mon[32893]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-09-21T08:24:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:39 smithi165 ceph-mon[32893]: osdmap e85: 8 total, 8 up, 7 in 2024-09-21T08:24:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:39 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1574150494' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:39 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3100918956' entity='client.0' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]: dispatch 2024-09-21T08:24:40.014 INFO:tasks.rados.rados.0.smithi137.stdout:38: writing smithi13779221-38 from 1865716 to 2621859 tid 2 2024-09-21T08:24:40.014 INFO:tasks.rados.rados.0.smithi137.stdout:39: write initial oid 39 2024-09-21T08:24:40.014 INFO:tasks.rados.rados.0.smithi137.stdout:39: seq_num 38 ranges {638666=524874,1924198=637005} 2024-09-21T08:24:40.020 INFO:tasks.rados.rados.0.smithi137.stdout:39: writing smithi13779221-39 from 638666 to 1163540 tid 1 2024-09-21T08:24:40.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:40.029 INFO:tasks.rados.rados.0.smithi137.stdout:39: writing smithi13779221-39 from 1924198 to 2561203 tid 2 2024-09-21T08:24:40.029 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:24: finishing write tid 2 to smithi13779221-24 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:24: finishing write tid 3 to smithi13779221-24 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:24: finishing write tid 4 to smithi13779221-24 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:24: oid 24 updating version 0 to 5 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:24: oid 24 updating version 5 to 6 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:24: oid 24 version 6 is already newer than 4 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 6 (ObjNum 23 snap 0 seq_num 23) dirty exists 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:24: left oid 24 (ObjNum 23 snap 0 seq_num 23) 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:25: finishing write tid 1 to smithi13779221-25 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:25: finishing write tid 2 to smithi13779221-25 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:24: done (15 left) 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:40: write initial oid 40 2024-09-21T08:24:40.030 INFO:tasks.rados.rados.0.smithi137.stdout:40: seq_num 39 ranges {655473=716476,2099033=556440} 2024-09-21T08:24:40.036 INFO:tasks.rados.rados.0.smithi137.stdout:40: writing smithi13779221-40 from 655473 to 1371949 tid 1 2024-09-21T08:24:40.044 INFO:tasks.rados.rados.0.smithi137.stdout:40: writing smithi13779221-40 from 2099033 to 2655473 tid 2 2024-09-21T08:24:40.044 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:40.044 INFO:tasks.rados.rados.0.smithi137.stdout:25: finishing write tid 3 to smithi13779221-25 2024-09-21T08:24:40.044 INFO:tasks.rados.rados.0.smithi137.stdout:25: finishing write tid 4 to smithi13779221-25 2024-09-21T08:24:40.044 INFO:tasks.rados.rados.0.smithi137.stdout:25: oid 25 updating version 0 to 1 2024-09-21T08:24:40.044 INFO:tasks.rados.rados.0.smithi137.stdout:25: oid 25 updating version 1 to 3 2024-09-21T08:24:40.044 INFO:tasks.rados.rados.0.smithi137.stdout:25: oid 25 version 3 is already newer than 2 2024-09-21T08:24:40.044 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 3 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-21T08:24:40.044 INFO:tasks.rados.rados.0.smithi137.stdout:25: left oid 25 (ObjNum 24 snap 0 seq_num 24) 2024-09-21T08:24:40.044 INFO:tasks.rados.rados.0.smithi137.stdout:26: finishing write tid 1 to smithi13779221-26 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:26: finishing write tid 2 to smithi13779221-26 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:26: finishing write tid 3 to smithi13779221-26 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:26: finishing write tid 4 to smithi13779221-26 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:26: oid 26 updating version 0 to 2 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:26: oid 26 updating version 2 to 3 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:26: oid 26 version 3 is already newer than 1 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 3 (ObjNum 25 snap 0 seq_num 25) dirty exists 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:26: left oid 26 (ObjNum 25 snap 0 seq_num 25) 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:27: finishing write tid 1 to smithi13779221-27 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:27: finishing write tid 2 to smithi13779221-27 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:25: done (15 left) 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:26: done (14 left) 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:41: write initial oid 41 2024-09-21T08:24:40.045 INFO:tasks.rados.rados.0.smithi137.stdout:41: seq_num 40 ranges {672280=791725,2157515=514765} 2024-09-21T08:24:40.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:40.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:40.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:40.051 INFO:tasks.rados.rados.0.smithi137.stdout:41: writing smithi13779221-41 from 672280 to 1464005 tid 1 2024-09-21T08:24:40.058 INFO:tasks.rados.rados.0.smithi137.stdout:41: writing smithi13779221-41 from 2157515 to 2672280 tid 2 2024-09-21T08:24:40.058 INFO:tasks.rados.rados.0.smithi137.stdout:42: write initial oid 42 2024-09-21T08:24:40.059 INFO:tasks.rados.rados.0.smithi137.stdout:42: seq_num 41 ranges {689087=466974,1699644=784332} 2024-09-21T08:24:40.064 INFO:tasks.rados.rados.0.smithi137.stdout:42: writing smithi13779221-42 from 689087 to 1156061 tid 1 2024-09-21T08:24:40.072 INFO:tasks.rados.rados.0.smithi137.stdout:42: writing smithi13779221-42 from 1699644 to 2483976 tid 2 2024-09-21T08:24:40.072 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:40.072 INFO:tasks.rados.rados.0.smithi137.stdout:27: finishing write tid 3 to smithi13779221-27 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:27: finishing write tid 4 to smithi13779221-27 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:27: oid 27 updating version 0 to 7 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:27: oid 27 updating version 7 to 9 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:27: oid 27 version 9 is already newer than 8 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 9 (ObjNum 26 snap 0 seq_num 26) dirty exists 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:27: left oid 27 (ObjNum 26 snap 0 seq_num 26) 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:28: finishing write tid 1 to smithi13779221-28 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:28: finishing write tid 2 to smithi13779221-28 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:28: finishing write tid 3 to smithi13779221-28 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:28: finishing write tid 4 to smithi13779221-28 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:28: oid 28 updating version 0 to 4 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:28: oid 28 updating version 4 to 6 2024-09-21T08:24:40.073 INFO:tasks.rados.rados.0.smithi137.stdout:28: oid 28 version 6 is already newer than 5 2024-09-21T08:24:40.074 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T08:24:40.074 INFO:tasks.rados.rados.0.smithi137.stdout:28: left oid 28 (ObjNum 27 snap 0 seq_num 27) 2024-09-21T08:24:40.074 INFO:tasks.rados.rados.0.smithi137.stdout:29: finishing write tid 1 to smithi13779221-29 2024-09-21T08:24:40.074 INFO:tasks.rados.rados.0.smithi137.stdout:29: finishing write tid 2 to smithi13779221-29 2024-09-21T08:24:40.074 INFO:tasks.rados.rados.0.smithi137.stdout:27: done (15 left) 2024-09-21T08:24:40.074 INFO:tasks.rados.rados.0.smithi137.stdout:28: done (14 left) 2024-09-21T08:24:40.074 INFO:tasks.rados.rados.0.smithi137.stdout:43: write initial oid 43 2024-09-21T08:24:40.074 INFO:tasks.rados.rados.0.smithi137.stdout:43: seq_num 42 ranges {705894=542223,1758126=644343} 2024-09-21T08:24:40.079 INFO:tasks.rados.rados.0.smithi137.stdout:43: writing smithi13779221-43 from 705894 to 1248117 tid 1 2024-09-21T08:24:40.085 INFO:tasks.rados.rados.0.smithi137.stdout:43: writing smithi13779221-43 from 1758126 to 2402469 tid 2 2024-09-21T08:24:40.085 INFO:tasks.rados.rados.0.smithi137.stdout:44: write initial oid 44 2024-09-21T08:24:40.085 INFO:tasks.rados.rados.0.smithi137.stdout:44: seq_num 43 ranges {722701=617472,1816608=788001} 2024-09-21T08:24:40.092 INFO:tasks.rados.rados.0.smithi137.stdout:44: writing smithi13779221-44 from 722701 to 1340173 tid 1 2024-09-21T08:24:40.100 INFO:tasks.rados.rados.0.smithi137.stdout:44: writing smithi13779221-44 from 1816608 to 2604609 tid 2 2024-09-21T08:24:40.100 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:40.100 INFO:tasks.rados.rados.0.smithi137.stdout:29: finishing write tid 3 to smithi13779221-29 2024-09-21T08:24:40.100 INFO:tasks.rados.rados.0.smithi137.stdout:29: finishing write tid 4 to smithi13779221-29 2024-09-21T08:24:40.100 INFO:tasks.rados.rados.0.smithi137.stdout:29: oid 29 updating version 0 to 8 2024-09-21T08:24:40.100 INFO:tasks.rados.rados.0.smithi137.stdout:29: oid 29 version 8 is already newer than 7 2024-09-21T08:24:40.100 INFO:tasks.rados.rados.0.smithi137.stdout:29: oid 29 updating version 8 to 9 2024-09-21T08:24:40.100 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 9 (ObjNum 28 snap 0 seq_num 28) dirty exists 2024-09-21T08:24:40.100 INFO:tasks.rados.rados.0.smithi137.stdout:29: left oid 29 (ObjNum 28 snap 0 seq_num 28) 2024-09-21T08:24:40.100 INFO:tasks.rados.rados.0.smithi137.stdout:30: finishing write tid 1 to smithi13779221-30 2024-09-21T08:24:40.101 INFO:tasks.rados.rados.0.smithi137.stdout:30: finishing write tid 2 to smithi13779221-30 2024-09-21T08:24:40.101 INFO:tasks.rados.rados.0.smithi137.stdout:29: done (15 left) 2024-09-21T08:24:40.101 INFO:tasks.rados.rados.0.smithi137.stdout:45: write initial oid 45 2024-09-21T08:24:40.101 INFO:tasks.rados.rados.0.smithi137.stdout:45: seq_num 44 ranges {739508=692721,1875090=648012} 2024-09-21T08:24:40.108 INFO:tasks.rados.rados.0.smithi137.stdout:45: writing smithi13779221-45 from 739508 to 1432229 tid 1 2024-09-21T08:24:40.115 INFO:tasks.rados.rados.0.smithi137.stdout:45: writing smithi13779221-45 from 1875090 to 2523102 tid 2 2024-09-21T08:24:40.115 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:40.115 INFO:tasks.rados.rados.0.smithi137.stdout:30: finishing write tid 3 to smithi13779221-30 2024-09-21T08:24:40.115 INFO:tasks.rados.rados.0.smithi137.stdout:30: finishing write tid 4 to smithi13779221-30 2024-09-21T08:24:40.115 INFO:tasks.rados.rados.0.smithi137.stdout:30: oid 30 updating version 0 to 11 2024-09-21T08:24:40.115 INFO:tasks.rados.rados.0.smithi137.stdout:30: oid 30 version 11 is already newer than 10 2024-09-21T08:24:40.115 INFO:tasks.rados.rados.0.smithi137.stdout:30: oid 30 updating version 11 to 12 2024-09-21T08:24:40.115 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 12 (ObjNum 29 snap 0 seq_num 29) dirty exists 2024-09-21T08:24:40.115 INFO:tasks.rados.rados.0.smithi137.stdout:30: left oid 30 (ObjNum 29 snap 0 seq_num 29) 2024-09-21T08:24:40.115 INFO:tasks.rados.rados.0.smithi137.stdout:31: finishing write tid 1 to smithi13779221-31 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:31: finishing write tid 2 to smithi13779221-31 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:31: finishing write tid 3 to smithi13779221-31 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:31: finishing write tid 4 to smithi13779221-31 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:31: oid 31 updating version 0 to 12 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:31: oid 31 version 12 is already newer than 10 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:31: oid 31 version 12 is already newer than 11 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 12 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:31: left oid 31 (ObjNum 30 snap 0 seq_num 30) 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:32: finishing write tid 1 to smithi13779221-32 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:32: finishing write tid 2 to smithi13779221-32 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:32: finishing write tid 3 to smithi13779221-32 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:32: finishing write tid 4 to smithi13779221-32 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:32: oid 32 updating version 0 to 5 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:32: oid 32 version 5 is already newer than 4 2024-09-21T08:24:40.116 INFO:tasks.rados.rados.0.smithi137.stdout:32: oid 32 updating version 5 to 6 2024-09-21T08:24:40.117 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 6 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-21T08:24:40.117 INFO:tasks.rados.rados.0.smithi137.stdout:32: left oid 32 (ObjNum 31 snap 0 seq_num 31) 2024-09-21T08:24:40.117 INFO:tasks.rados.rados.0.smithi137.stdout:33: finishing write tid 1 to smithi13779221-33 2024-09-21T08:24:40.117 INFO:tasks.rados.rados.0.smithi137.stdout:33: finishing write tid 2 to smithi13779221-33 2024-09-21T08:24:40.117 INFO:tasks.rados.rados.0.smithi137.stdout:30: done (15 left) 2024-09-21T08:24:40.117 INFO:tasks.rados.rados.0.smithi137.stdout:31: done (14 left) 2024-09-21T08:24:40.117 INFO:tasks.rados.rados.0.smithi137.stdout:32: done (13 left) 2024-09-21T08:24:40.117 INFO:tasks.rados.rados.0.smithi137.stdout:46: write initial oid 46 2024-09-21T08:24:40.117 INFO:tasks.rados.rados.0.smithi137.stdout:46: seq_num 45 ranges {756315=767970,1933572=791670} 2024-09-21T08:24:40.122 INFO:tasks.rados.rados.0.smithi137.stdout:46: writing smithi13779221-46 from 756315 to 1524285 tid 1 2024-09-21T08:24:40.129 INFO:tasks.rados.rados.0.smithi137.stdout:46: writing smithi13779221-46 from 1933572 to 2725242 tid 2 2024-09-21T08:24:40.129 INFO:tasks.rados.rados.0.smithi137.stdout:47: write initial oid 47 2024-09-21T08:24:40.129 INFO:tasks.rados.rados.0.smithi137.stdout:47: seq_num 46 ranges {773122=559572,1992054=651681} 2024-09-21T08:24:40.136 INFO:tasks.rados.rados.0.smithi137.stdout:47: writing smithi13779221-47 from 773122 to 1332694 tid 1 2024-09-21T08:24:40.143 INFO:tasks.rados.rados.0.smithi137.stdout:47: writing smithi13779221-47 from 1992054 to 2643735 tid 2 2024-09-21T08:24:40.143 INFO:tasks.rados.rados.0.smithi137.stdout:48: write initial oid 48 2024-09-21T08:24:40.143 INFO:tasks.rados.rados.0.smithi137.stdout:48: seq_num 47 ranges {789929=634821,2050536=739393} 2024-09-21T08:24:40.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:40.151 INFO:tasks.rados.rados.0.smithi137.stdout:48: writing smithi13779221-48 from 789929 to 1424750 tid 1 2024-09-21T08:24:40.159 INFO:tasks.rados.rados.0.smithi137.stdout:48: writing smithi13779221-48 from 2050536 to 2789929 tid 2 2024-09-21T08:24:40.159 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:40.159 INFO:tasks.rados.rados.0.smithi137.stdout:33: finishing write tid 3 to smithi13779221-33 2024-09-21T08:24:40.159 INFO:tasks.rados.rados.0.smithi137.stdout:33: finishing write tid 4 to smithi13779221-33 2024-09-21T08:24:40.160 INFO:tasks.rados.rados.0.smithi137.stdout:33: oid 33 updating version 0 to 3 2024-09-21T08:24:40.160 INFO:tasks.rados.rados.0.smithi137.stdout:33: oid 33 version 3 is already newer than 1 2024-09-21T08:24:40.160 INFO:tasks.rados.rados.0.smithi137.stdout:33: oid 33 version 3 is already newer than 2 2024-09-21T08:24:40.160 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 3 (ObjNum 32 snap 0 seq_num 32) dirty exists 2024-09-21T08:24:40.160 INFO:tasks.rados.rados.0.smithi137.stdout:33: left oid 33 (ObjNum 32 snap 0 seq_num 32) 2024-09-21T08:24:40.160 INFO:tasks.rados.rados.0.smithi137.stdout:34: finishing write tid 1 to smithi13779221-34 2024-09-21T08:24:40.160 INFO:tasks.rados.rados.0.smithi137.stdout:34: finishing write tid 2 to smithi13779221-34 2024-09-21T08:24:40.160 INFO:tasks.rados.rados.0.smithi137.stdout:33: done (15 left) 2024-09-21T08:24:40.160 INFO:tasks.rados.rados.0.smithi137.stdout:49: write initial oid 49 2024-09-21T08:24:40.160 INFO:tasks.rados.rados.0.smithi137.stdout:49: seq_num 48 ranges {406736=710070,1709018=655350} 2024-09-21T08:24:40.165 INFO:tasks.rados.rados.0.smithi137.stdout:49: writing smithi13779221-49 from 406736 to 1116806 tid 1 2024-09-21T08:24:40.173 INFO:tasks.rados.rados.0.smithi137.stdout:49: writing smithi13779221-49 from 1709018 to 2364368 tid 2 2024-09-21T08:24:40.173 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:40.173 INFO:tasks.rados.rados.0.smithi137.stdout:34: finishing write tid 3 to smithi13779221-34 2024-09-21T08:24:40.173 INFO:tasks.rados.rados.0.smithi137.stdout:34: finishing write tid 4 to smithi13779221-34 2024-09-21T08:24:40.173 INFO:tasks.rados.rados.0.smithi137.stdout:34: oid 34 updating version 0 to 4 2024-09-21T08:24:40.173 INFO:tasks.rados.rados.0.smithi137.stdout:34: oid 34 updating version 4 to 6 2024-09-21T08:24:40.173 INFO:tasks.rados.rados.0.smithi137.stdout:34: oid 34 version 6 is already newer than 5 2024-09-21T08:24:40.173 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 6 (ObjNum 33 snap 0 seq_num 33) dirty exists 2024-09-21T08:24:40.173 INFO:tasks.rados.rados.0.smithi137.stdout:34: left oid 34 (ObjNum 33 snap 0 seq_num 33) 2024-09-21T08:24:40.173 INFO:tasks.rados.rados.0.smithi137.stdout:35: finishing write tid 1 to smithi13779221-35 2024-09-21T08:24:40.174 INFO:tasks.rados.rados.0.smithi137.stdout:35: finishing write tid 2 to smithi13779221-35 2024-09-21T08:24:40.174 INFO:tasks.rados.rados.0.smithi137.stdout:34: done (15 left) 2024-09-21T08:24:40.174 INFO:tasks.rados.rados.0.smithi137.stdout:50: write initial oid 50 2024-09-21T08:24:40.174 INFO:tasks.rados.rados.0.smithi137.stdout:50: seq_num 49 ranges {423543=785319,1767500=799008} 2024-09-21T08:24:40.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:40.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:40.175 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:40.179 INFO:tasks.rados.rados.0.smithi137.stdout:50: writing smithi13779221-50 from 423543 to 1208862 tid 1 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:50: writing smithi13779221-50 from 1767500 to 2566508 tid 2 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout: waiting on 16 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:35: finishing write tid 3 to smithi13779221-35 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:35: finishing write tid 4 to smithi13779221-35 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:35: oid 35 updating version 0 to 12 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:35: oid 35 version 12 is already newer than 10 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:35: oid 35 version 12 is already newer than 11 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 12 (ObjNum 34 snap 0 seq_num 34) dirty exists 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:35: left oid 35 (ObjNum 34 snap 0 seq_num 34) 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:36: finishing write tid 1 to smithi13779221-36 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:36: finishing write tid 2 to smithi13779221-36 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:35: done (15 left) 2024-09-21T08:24:40.188 INFO:tasks.rados.rados.0.smithi137.stdout:51: copy_from oid 2 from oid 11 current snap is 0 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:36: finishing write tid 3 to smithi13779221-36 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:36: finishing write tid 4 to smithi13779221-36 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:36: oid 36 updating version 0 to 15 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:36: oid 36 version 15 is already newer than 13 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:36: oid 36 version 15 is already newer than 14 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 15 (ObjNum 35 snap 0 seq_num 35) dirty exists 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:36: left oid 36 (ObjNum 35 snap 0 seq_num 35) 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:37: finishing write tid 1 to smithi13779221-37 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:37: finishing write tid 2 to smithi13779221-37 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:37: finishing write tid 3 to smithi13779221-37 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:37: finishing write tid 4 to smithi13779221-37 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:37: oid 37 updating version 0 to 17 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:37: oid 37 version 17 is already newer than 16 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:37: oid 37 updating version 17 to 18 2024-09-21T08:24:40.189 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 18 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-21T08:24:40.190 INFO:tasks.rados.rados.0.smithi137.stdout:37: left oid 37 (ObjNum 36 snap 0 seq_num 36) 2024-09-21T08:24:40.190 INFO:tasks.rados.rados.0.smithi137.stdout:38: finishing write tid 1 to smithi13779221-38 2024-09-21T08:24:40.190 INFO:tasks.rados.rados.0.smithi137.stdout:38: finishing write tid 2 to smithi13779221-38 2024-09-21T08:24:40.190 INFO:tasks.rados.rados.0.smithi137.stdout:36: done (15 left) 2024-09-21T08:24:40.190 INFO:tasks.rados.rados.0.smithi137.stdout:37: done (14 left) 2024-09-21T08:24:40.190 INFO:tasks.rados.rados.0.smithi137.stdout:52: write oid 18 current snap is 0 2024-09-21T08:24:40.190 INFO:tasks.rados.rados.0.smithi137.stdout:38: finishing write tid 3 to smithi13779221-38 2024-09-21T08:24:40.190 INFO:tasks.rados.rados.0.smithi137.stdout:52: seq_num 50 ranges {440350=460568,1309629=542666,2323973=516377} 2024-09-21T08:24:40.192 INFO:tasks.rados.rados.0.smithi137.stdout:52: writing smithi13779221-18 from 440350 to 900918 tid 1 2024-09-21T08:24:40.197 INFO:tasks.rados.rados.0.smithi137.stdout:52: writing smithi13779221-18 from 1309629 to 1852295 tid 2 2024-09-21T08:24:40.203 INFO:tasks.rados.rados.0.smithi137.stdout:52: writing smithi13779221-18 from 2323973 to 2840350 tid 3 2024-09-21T08:24:40.203 INFO:tasks.rados.rados.0.smithi137.stdout:53: rollback oid 30 current snap is 0 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:53: done (15 left) 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:54: write (excl) oid 22 current snap is 0 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:38: finishing write tid 4 to smithi13779221-38 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:38: oid 38 updating version 0 to 19 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:38: oid 38 updating version 19 to 21 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:38: oid 38 version 21 is already newer than 20 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 21 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:38: left oid 38 (ObjNum 37 snap 0 seq_num 37) 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:39: finishing write tid 1 to smithi13779221-39 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:39: finishing write tid 2 to smithi13779221-39 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:39: finishing write tid 3 to smithi13779221-39 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:39: finishing write tid 4 to smithi13779221-39 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:39: oid 39 updating version 0 to 1 2024-09-21T08:24:40.204 INFO:tasks.rados.rados.0.smithi137.stdout:39: oid 39 updating version 1 to 2 2024-09-21T08:24:40.205 INFO:tasks.rados.rados.0.smithi137.stdout:39: oid 39 updating version 2 to 3 2024-09-21T08:24:40.205 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 3 (ObjNum 38 snap 0 seq_num 38) dirty exists 2024-09-21T08:24:40.205 INFO:tasks.rados.rados.0.smithi137.stdout:39: left oid 39 (ObjNum 38 snap 0 seq_num 38) 2024-09-21T08:24:40.205 INFO:tasks.rados.rados.0.smithi137.stdout:40: finishing write tid 1 to smithi13779221-40 2024-09-21T08:24:40.205 INFO:tasks.rados.rados.0.smithi137.stdout:54: seq_num 51 ranges {457157=535817,1768111=402677} 2024-09-21T08:24:40.209 INFO:tasks.rados.rados.0.smithi137.stdout:54: writing smithi13779221-22 from 457157 to 992974 tid 1 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:54: writing smithi13779221-22 from 1768111 to 2170788 tid 2 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:38: done (15 left) 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:39: done (14 left) 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:55: snap_create 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:40: finishing write tid 2 to smithi13779221-40 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:40: finishing write tid 3 to smithi13779221-40 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:40: finishing write tid 4 to smithi13779221-40 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:40: oid 40 updating version 0 to 9 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:40: oid 40 version 9 is already newer than 8 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:40: oid 40 version 9 is already newer than 7 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 9 (ObjNum 39 snap 0 seq_num 39) dirty exists 2024-09-21T08:24:40.219 INFO:tasks.rados.rados.0.smithi137.stdout:40: left oid 40 (ObjNum 39 snap 0 seq_num 39) 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:41: finishing write tid 1 to smithi13779221-41 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:41: finishing write tid 2 to smithi13779221-41 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:41: finishing write tid 3 to smithi13779221-41 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:41: finishing write tid 4 to smithi13779221-41 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:41: oid 41 updating version 0 to 6 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:41: oid 41 version 6 is already newer than 4 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:41: oid 41 version 6 is already newer than 5 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:41: left oid 41 (ObjNum 40 snap 0 seq_num 40) 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:42: finishing write tid 1 to smithi13779221-42 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:42: finishing write tid 2 to smithi13779221-42 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:42: finishing write tid 3 to smithi13779221-42 2024-09-21T08:24:40.220 INFO:tasks.rados.rados.0.smithi137.stdout:42: finishing write tid 4 to smithi13779221-42 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:42: oid 42 updating version 0 to 9 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:42: oid 42 version 9 is already newer than 8 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:42: oid 42 version 9 is already newer than 7 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 9 (ObjNum 41 snap 0 seq_num 41) dirty exists 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:42: left oid 42 (ObjNum 41 snap 0 seq_num 41) 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:43: finishing write tid 1 to smithi13779221-43 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:43: finishing write tid 2 to smithi13779221-43 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:43: finishing write tid 3 to smithi13779221-43 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:43: finishing write tid 4 to smithi13779221-43 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:43: oid 43 updating version 0 to 9 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:43: oid 43 version 9 is already newer than 7 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:43: oid 43 version 9 is already newer than 8 2024-09-21T08:24:40.221 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 9 (ObjNum 42 snap 0 seq_num 42) dirty exists 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:43: left oid 43 (ObjNum 42 snap 0 seq_num 42) 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:44: finishing write tid 1 to smithi13779221-44 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:44: finishing write tid 2 to smithi13779221-44 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:44: finishing write tid 3 to smithi13779221-44 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:44: finishing write tid 4 to smithi13779221-44 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:44: oid 44 updating version 0 to 15 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:44: oid 44 version 15 is already newer than 13 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:44: oid 44 version 15 is already newer than 14 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 15 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:44: left oid 44 (ObjNum 43 snap 0 seq_num 43) 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:45: finishing write tid 1 to smithi13779221-45 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:45: finishing write tid 2 to smithi13779221-45 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:45: finishing write tid 3 to smithi13779221-45 2024-09-21T08:24:40.222 INFO:tasks.rados.rados.0.smithi137.stdout:45: finishing write tid 4 to smithi13779221-45 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:45: oid 45 updating version 0 to 15 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:45: oid 45 version 15 is already newer than 13 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:45: oid 45 version 15 is already newer than 14 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 15 (ObjNum 44 snap 0 seq_num 44) dirty exists 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:45: left oid 45 (ObjNum 44 snap 0 seq_num 44) 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:46: finishing write tid 1 to smithi13779221-46 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:46: finishing write tid 2 to smithi13779221-46 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:46: finishing write tid 3 to smithi13779221-46 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:46: finishing write tid 4 to smithi13779221-46 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:46: oid 46 updating version 0 to 24 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:46: oid 46 version 24 is already newer than 22 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:46: oid 46 version 24 is already newer than 23 2024-09-21T08:24:40.223 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 24 (ObjNum 45 snap 0 seq_num 45) dirty exists 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:46: left oid 46 (ObjNum 45 snap 0 seq_num 45) 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:47: finishing write tid 1 to smithi13779221-47 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:47: finishing write tid 2 to smithi13779221-47 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:47: finishing write tid 3 to smithi13779221-47 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:47: finishing write tid 4 to smithi13779221-47 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:47: oid 47 updating version 0 to 10 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:47: oid 47 updating version 10 to 12 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:47: oid 47 version 12 is already newer than 11 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 12 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:47: left oid 47 (ObjNum 46 snap 0 seq_num 46) 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:48: finishing write tid 1 to smithi13779221-48 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:48: finishing write tid 2 to smithi13779221-48 2024-09-21T08:24:40.224 INFO:tasks.rados.rados.0.smithi137.stdout:48: finishing write tid 3 to smithi13779221-48 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:48: finishing write tid 4 to smithi13779221-48 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:48: oid 48 updating version 0 to 15 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:48: oid 48 version 15 is already newer than 14 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:48: oid 48 version 15 is already newer than 13 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 15 (ObjNum 47 snap 0 seq_num 47) dirty exists 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:48: left oid 48 (ObjNum 47 snap 0 seq_num 47) 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:49: finishing write tid 1 to smithi13779221-49 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:49: finishing write tid 2 to smithi13779221-49 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:49: finishing write tid 3 to smithi13779221-49 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:49: finishing write tid 4 to smithi13779221-49 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:49: oid 49 updating version 0 to 14 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:49: oid 49 updating version 14 to 15 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:49: oid 49 version 15 is already newer than 13 2024-09-21T08:24:40.225 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:49: left oid 49 (ObjNum 48 snap 0 seq_num 48) 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:50: finishing write tid 1 to smithi13779221-50 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:50: finishing write tid 2 to smithi13779221-50 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:50: finishing write tid 3 to smithi13779221-50 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:50: finishing write tid 4 to smithi13779221-50 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:50: oid 50 updating version 0 to 17 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:50: oid 50 version 17 is already newer than 16 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:50: oid 50 updating version 17 to 18 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 18 (ObjNum 49 snap 0 seq_num 49) dirty exists 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:50: left oid 50 (ObjNum 49 snap 0 seq_num 49) 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:51: finishing copy_from racing read to smithi13779221-2 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:51: finishing copy_from to smithi13779221-2 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 3 (ObjNum 10 snap 0 seq_num 10) dirty exists 2024-09-21T08:24:40.226 INFO:tasks.rados.rados.0.smithi137.stdout:52: finishing write tid 1 to smithi13779221-18 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:52: finishing write tid 2 to smithi13779221-18 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:52: finishing write tid 3 to smithi13779221-18 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:52: finishing write tid 4 to smithi13779221-18 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:52: finishing write tid 5 to smithi13779221-18 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:52: oid 18 updating version 0 to 14 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:52: oid 18 version 14 is already newer than 11 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:52: oid 18 version 14 is already newer than 13 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:52: oid 18 version 14 is already newer than 12 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 14 (ObjNum 50 snap 0 seq_num 50) dirty exists 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:52: left oid 18 (ObjNum 50 snap 0 seq_num 50) 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:54: finishing write tid 1 to smithi13779221-22 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:54: finishing write tid 2 to smithi13779221-22 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:54: finishing write tid 3 to smithi13779221-22 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:54: finishing write tid 4 to smithi13779221-22 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:54: oid 22 updating version 0 to 16 2024-09-21T08:24:40.227 INFO:tasks.rados.rados.0.smithi137.stdout:54: oid 22 updating version 16 to 18 2024-09-21T08:24:40.228 INFO:tasks.rados.rados.0.smithi137.stdout:54: oid 22 version 18 is already newer than 17 2024-09-21T08:24:40.228 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 18 (ObjNum 51 snap 0 seq_num 51) dirty exists 2024-09-21T08:24:40.228 INFO:tasks.rados.rados.0.smithi137.stdout:54: left oid 22 (ObjNum 51 snap 0 seq_num 51) 2024-09-21T08:24:40.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:40.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:40.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:40.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:40.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:40.420 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:40.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:40.421 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:40.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:40.542 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:40.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:40.543 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:40.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:40.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:40.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:40.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:40.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3100918956' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-21T08:24:40.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-mon[29634]: osdmap e86: 8 total, 8 up, 7 in 2024-09-21T08:24:40.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:40.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/585330273' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:40.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-mon[29634]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:40.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3100918956' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-21T08:24:40.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-mon[41044]: osdmap e86: 8 total, 8 up, 7 in 2024-09-21T08:24:40.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:40.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/585330273' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:40.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-mon[41044]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:40.765 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:24:40.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:40: done (14 left) 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:41: done (13 left) 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:42: done (12 left) 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:43: done (11 left) 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:44: done (10 left) 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:45: done (9 left) 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:46: done (8 left) 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:47: done (7 left) 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:48: done (6 left) 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:49: done (5 left) 2024-09-21T08:24:40.769 INFO:tasks.rados.rados.0.smithi137.stdout:50: done (4 left) 2024-09-21T08:24:40.770 INFO:tasks.rados.rados.0.smithi137.stdout:51: done (3 left) 2024-09-21T08:24:40.770 INFO:tasks.rados.rados.0.smithi137.stdout:52: done (2 left) 2024-09-21T08:24:40.770 INFO:tasks.rados.rados.0.smithi137.stdout:54: done (1 left) 2024-09-21T08:24:40.770 INFO:tasks.rados.rados.0.smithi137.stdout:55: done (0 left) 2024-09-21T08:24:40.770 INFO:tasks.rados.rados.0.smithi137.stdout:56: delete oid 34 current snap is 1 2024-09-21T08:24:40.772 INFO:tasks.rados.rados.0.smithi137.stdout:56: done (0 left) 2024-09-21T08:24:40.772 INFO:tasks.rados.rados.0.smithi137.stdout:57: snap_create 2024-09-21T08:24:40.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:40.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:40.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:40.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:40.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:40.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:40.912 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:40.978 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:40 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:24:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:24:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:40 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3100918956' entity='client.0' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "write_fadvise_dontneed", "val": "false"}]': finished 2024-09-21T08:24:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:40 smithi165 ceph-mon[32893]: osdmap e86: 8 total, 8 up, 7 in 2024-09-21T08:24:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:40 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:40 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/585330273' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:40 smithi165 ceph-mon[32893]: from='client.24748 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:41.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:41.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:41.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:41.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:41.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:41.158 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:41.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:41.159 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:41.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:41.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:41.283 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:41.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:41.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T08:24:41.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:41.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:41.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:41.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:41.485 INFO:tasks.rados.rados.0.smithi137.stdout:57: done (0 left) 2024-09-21T08:24:41.485 INFO:tasks.rados.rados.0.smithi137.stdout:58: write oid 33 current snap is 2 2024-09-21T08:24:41.485 INFO:tasks.rados.rados.0.smithi137.stdout:58: seq_num 53 ranges {490771=686315,1885075=406346} 2024-09-21T08:24:41.491 INFO:tasks.rados.rados.0.smithi137.stdout:58: writing smithi13779221-33 from 490771 to 1177086 tid 1 2024-09-21T08:24:41.497 INFO:tasks.rados.rados.0.smithi137.stdout:58: writing smithi13779221-33 from 1885075 to 2291421 tid 2 2024-09-21T08:24:41.497 INFO:tasks.rados.rados.0.smithi137.stdout:59: delete oid 3 current snap is 2 2024-09-21T08:24:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:58: finishing write tid 1 to smithi13779221-33 2024-09-21T08:24:41.499 INFO:tasks.rados.rados.0.smithi137.stdout:58: finishing write tid 2 to smithi13779221-33 2024-09-21T08:24:41.500 INFO:tasks.rados.rados.0.smithi137.stdout:59: done (1 left) 2024-09-21T08:24:41.500 INFO:tasks.rados.rados.0.smithi137.stdout:60: write oid 12 current snap is 2 2024-09-21T08:24:41.500 INFO:tasks.rados.rados.0.smithi137.stdout:60: seq_num 55 ranges {524385=553166,1602039=410015,2577441=346944} 2024-09-21T08:24:41.505 INFO:tasks.rados.rados.0.smithi137.stdout:60: writing smithi13779221-12 from 524385 to 1077551 tid 1 2024-09-21T08:24:41.511 INFO:tasks.rados.rados.0.smithi137.stdout:60: writing smithi13779221-12 from 1602039 to 2012054 tid 2 2024-09-21T08:24:41.516 INFO:tasks.rados.rados.0.smithi137.stdout:60: writing smithi13779221-12 from 2577441 to 2924385 tid 3 2024-09-21T08:24:41.516 INFO:tasks.rados.rados.0.smithi137.stdout:61: rollback oid 47 current snap is 2 2024-09-21T08:24:41.516 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 47 to 1 2024-09-21T08:24:41.516 INFO:tasks.rados.rados.0.smithi137.stdout:58: finishing write tid 3 to smithi13779221-33 2024-09-21T08:24:41.516 INFO:tasks.rados.rados.0.smithi137.stdout:58: finishing write tid 4 to smithi13779221-33 2024-09-21T08:24:41.517 INFO:tasks.rados.rados.0.smithi137.stdout:58: oid 33 updating version 0 to 6 2024-09-21T08:24:41.517 INFO:tasks.rados.rados.0.smithi137.stdout:58: oid 33 updating version 6 to 7 2024-09-21T08:24:41.517 INFO:tasks.rados.rados.0.smithi137.stdout:58: oid 33 version 7 is already newer than 5 2024-09-21T08:24:41.517 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 7 (ObjNum 53 snap 2 seq_num 53) dirty exists 2024-09-21T08:24:41.517 INFO:tasks.rados.rados.0.smithi137.stdout:58: left oid 33 (ObjNum 53 snap 2 seq_num 53) 2024-09-21T08:24:41.517 INFO:tasks.rados.rados.0.smithi137.stdout:58: done (2 left) 2024-09-21T08:24:41.517 INFO:tasks.rados.rados.0.smithi137.stdout:62: snap_remove snap 0 2024-09-21T08:24:41.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:41.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:41.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:41.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:41.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:41 smithi137 ceph-mon[29634]: pgmap v34: 148 pgs: 148 active+clean; 583 KiB data, 195 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:24:41.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:41 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:24:41.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:41 smithi137 ceph-mon[29634]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:24:41.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:41 smithi137 ceph-mon[29634]: osdmap e87: 8 total, 8 up, 7 in 2024-09-21T08:24:41.635 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:41 smithi137 ceph-mon[29634]: osdmap e88: 8 total, 8 up, 7 in 2024-09-21T08:24:41.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:41 smithi137 ceph-mon[41044]: pgmap v34: 148 pgs: 148 active+clean; 583 KiB data, 195 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:24:41.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:41 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:24:41.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:41 smithi137 ceph-mon[41044]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:24:41.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:41 smithi137 ceph-mon[41044]: osdmap e87: 8 total, 8 up, 7 in 2024-09-21T08:24:41.635 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:41 smithi137 ceph-mon[41044]: osdmap e88: 8 total, 8 up, 7 in 2024-09-21T08:24:41.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:41.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:41.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:41.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:41.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:41.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:41.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:41.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:41.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:41.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:41.922 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:41.923 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:42.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:24:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:24:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:41 smithi165 ceph-mon[32893]: pgmap v34: 148 pgs: 148 active+clean; 583 KiB data, 195 MiB used, 626 GiB / 626 GiB avail 2024-09-21T08:24:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:41 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:24:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:41 smithi165 ceph-mon[32893]: from='client.24748 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:24:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:41 smithi165 ceph-mon[32893]: osdmap e87: 8 total, 8 up, 7 in 2024-09-21T08:24:42.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:41 smithi165 ceph-mon[32893]: osdmap e88: 8 total, 8 up, 7 in 2024-09-21T08:24:42.023 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:42.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:42.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:42.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:42.055 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:42.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:42.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:42.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:42.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:42.176 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:24:42 smithi137 bash[79543]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T08:24:42.269 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:42.291 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:42.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:42.292 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:42.393 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:42.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:42.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:42.416 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:42.487 INFO:tasks.rados.rados.0.smithi137.stdout:62: done (2 left) 2024-09-21T08:24:42.487 INFO:tasks.rados.rados.0.smithi137.stdout:63: read oid 28 snap -1 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:64: read oid 33 snap -1 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:64: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:65: delete oid 2 current snap is 2 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:60: finishing write tid 1 to smithi13779221-12 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:60: finishing write tid 2 to smithi13779221-12 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:60: finishing write tid 3 to smithi13779221-12 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:60: finishing write tid 4 to smithi13779221-12 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:60: finishing write tid 5 to smithi13779221-12 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:60: oid 12 updating version 0 to 20 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:60: oid 12 updating version 20 to 23 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:60: oid 12 version 23 is already newer than 21 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:60: oid 12 version 23 is already newer than 22 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 23 (ObjNum 55 snap 2 seq_num 55) dirty exists 2024-09-21T08:24:42.488 INFO:tasks.rados.rados.0.smithi137.stdout:60: left oid 12 (ObjNum 55 snap 2 seq_num 55) 2024-09-21T08:24:42.489 INFO:tasks.rados.rados.0.smithi137.stdout:61: finishing rollback tid 0 to smithi13779221-47 2024-09-21T08:24:42.489 INFO:tasks.rados.rados.0.smithi137.stdout:61: finishing rollback tid 1 to smithi13779221-47 2024-09-21T08:24:42.489 INFO:tasks.rados.rados.0.smithi137.stdout:61: finishing rollback tid 2 to smithi13779221-47 2024-09-21T08:24:42.489 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 21 (ObjNum 46 snap 0 seq_num 46) dirty exists 2024-09-21T08:24:42.495 INFO:tasks.rados.rados.0.smithi137.stdout:60: done (4 left) 2024-09-21T08:24:42.495 INFO:tasks.rados.rados.0.smithi137.stdout:61: done (3 left) 2024-09-21T08:24:42.495 INFO:tasks.rados.rados.0.smithi137.stdout:65: done (2 left) 2024-09-21T08:24:42.495 INFO:tasks.rados.rados.0.smithi137.stdout:66: copy_from oid 3 from oid 31 current snap is 2 2024-09-21T08:24:42.495 INFO:tasks.rados.rados.0.smithi137.stdout:67: snap_create 2024-09-21T08:24:42.497 INFO:tasks.rados.rados.0.smithi137.stdout:63: expect (ObjNum 27 snap 0 seq_num 27) 2024-09-21T08:24:42.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:42.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:42.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:42.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:42.546 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:24:42.546 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:24:42.580 INFO:tasks.rados.rados.0.smithi137.stdout:64: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-21T08:24:42.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:42.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:42.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:42.666 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:42.715 INFO:tasks.rados.rados.0.smithi137.stdout:66: finishing copy_from racing read to smithi13779221-3 2024-09-21T08:24:42.715 INFO:tasks.rados.rados.0.smithi137.stdout:66: finishing copy_from to smithi13779221-3 2024-09-21T08:24:42.715 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 12 (ObjNum 30 snap 0 seq_num 30) dirty exists 2024-09-21T08:24:42.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:42.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:42 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:42.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:42 smithi137 ceph-mon[29634]: osdmap e89: 8 total, 8 up, 7 in 2024-09-21T08:24:42.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:42 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:42.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:42 smithi137 ceph-mon[41044]: osdmap e89: 8 total, 8 up, 7 in 2024-09-21T08:24:42.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:42.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:42.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:42.911 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:42.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:42.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:43.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:24:43.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:42 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:42 smithi165 ceph-mon[32893]: osdmap e89: 8 total, 8 up, 7 in 2024-09-21T08:24:43.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:43.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:43.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:43.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T08:24:43.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:43.198 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:43.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:43.199 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:43.268 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:43.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:43.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:43.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:43.322 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:43.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:43.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:43.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:43.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:43.484 INFO:tasks.rados.rados.0.smithi137.stdout:63: done (3 left) 2024-09-21T08:24:43.484 INFO:tasks.rados.rados.0.smithi137.stdout:64: done (2 left) 2024-09-21T08:24:43.484 INFO:tasks.rados.rados.0.smithi137.stdout:66: done (1 left) 2024-09-21T08:24:43.485 INFO:tasks.rados.rados.0.smithi137.stdout:67: done (0 left) 2024-09-21T08:24:43.485 INFO:tasks.rados.rados.0.smithi137.stdout:68: snap_remove snap 1 2024-09-21T08:24:43.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:43.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:43.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:43.573 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:43.666 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:43 smithi137 ceph-mon[29634]: pgmap v37: 148 pgs: 148 active+clean; 117 MiB data, 321 MiB used, 626 GiB / 626 GiB avail; 12 KiB/s rd, 15 MiB/s wr, 77 op/s; 662 B/s, 20 objects/s recovering 2024-09-21T08:24:43.666 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:43 smithi137 ceph-mon[29634]: osdmap e90: 8 total, 8 up, 7 in 2024-09-21T08:24:43.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:43 smithi137 ceph-mon[41044]: pgmap v37: 148 pgs: 148 active+clean; 117 MiB data, 321 MiB used, 626 GiB / 626 GiB avail; 12 KiB/s rd, 15 MiB/s wr, 77 op/s; 662 B/s, 20 objects/s recovering 2024-09-21T08:24:43.666 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:43 smithi137 ceph-mon[41044]: osdmap e90: 8 total, 8 up, 7 in 2024-09-21T08:24:43.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:43.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:43.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:43.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:43.774 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:24:43.774 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:24:43.786 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:24:43.786 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:24:43.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:43.806 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:24:43.806 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:24:43.811 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:43.822 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:24:43.822 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:24:43.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:43.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:43.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:43.840 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:24:43.841 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:24:43.857 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:24:43.858 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:24:43.874 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:24:43.874 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:24:43.890 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:24:43.890 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:24:43.901 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 3, 5, 6, 7] out_osds: [4] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:24:43.902 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:24:43.902 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 1, 2, 3, 5, 6, 7] 2024-09-21T08:24:43.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd out 3 2024-09-21T08:24:43.920 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:24:43.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:43.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:43.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:43.952 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:43 smithi165 ceph-mon[32893]: pgmap v37: 148 pgs: 148 active+clean; 117 MiB data, 321 MiB used, 626 GiB / 626 GiB avail; 12 KiB/s rd, 15 MiB/s wr, 77 op/s; 662 B/s, 20 objects/s recovering 2024-09-21T08:24:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:43 smithi165 ceph-mon[32893]: osdmap e90: 8 total, 8 up, 7 in 2024-09-21T08:24:44.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:44.076 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:44.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:44.077 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:44.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:44.203 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:44.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:44.204 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:44.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:44.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:44.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:44.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:44.429 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:44.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:44.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:44.451 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:44.487 INFO:tasks.rados.rados.0.smithi137.stdout:68: done (0 left) 2024-09-21T08:24:44.487 INFO:tasks.rados.rados.0.smithi137.stdout:69: read oid 23 snap -1 2024-09-21T08:24:44.487 INFO:tasks.rados.rados.0.smithi137.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-21T08:24:44.487 INFO:tasks.rados.rados.0.smithi137.stdout:70: read oid 24 snap -1 2024-09-21T08:24:44.487 INFO:tasks.rados.rados.0.smithi137.stdout:70: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-21T08:24:44.487 INFO:tasks.rados.rados.0.smithi137.stdout:71: read oid 41 snap -1 2024-09-21T08:24:44.487 INFO:tasks.rados.rados.0.smithi137.stdout:71: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-21T08:24:44.487 INFO:tasks.rados.rados.0.smithi137.stdout:72: snap_remove snap 2 2024-09-21T08:24:44.519 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:24:44.519 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:24:44.519 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:24:44.520 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:24:44.520 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:24:44.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:44.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:44.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:44.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:44.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:44.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:44.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:44.702 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:44.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:44.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:44.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:44.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:44.884 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:44 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:24:44.884 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:44 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:24:44.884 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:44 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3668906400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:44.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:44 smithi137 ceph-mon[41044]: osdmap e91: 8 total, 8 up, 7 in 2024-09-21T08:24:44.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:44 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:24:44.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:44 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:24:44.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:44 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3668906400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:44.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:44 smithi137 ceph-mon[29634]: osdmap e91: 8 total, 8 up, 7 in 2024-09-21T08:24:44.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:44.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:44.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:44.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:44 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:24:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:44 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:24:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:44 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3668906400' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:44 smithi165 ceph-mon[32893]: osdmap e91: 8 total, 8 up, 7 in 2024-09-21T08:24:45.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:45.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:45.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:45.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:45.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:45.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:45.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:45.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:45.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:45.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:45.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:45.326 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:45.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:45.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:45.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:45.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:45.490 INFO:tasks.rados.rados.0.smithi137.stdout:72: done (3 left) 2024-09-21T08:24:45.490 INFO:tasks.rados.rados.0.smithi137.stdout:73: write oid 47 current snap is 3 2024-09-21T08:24:45.490 INFO:tasks.rados.rados.0.smithi137.stdout:73: seq_num 57 ranges {557999=703664,1719003=413684,2632287=325712} 2024-09-21T08:24:45.497 INFO:tasks.rados.rados.0.smithi137.stdout:73: writing smithi13779221-47 from 557999 to 1261663 tid 1 2024-09-21T08:24:45.502 INFO:tasks.rados.rados.0.smithi137.stdout:73: writing smithi13779221-47 from 1719003 to 2132687 tid 2 2024-09-21T08:24:45.507 INFO:tasks.rados.rados.0.smithi137.stdout:73: writing smithi13779221-47 from 2632287 to 2957999 tid 3 2024-09-21T08:24:45.508 INFO:tasks.rados.rados.0.smithi137.stdout:74: copy_from oid 48 from oid 25 current snap is 3 2024-09-21T08:24:45.508 INFO:tasks.rados.rados.0.smithi137.stdout:70: expect (ObjNum 23 snap 0 seq_num 23) 2024-09-21T08:24:45.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:45.576 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:45.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:45.577 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:45.591 INFO:tasks.rados.rados.0.smithi137.stdout:69: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-21T08:24:45.670 INFO:tasks.rados.rados.0.smithi137.stdout:71: expect (ObjNum 40 snap 0 seq_num 40) 2024-09-21T08:24:45.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:45.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:45.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:45.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:45.758 INFO:tasks.rados.rados.0.smithi137.stdout:73: finishing write tid 1 to smithi13779221-47 2024-09-21T08:24:45.758 INFO:tasks.rados.rados.0.smithi137.stdout:73: finishing write tid 2 to smithi13779221-47 2024-09-21T08:24:45.758 INFO:tasks.rados.rados.0.smithi137.stdout:73: finishing write tid 3 to smithi13779221-47 2024-09-21T08:24:45.758 INFO:tasks.rados.rados.0.smithi137.stdout:73: finishing write tid 4 to smithi13779221-47 2024-09-21T08:24:45.758 INFO:tasks.rados.rados.0.smithi137.stdout:69: done (4 left) 2024-09-21T08:24:45.758 INFO:tasks.rados.rados.0.smithi137.stdout:70: done (3 left) 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:71: done (2 left) 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:75: copy_from oid 24 from oid 1 current snap is 3 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:73: finishing write tid 5 to smithi13779221-47 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:73: oid 47 updating version 0 to 30 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:73: oid 47 version 30 is already newer than 28 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:73: oid 47 version 30 is already newer than 29 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:73: oid 47 updating version 30 to 31 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 31 (ObjNum 57 snap 3 seq_num 57) dirty exists 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:73: left oid 47 (ObjNum 57 snap 3 seq_num 57) 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:74: finishing copy_from racing read to smithi13779221-48 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:74: finishing copy_from to smithi13779221-48 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 3 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:73: done (2 left) 2024-09-21T08:24:45.759 INFO:tasks.rados.rados.0.smithi137.stdout:74: done (1 left) 2024-09-21T08:24:45.760 INFO:tasks.rados.rados.0.smithi137.stdout:76: snap_create 2024-09-21T08:24:45.764 INFO:tasks.rados.rados.0.smithi137.stdout:75: finishing copy_from racing read to smithi13779221-24 2024-09-21T08:24:45.765 INFO:tasks.rados.rados.0.smithi137.stdout:75: finishing copy_from to smithi13779221-24 2024-09-21T08:24:45.765 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 3 (ObjNum 0 snap 0 seq_num 0) dirty exists 2024-09-21T08:24:45.788 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:45.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:45.814 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[29634]: pgmap v40: 148 pgs: 148 active+clean; 117 MiB data, 321 MiB used, 626 GiB / 626 GiB avail; 12 KiB/s rd, 15 MiB/s wr, 77 op/s; 662 B/s, 20 objects/s recovering 2024-09-21T08:24:45.814 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[29634]: osdmap e92: 8 total, 8 up, 7 in 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[41044]: pgmap v40: 148 pgs: 148 active+clean; 117 MiB data, 321 MiB used, 626 GiB / 626 GiB avail; 12 KiB/s rd, 15 MiB/s wr, 77 op/s; 662 B/s, 20 objects/s recovering 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:45.815 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:45.816 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:45 smithi137 ceph-mon[41044]: osdmap e92: 8 total, 8 up, 7 in 2024-09-21T08:24:45.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T08:24:45.826 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:45.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:45.827 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:45.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:45.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:45.951 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:45 smithi165 ceph-mon[32893]: pgmap v40: 148 pgs: 148 active+clean; 117 MiB data, 321 MiB used, 626 GiB / 626 GiB avail; 12 KiB/s rd, 15 MiB/s wr, 77 op/s; 662 B/s, 20 objects/s recovering 2024-09-21T08:24:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:45 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:24:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:45 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:24:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:45 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:45 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:45 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:24:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:45 smithi165 ceph-mon[32893]: osdmap e92: 8 total, 8 up, 7 in 2024-09-21T08:24:46.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:46.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:46.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:46.075 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:46.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:46.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:46.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:46.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:46.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:46.302 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:46.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:46.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:46.329 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:46.340 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:46.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:46.451 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:46.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:46.452 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:46.549 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:24:46.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:46.554 INFO:tasks.rados.rados.0.smithi137.stdout:75: done (1 left) 2024-09-21T08:24:46.554 INFO:tasks.rados.rados.0.smithi137.stdout:76: done (0 left) 2024-09-21T08:24:46.554 INFO:tasks.rados.rados.0.smithi137.stdout:77: copy_from oid 22 from oid 49 current snap is 4 2024-09-21T08:24:46.555 INFO:tasks.rados.rados.0.smithi137.stdout:78: read oid 39 snap 3 2024-09-21T08:24:46.555 INFO:tasks.rados.rados.0.smithi137.stdout:78: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-21T08:24:46.555 INFO:tasks.rados.rados.0.smithi137.stdout:79: write oid 30 current snap is 4 2024-09-21T08:24:46.555 INFO:tasks.rados.rados.0.smithi137.stdout:79: seq_num 58 ranges {574806=778913,2061132=557342} 2024-09-21T08:24:46.562 INFO:tasks.rados.rados.0.smithi137.stdout:79: writing smithi13779221-30 from 574806 to 1353719 tid 1 2024-09-21T08:24:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:79: writing smithi13779221-30 from 2061132 to 2618474 tid 2 2024-09-21T08:24:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:80: write (excl) oid 29 current snap is 4 2024-09-21T08:24:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:80: seq_num 59 ranges {591613=454162,1719614=417353,2570780=420833} 2024-09-21T08:24:46.578 INFO:tasks.rados.rados.0.smithi137.stdout:80: writing smithi13779221-29 from 591613 to 1045775 tid 1 2024-09-21T08:24:46.583 INFO:tasks.rados.rados.0.smithi137.stdout:80: writing smithi13779221-29 from 1719614 to 2136967 tid 2 2024-09-21T08:24:46.588 INFO:tasks.rados.rados.0.smithi137.stdout:80: writing smithi13779221-29 from 2570780 to 2991613 tid 3 2024-09-21T08:24:46.588 INFO:tasks.rados.rados.0.smithi137.stdout:81: read oid 36 snap 3 2024-09-21T08:24:46.589 INFO:tasks.rados.rados.0.smithi137.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-21T08:24:46.589 INFO:tasks.rados.rados.0.smithi137.stdout:82: copy_from oid 42 from oid 28 current snap is 4 2024-09-21T08:24:46.589 INFO:tasks.rados.rados.0.smithi137.stdout:78: expect (ObjNum 38 snap 0 seq_num 38) 2024-09-21T08:24:46.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:46.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:46.592 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:46.679 INFO:tasks.rados.rados.0.smithi137.stdout:79: finishing write tid 1 to smithi13779221-30 2024-09-21T08:24:46.679 INFO:tasks.rados.rados.0.smithi137.stdout:77: finishing copy_from racing read to smithi13779221-22 2024-09-21T08:24:46.679 INFO:tasks.rados.rados.0.smithi137.stdout:79: finishing write tid 2 to smithi13779221-30 2024-09-21T08:24:46.679 INFO:tasks.rados.rados.0.smithi137.stdout:78: done (5 left) 2024-09-21T08:24:46.679 INFO:tasks.rados.rados.0.smithi137.stdout:83: rollback oid 13 current snap is 4 2024-09-21T08:24:46.679 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 13 to 3 2024-09-21T08:24:46.679 INFO:tasks.rados.rados.0.smithi137.stdout:79: finishing write tid 3 to smithi13779221-30 2024-09-21T08:24:46.679 INFO:tasks.rados.rados.0.smithi137.stdout:77: finishing copy_from to smithi13779221-22 2024-09-21T08:24:46.679 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 15 (ObjNum 48 snap 0 seq_num 48) dirty exists 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:79: finishing write tid 4 to smithi13779221-30 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:79: oid 30 updating version 0 to 27 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:79: oid 30 updating version 27 to 28 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:79: oid 30 version 28 is already newer than 26 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 28 (ObjNum 58 snap 4 seq_num 58) dirty exists 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:79: left oid 30 (ObjNum 58 snap 4 seq_num 58) 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:80: finishing write tid 1 to smithi13779221-29 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:80: finishing write tid 2 to smithi13779221-29 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:80: finishing write tid 3 to smithi13779221-29 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:80: finishing write tid 4 to smithi13779221-29 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:80: finishing write tid 5 to smithi13779221-29 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:80: oid 29 updating version 0 to 32 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:80: oid 29 version 32 is already newer than 31 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:80: oid 29 updating version 32 to 33 2024-09-21T08:24:46.680 INFO:tasks.rados.rados.0.smithi137.stdout:80: oid 29 version 33 is already newer than 30 2024-09-21T08:24:46.681 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 33 (ObjNum 59 snap 4 seq_num 59) dirty exists 2024-09-21T08:24:46.681 INFO:tasks.rados.rados.0.smithi137.stdout:80: left oid 29 (ObjNum 59 snap 4 seq_num 59) 2024-09-21T08:24:46.681 INFO:tasks.rados.rados.0.smithi137.stdout:81: expect (ObjNum 35 snap 0 seq_num 35) 2024-09-21T08:24:46.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:46.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:46.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:46.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:46.762 INFO:tasks.rados.rados.0.smithi137.stdout:82: finishing copy_from racing read to smithi13779221-42 2024-09-21T08:24:46.762 INFO:tasks.rados.rados.0.smithi137.stdout:82: finishing copy_from to smithi13779221-42 2024-09-21T08:24:46.762 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 6 (ObjNum 27 snap 0 seq_num 27) dirty exists 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:83: finishing rollback tid 0 to smithi13779221-13 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:83: finishing rollback tid 1 to smithi13779221-13 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:83: finishing rollback tid 2 to smithi13779221-13 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 37 (ObjNum 12 snap 0 seq_num 12) dirty exists 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:77: done (5 left) 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:79: done (4 left) 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:80: done (3 left) 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:81: done (2 left) 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:82: done (1 left) 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:83: done (0 left) 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:84: read oid 10 snap -1 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:84: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:85: read oid 24 snap -1 2024-09-21T08:24:46.763 INFO:tasks.rados.rados.0.smithi137.stdout:85: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:86: read oid 49 snap 3 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:86: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:87: read oid 37 snap -1 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:87: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:88: copy_from oid 9 from oid 5 current snap is 4 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:89: read oid 2 snap -1 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:89: expect deleted 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:90: read oid 34 snap -1 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:90: expect deleted 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:91: write oid 35 current snap is 4 2024-09-21T08:24:46.764 INFO:tasks.rados.rados.0.smithi137.stdout:91: seq_num 60 ranges {608420=529411,1778096=561011,2881850=126570} 2024-09-21T08:24:46.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:46 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1043161239' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:24:46.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:46 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1043161239' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:24:46.769 INFO:tasks.rados.rados.0.smithi137.stdout:91: writing smithi13779221-35 from 608420 to 1137831 tid 1 2024-09-21T08:24:46.776 INFO:tasks.rados.rados.0.smithi137.stdout:91: writing smithi13779221-35 from 1778096 to 2339107 tid 2 2024-09-21T08:24:46.780 INFO:tasks.rados.rados.0.smithi137.stdout:91: writing smithi13779221-35 from 2881850 to 3008420 tid 3 2024-09-21T08:24:46.780 INFO:tasks.rados.rados.0.smithi137.stdout:92: read oid 17 snap -1 2024-09-21T08:24:46.780 INFO:tasks.rados.rados.0.smithi137.stdout:92: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-21T08:24:46.781 INFO:tasks.rados.rados.0.smithi137.stdout:93: write (excl) oid 3 current snap is 4 2024-09-21T08:24:46.781 INFO:tasks.rados.rados.0.smithi137.stdout:93: seq_num 61 ranges {625227=721013,1952931=704669} 2024-09-21T08:24:46.787 INFO:tasks.rados.rados.0.smithi137.stdout:93: writing smithi13779221-3 from 625227 to 1346240 tid 1 2024-09-21T08:24:46.795 INFO:tasks.rados.rados.0.smithi137.stdout:93: writing smithi13779221-3 from 1952931 to 2657600 tid 2 2024-09-21T08:24:46.795 INFO:tasks.rados.rados.0.smithi137.stdout:94: write oid 43 current snap is 4 2024-09-21T08:24:46.796 INFO:tasks.rados.rados.0.smithi137.stdout:94: seq_num 62 ranges {642034=796262,1895060=564680} 2024-09-21T08:24:46.804 INFO:tasks.rados.rados.0.smithi137.stdout:94: writing smithi13779221-43 from 642034 to 1438296 tid 1 2024-09-21T08:24:46.809 INFO:tasks.rados.rados.0.smithi137.stdout:94: writing smithi13779221-43 from 1895060 to 2459740 tid 2 2024-09-21T08:24:46.809 INFO:tasks.rados.rados.0.smithi137.stdout:95: read oid 31 snap -1 2024-09-21T08:24:46.809 INFO:tasks.rados.rados.0.smithi137.stdout:95: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-21T08:24:46.809 INFO:tasks.rados.rados.0.smithi137.stdout:96: copy_from oid 28 from oid 41 current snap is 4 2024-09-21T08:24:46.809 INFO:tasks.rados.rados.0.smithi137.stdout:85: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-21T08:24:46.817 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:46.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:46.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:46.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:46.876 INFO:tasks.rados.rados.0.smithi137.stdout:87: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-21T08:24:46.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:46.958 INFO:tasks.rados.rados.0.smithi137.stdout:84: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-21T08:24:46.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:46.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:46.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:46 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1043161239' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:24:47.027 INFO:tasks.rados.rados.0.smithi137.stdout:91: finishing write tid 1 to smithi13779221-35 2024-09-21T08:24:47.027 INFO:tasks.rados.rados.0.smithi137.stdout:86: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-21T08:24:47.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:47.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:47.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:47.083 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:47.117 INFO:tasks.rados.rados.0.smithi137.stdout:88: finishing copy_from racing read to smithi13779221-9 2024-09-21T08:24:47.117 INFO:tasks.rados.rados.0.smithi137.stdout:88: finishing copy_from to smithi13779221-9 2024-09-21T08:24:47.117 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 9 v 3 (ObjNum 4 snap 0 seq_num 4) dirty exists 2024-09-21T08:24:47.117 INFO:tasks.rados.rados.0.smithi137.stdout:91: finishing write tid 2 to smithi13779221-35 2024-09-21T08:24:47.118 INFO:tasks.rados.rados.0.smithi137.stdout:91: finishing write tid 3 to smithi13779221-35 2024-09-21T08:24:47.118 INFO:tasks.rados.rados.0.smithi137.stdout:91: finishing write tid 4 to smithi13779221-35 2024-09-21T08:24:47.118 INFO:tasks.rados.rados.0.smithi137.stdout:91: finishing write tid 5 to smithi13779221-35 2024-09-21T08:24:47.118 INFO:tasks.rados.rados.0.smithi137.stdout:91: oid 35 updating version 0 to 21 2024-09-21T08:24:47.118 INFO:tasks.rados.rados.0.smithi137.stdout:91: oid 35 version 21 is already newer than 18 2024-09-21T08:24:47.118 INFO:tasks.rados.rados.0.smithi137.stdout:91: oid 35 version 21 is already newer than 20 2024-09-21T08:24:47.118 INFO:tasks.rados.rados.0.smithi137.stdout:91: oid 35 version 21 is already newer than 19 2024-09-21T08:24:47.118 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 21 (ObjNum 60 snap 4 seq_num 60) dirty exists 2024-09-21T08:24:47.118 INFO:tasks.rados.rados.0.smithi137.stdout:91: left oid 35 (ObjNum 60 snap 4 seq_num 60) 2024-09-21T08:24:47.118 INFO:tasks.rados.rados.0.smithi137.stdout:92: expect (ObjNum 16 snap 0 seq_num 16) 2024-09-21T08:24:47.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:47.201 INFO:tasks.rados.rados.0.smithi137.stdout:93: finishing write tid 1 to smithi13779221-3 2024-09-21T08:24:47.201 INFO:tasks.rados.rados.0.smithi137.stdout:93: finishing write tid 2 to smithi13779221-3 2024-09-21T08:24:47.201 INFO:tasks.rados.rados.0.smithi137.stdout:84: done (12 left) 2024-09-21T08:24:47.202 INFO:tasks.rados.rados.0.smithi137.stdout:85: done (11 left) 2024-09-21T08:24:47.202 INFO:tasks.rados.rados.0.smithi137.stdout:86: done (10 left) 2024-09-21T08:24:47.202 INFO:tasks.rados.rados.0.smithi137.stdout:87: done (9 left) 2024-09-21T08:24:47.202 INFO:tasks.rados.rados.0.smithi137.stdout:88: done (8 left) 2024-09-21T08:24:47.202 INFO:tasks.rados.rados.0.smithi137.stdout:89: done (7 left) 2024-09-21T08:24:47.202 INFO:tasks.rados.rados.0.smithi137.stdout:90: done (6 left) 2024-09-21T08:24:47.202 INFO:tasks.rados.rados.0.smithi137.stdout:91: done (5 left) 2024-09-21T08:24:47.202 INFO:tasks.rados.rados.0.smithi137.stdout:92: done (4 left) 2024-09-21T08:24:47.202 INFO:tasks.rados.rados.0.smithi137.stdout:97: snap_remove snap 3 2024-09-21T08:24:47.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:47.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:47.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:47.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:24:47.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:47.349 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:47.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:47.352 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:47.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:47.475 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:47.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:47.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:47.570 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.3. 2024-09-21T08:24:47.575 INFO:tasks.rados.rados.0.smithi137.stdout:97: done (4 left) 2024-09-21T08:24:47.575 INFO:tasks.rados.rados.0.smithi137.stdout:98: write oid 13 current snap is 4 2024-09-21T08:24:47.575 INFO:tasks.rados.rados.0.smithi137.stdout:98: seq_num 63 ranges {658841=471511,1553542=708338,2964119=94722} 2024-09-21T08:24:47.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:47.581 INFO:tasks.rados.rados.0.smithi137.stdout:98: writing smithi13779221-13 from 658841 to 1130352 tid 1 2024-09-21T08:24:47.593 INFO:tasks.rados.rados.0.smithi137.stdout:98: writing smithi13779221-13 from 1553542 to 2261880 tid 2 2024-09-21T08:24:47.597 INFO:tasks.rados.rados.0.smithi137.stdout:98: writing smithi13779221-13 from 2964119 to 3058841 tid 3 2024-09-21T08:24:47.598 INFO:tasks.rados.rados.0.smithi137.stdout:99: read oid 40 snap -1 2024-09-21T08:24:47.598 INFO:tasks.rados.rados.0.smithi137.stdout:99: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-21T08:24:47.598 INFO:tasks.rados.rados.0.smithi137.stdout:93: finishing write tid 3 to smithi13779221-3 2024-09-21T08:24:47.598 INFO:tasks.rados.rados.0.smithi137.stdout:93: finishing write tid 4 to smithi13779221-3 2024-09-21T08:24:47.598 INFO:tasks.rados.rados.0.smithi137.stdout:93: oid 3 updating version 0 to 15 2024-09-21T08:24:47.598 INFO:tasks.rados.rados.0.smithi137.stdout:93: oid 3 updating version 15 to 17 2024-09-21T08:24:47.598 INFO:tasks.rados.rados.0.smithi137.stdout:93: oid 3 version 17 is already newer than 16 2024-09-21T08:24:47.598 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 17 (ObjNum 61 snap 4 seq_num 61) dirty exists 2024-09-21T08:24:47.598 INFO:tasks.rados.rados.0.smithi137.stdout:93: left oid 3 (ObjNum 61 snap 4 seq_num 61) 2024-09-21T08:24:47.599 INFO:tasks.rados.rados.0.smithi137.stdout:94: finishing write tid 1 to smithi13779221-43 2024-09-21T08:24:47.599 INFO:tasks.rados.rados.0.smithi137.stdout:95: expect (ObjNum 30 snap 0 seq_num 30) 2024-09-21T08:24:47.617 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:47.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:47.618 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:47.687 INFO:tasks.rados.rados.0.smithi137.stdout:94: finishing write tid 2 to smithi13779221-43 2024-09-21T08:24:47.687 INFO:tasks.rados.rados.0.smithi137.stdout:94: finishing write tid 3 to smithi13779221-43 2024-09-21T08:24:47.687 INFO:tasks.rados.rados.0.smithi137.stdout:94: finishing write tid 4 to smithi13779221-43 2024-09-21T08:24:47.687 INFO:tasks.rados.rados.0.smithi137.stdout:94: oid 43 updating version 0 to 11 2024-09-21T08:24:47.687 INFO:tasks.rados.rados.0.smithi137.stdout:94: oid 43 updating version 11 to 12 2024-09-21T08:24:47.687 INFO:tasks.rados.rados.0.smithi137.stdout:94: oid 43 updating version 12 to 13 2024-09-21T08:24:47.687 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 13 (ObjNum 62 snap 4 seq_num 62) dirty exists 2024-09-21T08:24:47.687 INFO:tasks.rados.rados.0.smithi137.stdout:94: left oid 43 (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:24:47.687 INFO:tasks.rados.rados.0.smithi137.stdout:96: finishing copy_from racing read to smithi13779221-28 2024-09-21T08:24:47.687 INFO:tasks.rados.rados.0.smithi137.stdout:96: finishing copy_from to smithi13779221-28 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:93: done (5 left) 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:94: done (4 left) 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:95: done (3 left) 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:96: done (2 left) 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:100: rollback oid 22 current snap is 4 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:100: done (2 left) 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:101: read oid 25 snap -1 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:101: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:102: snap_create 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:98: finishing write tid 1 to smithi13779221-13 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:98: finishing write tid 2 to smithi13779221-13 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:98: finishing write tid 3 to smithi13779221-13 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:98: finishing write tid 4 to smithi13779221-13 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:98: finishing write tid 5 to smithi13779221-13 2024-09-21T08:24:47.688 INFO:tasks.rados.rados.0.smithi137.stdout:98: oid 13 updating version 0 to 43 2024-09-21T08:24:47.689 INFO:tasks.rados.rados.0.smithi137.stdout:98: oid 13 updating version 43 to 44 2024-09-21T08:24:47.689 INFO:tasks.rados.rados.0.smithi137.stdout:98: oid 13 version 44 is already newer than 42 2024-09-21T08:24:47.689 INFO:tasks.rados.rados.0.smithi137.stdout:98: oid 13 updating version 44 to 45 2024-09-21T08:24:47.689 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 45 (ObjNum 63 snap 4 seq_num 63) dirty exists 2024-09-21T08:24:47.689 INFO:tasks.rados.rados.0.smithi137.stdout:98: left oid 13 (ObjNum 63 snap 4 seq_num 63) 2024-09-21T08:24:47.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:47.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:47.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:47.742 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:47.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[41044]: pgmap v43: 148 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 114 MiB data, 369 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-21T08:24:47.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:24:47.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1043161239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:24:47.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[41044]: osdmap e93: 8 total, 8 up, 7 in 2024-09-21T08:24:47.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[41044]: 5.11 scrub starts 2024-09-21T08:24:47.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[41044]: 5.11 scrub ok 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2444536731' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[41044]: from='client.44759 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[41044]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29611]: 2024-09-21T08:24:47.498+0000 7f1058bb7640 -1 mon.a@0(leader).osd e93 definitely_dead 0 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[29634]: pgmap v43: 148 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 114 MiB data, 369 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1043161239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[29634]: osdmap e93: 8 total, 8 up, 7 in 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[29634]: 5.11 scrub starts 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[29634]: 5.11 scrub ok 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2444536731' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[29634]: from='client.44759 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-21T08:24:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:47 smithi137 ceph-mon[29634]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T08:24:47.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:47.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:47.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:47.865 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:47.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:47.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:47.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:47.988 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:47 smithi165 ceph-mon[32893]: pgmap v43: 148 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 114 MiB data, 369 MiB used, 625 GiB / 626 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-21T08:24:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:47 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:24:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:47 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1043161239' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:24:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:47 smithi165 ceph-mon[32893]: osdmap e93: 8 total, 8 up, 7 in 2024-09-21T08:24:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:47 smithi165 ceph-mon[32893]: 5.11 scrub starts 2024-09-21T08:24:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:47 smithi165 ceph-mon[32893]: 5.11 scrub ok 2024-09-21T08:24:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:47 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2444536731' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-21T08:24:48.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:47 smithi165 ceph-mon[32893]: from='client.44759 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-21T08:24:48.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:47 smithi165 ceph-mon[32893]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T08:24:48.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:48.115 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:48.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:48.116 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:48.119 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:24:48.119 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:24:48.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:24:48.119 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:24:48.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.119 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T08:24:48.119 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:24:48.119 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 211.64333638299999, 2024-09-21T08:24:48.119 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27973036899999998, 2024-09-21T08:24:48.119 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473701+0000", 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1320000000000001e-06 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473716+0000", 2024-09-21T08:24:48.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5622999999999999e-05 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473730+0000", 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3259e-05 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473740+0000", 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0746e-05 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473802+0000", 2024-09-21T08:24:48.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2043000000000002e-05 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473810+0000", 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1500000000000002e-06 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752339+0000", 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27852951199999998 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752485+0000", 2024-09-21T08:24:48.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014628699999999999 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752609+0000", 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012376499999999999 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753145+0000", 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053571000000000001 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753326+0000", 2024-09-21T08:24:48.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018102799999999999 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753367+0000", 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1253e-05 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753400+0000", 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3065999999999998e-05 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753429+0000", 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8795000000000001e-05 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:117 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [setxattr striper.size (16) in=12b,setxattr striper.version (16) in=15b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:17.332236+0000", 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 210.78479949300001, 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072995509999999996, 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.126 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 117 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.332236+0000", 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.332236+0000", 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.332236+0000", 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5600000000000005e-07 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.332240+0000", 2024-09-21T08:24:48.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7019999999999999e-06 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.332246+0000", 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8649999999999998e-06 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.332251+0000", 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7940000000000002e-06 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.337566+0000", 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053151409999999998 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.337636+0000", 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0210999999999996e-05 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.337739+0000", 2024-09-21T08:24:48.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010347499999999999 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.338086+0000", 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000346919 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.338335+0000", 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024842899999999999 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.339485+0000", 2024-09-21T08:24:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001149979 2024-09-21T08:24:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.339513+0000", 2024-09-21T08:24:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8646999999999999e-05 2024-09-21T08:24:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.339535+0000", 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1633000000000001e-05 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:130 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 512~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.430667+0000", 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68636784, 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082449310000000005, 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.133 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 130 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430667+0000", 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430667+0000", 2024-09-21T08:24:48.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430668+0000", 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8100000000000001e-07 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430674+0000", 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.789e-06 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430681+0000", 2024-09-21T08:24:48.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.114e-06 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430686+0000", 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2700000000000004e-06 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431162+0000", 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047528099999999998 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431229+0000", 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7081000000000004e-05 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431324+0000", 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5581000000000005e-05 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432756+0000", 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014314169999999999 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432790+0000", 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4116000000000003e-05 2024-09-21T08:24:48.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.438802+0000", 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0060123939999999999 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.438864+0000", 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2181000000000007e-05 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.438912+0000", 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7825999999999999e-05 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:131 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 516~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.430697+0000", 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68633753500001, 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0083619920000000004, 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.139 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 131 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430697+0000", 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430697+0000", 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430698+0000", 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1200000000000002e-07 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430747+0000", 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9162999999999997e-05 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430753+0000", 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5210000000000002e-06 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430756+0000", 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.461e-06 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431593+0000", 2024-09-21T08:24:48.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00083617000000000001 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431687+0000", 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4613999999999999e-05 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431806+0000", 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000118558 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432822+0000", 2024-09-21T08:24:48.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010164620000000001 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433891+0000", 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010690140000000001 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.438962+0000", 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050712259999999999 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439019+0000", 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6709000000000002e-05 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439059+0000", 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0281999999999998e-05 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:132 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66052~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.430762+0000", 2024-09-21T08:24:48.145 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68627297800001, 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0083799570000000004, 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 132 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430762+0000", 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430762+0000", 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430763+0000", 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7300000000000001e-07 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.147 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430766+0000", 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.596e-06 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430769+0000", 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9550000000000001e-06 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.430772+0000", 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.576e-06 2024-09-21T08:24:48.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432213+0000", 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001441667 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432254+0000", 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1121999999999997e-05 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432337+0000", 2024-09-21T08:24:48.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2552000000000005e-05 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433854+0000", 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0015165879999999999 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.438952+0000", 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005098399 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439069+0000", 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117244 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439110+0000", 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0572999999999999e-05 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439142+0000", 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2212000000000003e-05 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.151 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:133 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66056~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.431143+0000", 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.685892303, 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081278589999999994, 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 133 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.152 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431143+0000", 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431143+0000", 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431144+0000", 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6700000000000001e-06 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431150+0000", 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6579999999999999e-06 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431161+0000", 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.116e-05 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431170+0000", 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5930000000000006e-06 2024-09-21T08:24:48.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432859+0000", 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016894119999999999 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432912+0000", 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3124999999999997e-05 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432977+0000", 2024-09-21T08:24:48.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4523000000000005e-05 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433927+0000", 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00095026599999999996 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439146+0000", 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052193389999999999 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439196+0000", 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9709000000000001e-05 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439238+0000", 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1999999999999998e-05 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439271+0000", 2024-09-21T08:24:48.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2403999999999997e-05 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:134 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 66060~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.431181+0000", 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68585412499999, 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082150079999999993, 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.158 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 134 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431181+0000", 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431181+0000", 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431182+0000", 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1999999999999998e-07 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431244+0000", 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2637000000000004e-05 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431253+0000", 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4759999999999998e-06 2024-09-21T08:24:48.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431261+0000", 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9149999999999996e-06 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433219+0000", 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0019581590000000001 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433292+0000", 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2602999999999996e-05 2024-09-21T08:24:48.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433413+0000", 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000121334 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.436702+0000", 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032895369999999999 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439149+0000", 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024464209999999998 2024-09-21T08:24:48.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439313+0000", 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000163769 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439358+0000", 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4867999999999998e-05 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439396+0000", 2024-09-21T08:24:48.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8368999999999998e-05 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:135 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131596~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.431270+0000", 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68576537800001, 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008249925, 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.164 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 135 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431270+0000", 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431270+0000", 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431270+0000", 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.16e-07 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431277+0000", 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3239999999999999e-06 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431290+0000", 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3144e-05 2024-09-21T08:24:48.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431295+0000", 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3290000000000001e-06 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433955+0000", 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026593789999999999 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.434031+0000", 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6416999999999997e-05 2024-09-21T08:24:48.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.434143+0000", 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011240500000000001 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.436743+0000", 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025998359999999999 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439151+0000", 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024079660000000001 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439434+0000", 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000283035 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439480+0000", 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5367000000000001e-05 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439520+0000", 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9907000000000003e-05 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:136 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131600~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.431301+0000", 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68573412999999, 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0084217849999999993, 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 136 2024-09-21T08:24:48.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431301+0000", 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431301+0000", 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431301+0000", 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8599999999999998e-07 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431307+0000", 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5280000000000003e-06 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431313+0000", 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.049e-06 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431328+0000", 2024-09-21T08:24:48.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5013e-05 2024-09-21T08:24:48.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.434387+0000", 2024-09-21T08:24:48.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030593970000000002 2024-09-21T08:24:48.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.434453+0000", 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5705999999999997e-05 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.434567+0000", 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011351199999999999 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.436781+0000", 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002214461 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439154+0000", 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023726179999999999 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439629+0000", 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047560099999999998 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439684+0000", 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4199000000000001e-05 2024-09-21T08:24:48.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439723+0000", 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9115000000000001e-05 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:137 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 131604~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.431334+0000", 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68570073399999, 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0085770289999999999, 2024-09-21T08:24:48.176 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 137 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431334+0000", 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.177 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431334+0000", 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431335+0000", 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3399999999999999e-07 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431379+0000", 2024-09-21T08:24:48.178 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4526e-05 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431387+0000", 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2130000000000002e-06 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431392+0000", 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1869999999999996e-06 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.434799+0000", 2024-09-21T08:24:48.179 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034075239999999999 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.434874+0000", 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4966e-05 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.434986+0000", 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111618 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.180 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.436819+0000", 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001833229 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439156+0000", 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023368 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439820+0000", 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00066386899999999998 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.181 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439877+0000", 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6684999999999997e-05 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439911+0000", 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4777999999999998e-05 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.182 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:138 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197140~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.431399+0000", 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68563608599999, 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0086774899999999995, 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 138 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.183 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431399+0000", 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431399+0000", 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431400+0000", 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.75e-07 2024-09-21T08:24:48.184 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431405+0000", 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9910000000000002e-06 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431411+0000", 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.981e-06 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431417+0000", 2024-09-21T08:24:48.185 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8730000000000001e-06 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.435901+0000", 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044839570000000002 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.436183+0000", 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028181700000000002 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.186 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.436358+0000", 2024-09-21T08:24:48.187 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017433999999999999 2024-09-21T08:24:48.187 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.187 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439159+0000", 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028015309999999999 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439580+0000", 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042094700000000002 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.188 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439990+0000", 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00041031100000000001 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440041+0000", 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0637000000000003e-05 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440076+0000", 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5429999999999998e-05 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.189 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:139 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197144~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.431423+0000", 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68561188800001, 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088066859999999993, 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 139 2024-09-21T08:24:48.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431423+0000", 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431423+0000", 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431424+0000", 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7299999999999995e-07 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431429+0000", 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2569999999999998e-06 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431435+0000", 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5289999999999999e-06 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431439+0000", 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4580000000000003e-06 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.436857+0000", 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054177979999999997 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.436936+0000", 2024-09-21T08:24:48.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9258999999999994e-05 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.437060+0000", 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123573 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439163+0000", 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021033969999999999 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439771+0000", 2024-09-21T08:24:48.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00060808399999999995 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440148+0000", 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037700199999999998 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440196+0000", 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8063999999999999e-05 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440230+0000", 2024-09-21T08:24:48.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3592e-05 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:140 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 197148~65536] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.431444+0000", 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68559063800001, 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088695430000000006, 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.196 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 140 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431444+0000", 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431444+0000", 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431445+0000", 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5799999999999999e-07 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431504+0000", 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9147000000000001e-05 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.198 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431516+0000", 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2094e-05 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431524+0000", 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6969999999999993e-06 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.437370+0000", 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058459150000000001 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.437445+0000", 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5419000000000004e-05 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.437566+0000", 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120937 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439167+0000", 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016005559999999999 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439949+0000", 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000782535 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440259+0000", 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030997700000000002 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440291+0000", 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1541000000000002e-05 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440314+0000", 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3167e-05 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:141 1.0 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head [write 262684~4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e22)", 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.431532+0000", 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 208.68550345899999, 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088989259999999997, 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.202 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 141 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431532+0000", 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431532+0000", 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431532+0000", 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5099999999999999e-07 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431544+0000", 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1610000000000001e-05 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431550+0000", 2024-09-21T08:24:48.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7470000000000002e-06 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.431554+0000", 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6410000000000001e-06 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.438245+0000", 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066908549999999999 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.438344+0000", 2024-09-21T08:24:48.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9261999999999999e-05 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.438484+0000", 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013951999999999999 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.439279+0000", 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00079506599999999998 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440107+0000", 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082846999999999999 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440353+0000", 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024544299999999997 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440397+0000", 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4042999999999998e-05 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.440430+0000", 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3518e-05 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44417.0:43 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e68)", 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:32.549119+0000", 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.567915806000002, 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093677340000000008, 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.208 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44417", 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 43 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549119+0000", 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549119+0000", 2024-09-21T08:24:48.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549122+0000", 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.717e-06 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549132+0000", 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0522e-05 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549148+0000", 2024-09-21T08:24:48.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5085999999999999e-05 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549160+0000", 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2232000000000001e-05 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549235+0000", 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5610000000000003e-05 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549248+0000", 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2591e-05 2024-09-21T08:24:48.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.558395+0000", 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.009147367 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.558487+0000", 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1608999999999997e-05 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.212 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:43 4.2 4:4347d321:::bucket.sync-source-hints.:head [call,read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e68)", 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:32.549223+0000", 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.567811938000006, 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0091367489999999996, 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 43 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549223+0000", 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549223+0000", 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549225+0000", 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3499999999999999e-06 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549236+0000", 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0475999999999999e-05 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549250+0000", 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3723999999999999e-05 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549259+0000", 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5000000000000005e-06 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549334+0000", 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5253999999999995e-05 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.549340+0000", 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1909999999999999e-06 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.558247+0000", 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0089070529999999998 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.558360+0000", 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113201 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.24394.0:7 2.2 e68/64 2:5cea7035:::rbd_info:head v 68'2, mlcod=0'0)", 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:32.550758+0000", 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.566276798000004, 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077531680000000004, 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.217 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.550758+0000", 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.550758+0000", 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.550760+0000", 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8950000000000001e-06 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.550771+0000", 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0651e-05 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.550784+0000", 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2767e-05 2024-09-21T08:24:48.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.550792+0000", 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6479999999999992e-06 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.550865+0000", 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3016999999999996e-05 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.550888+0000", 2024-09-21T08:24:48.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2846000000000001e-05 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.556684+0000", 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057955919999999996 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:32.558511+0000", 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001827752 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:267 7.b 7:dd97af7e:::smithi13779221-30:head [setxattr _header (56) in=7b,truncate 2974806] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.571814+0000", 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.545221253, 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073730890000000002, 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 267 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.222 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571814+0000", 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571814+0000", 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571815+0000", 2024-09-21T08:24:48.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.519e-06 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571820+0000", 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7609999999999998e-06 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.572424+0000", 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00060438099999999997 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.572435+0000", 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0302999999999999e-05 2024-09-21T08:24:48.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.573121+0000", 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00068617400000000001 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.573388+0000", 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026716499999999998 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:24:48.225 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.573541+0000", 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015310599999999999 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574758+0000", 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012166569999999999 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576435+0000", 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016772600000000001 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579056+0000", 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002621043 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579103+0000", 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7355000000000003e-05 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579187+0000", 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3364999999999999e-05 2024-09-21T08:24:48.227 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:294 7.2 e93/80 7:4d793d31:::smithi13779221-9:head v 93'30, mlcod=91'27)", 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.771697+0000", 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.345338471, 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075355129999999998, 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.771697+0000", 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.771697+0000", 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.771697+0000", 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8400000000000004e-07 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.229 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.773467+0000", 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001770202 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.773489+0000", 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1532999999999999e-05 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.773498+0000", 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9379999999999998e-06 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.773581+0000", 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2619000000000003e-05 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.773605+0000", 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4287e-05 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779183+0000", 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055779189999999998 2024-09-21T08:24:48.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779232+0000", 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9330999999999999e-05 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:324 7.b 7:d7506421:::smithi13779221-13:head [write 658841~471511] snapc 5=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.585914+0000", 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.53112088899999998, 2024-09-21T08:24:48.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013763888, 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 324 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.585914+0000", 2024-09-21T08:24:48.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.585914+0000", 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.585916+0000", 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9099999999999999e-06 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:48.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.586227+0000", 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000311043 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.586239+0000", 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.15e-05 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.586249+0000", 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0713999999999999e-05 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.593696+0000", 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0074462340000000004 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.593795+0000", 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9533999999999999e-05 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.593950+0000", 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015500999999999999 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.595433+0000", 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014831639999999999 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.598311+0000", 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028782439999999999 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.599578+0000", 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001266238 2024-09-21T08:24:48.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.599631+0000", 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3742999999999997e-05 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.599678+0000", 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6554000000000002e-05 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:48.238 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:48.239 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:24:48.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:48.261 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:48.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:48.261 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:48.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:48.387 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:48.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:48.388 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:48.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:48.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:48.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:48.511 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:48.566 INFO:tasks.rados.rados.0.smithi137.stdout:98: done (3 left) 2024-09-21T08:24:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:102: done (2 left) 2024-09-21T08:24:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:103: copy_from oid 27 from oid 33 current snap is 5 2024-09-21T08:24:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:104: read oid 21 snap -1 2024-09-21T08:24:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:104: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-21T08:24:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:105: read oid 49 snap -1 2024-09-21T08:24:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:105: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-21T08:24:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:106: snap_remove snap 4 2024-09-21T08:24:48.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:48.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:48.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:48.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:48.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:48.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:48.764 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:48.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:48.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:48.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:48.887 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:48.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:49.010 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:49.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:49.011 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:48 smithi165 ceph-mon[32893]: 4.8 scrub starts 2024-09-21T08:24:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:48 smithi165 ceph-mon[32893]: 4.8 scrub ok 2024-09-21T08:24:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:48 smithi165 ceph-mon[32893]: 3.d scrub starts 2024-09-21T08:24:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:48 smithi165 ceph-mon[32893]: 3.d scrub ok 2024-09-21T08:24:49.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:48 smithi165 ceph-mon[32893]: from='client.44759 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T08:24:49.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:48 smithi165 ceph-mon[32893]: osdmap e94: 8 total, 8 up, 6 in 2024-09-21T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[29634]: 4.8 scrub starts 2024-09-21T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[29634]: 4.8 scrub ok 2024-09-21T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[29634]: 3.d scrub starts 2024-09-21T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[29634]: 3.d scrub ok 2024-09-21T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[29634]: from='client.44759 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T08:24:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[29634]: osdmap e94: 8 total, 8 up, 6 in 2024-09-21T08:24:49.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[41044]: 4.8 scrub starts 2024-09-21T08:24:49.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[41044]: 4.8 scrub ok 2024-09-21T08:24:49.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[41044]: 3.d scrub starts 2024-09-21T08:24:49.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[41044]: 3.d scrub ok 2024-09-21T08:24:49.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[41044]: from='client.44759 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T08:24:49.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:48 smithi137 ceph-mon[41044]: osdmap e94: 8 total, 8 up, 6 in 2024-09-21T08:24:49.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:49.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:49.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:49.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:49.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:24:49.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:49.256 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:49.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:49.257 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:49.358 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:49.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:49.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:49.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:49.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:49.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:49.505 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:49.567 INFO:tasks.rados.rados.0.smithi137.stdout:106: done (5 left) 2024-09-21T08:24:49.567 INFO:tasks.rados.rados.0.smithi137.stdout:107: read oid 46 snap -1 2024-09-21T08:24:49.567 INFO:tasks.rados.rados.0.smithi137.stdout:107: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-21T08:24:49.567 INFO:tasks.rados.rados.0.smithi137.stdout:108: copy_from oid 35 from oid 29 current snap is 5 2024-09-21T08:24:49.567 INFO:tasks.rados.rados.0.smithi137.stdout:105: expect (ObjNum 48 snap 0 seq_num 48) 2024-09-21T08:24:49.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:49.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:49.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:49.628 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:49.662 INFO:tasks.rados.rados.0.smithi137.stdout:104: expect (ObjNum 20 snap 0 seq_num 20) 2024-09-21T08:24:49.729 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:49.741 INFO:tasks.rados.rados.0.smithi137.stdout:101: expect (ObjNum 24 snap 0 seq_num 24) 2024-09-21T08:24:49.750 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:49.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:49.751 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:49.816 INFO:tasks.rados.rados.0.smithi137.stdout:99: expect (ObjNum 39 snap 0 seq_num 39) 2024-09-21T08:24:49.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:49.862 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[29634]: 2.0 scrub starts 2024-09-21T08:24:49.862 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[29634]: 2.0 scrub ok 2024-09-21T08:24:49.862 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[29634]: pgmap v46: 148 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 114 MiB data, 312 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-21T08:24:49.862 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[29634]: 4.5 scrub starts 2024-09-21T08:24:49.862 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[29634]: osdmap e95: 8 total, 8 up, 6 in 2024-09-21T08:24:49.862 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[29634]: 4.5 scrub ok 2024-09-21T08:24:49.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[41044]: 2.0 scrub starts 2024-09-21T08:24:49.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[41044]: 2.0 scrub ok 2024-09-21T08:24:49.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[41044]: pgmap v46: 148 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 114 MiB data, 312 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-21T08:24:49.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[41044]: 4.5 scrub starts 2024-09-21T08:24:49.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[41044]: osdmap e95: 8 total, 8 up, 6 in 2024-09-21T08:24:49.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:49 smithi137 ceph-mon[41044]: 4.5 scrub ok 2024-09-21T08:24:49.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:49.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:49.874 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:49.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T08:24:49.902 INFO:tasks.rados.rados.0.smithi137.stdout:103: finishing copy_from racing read to smithi13779221-27 2024-09-21T08:24:49.902 INFO:tasks.rados.rados.0.smithi137.stdout:103: finishing copy_from to smithi13779221-27 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 7 (ObjNum 53 snap 2 seq_num 53) dirty exists 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:99: done (6 left) 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:101: done (5 left) 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:103: done (4 left) 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:104: done (3 left) 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:105: done (2 left) 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:109: read oid 16 snap -1 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:109: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:110: copy_from oid 40 from oid 33 current snap is 5 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:108: finishing copy_from racing read to smithi13779221-35 2024-09-21T08:24:49.903 INFO:tasks.rados.rados.0.smithi137.stdout:107: expect (ObjNum 45 snap 0 seq_num 45) 2024-09-21T08:24:49.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:49.992 INFO:tasks.rados.rados.0.smithi137.stdout:108: finishing copy_from to smithi13779221-35 2024-09-21T08:24:49.992 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 33 (ObjNum 59 snap 4 seq_num 59) dirty exists 2024-09-21T08:24:49.992 INFO:tasks.rados.rados.0.smithi137.stdout:107: done (3 left) 2024-09-21T08:24:49.992 INFO:tasks.rados.rados.0.smithi137.stdout:108: done (2 left) 2024-09-21T08:24:49.992 INFO:tasks.rados.rados.0.smithi137.stdout:111: write oid 7 current snap is 5 2024-09-21T08:24:49.992 INFO:tasks.rados.rados.0.smithi137.stdout:109: expect (ObjNum 15 snap 0 seq_num 15) 2024-09-21T08:24:49.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:49.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:49.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:49 smithi165 ceph-mon[32893]: 2.0 scrub starts 2024-09-21T08:24:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:49 smithi165 ceph-mon[32893]: 2.0 scrub ok 2024-09-21T08:24:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:49 smithi165 ceph-mon[32893]: pgmap v46: 148 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 114 MiB data, 312 MiB used, 536 GiB / 536 GiB avail; 11 MiB/s rd, 2.1 MiB/s wr, 18 op/s 2024-09-21T08:24:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:49 smithi165 ceph-mon[32893]: 4.5 scrub starts 2024-09-21T08:24:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:49 smithi165 ceph-mon[32893]: osdmap e95: 8 total, 8 up, 6 in 2024-09-21T08:24:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:49 smithi165 ceph-mon[32893]: 4.5 scrub ok 2024-09-21T08:24:50.070 INFO:tasks.rados.rados.0.smithi137.stdout:110: finishing copy_from racing read to smithi13779221-40 2024-09-21T08:24:50.070 INFO:tasks.rados.rados.0.smithi137.stdout:110: finishing copy_from to smithi13779221-40 2024-09-21T08:24:50.070 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 7 (ObjNum 53 snap 2 seq_num 53) dirty exists 2024-09-21T08:24:50.070 INFO:tasks.rados.rados.0.smithi137.stdout:111: seq_num 64 ranges {675648=546760,2012024=568349} 2024-09-21T08:24:50.077 INFO:tasks.rados.rados.0.smithi137.stdout:111: writing smithi13779221-7 from 675648 to 1222408 tid 1 2024-09-21T08:24:50.086 INFO:tasks.rados.rados.0.smithi137.stdout:111: writing smithi13779221-7 from 2012024 to 2580373 tid 2 2024-09-21T08:24:50.086 INFO:tasks.rados.rados.0.smithi137.stdout:111: finishing write tid 1 to smithi13779221-7 2024-09-21T08:24:50.086 INFO:tasks.rados.rados.0.smithi137.stdout:109: done (2 left) 2024-09-21T08:24:50.086 INFO:tasks.rados.rados.0.smithi137.stdout:110: done (1 left) 2024-09-21T08:24:50.086 INFO:tasks.rados.rados.0.smithi137.stdout:112: copy_from oid 28 from oid 34 current snap is 5 2024-09-21T08:24:50.086 INFO:tasks.rados.rados.0.smithi137.stdout:113: write (excl) oid 48 current snap is 5 2024-09-21T08:24:50.086 INFO:tasks.rados.rados.0.smithi137.stdout:113: seq_num 65 ranges {692455=622009,2070506=712007} 2024-09-21T08:24:50.088 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:24:50.088 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:24:50.093 INFO:tasks.rados.rados.0.smithi137.stdout:113: writing smithi13779221-48 from 692455 to 1314464 tid 1 2024-09-21T08:24:50.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:50.102 INFO:tasks.rados.rados.0.smithi137.stdout:113: writing smithi13779221-48 from 2070506 to 2782513 tid 2 2024-09-21T08:24:50.102 INFO:tasks.rados.rados.0.smithi137.stdout:112: finishing copy_from to smithi13779221-28 2024-09-21T08:24:50.102 INFO:tasks.rados.rados.0.smithi137.stdout:112: got expected ENOENT (src dne) 2024-09-21T08:24:50.102 INFO:tasks.rados.rados.0.smithi137.stdout:112: finishing copy_from racing read to smithi13779221-28 2024-09-21T08:24:50.102 INFO:tasks.rados.rados.0.smithi137.stdout:111: finishing write tid 2 to smithi13779221-7 2024-09-21T08:24:50.102 INFO:tasks.rados.rados.0.smithi137.stdout:111: finishing write tid 3 to smithi13779221-7 2024-09-21T08:24:50.102 INFO:tasks.rados.rados.0.smithi137.stdout:111: finishing write tid 4 to smithi13779221-7 2024-09-21T08:24:50.102 INFO:tasks.rados.rados.0.smithi137.stdout:111: oid 7 updating version 0 to 49 2024-09-21T08:24:50.102 INFO:tasks.rados.rados.0.smithi137.stdout:111: oid 7 version 49 is already newer than 48 2024-09-21T08:24:50.103 INFO:tasks.rados.rados.0.smithi137.stdout:111: oid 7 updating version 49 to 50 2024-09-21T08:24:50.103 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 50 (ObjNum 64 snap 5 seq_num 64) dirty exists 2024-09-21T08:24:50.103 INFO:tasks.rados.rados.0.smithi137.stdout:111: left oid 7 (ObjNum 64 snap 5 seq_num 64) 2024-09-21T08:24:50.103 INFO:tasks.rados.rados.0.smithi137.stdout:113: finishing write tid 1 to smithi13779221-48 2024-09-21T08:24:50.103 INFO:tasks.rados.rados.0.smithi137.stdout:111: done (2 left) 2024-09-21T08:24:50.103 INFO:tasks.rados.rados.0.smithi137.stdout:112: done (1 left) 2024-09-21T08:24:50.103 INFO:tasks.rados.rados.0.smithi137.stdout:114: snap_create 2024-09-21T08:24:50.105 INFO:tasks.rados.rados.0.smithi137.stdout:113: finishing write tid 2 to smithi13779221-48 2024-09-21T08:24:50.106 INFO:tasks.rados.rados.0.smithi137.stdout:113: finishing write tid 3 to smithi13779221-48 2024-09-21T08:24:50.107 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:24:50.107 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:24:50.107 INFO:tasks.rados.rados.0.smithi137.stdout:113: finishing write tid 4 to smithi13779221-48 2024-09-21T08:24:50.107 INFO:tasks.rados.rados.0.smithi137.stdout:113: oid 48 updating version 0 to 35 2024-09-21T08:24:50.107 INFO:tasks.rados.rados.0.smithi137.stdout:113: oid 48 version 35 is already newer than 34 2024-09-21T08:24:50.107 INFO:tasks.rados.rados.0.smithi137.stdout:113: oid 48 version 35 is already newer than 33 2024-09-21T08:24:50.107 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 35 (ObjNum 65 snap 5 seq_num 65) dirty exists 2024-09-21T08:24:50.108 INFO:tasks.rados.rados.0.smithi137.stdout:113: left oid 48 (ObjNum 65 snap 5 seq_num 65) 2024-09-21T08:24:50.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:50.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:50.120 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:50.128 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:50.137 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:24:50.137 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:24:50.160 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:24:50.160 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:24:50.177 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:24:50.177 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:24:50.193 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:24:50.194 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:24:50.210 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:24:50.210 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:24:50.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:50.226 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:24:50.226 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:24:50.241 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:50.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:50.242 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:50.343 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:50.364 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:50.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:50.365 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:50.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:50.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:50.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:50.487 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:50.577 INFO:tasks.rados.rados.0.smithi137.stdout:113: done (1 left) 2024-09-21T08:24:50.578 INFO:tasks.rados.rados.0.smithi137.stdout:114: done (0 left) 2024-09-21T08:24:50.578 INFO:tasks.rados.rados.0.smithi137.stdout:115: snap_create 2024-09-21T08:24:50.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:50.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:50.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:50.612 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:50.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[29634]: 6.7 scrub starts 2024-09-21T08:24:50.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[29634]: 6.7 scrub ok 2024-09-21T08:24:50.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[29634]: 3.9 scrub starts 2024-09-21T08:24:50.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[29634]: 3.9 scrub ok 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[29634]: osdmap e96: 8 total, 8 up, 6 in 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/487974102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[41044]: 6.7 scrub starts 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[41044]: 6.7 scrub ok 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[41044]: 3.9 scrub starts 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[41044]: 3.9 scrub ok 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[41044]: osdmap e96: 8 total, 8 up, 6 in 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:24:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:24:50.680 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/487974102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:50.713 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:50.734 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:50.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:50.735 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:50.836 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:50.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:50.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:50.858 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:50.948 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:50.959 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:50.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:50.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:50.980 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:50 smithi165 ceph-mon[32893]: 6.7 scrub starts 2024-09-21T08:24:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:50 smithi165 ceph-mon[32893]: 6.7 scrub ok 2024-09-21T08:24:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:50 smithi165 ceph-mon[32893]: 3.9 scrub starts 2024-09-21T08:24:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:50 smithi165 ceph-mon[32893]: 3.9 scrub ok 2024-09-21T08:24:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:50 smithi165 ceph-mon[32893]: osdmap e96: 8 total, 8 up, 6 in 2024-09-21T08:24:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:50 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:24:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:50 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:24:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:50 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/487974102' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:51.017 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:50 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:24:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:24:51.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:51.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:51.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:51.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:51.205 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:51.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:51.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:51.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:51.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:51.349 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:51.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:51.350 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:51.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:51.473 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:51.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:51.474 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:51.488 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:24:51.489 INFO:tasks.rados.rados.0.smithi137.stdout:115: done (0 left) 2024-09-21T08:24:51.489 INFO:tasks.rados.rados.0.smithi137.stdout:116: delete oid 39 current snap is 7 2024-09-21T08:24:51.496 INFO:tasks.rados.rados.0.smithi137.stdout:116: done (0 left) 2024-09-21T08:24:51.496 INFO:tasks.rados.rados.0.smithi137.stdout:117: snap_remove snap 6 2024-09-21T08:24:51.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:51.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:51.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:51.598 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: pgmap v49: 148 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 114 MiB data, 312 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: 4.18 scrub starts 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: 4.18 scrub ok 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: osdmap e97: 8 total, 8 up, 6 in 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: 3.7 scrub starts 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: 3.7 scrub ok 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: 2.1 scrub starts 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: 2.1 scrub ok 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4123100278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: from='client.44771 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:24:51.652 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: from='client.44771 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[29634]: osdmap e98: 8 total, 8 up, 6 in 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: pgmap v49: 148 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 114 MiB data, 312 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: 4.18 scrub starts 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: 4.18 scrub ok 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: osdmap e97: 8 total, 8 up, 6 in 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: 3.7 scrub starts 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: 3.7 scrub ok 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: 2.1 scrub starts 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: 2.1 scrub ok 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4123100278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: from='client.44771 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:51.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:24:51.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: from='client.44771 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:24:51.654 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:51 smithi137 ceph-mon[41044]: osdmap e98: 8 total, 8 up, 6 in 2024-09-21T08:24:51.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:51.720 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:51.721 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:51.721 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:51.822 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:51.844 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:51.844 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:51.844 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:51.891 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:24:51.892 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:24:51.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:51.966 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:51.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:51.967 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:52.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:24:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: pgmap v49: 148 pgs: 2 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 142 active+clean; 114 MiB data, 312 MiB used, 536 GiB / 536 GiB avail 2024-09-21T08:24:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: 4.18 scrub starts 2024-09-21T08:24:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: 4.18 scrub ok 2024-09-21T08:24:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: osdmap e97: 8 total, 8 up, 6 in 2024-09-21T08:24:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: 3.7 scrub starts 2024-09-21T08:24:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: 3.7 scrub ok 2024-09-21T08:24:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: 2.1 scrub starts 2024-09-21T08:24:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: 2.1 scrub ok 2024-09-21T08:24:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4123100278' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: from='client.44771 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:24:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:24:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: from='client.44771 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:24:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:51 smithi165 ceph-mon[32893]: osdmap e98: 8 total, 8 up, 6 in 2024-09-21T08:24:52.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:52.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:52.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:52.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:52.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:52.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:52.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:52.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:52.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:52.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:52.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:52.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:52.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:52.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:52.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:52.459 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:52.498 INFO:tasks.rados.rados.0.smithi137.stdout:117: done (0 left) 2024-09-21T08:24:52.499 INFO:tasks.rados.rados.0.smithi137.stdout:118: read oid 14 snap 5 2024-09-21T08:24:52.499 INFO:tasks.rados.rados.0.smithi137.stdout:118: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-21T08:24:52.499 INFO:tasks.rados.rados.0.smithi137.stdout:119: write oid 47 current snap is 7 2024-09-21T08:24:52.499 INFO:tasks.rados.rados.0.smithi137.stdout:119: seq_num 67 ranges {726069=772507,2071117=715676} 2024-09-21T08:24:52.507 INFO:tasks.rados.rados.0.smithi137.stdout:119: writing smithi13779221-47 from 726069 to 1498576 tid 1 2024-09-21T08:24:52.507 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:24:52 smithi137 bash[79543]: Error: initializing source docker://quay.io/prometheus/node-exporter:v1.5.0: can't talk to a V1 container registry 2024-09-21T08:24:52.507 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:24:52 smithi137 podman[79543]: 2024-09-21 08:24:52.169662777 +0000 UTC m=+11.724604204 image pull-error quay.io/prometheus/node-exporter:v1.5.0 initializing source docker://quay.io/prometheus/node-exporter:v1.5.0: can't talk to a V1 container registry 2024-09-21T08:24:52.507 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:24:52 smithi137 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service: Control process exited, code=exited, status=125/n/a 2024-09-21T08:24:52.515 INFO:tasks.rados.rados.0.smithi137.stdout:119: writing smithi13779221-47 from 2071117 to 2786793 tid 2 2024-09-21T08:24:52.515 INFO:tasks.rados.rados.0.smithi137.stdout:120: read oid 13 snap -1 2024-09-21T08:24:52.515 INFO:tasks.rados.rados.0.smithi137.stdout:120: expect (ObjNum 63 snap 4 seq_num 63) 2024-09-21T08:24:52.515 INFO:tasks.rados.rados.0.smithi137.stdout:121: write (excl) oid 30 current snap is 7 2024-09-21T08:24:52.515 INFO:tasks.rados.rados.0.smithi137.stdout:121: seq_num 68 ranges {742876=447756,1729599=575687,3101234=41642} 2024-09-21T08:24:52.522 INFO:tasks.rados.rados.0.smithi137.stdout:121: writing smithi13779221-30 from 742876 to 1190632 tid 1 2024-09-21T08:24:52.528 INFO:tasks.rados.rados.0.smithi137.stdout:121: writing smithi13779221-30 from 1729599 to 2305286 tid 2 2024-09-21T08:24:52.533 INFO:tasks.rados.rados.0.smithi137.stdout:121: writing smithi13779221-30 from 3101234 to 3142876 tid 3 2024-09-21T08:24:52.534 INFO:tasks.rados.rados.0.smithi137.stdout:122: write (excl) oid 39 current snap is 7 2024-09-21T08:24:52.534 INFO:tasks.rados.rados.0.smithi137.stdout:122: seq_num 69 ranges {759683=639358,1904434=719345,3128657=31026} 2024-09-21T08:24:52.541 INFO:tasks.rados.rados.0.smithi137.stdout:122: writing smithi13779221-39 from 759683 to 1399041 tid 1 2024-09-21T08:24:52.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T08:24:52.548 INFO:tasks.rados.rados.0.smithi137.stdout:122: writing smithi13779221-39 from 1904434 to 2623779 tid 2 2024-09-21T08:24:52.551 INFO:tasks.rados.rados.0.smithi137.stdout:122: writing smithi13779221-39 from 3128657 to 3159683 tid 3 2024-09-21T08:24:52.552 INFO:tasks.rados.rados.0.smithi137.stdout:123: write oid 36 current snap is 7 2024-09-21T08:24:52.552 INFO:tasks.rados.rados.0.smithi137.stdout:118: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-21T08:24:52.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:52.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:52.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:52.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:52.629 INFO:tasks.rados.rados.0.smithi137.stdout:119: finishing write tid 1 to smithi13779221-47 2024-09-21T08:24:52.629 INFO:tasks.rados.rados.0.smithi137.stdout:123: seq_num 70 ranges {776490=714607,2246563=579356} 2024-09-21T08:24:52.637 INFO:tasks.rados.rados.0.smithi137.stdout:123: writing smithi13779221-36 from 776490 to 1491097 tid 1 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:123: writing smithi13779221-36 from 2246563 to 2825919 tid 2 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:118: done (5 left) 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:124: rollback oid 3 current snap is 7 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 3 to 5 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:119: finishing write tid 2 to smithi13779221-47 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:119: finishing write tid 3 to smithi13779221-47 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:119: finishing write tid 4 to smithi13779221-47 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:119: oid 47 updating version 0 to 37 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:119: oid 47 updating version 37 to 38 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:119: oid 47 updating version 38 to 39 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 39 (ObjNum 67 snap 7 seq_num 67) dirty exists 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:119: left oid 47 (ObjNum 67 snap 7 seq_num 67) 2024-09-21T08:24:52.645 INFO:tasks.rados.rados.0.smithi137.stdout:120: expect (ObjNum 63 snap 4 seq_num 63) 2024-09-21T08:24:52.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:52.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:52.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:52.706 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:52.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[29634]: 5.1f deep-scrub starts 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[29634]: 5.1f deep-scrub ok 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[29634]: 5.d scrub starts 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[29634]: 5.d scrub ok 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[29634]: 3.18 scrub starts 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[29634]: 3.18 scrub ok 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[29634]: osdmap e99: 8 total, 8 up, 6 in 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[41044]: 5.1f deep-scrub starts 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[41044]: 5.1f deep-scrub ok 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[41044]: 5.d scrub starts 2024-09-21T08:24:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[41044]: 5.d scrub ok 2024-09-21T08:24:52.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[41044]: 3.18 scrub starts 2024-09-21T08:24:52.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[41044]: 3.18 scrub ok 2024-09-21T08:24:52.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:52 smithi137 ceph-mon[41044]: osdmap e99: 8 total, 8 up, 6 in 2024-09-21T08:24:52.783 INFO:tasks.rados.rados.0.smithi137.stdout:121: finishing write tid 1 to smithi13779221-30 2024-09-21T08:24:52.783 INFO:tasks.rados.rados.0.smithi137.stdout:121: finishing write tid 2 to smithi13779221-30 2024-09-21T08:24:52.783 INFO:tasks.rados.rados.0.smithi137.stdout:121: finishing write tid 3 to smithi13779221-30 2024-09-21T08:24:52.783 INFO:tasks.rados.rados.0.smithi137.stdout:121: finishing write tid 4 to smithi13779221-30 2024-09-21T08:24:52.784 INFO:tasks.rados.rados.0.smithi137.stdout:121: finishing write tid 5 to smithi13779221-30 2024-09-21T08:24:52.784 INFO:tasks.rados.rados.0.smithi137.stdout:121: oid 30 updating version 0 to 55 2024-09-21T08:24:52.784 INFO:tasks.rados.rados.0.smithi137.stdout:121: oid 30 version 55 is already newer than 53 2024-09-21T08:24:52.784 INFO:tasks.rados.rados.0.smithi137.stdout:121: oid 30 version 55 is already newer than 54 2024-09-21T08:24:52.784 INFO:tasks.rados.rados.0.smithi137.stdout:121: oid 30 version 55 is already newer than 52 2024-09-21T08:24:52.784 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 55 (ObjNum 68 snap 7 seq_num 68) dirty exists 2024-09-21T08:24:52.784 INFO:tasks.rados.rados.0.smithi137.stdout:121: left oid 30 (ObjNum 68 snap 7 seq_num 68) 2024-09-21T08:24:52.785 INFO:tasks.rados.rados.0.smithi137.stdout:122: finishing write tid 1 to smithi13779221-39 2024-09-21T08:24:52.785 INFO:tasks.rados.rados.0.smithi137.stdout:122: finishing write tid 2 to smithi13779221-39 2024-09-21T08:24:52.785 INFO:tasks.rados.rados.0.smithi137.stdout:122: finishing write tid 3 to smithi13779221-39 2024-09-21T08:24:52.785 INFO:tasks.rados.rados.0.smithi137.stdout:119: done (5 left) 2024-09-21T08:24:52.785 INFO:tasks.rados.rados.0.smithi137.stdout:120: done (4 left) 2024-09-21T08:24:52.785 INFO:tasks.rados.rados.0.smithi137.stdout:121: done (3 left) 2024-09-21T08:24:52.785 INFO:tasks.rados.rados.0.smithi137.stdout:125: delete oid 26 current snap is 7 2024-09-21T08:24:52.786 INFO:tasks.rados.rados.0.smithi137.stdout:122: finishing write tid 4 to smithi13779221-39 2024-09-21T08:24:52.786 INFO:tasks.rados.rados.0.smithi137.stdout:122: finishing write tid 5 to smithi13779221-39 2024-09-21T08:24:52.786 INFO:tasks.rados.rados.0.smithi137.stdout:122: oid 39 updating version 0 to 8 2024-09-21T08:24:52.786 INFO:tasks.rados.rados.0.smithi137.stdout:122: oid 39 updating version 8 to 10 2024-09-21T08:24:52.786 INFO:tasks.rados.rados.0.smithi137.stdout:122: oid 39 version 10 is already newer than 9 2024-09-21T08:24:52.786 INFO:tasks.rados.rados.0.smithi137.stdout:122: oid 39 updating version 10 to 11 2024-09-21T08:24:52.786 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 11 (ObjNum 69 snap 7 seq_num 69) dirty exists 2024-09-21T08:24:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:122: left oid 39 (ObjNum 69 snap 7 seq_num 69) 2024-09-21T08:24:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:123: finishing write tid 1 to smithi13779221-36 2024-09-21T08:24:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:124: finishing rollback tid 0 to smithi13779221-3 2024-09-21T08:24:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:124: finishing rollback tid 1 to smithi13779221-3 2024-09-21T08:24:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:124: finishing rollback tid 2 to smithi13779221-3 2024-09-21T08:24:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 23 (ObjNum 61 snap 4 seq_num 61) dirty exists 2024-09-21T08:24:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:123: finishing write tid 2 to smithi13779221-36 2024-09-21T08:24:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:123: finishing write tid 3 to smithi13779221-36 2024-09-21T08:24:52.788 INFO:tasks.rados.rados.0.smithi137.stdout:123: finishing write tid 4 to smithi13779221-36 2024-09-21T08:24:52.788 INFO:tasks.rados.rados.0.smithi137.stdout:123: oid 36 updating version 0 to 59 2024-09-21T08:24:52.788 INFO:tasks.rados.rados.0.smithi137.stdout:123: oid 36 version 59 is already newer than 57 2024-09-21T08:24:52.788 INFO:tasks.rados.rados.0.smithi137.stdout:123: oid 36 version 59 is already newer than 58 2024-09-21T08:24:52.788 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 59 (ObjNum 70 snap 7 seq_num 70) dirty exists 2024-09-21T08:24:52.788 INFO:tasks.rados.rados.0.smithi137.stdout:123: left oid 36 (ObjNum 70 snap 7 seq_num 70) 2024-09-21T08:24:52.789 INFO:tasks.rados.rados.0.smithi137.stdout:122: done (3 left) 2024-09-21T08:24:52.789 INFO:tasks.rados.rados.0.smithi137.stdout:123: done (2 left) 2024-09-21T08:24:52.789 INFO:tasks.rados.rados.0.smithi137.stdout:124: done (1 left) 2024-09-21T08:24:52.790 INFO:tasks.rados.rados.0.smithi137.stdout:125: done (0 left) 2024-09-21T08:24:52.790 INFO:tasks.rados.rados.0.smithi137.stdout:126: write oid 17 current snap is 7 2024-09-21T08:24:52.790 INFO:tasks.rados.rados.0.smithi137.stdout:126: seq_num 72 ranges {410104=465105,1563527=583025,2810926=399178} 2024-09-21T08:24:52.790 INFO:tasks.rados.rados.0.smithi137.stdout:126: writing smithi13779221-17 from 410104 to 875209 tid 1 2024-09-21T08:24:52.795 INFO:tasks.rados.rados.0.smithi137.stdout:126: writing smithi13779221-17 from 1563527 to 2146552 tid 2 2024-09-21T08:24:52.802 INFO:tasks.rados.rados.0.smithi137.stdout:126: writing smithi13779221-17 from 2810926 to 3210104 tid 3 2024-09-21T08:24:52.802 INFO:tasks.rados.rados.0.smithi137.stdout:127: rollback oid 44 current snap is 7 2024-09-21T08:24:52.802 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 44 to 5 2024-09-21T08:24:52.802 INFO:tasks.rados.rados.0.smithi137.stdout:126: finishing write tid 1 to smithi13779221-17 2024-09-21T08:24:52.802 INFO:tasks.rados.rados.0.smithi137.stdout:126: finishing write tid 2 to smithi13779221-17 2024-09-21T08:24:52.802 INFO:tasks.rados.rados.0.smithi137.stdout:128: rollback oid 32 current snap is 7 2024-09-21T08:24:52.802 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 32 to 5 2024-09-21T08:24:52.803 INFO:tasks.rados.rados.0.smithi137.stdout:129: write (excl) oid 31 current snap is 7 2024-09-21T08:24:52.803 INFO:tasks.rados.rados.0.smithi137.stdout:129: seq_num 73 ranges {426911=540354,1622009=726683,3121996=104915} 2024-09-21T08:24:52.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:52.807 INFO:tasks.rados.rados.0.smithi137.stdout:129: writing smithi13779221-31 from 426911 to 967265 tid 1 2024-09-21T08:24:52.814 INFO:tasks.rados.rados.0.smithi137.stdout:129: writing smithi13779221-31 from 1622009 to 2348692 tid 2 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:129: writing smithi13779221-31 from 3121996 to 3226911 tid 3 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:130: copy_from oid 1 from oid 23 current snap is 7 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:126: finishing write tid 3 to smithi13779221-17 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:128: finishing rollback tid 0 to smithi13779221-32 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:128: finishing rollback tid 1 to smithi13779221-32 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:128: finishing rollback tid 2 to smithi13779221-32 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 10 (ObjNum 31 snap 0 seq_num 31) dirty exists 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:126: finishing write tid 4 to smithi13779221-17 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:126: finishing write tid 5 to smithi13779221-17 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:126: oid 17 updating version 0 to 34 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:126: oid 17 updating version 34 to 36 2024-09-21T08:24:52.821 INFO:tasks.rados.rados.0.smithi137.stdout:126: oid 17 updating version 36 to 37 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:126: oid 17 version 37 is already newer than 35 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 37 (ObjNum 72 snap 7 seq_num 72) dirty exists 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:126: left oid 17 (ObjNum 72 snap 7 seq_num 72) 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:127: finishing rollback tid 0 to smithi13779221-44 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:127: finishing rollback tid 1 to smithi13779221-44 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:127: finishing rollback tid 2 to smithi13779221-44 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 40 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:129: finishing write tid 1 to smithi13779221-31 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:129: finishing write tid 2 to smithi13779221-31 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:126: done (4 left) 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:127: done (3 left) 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:128: done (2 left) 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:131: read oid 48 snap -1 2024-09-21T08:24:52.822 INFO:tasks.rados.rados.0.smithi137.stdout:131: expect (ObjNum 65 snap 5 seq_num 65) 2024-09-21T08:24:52.823 INFO:tasks.rados.rados.0.smithi137.stdout:132: snap_remove snap 5 2024-09-21T08:24:52.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:52.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:52.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:52.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:52 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:24:52.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:52 smithi165 ceph-mon[32893]: 5.1f deep-scrub starts 2024-09-21T08:24:52.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:52 smithi165 ceph-mon[32893]: 5.1f deep-scrub ok 2024-09-21T08:24:52.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:52 smithi165 ceph-mon[32893]: 5.d scrub starts 2024-09-21T08:24:52.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:52 smithi165 ceph-mon[32893]: 5.d scrub ok 2024-09-21T08:24:52.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:52 smithi165 ceph-mon[32893]: 3.18 scrub starts 2024-09-21T08:24:52.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:52 smithi165 ceph-mon[32893]: 3.18 scrub ok 2024-09-21T08:24:52.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:52 smithi165 ceph-mon[32893]: osdmap e99: 8 total, 8 up, 6 in 2024-09-21T08:24:52.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:52.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:52.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:52.955 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:53.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:53.078 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:53.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:53.079 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:53.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:53.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:53.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:53.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:53.242 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:53.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:53.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:53.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:53.326 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:53.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:53.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:53.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:53.450 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:53.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:53.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:53.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:53.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:53.585 INFO:tasks.rados.rados.0.smithi137.stdout:132: done (3 left) 2024-09-21T08:24:53.585 INFO:tasks.rados.rados.0.smithi137.stdout:133: rollback oid 27 current snap is 7 2024-09-21T08:24:53.585 INFO:tasks.rados.rados.0.smithi137.stdout:133: done (3 left) 2024-09-21T08:24:53.585 INFO:tasks.rados.rados.0.smithi137.stdout:134: snap_create 2024-09-21T08:24:53.586 INFO:tasks.rados.rados.0.smithi137.stdout:129: finishing write tid 3 to smithi13779221-31 2024-09-21T08:24:53.586 INFO:tasks.rados.rados.0.smithi137.stdout:129: finishing write tid 4 to smithi13779221-31 2024-09-21T08:24:53.586 INFO:tasks.rados.rados.0.smithi137.stdout:129: finishing write tid 5 to smithi13779221-31 2024-09-21T08:24:53.586 INFO:tasks.rados.rados.0.smithi137.stdout:129: oid 31 updating version 0 to 25 2024-09-21T08:24:53.586 INFO:tasks.rados.rados.0.smithi137.stdout:129: oid 31 updating version 25 to 27 2024-09-21T08:24:53.586 INFO:tasks.rados.rados.0.smithi137.stdout:129: oid 31 version 27 is already newer than 24 2024-09-21T08:24:53.586 INFO:tasks.rados.rados.0.smithi137.stdout:129: oid 31 version 27 is already newer than 26 2024-09-21T08:24:53.586 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 27 (ObjNum 73 snap 7 seq_num 73) dirty exists 2024-09-21T08:24:53.586 INFO:tasks.rados.rados.0.smithi137.stdout:129: left oid 31 (ObjNum 73 snap 7 seq_num 73) 2024-09-21T08:24:53.586 INFO:tasks.rados.rados.0.smithi137.stdout:131: expect (ObjNum 65 snap 5 seq_num 65) 2024-09-21T08:24:53.647 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[29634]: pgmap v52: 148 pgs: 2 active+clean+snaptrim, 146 active+clean; 118 MiB data, 398 MiB used, 536 GiB / 536 GiB avail; 25 MiB/s rd, 7.4 MiB/s wr, 37 op/s; 9.8 MiB/s, 30 objects/s recovering 2024-09-21T08:24:53.647 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[29634]: 3.4 scrub starts 2024-09-21T08:24:53.647 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[29634]: 3.4 scrub ok 2024-09-21T08:24:53.647 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[29634]: 5.18 scrub starts 2024-09-21T08:24:53.647 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[29634]: 5.18 scrub ok 2024-09-21T08:24:53.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[29634]: 4.11 scrub starts 2024-09-21T08:24:53.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[29634]: 4.11 scrub ok 2024-09-21T08:24:53.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[41044]: pgmap v52: 148 pgs: 2 active+clean+snaptrim, 146 active+clean; 118 MiB data, 398 MiB used, 536 GiB / 536 GiB avail; 25 MiB/s rd, 7.4 MiB/s wr, 37 op/s; 9.8 MiB/s, 30 objects/s recovering 2024-09-21T08:24:53.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[41044]: 3.4 scrub starts 2024-09-21T08:24:53.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[41044]: 3.4 scrub ok 2024-09-21T08:24:53.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[41044]: 5.18 scrub starts 2024-09-21T08:24:53.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[41044]: 5.18 scrub ok 2024-09-21T08:24:53.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[41044]: 4.11 scrub starts 2024-09-21T08:24:53.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-mon[41044]: 4.11 scrub ok 2024-09-21T08:24:53.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:53.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:53.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:53.696 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:53.724 INFO:tasks.rados.rados.0.smithi137.stdout:130: finishing copy_from racing read to smithi13779221-1 2024-09-21T08:24:53.724 INFO:tasks.rados.rados.0.smithi137.stdout:130: finishing copy_from to smithi13779221-1 2024-09-21T08:24:53.724 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 6 (ObjNum 22 snap 0 seq_num 22) dirty exists 2024-09-21T08:24:53.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:53.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:53.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:53.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:53.885 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:24:53.886 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:24:53.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:53.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:24:53.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:53.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:53.942 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:53 smithi165 ceph-mon[32893]: pgmap v52: 148 pgs: 2 active+clean+snaptrim, 146 active+clean; 118 MiB data, 398 MiB used, 536 GiB / 536 GiB avail; 25 MiB/s rd, 7.4 MiB/s wr, 37 op/s; 9.8 MiB/s, 30 objects/s recovering 2024-09-21T08:24:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:53 smithi165 ceph-mon[32893]: 3.4 scrub starts 2024-09-21T08:24:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:53 smithi165 ceph-mon[32893]: 3.4 scrub ok 2024-09-21T08:24:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:53 smithi165 ceph-mon[32893]: 5.18 scrub starts 2024-09-21T08:24:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:53 smithi165 ceph-mon[32893]: 5.18 scrub ok 2024-09-21T08:24:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:53 smithi165 ceph-mon[32893]: 4.11 scrub starts 2024-09-21T08:24:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:53 smithi165 ceph-mon[32893]: 4.11 scrub ok 2024-09-21T08:24:54.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:54.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:54.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:54.065 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:54.074 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 6, 7] out_osds: [4, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:24:54.074 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:24:54.075 INFO:tasks.thrashosds.thrasher:Removing osd 6, in_osds are: [0, 1, 2, 5, 6, 7] 2024-09-21T08:24:54.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd out 6 2024-09-21T08:24:54.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:54.178 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:24:53 smithi137 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T08:24:54.178 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:24:53 smithi137 systemd[1]: Failed to start Ceph node-exporter.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:24:54.179 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:24:53.941+0000 7f534001e640 -1 log_channel(cephadm) log [ERR] : Failed while placing node-exporter.a on smithi137: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T08:24:54.179 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T08:24:54.179 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter.a 2024-09-21T08:24:54.179 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter.a 2024-09-21T08:24:54.179 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Deploy daemon node-exporter.a ... 2024-09-21T08:24:54.179 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Verifying port 0.0.0.0:9100 ... 2024-09-21T08:24:54.179 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Non-zero exit code 1 from systemctl start ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: systemctl: stderr Job for ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service failed because the control process exited with error code. 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: systemctl: stderr See "systemctl status ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" and "journalctl -xeu ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" for details. 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: Traceback (most recent call last): 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: return _run_code(code, main_globals, None, 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: exec(code, run_globals) 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-21T08:24:54.180 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-21T08:24:54.181 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-21T08:24:54.181 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1088, in deploy_daemon 2024-09-21T08:24:54.181 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1224, in deploy_daemon_units 2024-09-21T08:24:54.181 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T08:24:54.181 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: RuntimeError: Failed command: systemctl start ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a: Job for ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service failed because the control process exited with error code. 2024-09-21T08:24:54.181 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:24:53 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: See "systemctl status ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" and "journalctl -xeu ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" for details. 2024-09-21T08:24:54.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:54.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:54.188 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:54.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:54.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:54.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:54.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:54.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:54.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:54.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:54.436 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:54.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:54.563 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:54.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:54.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:54.594 INFO:tasks.rados.rados.0.smithi137.stdout:129: done (3 left) 2024-09-21T08:24:54.595 INFO:tasks.rados.rados.0.smithi137.stdout:130: done (2 left) 2024-09-21T08:24:54.595 INFO:tasks.rados.rados.0.smithi137.stdout:131: done (1 left) 2024-09-21T08:24:54.595 INFO:tasks.rados.rados.0.smithi137.stdout:134: done (0 left) 2024-09-21T08:24:54.595 INFO:tasks.rados.rados.0.smithi137.stdout:135: snap_create 2024-09-21T08:24:54.665 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:54.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T08:24:54.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:54.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:54.688 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:54.757 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:54.789 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:54.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:54.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:54.811 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:54.880 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:54 smithi137 ceph-mon[29634]: osdmap e100: 8 total, 8 up, 6 in 2024-09-21T08:24:54.880 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:54 smithi137 ceph-mon[29634]: 6.8 scrub starts 2024-09-21T08:24:54.880 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:54 smithi137 ceph-mon[29634]: 6.8 scrub ok 2024-09-21T08:24:54.880 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:54 smithi137 ceph-mon[29634]: 6.14 scrub starts 2024-09-21T08:24:54.880 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:54 smithi137 ceph-mon[29634]: 6.14 scrub ok 2024-09-21T08:24:54.880 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:54 smithi137 ceph-mon[41044]: osdmap e100: 8 total, 8 up, 6 in 2024-09-21T08:24:54.881 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:54 smithi137 ceph-mon[41044]: 6.8 scrub starts 2024-09-21T08:24:54.881 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:54 smithi137 ceph-mon[41044]: 6.8 scrub ok 2024-09-21T08:24:54.881 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:54 smithi137 ceph-mon[41044]: 6.14 scrub starts 2024-09-21T08:24:54.881 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:54 smithi137 ceph-mon[41044]: 6.14 scrub ok 2024-09-21T08:24:54.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:54 smithi165 ceph-mon[32893]: osdmap e100: 8 total, 8 up, 6 in 2024-09-21T08:24:54.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:54 smithi165 ceph-mon[32893]: 6.8 scrub starts 2024-09-21T08:24:54.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:54 smithi165 ceph-mon[32893]: 6.8 scrub ok 2024-09-21T08:24:54.909 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:54 smithi165 ceph-mon[32893]: 6.14 scrub starts 2024-09-21T08:24:54.909 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:54 smithi165 ceph-mon[32893]: 6.14 scrub ok 2024-09-21T08:24:54.909 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:54 smithi165 systemd[1]: Starting Ceph node-exporter.b for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:24:54.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:54.935 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:54.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:54.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:55.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:55.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:55.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:55.060 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:55.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:24:55.160 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:55.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:55.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:55.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:55.221 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:54 smithi165 bash[66232]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T08:24:55.221 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:55 smithi165 bash[66232]: Getting image source signatures 2024-09-21T08:24:55.221 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:55 smithi165 bash[66232]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T08:24:55.221 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:55 smithi165 bash[66232]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-21T08:24:55.221 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:55 smithi165 bash[66232]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-21T08:24:55.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:55.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:55.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:55.311 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:55.367 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:55.400 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:55.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:55.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:55.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:55.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:55.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:55.556 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:55.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:55.557 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:55.600 INFO:tasks.rados.rados.0.smithi137.stdout:135: done (0 left) 2024-09-21T08:24:55.600 INFO:tasks.rados.rados.0.smithi137.stdout:136: snap_remove snap 8 2024-09-21T08:24:55.600 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.6. 2024-09-21T08:24:55.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:55.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:55.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:55.684 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29611]: 2024-09-21T08:24:55.540+0000 7f1058bb7640 -1 mon.a@0(leader).osd e101 definitely_dead 0 2024-09-21T08:24:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: pgmap v55: 148 pgs: 2 active+clean+snaptrim, 146 active+clean; 118 MiB data, 398 MiB used, 536 GiB / 536 GiB avail; 25 MiB/s rd, 7.4 MiB/s wr, 37 op/s; 9.8 MiB/s, 30 objects/s recovering 2024-09-21T08:24:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: Failed while placing node-exporter.a on smithi137: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T08:24:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T08:24:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter.a 2024-09-21T08:24:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter.a 2024-09-21T08:24:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout: Deploy daemon node-exporter.a ... 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: Verifying port 0.0.0.0:9100 ... 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: Non-zero exit code 1 from systemctl start ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: systemctl: stderr Job for ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service failed because the control process exited with error code. 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: systemctl: stderr See "systemctl status ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" and "journalctl -xeu ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" for details. 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: Traceback (most recent call last): 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: return _run_code(code, main_globals, None, 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: exec(code, run_globals) 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-21T08:24:55.769 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-21T08:24:55.770 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-21T08:24:55.770 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-21T08:24:55.770 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1088, in deploy_daemon 2024-09-21T08:24:55.770 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1224, in deploy_daemon_units 2024-09-21T08:24:55.770 INFO:journalctl@ceph.mon.a.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T08:24:55.770 INFO:journalctl@ceph.mon.a.smithi137.stdout: RuntimeError: Failed command: systemctl start ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a: Job for ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service failed because the control process exited with error code. 2024-09-21T08:24:55.770 INFO:journalctl@ceph.mon.a.smithi137.stdout: See "systemctl status ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" and "journalctl -xeu ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" for details. 2024-09-21T08:24:55.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: Deploying daemon node-exporter.b on smithi165 2024-09-21T08:24:55.771 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: pgmap v55: 148 pgs: 2 active+clean+snaptrim, 146 active+clean; 118 MiB data, 398 MiB used, 536 GiB / 536 GiB avail; 25 MiB/s rd, 7.4 MiB/s wr, 37 op/s; 9.8 MiB/s, 30 objects/s recovering 2024-09-21T08:24:55.771 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: Failed while placing node-exporter.a on smithi137: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T08:24:55.771 INFO:journalctl@ceph.mon.c.smithi137.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T08:24:55.771 INFO:journalctl@ceph.mon.c.smithi137.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter.a 2024-09-21T08:24:55.771 INFO:journalctl@ceph.mon.c.smithi137.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter.a 2024-09-21T08:24:55.771 INFO:journalctl@ceph.mon.c.smithi137.stdout: Deploy daemon node-exporter.a ... 2024-09-21T08:24:55.771 INFO:journalctl@ceph.mon.c.smithi137.stdout: Verifying port 0.0.0.0:9100 ... 2024-09-21T08:24:55.771 INFO:journalctl@ceph.mon.c.smithi137.stdout: Non-zero exit code 1 from systemctl start ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a 2024-09-21T08:24:55.771 INFO:journalctl@ceph.mon.c.smithi137.stdout: systemctl: stderr Job for ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service failed because the control process exited with error code. 2024-09-21T08:24:55.771 INFO:journalctl@ceph.mon.c.smithi137.stdout: systemctl: stderr See "systemctl status ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" and "journalctl -xeu ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" for details. 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: Traceback (most recent call last): 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: return _run_code(code, main_globals, None, 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: exec(code, run_globals) 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1088, in deploy_daemon 2024-09-21T08:24:55.772 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1224, in deploy_daemon_units 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout: RuntimeError: Failed command: systemctl start ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a: Job for ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service failed because the control process exited with error code. 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout: See "systemctl status ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" and "journalctl -xeu ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" for details. 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: Deploying daemon node-exporter.b on smithi165 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: osdmap e101: 8 total, 8 up, 6 in 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: 4.1f scrub starts 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: 4.1f scrub ok 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: 5.14 deep-scrub starts 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: 5.14 deep-scrub ok 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: 3.1b scrub starts 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: 3.1b scrub ok 2024-09-21T08:24:55.773 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/554759833' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-21T08:24:55.774 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: from='client.44777 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-21T08:24:55.774 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[41044]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-21T08:24:55.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: osdmap e101: 8 total, 8 up, 6 in 2024-09-21T08:24:55.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: 4.1f scrub starts 2024-09-21T08:24:55.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: 4.1f scrub ok 2024-09-21T08:24:55.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: 5.14 deep-scrub starts 2024-09-21T08:24:55.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: 5.14 deep-scrub ok 2024-09-21T08:24:55.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: 3.1b scrub starts 2024-09-21T08:24:55.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: 3.1b scrub ok 2024-09-21T08:24:55.774 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/554759833' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-21T08:24:55.775 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: from='client.44777 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-21T08:24:55.775 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:55 smithi137 ceph-mon[29634]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-21T08:24:55.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:55.807 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:55.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:55.808 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:55.909 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:55.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:55.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:55.931 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: pgmap v55: 148 pgs: 2 active+clean+snaptrim, 146 active+clean; 118 MiB data, 398 MiB used, 536 GiB / 536 GiB avail; 25 MiB/s rd, 7.4 MiB/s wr, 37 op/s; 9.8 MiB/s, 30 objects/s recovering 2024-09-21T08:24:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: Failed while placing node-exporter.a on smithi137: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T08:24:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter.a 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: /usr/bin/podman: stderr Error: no such container ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter.a 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: Deploy daemon node-exporter.a ... 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: Verifying port 0.0.0.0:9100 ... 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: Non-zero exit code 1 from systemctl start ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: systemctl: stderr Job for ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service failed because the control process exited with error code. 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: systemctl: stderr See "systemctl status ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" and "journalctl -xeu ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" for details. 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: Traceback (most recent call last): 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/usr/lib64/python3.9/runpy.py", line 197, in _run_module_as_main 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: return _run_code(code, main_globals, None, 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/usr/lib64/python3.9/runpy.py", line 87, in _run_code 2024-09-21T08:24:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout: exec(code, run_globals) 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5579, in 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 5567, in main 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3049, in command_deploy_from 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3084, in _common_deploy 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 3104, in _deploy_daemon_container 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1088, in deploy_daemon 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/__main__.py", line 1224, in deploy_daemon_units 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout: File "/tmp/tmp3ebdx_ug.cephadm.build/app/cephadmlib/call_wrappers.py", line 307, in call_throws 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout: RuntimeError: Failed command: systemctl start ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a: Job for ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service failed because the control process exited with error code. 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout: See "systemctl status ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" and "journalctl -xeu ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service" for details. 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: Deploying daemon node-exporter.b on smithi165 2024-09-21T08:24:56.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: osdmap e101: 8 total, 8 up, 6 in 2024-09-21T08:24:56.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: 4.1f scrub starts 2024-09-21T08:24:56.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: 4.1f scrub ok 2024-09-21T08:24:56.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: 5.14 deep-scrub starts 2024-09-21T08:24:56.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: 5.14 deep-scrub ok 2024-09-21T08:24:56.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: 3.1b scrub starts 2024-09-21T08:24:56.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: 3.1b scrub ok 2024-09-21T08:24:56.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/554759833' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-21T08:24:56.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: from='client.44777 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["6"]}]: dispatch 2024-09-21T08:24:56.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:55 smithi165 ceph-mon[32893]: Client client.admin marked osd.6 out, while it was still marked up 2024-09-21T08:24:56.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:56.051 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:24:56.051 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:24:56.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:56.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:56.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:56.069 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:24:56.069 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:24:56.085 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:24:56.085 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:24:56.101 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:24:56.101 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:24:56.117 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:24:56.118 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:24:56.134 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:24:56.134 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:24:56.150 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:24:56.151 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:24:56.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:56.166 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:24:56.167 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:24:56.177 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:56.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:56.177 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:56.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:56.300 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:56.301 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:56.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:56.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:56.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:56.424 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:56.495 INFO:tasks.rados.rados.0.smithi137.stdout:136: done (0 left) 2024-09-21T08:24:56.495 INFO:tasks.rados.rados.0.smithi137.stdout:137: write oid 21 current snap is 9 2024-09-21T08:24:56.495 INFO:tasks.rados.rados.0.smithi137.stdout:137: seq_num 74 ranges {443718=615603,1564138=470341,2633066=610652} 2024-09-21T08:24:56.495 INFO:tasks.rados.rados.0.smithi137.stdout:137: writing smithi13779221-21 from 443718 to 1059321 tid 1 2024-09-21T08:24:56.501 INFO:tasks.rados.rados.0.smithi137.stdout:137: writing smithi13779221-21 from 1564138 to 2034479 tid 2 2024-09-21T08:24:56.509 INFO:tasks.rados.rados.0.smithi137.stdout:137: writing smithi13779221-21 from 2633066 to 3243718 tid 3 2024-09-21T08:24:56.509 INFO:tasks.rados.rados.0.smithi137.stdout:138: read oid 47 snap -1 2024-09-21T08:24:56.509 INFO:tasks.rados.rados.0.smithi137.stdout:138: expect (ObjNum 67 snap 7 seq_num 67) 2024-09-21T08:24:56.509 INFO:tasks.rados.rados.0.smithi137.stdout:139: delete oid 19 current snap is 9 2024-09-21T08:24:56.524 INFO:tasks.rados.rados.0.smithi137.stdout:138: expect (ObjNum 67 snap 7 seq_num 67) 2024-09-21T08:24:56.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:56.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:56.561 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:56.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:56.682 INFO:tasks.rados.rados.0.smithi137.stdout:137: finishing write tid 1 to smithi13779221-21 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:137: finishing write tid 2 to smithi13779221-21 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:137: finishing write tid 3 to smithi13779221-21 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:137: finishing write tid 4 to smithi13779221-21 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:137: finishing write tid 5 to smithi13779221-21 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:137: oid 21 updating version 0 to 11 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:137: oid 21 version 11 is already newer than 8 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:137: oid 21 version 11 is already newer than 9 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:137: oid 21 version 11 is already newer than 10 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 11 (ObjNum 74 snap 9 seq_num 74) dirty exists 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:137: left oid 21 (ObjNum 74 snap 9 seq_num 74) 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:137: done (2 left) 2024-09-21T08:24:56.683 INFO:tasks.rados.rados.0.smithi137.stdout:138: done (1 left) 2024-09-21T08:24:56.684 INFO:tasks.rados.rados.0.smithi137.stdout:139: done (0 left) 2024-09-21T08:24:56.684 INFO:tasks.rados.rados.0.smithi137.stdout:140: write oid 48 current snap is 9 2024-09-21T08:24:56.684 INFO:tasks.rados.rados.0.smithi137.stdout:140: seq_num 76 ranges {477332=766101,1681102=474010,2687912=589420} 2024-09-21T08:24:56.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:56.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:56.685 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:56.689 INFO:tasks.rados.rados.0.smithi137.stdout:140: writing smithi13779221-48 from 477332 to 1243433 tid 1 2024-09-21T08:24:56.694 INFO:tasks.rados.rados.0.smithi137.stdout:140: writing smithi13779221-48 from 1681102 to 2155112 tid 2 2024-09-21T08:24:56.701 INFO:tasks.rados.rados.0.smithi137.stdout:140: writing smithi13779221-48 from 2687912 to 3277332 tid 3 2024-09-21T08:24:56.701 INFO:tasks.rados.rados.0.smithi137.stdout:141: write (excl) oid 47 current snap is 9 2024-09-21T08:24:56.701 INFO:tasks.rados.rados.0.smithi137.stdout:140: finishing write tid 1 to smithi13779221-48 2024-09-21T08:24:56.701 INFO:tasks.rados.rados.0.smithi137.stdout:140: finishing write tid 2 to smithi13779221-48 2024-09-21T08:24:56.701 INFO:tasks.rados.rados.0.smithi137.stdout:141: seq_num 77 ranges {494139=557703,1455937=734021,2948041=346098} 2024-09-21T08:24:56.706 INFO:tasks.rados.rados.0.smithi137.stdout:141: writing smithi13779221-47 from 494139 to 1051842 tid 1 2024-09-21T08:24:56.713 INFO:tasks.rados.rados.0.smithi137.stdout:141: writing smithi13779221-47 from 1455937 to 2189958 tid 2 2024-09-21T08:24:56.719 INFO:tasks.rados.rados.0.smithi137.stdout:141: writing smithi13779221-47 from 2948041 to 3294139 tid 3 2024-09-21T08:24:56.719 INFO:tasks.rados.rados.0.smithi137.stdout:142: write (excl) oid 20 current snap is 9 2024-09-21T08:24:56.719 INFO:tasks.rados.rados.0.smithi137.stdout:142: seq_num 78 ranges {510946=632952,1798066=477679,2742758=457217} 2024-09-21T08:24:56.725 INFO:tasks.rados.rados.0.smithi137.stdout:142: writing smithi13779221-20 from 510946 to 1143898 tid 1 2024-09-21T08:24:56.732 INFO:tasks.rados.rados.0.smithi137.stdout:142: writing smithi13779221-20 from 1798066 to 2275745 tid 2 2024-09-21T08:24:56.737 INFO:tasks.rados.rados.0.smithi137.stdout:142: writing smithi13779221-20 from 2742758 to 3199975 tid 3 2024-09-21T08:24:56.737 INFO:tasks.rados.rados.0.smithi137.stdout:143: delete oid 19 current snap is 9 2024-09-21T08:24:56.737 INFO:tasks.rados.rados.0.smithi137.stdout:140: finishing write tid 3 to smithi13779221-48 2024-09-21T08:24:56.737 INFO:tasks.rados.rados.0.smithi137.stdout:140: finishing write tid 4 to smithi13779221-48 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:140: finishing write tid 5 to smithi13779221-48 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:140: oid 48 updating version 0 to 38 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:140: oid 48 updating version 38 to 40 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:140: oid 48 version 40 is already newer than 37 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:140: oid 48 version 40 is already newer than 39 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 40 (ObjNum 76 snap 9 seq_num 76) dirty exists 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:140: left oid 48 (ObjNum 76 snap 9 seq_num 76) 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:141: finishing write tid 1 to smithi13779221-47 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:141: finishing write tid 2 to smithi13779221-47 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:141: finishing write tid 3 to smithi13779221-47 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:141: finishing write tid 4 to smithi13779221-47 2024-09-21T08:24:56.738 INFO:tasks.rados.rados.0.smithi137.stdout:141: finishing write tid 5 to smithi13779221-47 2024-09-21T08:24:56.739 INFO:tasks.rados.rados.0.smithi137.stdout:141: oid 47 updating version 0 to 44 2024-09-21T08:24:56.739 INFO:tasks.rados.rados.0.smithi137.stdout:141: oid 47 updating version 44 to 45 2024-09-21T08:24:56.739 INFO:tasks.rados.rados.0.smithi137.stdout:141: oid 47 version 45 is already newer than 43 2024-09-21T08:24:56.739 INFO:tasks.rados.rados.0.smithi137.stdout:141: oid 47 updating version 45 to 46 2024-09-21T08:24:56.739 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 46 (ObjNum 77 snap 9 seq_num 77) dirty exists 2024-09-21T08:24:56.739 INFO:tasks.rados.rados.0.smithi137.stdout:141: left oid 47 (ObjNum 77 snap 9 seq_num 77) 2024-09-21T08:24:56.739 INFO:tasks.rados.rados.0.smithi137.stdout:142: finishing write tid 1 to smithi13779221-20 2024-09-21T08:24:56.739 INFO:tasks.rados.rados.0.smithi137.stdout:142: finishing write tid 2 to smithi13779221-20 2024-09-21T08:24:56.740 INFO:tasks.rados.rados.0.smithi137.stdout:140: done (3 left) 2024-09-21T08:24:56.740 INFO:tasks.rados.rados.0.smithi137.stdout:141: done (2 left) 2024-09-21T08:24:56.740 INFO:tasks.rados.rados.0.smithi137.stdout:143: done (1 left) 2024-09-21T08:24:56.740 INFO:tasks.rados.rados.0.smithi137.stdout:144: rollback oid 10 current snap is 9 2024-09-21T08:24:56.740 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 10 to 7 2024-09-21T08:24:56.740 INFO:tasks.rados.rados.0.smithi137.stdout:145: read oid 35 snap -1 2024-09-21T08:24:56.740 INFO:tasks.rados.rados.0.smithi137.stdout:145: expect (ObjNum 59 snap 4 seq_num 59) 2024-09-21T08:24:56.740 INFO:tasks.rados.rados.0.smithi137.stdout:146: write (excl) oid 9 current snap is 9 2024-09-21T08:24:56.740 INFO:tasks.rados.rados.0.smithi137.stdout:146: seq_num 80 ranges {544560=783450,1915030=481348,2797604=479761} 2024-09-21T08:24:56.746 INFO:tasks.rados.rados.0.smithi137.stdout:146: writing smithi13779221-9 from 544560 to 1328010 tid 1 2024-09-21T08:24:56.752 INFO:tasks.rados.rados.0.smithi137.stdout:146: writing smithi13779221-9 from 1915030 to 2396378 tid 2 2024-09-21T08:24:56.757 INFO:tasks.rados.rados.0.smithi137.stdout:146: writing smithi13779221-9 from 2797604 to 3277365 tid 3 2024-09-21T08:24:56.757 INFO:tasks.rados.rados.0.smithi137.stdout:147: write oid 48 current snap is 9 2024-09-21T08:24:56.757 INFO:tasks.rados.rados.0.smithi137.stdout:142: finishing write tid 3 to smithi13779221-20 2024-09-21T08:24:56.757 INFO:tasks.rados.rados.0.smithi137.stdout:147: seq_num 81 ranges {561367=458699,1573512=741359,2941380=419987} 2024-09-21T08:24:56.762 INFO:tasks.rados.rados.0.smithi137.stdout:147: writing smithi13779221-48 from 561367 to 1020066 tid 1 2024-09-21T08:24:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[29634]: 6.f scrub starts 2024-09-21T08:24:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[29634]: 6.f scrub ok 2024-09-21T08:24:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[29634]: from='client.44777 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-21T08:24:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[29634]: osdmap e102: 8 total, 8 up, 5 in 2024-09-21T08:24:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[29634]: 3.6 scrub starts 2024-09-21T08:24:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[29634]: 3.6 scrub ok 2024-09-21T08:24:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/13858152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:56.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[29634]: osdmap e103: 8 total, 8 up, 5 in 2024-09-21T08:24:56.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[41044]: 6.f scrub starts 2024-09-21T08:24:56.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[41044]: 6.f scrub ok 2024-09-21T08:24:56.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[41044]: from='client.44777 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-21T08:24:56.771 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[41044]: osdmap e102: 8 total, 8 up, 5 in 2024-09-21T08:24:56.771 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[41044]: 3.6 scrub starts 2024-09-21T08:24:56.771 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[41044]: 3.6 scrub ok 2024-09-21T08:24:56.771 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/13858152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:56.771 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:56 smithi137 ceph-mon[41044]: osdmap e103: 8 total, 8 up, 5 in 2024-09-21T08:24:56.771 INFO:tasks.rados.rados.0.smithi137.stdout:147: writing smithi13779221-48 from 1573512 to 2314871 tid 2 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:147: writing smithi13779221-48 from 2941380 to 3361367 tid 3 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:148: snap_create 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:142: finishing write tid 4 to smithi13779221-20 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:142: finishing write tid 5 to smithi13779221-20 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:142: oid 20 updating version 0 to 51 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:142: oid 20 version 51 is already newer than 49 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:142: oid 20 version 51 is already newer than 48 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:142: oid 20 version 51 is already newer than 50 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 51 (ObjNum 78 snap 9 seq_num 78) dirty exists 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:142: left oid 20 (ObjNum 78 snap 9 seq_num 78) 2024-09-21T08:24:56.776 INFO:tasks.rados.rados.0.smithi137.stdout:144: finishing rollback tid 0 to smithi13779221-10 2024-09-21T08:24:56.777 INFO:tasks.rados.rados.0.smithi137.stdout:144: finishing rollback tid 1 to smithi13779221-10 2024-09-21T08:24:56.777 INFO:tasks.rados.rados.0.smithi137.stdout:144: finishing rollback tid 2 to smithi13779221-10 2024-09-21T08:24:56.777 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 44 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-21T08:24:56.777 INFO:tasks.rados.rados.0.smithi137.stdout:145: expect (ObjNum 59 snap 4 seq_num 59) 2024-09-21T08:24:56.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:56.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:56.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:56.806 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:56.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:56.910 INFO:tasks.rados.rados.0.smithi137.stdout:146: finishing write tid 1 to smithi13779221-9 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:146: finishing write tid 2 to smithi13779221-9 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:146: finishing write tid 3 to smithi13779221-9 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:146: finishing write tid 4 to smithi13779221-9 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:146: finishing write tid 5 to smithi13779221-9 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:146: oid 9 updating version 0 to 46 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:146: oid 9 updating version 46 to 47 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:146: oid 9 updating version 47 to 49 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:146: oid 9 version 49 is already newer than 48 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 9 v 49 (ObjNum 80 snap 9 seq_num 80) dirty exists 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:146: left oid 9 (ObjNum 80 snap 9 seq_num 80) 2024-09-21T08:24:56.911 INFO:tasks.rados.rados.0.smithi137.stdout:147: finishing write tid 1 to smithi13779221-48 2024-09-21T08:24:56.912 INFO:tasks.rados.rados.0.smithi137.stdout:147: finishing write tid 2 to smithi13779221-48 2024-09-21T08:24:56.912 INFO:tasks.rados.rados.0.smithi137.stdout:147: finishing write tid 3 to smithi13779221-48 2024-09-21T08:24:56.912 INFO:tasks.rados.rados.0.smithi137.stdout:147: finishing write tid 4 to smithi13779221-48 2024-09-21T08:24:56.912 INFO:tasks.rados.rados.0.smithi137.stdout:147: finishing write tid 5 to smithi13779221-48 2024-09-21T08:24:56.912 INFO:tasks.rados.rados.0.smithi137.stdout:147: oid 48 updating version 0 to 51 2024-09-21T08:24:56.912 INFO:tasks.rados.rados.0.smithi137.stdout:147: oid 48 updating version 51 to 53 2024-09-21T08:24:56.912 INFO:tasks.rados.rados.0.smithi137.stdout:147: oid 48 version 53 is already newer than 52 2024-09-21T08:24:56.912 INFO:tasks.rados.rados.0.smithi137.stdout:147: oid 48 version 53 is already newer than 50 2024-09-21T08:24:56.912 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 53 (ObjNum 81 snap 9 seq_num 81) dirty exists 2024-09-21T08:24:56.912 INFO:tasks.rados.rados.0.smithi137.stdout:147: left oid 48 (ObjNum 81 snap 9 seq_num 81) 2024-09-21T08:24:56.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:56.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:56.929 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:57.013 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:56 smithi165 bash[66232]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T08:24:57.013 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:56 smithi165 bash[66232]: Writing manifest to image destination 2024-09-21T08:24:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:56 smithi165 ceph-mon[32893]: 6.f scrub starts 2024-09-21T08:24:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:56 smithi165 ceph-mon[32893]: 6.f scrub ok 2024-09-21T08:24:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:56 smithi165 ceph-mon[32893]: from='client.44777 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["6"]}]': finished 2024-09-21T08:24:57.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:56 smithi165 ceph-mon[32893]: osdmap e102: 8 total, 8 up, 5 in 2024-09-21T08:24:57.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:56 smithi165 ceph-mon[32893]: 3.6 scrub starts 2024-09-21T08:24:57.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:56 smithi165 ceph-mon[32893]: 3.6 scrub ok 2024-09-21T08:24:57.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:56 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/13858152' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:24:57.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:56 smithi165 ceph-mon[32893]: osdmap e103: 8 total, 8 up, 5 in 2024-09-21T08:24:57.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:57.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:57.051 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:57.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:57.152 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:57.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:57.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:57.175 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:57.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:57.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:57.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:57.347 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 podman[66232]: 2024-09-21 08:24:56.98645001 +0000 UTC m=+2.179217701 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T08:24:57.347 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 podman[66232]: 2024-09-21 08:24:57.105645435 +0000 UTC m=+2.298413168 container create 06e7d047a4e741191b0de40950b283f5f6d884c89a5b118975a59977500dbd4f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T08:24:57.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:57.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:57.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:57.487 INFO:tasks.rados.rados.0.smithi137.stdout:142: done (5 left) 2024-09-21T08:24:57.487 INFO:tasks.rados.rados.0.smithi137.stdout:144: done (4 left) 2024-09-21T08:24:57.488 INFO:tasks.rados.rados.0.smithi137.stdout:145: done (3 left) 2024-09-21T08:24:57.488 INFO:tasks.rados.rados.0.smithi137.stdout:146: done (2 left) 2024-09-21T08:24:57.488 INFO:tasks.rados.rados.0.smithi137.stdout:147: done (1 left) 2024-09-21T08:24:57.488 INFO:tasks.rados.rados.0.smithi137.stdout:148: done (0 left) 2024-09-21T08:24:57.488 INFO:tasks.rados.rados.0.smithi137.stdout:149: read oid 29 snap 9 2024-09-21T08:24:57.488 INFO:tasks.rados.rados.0.smithi137.stdout:149: expect (ObjNum 59 snap 4 seq_num 59) 2024-09-21T08:24:57.488 INFO:tasks.rados.rados.0.smithi137.stdout:150: write (excl) oid 21 current snap is 10 2024-09-21T08:24:57.488 INFO:tasks.rados.rados.0.smithi137.stdout:150: seq_num 82 ranges {578174=533948,1515641=485017,2736097=502305} 2024-09-21T08:24:57.494 INFO:tasks.rados.rados.0.smithi137.stdout:150: writing smithi13779221-21 from 578174 to 1112122 tid 1 2024-09-21T08:24:57.499 INFO:tasks.rados.rados.0.smithi137.stdout:150: writing smithi13779221-21 from 1515641 to 2000658 tid 2 2024-09-21T08:24:57.506 INFO:tasks.rados.rados.0.smithi137.stdout:150: writing smithi13779221-21 from 2736097 to 3238402 tid 3 2024-09-21T08:24:57.506 INFO:tasks.rados.rados.0.smithi137.stdout:151: delete oid 23 current snap is 10 2024-09-21T08:24:57.506 INFO:tasks.rados.rados.0.smithi137.stdout:149: expect (ObjNum 59 snap 4 seq_num 59) 2024-09-21T08:24:57.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:57.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:57.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:57.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:57.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:57.648 INFO:tasks.rados.rados.0.smithi137.stdout:150: finishing write tid 1 to smithi13779221-21 2024-09-21T08:24:57.648 INFO:tasks.rados.rados.0.smithi137.stdout:150: finishing write tid 2 to smithi13779221-21 2024-09-21T08:24:57.648 INFO:tasks.rados.rados.0.smithi137.stdout:149: done (2 left) 2024-09-21T08:24:57.648 INFO:tasks.rados.rados.0.smithi137.stdout:151: done (1 left) 2024-09-21T08:24:57.648 INFO:tasks.rados.rados.0.smithi137.stdout:152: read oid 14 snap 7 2024-09-21T08:24:57.648 INFO:tasks.rados.rados.0.smithi137.stdout:152: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-21T08:24:57.648 INFO:tasks.rados.rados.0.smithi137.stdout:153: snap_remove snap 9 2024-09-21T08:24:57.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:57.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:57.667 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:57.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-mon[32893]: 5.1c deep-scrub starts 2024-09-21T08:24:57.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-mon[32893]: 5.1c deep-scrub ok 2024-09-21T08:24:57.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-mon[32893]: pgmap v58: 148 pgs: 71 peering, 3 active+clean+snaptrim, 74 active+clean; 119 MiB data, 353 MiB used, 447 GiB / 447 GiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-21T08:24:57.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-mon[32893]: Health check failed: Reduced data availability: 6 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-21T08:24:57.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-mon[32893]: 3.1c scrub starts 2024-09-21T08:24:57.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-mon[32893]: 3.1c scrub ok 2024-09-21T08:24:57.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-mon[32893]: osdmap e104: 8 total, 8 up, 5 in 2024-09-21T08:24:57.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:57.764 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 podman[66232]: 2024-09-21 08:24:57.453378324 +0000 UTC m=+2.646146026 container init 06e7d047a4e741191b0de40950b283f5f6d884c89a5b118975a59977500dbd4f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T08:24:57.764 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 podman[66232]: 2024-09-21 08:24:57.458046323 +0000 UTC m=+2.650814014 container start 06e7d047a4e741191b0de40950b283f5f6d884c89a5b118975a59977500dbd4f (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b, maintainer=The Prometheus Authors ) 2024-09-21T08:24:57.764 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.461Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T08:24:57.764 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.462Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.462Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.462Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.462Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.462Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T08:24:57.765 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T08:24:57.766 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T08:24:57.767 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-b[66511]: ts=2024-09-21T08:24:57.463Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 bash[66232]: 06e7d047a4e741191b0de40950b283f5f6d884c89a5b118975a59977500dbd4f 2024-09-21T08:24:57.768 INFO:journalctl@ceph.node-exporter.b.smithi165.stdout:Sep 21 08:24:57 smithi165 systemd[1]: Started Ceph node-exporter.b for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:24:57.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:57.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:57.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:57.790 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:57.880 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[41044]: 5.1c deep-scrub starts 2024-09-21T08:24:57.880 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[41044]: 5.1c deep-scrub ok 2024-09-21T08:24:57.880 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[41044]: pgmap v58: 148 pgs: 71 peering, 3 active+clean+snaptrim, 74 active+clean; 119 MiB data, 353 MiB used, 447 GiB / 447 GiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-21T08:24:57.880 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[41044]: Health check failed: Reduced data availability: 6 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-21T08:24:57.880 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[41044]: 3.1c scrub starts 2024-09-21T08:24:57.880 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[41044]: 3.1c scrub ok 2024-09-21T08:24:57.880 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[41044]: osdmap e104: 8 total, 8 up, 5 in 2024-09-21T08:24:57.881 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:57.881 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[29634]: 5.1c deep-scrub starts 2024-09-21T08:24:57.881 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[29634]: 5.1c deep-scrub ok 2024-09-21T08:24:57.881 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[29634]: pgmap v58: 148 pgs: 71 peering, 3 active+clean+snaptrim, 74 active+clean; 119 MiB data, 353 MiB used, 447 GiB / 447 GiB avail; 3.5 MiB/s rd, 2.5 MiB/s wr, 13 op/s 2024-09-21T08:24:57.881 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[29634]: Health check failed: Reduced data availability: 6 pgs inactive, 13 pgs peering (PG_AVAILABILITY) 2024-09-21T08:24:57.881 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[29634]: 3.1c scrub starts 2024-09-21T08:24:57.881 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[29634]: 3.1c scrub ok 2024-09-21T08:24:57.881 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[29634]: osdmap e104: 8 total, 8 up, 5 in 2024-09-21T08:24:57.881 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:57 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:57.891 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:57.912 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:57.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:57.913 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:58.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:58.034 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:58.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:58.035 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14154.0:109 1.0 e20/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 20'71, mlcod=20'67)", 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:17.334521+0000", 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 219.91854029199999, 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037482370000000002, 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:58.064 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.334521+0000", 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.334521+0000", 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.334522+0000", 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4600000000000003e-07 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.334541+0000", 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9828000000000001e-05 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.334554+0000", 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3035e-05 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.334561+0000", 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1999999999999999e-06 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.334605+0000", 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4366999999999997e-05 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.334620+0000", 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5004000000000001e-05 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.338239+0000", 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036192289999999998 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:17.338269+0000", 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9628000000000002e-05 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.068 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14154.0:132 1.0 e22/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 22'90, mlcod=20'85)", 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.432614+0000", 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 217.82044712800001, 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053383140000000003, 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432614+0000", 2024-09-21T08:24:58.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432614+0000", 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432615+0000", 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.269e-06 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432638+0000", 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2952999999999999e-05 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432647+0000", 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7569999999999993e-06 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432652+0000", 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7370000000000003e-06 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432718+0000", 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5543000000000006e-05 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.432736+0000", 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8161000000000001e-05 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.437884+0000", 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051479309999999997 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.437952+0000", 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7963e-05 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14154.0:133 1.0 e22/19 1:1b0e96ae:devicehealth::main.db-journal.0000000000000000:head v 22'91, mlcod=22'86)", 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:19.433517+0000", 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 217.81954360399999, 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047579859999999996, 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.073 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:58.074 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433517+0000", 2024-09-21T08:24:58.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433517+0000", 2024-09-21T08:24:58.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433519+0000", 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.274e-06 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433529+0000", 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0594999999999999e-05 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433543+0000", 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3841e-05 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.433549+0000", 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2979999999999997e-06 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.436671+0000", 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003121239 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.436692+0000", 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1509e-05 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.438238+0000", 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0015458550000000001 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:19.438275+0000", 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7375e-05 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.077 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.24394.0:1 2.1 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:30.537490+0000", 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 86.715570821, 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010371768999999999, 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:58.078 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.24394", 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.165:0/2482038767", 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537490+0000", 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537490+0000", 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537499+0000", 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8969999999999997e-06 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537514+0000", 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4524e-05 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537527+0000", 2024-09-21T08:24:58.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3788000000000001e-05 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537537+0000", 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.007e-05 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537597+0000", 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9914000000000003e-05 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537603+0000", 2024-09-21T08:24:58.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1480000000000003e-06 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.544050+0000", 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064466499999999999 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.544174+0000", 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012360800000000001 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.544298+0000", 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124428 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.545421+0000", 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001123162 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.547715+0000", 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022933649999999999 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.547754+0000", 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9532e-05 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.547813+0000", 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8746999999999998e-05 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.547862+0000", 2024-09-21T08:24:58.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8936e-05 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:246 7.e 7:78d05d33:::smithi13779221-41:head [sparse-read 0~1306490,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:44.487069+0000", 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.765992308, 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058192540000000003, 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:24:58.085 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 246 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487069+0000", 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487069+0000", 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487072+0000", 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3869999999999999e-06 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487083+0000", 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.06e-05 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487105+0000", 2024-09-21T08:24:58.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2337999999999999e-05 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487115+0000", 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6690000000000001e-06 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487208+0000", 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3858999999999997e-05 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487215+0000", 2024-09-21T08:24:58.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9160000000000004e-06 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.490933+0000", 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003717162 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.491075+0000", 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014278500000000001 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.492888+0000", 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001812538 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:247 7.e 7:78d05d33:::smithi13779221-41:head [read 0~2672280,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:44.487285+0000", 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.765775845, 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075273149999999997, 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:58.090 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 247 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487285+0000", 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487285+0000", 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487286+0000", 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.223e-06 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487292+0000", 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5010000000000004e-06 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487300+0000", 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6589999999999996e-06 2024-09-21T08:24:58.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487307+0000", 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8820000000000003e-06 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487372+0000", 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4472000000000002e-05 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487376+0000", 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6299999999999997e-06 2024-09-21T08:24:58.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.492941+0000", 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055648579999999998 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.493033+0000", 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2142e-05 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494812+0000", 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001778948 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:248 7.e 7:78d05d33:::smithi13779221-41:head [sparse-read 0~1306490] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e91)", 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:44.487322+0000", 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.765738676, 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077760549999999996, 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:58.095 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:24:58.096 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:24:58.096 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 248 2024-09-21T08:24:58.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.096 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487322+0000", 2024-09-21T08:24:58.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487322+0000", 2024-09-21T08:24:58.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487323+0000", 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7299999999999995e-07 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487328+0000", 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7119999999999999e-06 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487333+0000", 2024-09-21T08:24:58.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7969999999999997e-06 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487339+0000", 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2630000000000003e-06 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487406+0000", 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7339e-05 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.487410+0000", 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.16e-06 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494876+0000", 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0074662299999999999 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494971+0000", 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4860999999999999e-05 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.495098+0000", 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012701999999999999 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:255 7.2 e92/80 7:4031afd1:::smithi13779221-48:head v 92'28, mlcod=89'25)", 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.516560+0000", 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 11.736500734, 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0061843549999999999, 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.304 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.516560+0000", 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.516560+0000", 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.516561+0000", 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7400000000000003e-07 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.518699+0000", 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0021377940000000002 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.518717+0000", 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.855e-05 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.518727+0000", 2024-09-21T08:24:58.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3179999999999996e-06 2024-09-21T08:24:58.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.518799+0000", 2024-09-21T08:24:58.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2057000000000005e-05 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.518825+0000", 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6792e-05 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522689+0000", 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038635430000000001 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522745+0000", 2024-09-21T08:24:58.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5627000000000001e-05 2024-09-21T08:24:58.309 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.309 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.309 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.309 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:260 7.8 7:1fd29771:::smithi13779221-22:head [copy-from ver 15] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:24:58.309 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:24:58.309 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.69873872, 2024-09-21T08:24:58.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025065418999999999, 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 260 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554325+0000", 2024-09-21T08:24:58.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.667e-06 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554332+0000", 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1600000000000001e-06 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554346+0000", 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4011e-05 2024-09-21T08:24:58.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554355+0000", 2024-09-21T08:24:58.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9199999999999993e-06 2024-09-21T08:24:58.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554524+0000", 2024-09-21T08:24:58.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000169469 2024-09-21T08:24:58.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:24:58.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554534+0000", 2024-09-21T08:24:58.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5489999999999995e-06 2024-09-21T08:24:58.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557746+0000", 2024-09-21T08:24:58.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003212164 2024-09-21T08:24:58.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557822+0000", 2024-09-21T08:24:58.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6108999999999999e-05 2024-09-21T08:24:58.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:24:58.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568552+0000", 2024-09-21T08:24:58.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01072972 2024-09-21T08:24:58.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:58.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574773+0000", 2024-09-21T08:24:58.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062208760000000002 2024-09-21T08:24:58.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:24:58.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576999+0000", 2024-09-21T08:24:58.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022266059999999999 2024-09-21T08:24:58.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:24:58.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579275+0000", 2024-09-21T08:24:58.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002276054 2024-09-21T08:24:58.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:58.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579335+0000", 2024-09-21T08:24:58.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9775000000000003e-05 2024-09-21T08:24:58.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579388+0000", 2024-09-21T08:24:58.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2339000000000002e-05 2024-09-21T08:24:58.318 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.318 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.319 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:261 7.8 7:1fd29771:::smithi13779221-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:24:58.319 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:24:58.319 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.698594159000001, 2024-09-21T08:24:58.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020259856, 2024-09-21T08:24:58.319 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.319 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 261 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554468+0000", 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0219999999999999e-06 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554475+0000", 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.81e-06 2024-09-21T08:24:58.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554484+0000", 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0550000000000005e-06 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554491+0000", 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2080000000000003e-06 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554607+0000", 2024-09-21T08:24:58.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011585400000000001 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554612+0000", 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0130000000000001e-06 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557969+0000", 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033576320000000002 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558007+0000", 2024-09-21T08:24:58.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.735e-05 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574538+0000", 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016531694 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574630+0000", 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1902000000000002e-05 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574727+0000", 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6316000000000004e-05 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:277 7.5 e93/80 7:a6c204bd:::smithi13779221-42:head v 93'33, mlcod=92'30)", 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.592392+0000", 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.660668955, 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.006355734, 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:58.325 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.592392+0000", 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.592392+0000", 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.592395+0000", 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.813e-06 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.596036+0000", 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036417260000000001 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.596054+0000", 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7791999999999998e-05 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.596066+0000", 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1769e-05 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.596348+0000", 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028206800000000001 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.596373+0000", 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5256e-05 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.598698+0000", 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023243949999999999 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.598748+0000", 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9914999999999999e-05 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:294 7.2 e93/80 7:4d793d31:::smithi13779221-9:head v 93'30, mlcod=91'27)", 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.772177+0000", 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.480883963, 2024-09-21T08:24:58.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0070560220000000003, 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.772177+0000", 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.772177+0000", 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.772181+0000", 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2069999999999997e-06 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.773370+0000", 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001189205 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.773397+0000", 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6811000000000001e-05 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.773409+0000", 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2157000000000001e-05 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.773494+0000", 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4544999999999998e-05 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.773518+0000", 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4383e-05 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779176+0000", 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056578389999999996 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779233+0000", 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6875e-05 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.333 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:332 7.3 7:c851e2c3:::smithi13779221-25:head [read 0~2403368,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.5669173379999997, 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90930888700000001, 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:24:58.334 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 332 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686147+0000", 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5370000000000002e-06 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686160+0000", 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2542e-05 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7927999999999998e-05 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686187+0000", 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3610000000000007e-06 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686261+0000", 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4197999999999997e-05 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686271+0000", 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5259999999999999e-06 2024-09-21T08:24:58.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.590700+0000", 2024-09-21T08:24:58.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90442903500000005 2024-09-21T08:24:58.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.590838+0000", 2024-09-21T08:24:58.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137863 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595452+0000", 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046148969999999997 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:333 7.3 7:c851e2c3:::smithi13779221-25:head [read 0~2403368,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.5668538089999995, 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91060290700000002, 2024-09-21T08:24:58.339 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.340 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.340 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:24:58.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:58.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:24:58.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:24:58.340 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 333 2024-09-21T08:24:58.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.340 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686208+0000", 2024-09-21T08:24:58.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4900000000000005e-07 2024-09-21T08:24:58.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686212+0000", 2024-09-21T08:24:58.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9940000000000002e-06 2024-09-21T08:24:58.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686221+0000", 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8789999999999992e-06 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686225+0000", 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5059999999999997e-06 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686303+0000", 2024-09-21T08:24:58.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7855000000000006e-05 2024-09-21T08:24:58.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:24:58.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686310+0000", 2024-09-21T08:24:58.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0829999999999998e-06 2024-09-21T08:24:58.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595485+0000", 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90917505300000001 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595590+0000", 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105059 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.596810+0000", 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012197289999999999 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.345 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.346 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.346 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:334 7.3 7:c851e2c3:::smithi13779221-25:head [sparse-read 0~1282710] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:24:58.346 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:24:58.346 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.566816803, 2024-09-21T08:24:58.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91097041300000003, 2024-09-21T08:24:58.346 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.346 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.346 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:24:58.347 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:24:58.347 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:24:58.347 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:24:58.347 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 334 2024-09-21T08:24:58.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.347 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:24:58.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686245+0000", 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2600000000000002e-07 2024-09-21T08:24:58.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686247+0000", 2024-09-21T08:24:58.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7429999999999998e-06 2024-09-21T08:24:58.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686255+0000", 2024-09-21T08:24:58.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8369999999999997e-06 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686259+0000", 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0509999999999998e-06 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686326+0000", 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6845000000000001e-05 2024-09-21T08:24:58.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686331+0000", 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.13e-06 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.596911+0000", 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.910579845 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.597015+0000", 2024-09-21T08:24:58.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103538 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.597215+0000", 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000199798 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "MOSDPGPush(7.0 95/94 [PushOp(7:0979c720:::smithi13779221-16:head, version: 86'6, data_included: [652105~445088,1728171~452988], data_size: 898076, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7:0979c720:::smithi13779221-16:head@86'6, size: 2252105, copy_subset: [(0, 2252105)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2252105, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:48.605014+0000", 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 8.6480473139999994, 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067181710000000002, 2024-09-21T08:24:58.352 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.605014+0000", 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.605014+0000", 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.605015+0000", 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3519999999999999e-06 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.610768+0000", 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057531250000000004 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.610788+0000", 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0268999999999999e-05 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.610803+0000", 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5160000000000001e-05 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.610954+0000", 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015005699999999999 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.610967+0000", 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3441000000000001e-05 2024-09-21T08:24:58.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.611732+0000", 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00076476699999999996 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:353 7.c e96/94 7:361ee62b:::smithi13779221-40:head v 96'14, mlcod=93'11)", 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:49.908003+0000", 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 7.3450576739999995, 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0049226180000000001, 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:24:58.356 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.908003+0000", 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.908003+0000", 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.908004+0000", 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.057e-06 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.910978+0000", 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002973991 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.910996+0000", 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8020999999999999e-05 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.911006+0000", 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0111e-05 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.911077+0000", 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0927999999999996e-05 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.911104+0000", 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6605e-05 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.912886+0000", 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017823310000000001 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.912926+0000", 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9573999999999998e-05 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "pg_update_log_missing(5.1f epoch 103/102 rep_tid 64 entries 103'11 (0'0) error 5:f95f44c2:::notify.0:head by client.44387.0:1132 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.518360+0000", 2024-09-21T08:24:58.360 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.73470054200000001, 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057295840000000002, 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518360+0000", 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518360+0000", 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518361+0000", 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6499999999999998e-07 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518368+0000", 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2119999999999997e-06 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518382+0000", 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3176999999999999e-05 2024-09-21T08:24:58.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518395+0000", 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3512e-05 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.520696+0000", 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023009340000000001 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.524090+0000", 2024-09-21T08:24:58.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033938839999999998 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "pg_update_log_missing(5.9 epoch 103/102 rep_tid 66 entries 103'10 (0'0) error 5:93e5b521:::notify.7:head by client.44387.0:1134 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.518407+0000", 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.73465388499999995, 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056408780000000002, 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518407+0000", 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518407+0000", 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518408+0000", 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6099999999999999e-07 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518413+0000", 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.775e-06 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518423+0000", 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0389e-05 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.518433+0000", 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0314e-05 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.519262+0000", 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082832399999999997 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.524048+0000", 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047862149999999999 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.368 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "MOSDPGPush(7.b 103/102 [PushOp(7:d8f73c7b:::smithi13779221-38:head, version: 86'21, data_included: [2464667~157192], data_size: 157192, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:d8f73c7b:::smithi13779221-38:head@86'21, size: 2621859, copy_subset: [(0, 2621859)], clone_subset: {}, snapset: 0=[]:{}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2621859, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2464667, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:24:58.368 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.578304+0000", 2024-09-21T08:24:58.368 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.67475715000000003, 2024-09-21T08:24:58.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033217429999999998, 2024-09-21T08:24:58.368 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:24:58.368 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:24:58.368 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:24:58.368 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:24:58.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.578304+0000", 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.578304+0000", 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.578305+0000", 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1090000000000001e-06 2024-09-21T08:24:58.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.578361+0000", 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.575e-05 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.578376+0000", 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.523e-05 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.578386+0000", 2024-09-21T08:24:58.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0159e-05 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.578466+0000", 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0081999999999997e-05 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.578481+0000", 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4419e-05 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.581625+0000", 2024-09-21T08:24:58.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031449939999999999 2024-09-21T08:24:58.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.372 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:24:58.372 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:24:58.372 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:24:58.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:58.394 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:58.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:58.395 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:58.406 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:58.489 INFO:tasks.rados.rados.0.smithi137.stdout:153: done (2 left) 2024-09-21T08:24:58.489 INFO:tasks.rados.rados.0.smithi137.stdout:154: write oid 46 current snap is 10 2024-09-21T08:24:58.489 INFO:tasks.rados.rados.0.smithi137.stdout:150: finishing write tid 3 to smithi13779221-21 2024-09-21T08:24:58.489 INFO:tasks.rados.rados.0.smithi137.stdout:150: finishing write tid 4 to smithi13779221-21 2024-09-21T08:24:58.489 INFO:tasks.rados.rados.0.smithi137.stdout:150: finishing write tid 5 to smithi13779221-21 2024-09-21T08:24:58.490 INFO:tasks.rados.rados.0.smithi137.stdout:150: oid 21 updating version 0 to 18 2024-09-21T08:24:58.490 INFO:tasks.rados.rados.0.smithi137.stdout:150: oid 21 version 18 is already newer than 17 2024-09-21T08:24:58.490 INFO:tasks.rados.rados.0.smithi137.stdout:150: oid 21 version 18 is already newer than 16 2024-09-21T08:24:58.490 INFO:tasks.rados.rados.0.smithi137.stdout:150: oid 21 updating version 18 to 19 2024-09-21T08:24:58.490 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 19 (ObjNum 82 snap 10 seq_num 82) dirty exists 2024-09-21T08:24:58.490 INFO:tasks.rados.rados.0.smithi137.stdout:150: left oid 21 (ObjNum 82 snap 10 seq_num 82) 2024-09-21T08:24:58.490 INFO:tasks.rados.rados.0.smithi137.stdout:152: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-21T08:24:58.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:58.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:58.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:58.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:58.566 INFO:tasks.rados.rados.0.smithi137.stdout:154: seq_num 84 ranges {611788=400799,1748958=488686,2907296=504492} 2024-09-21T08:24:58.571 INFO:tasks.rados.rados.0.smithi137.stdout:154: writing smithi13779221-46 from 611788 to 1012587 tid 1 2024-09-21T08:24:58.578 INFO:tasks.rados.rados.0.smithi137.stdout:154: writing smithi13779221-46 from 1748958 to 2237644 tid 2 2024-09-21T08:24:58.585 INFO:tasks.rados.rados.0.smithi137.stdout:154: writing smithi13779221-46 from 2907296 to 3411788 tid 3 2024-09-21T08:24:58.585 INFO:tasks.rados.rados.0.smithi137.stdout:150: done (2 left) 2024-09-21T08:24:58.585 INFO:tasks.rados.rados.0.smithi137.stdout:152: done (1 left) 2024-09-21T08:24:58.585 INFO:tasks.rados.rados.0.smithi137.stdout:155: snap_remove snap 7 2024-09-21T08:24:58.607 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: 3.13 scrub starts 2024-09-21T08:24:58.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:58.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:58.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:58.643 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:58.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:24:58.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:58.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:58.766 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: 3.13 scrub starts 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: 3.13 scrub ok 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: pgmap v61: 148 pgs: 71 peering, 3 active+clean+snaptrim, 74 active+clean; 119 MiB data, 353 MiB used, 447 GiB / 447 GiB avail; 3.8 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: 4.14 deep-scrub starts 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: 4.14 deep-scrub ok 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[29634]: osdmap e105: 8 total, 8 up, 5 in 2024-09-21T08:24:58.788 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: 3.13 scrub starts 2024-09-21T08:24:58.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: 3.13 scrub ok 2024-09-21T08:24:58.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:58.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:58.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: pgmap v61: 148 pgs: 71 peering, 3 active+clean+snaptrim, 74 active+clean; 119 MiB data, 353 MiB used, 447 GiB / 447 GiB avail; 3.8 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2024-09-21T08:24:58.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:58.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: 4.14 deep-scrub starts 2024-09-21T08:24:58.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:58.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: 4.14 deep-scrub ok 2024-09-21T08:24:58.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:24:58.789 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:58 smithi137 ceph-mon[41044]: osdmap e105: 8 total, 8 up, 5 in 2024-09-21T08:24:58.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:24:58.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:58.889 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:58.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:24:58.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:24:59.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:59.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:59.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:24:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: 3.13 scrub ok 2024-09-21T08:24:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: pgmap v61: 148 pgs: 71 peering, 3 active+clean+snaptrim, 74 active+clean; 119 MiB data, 353 MiB used, 447 GiB / 447 GiB avail; 3.8 MiB/s rd, 2.7 MiB/s wr, 14 op/s 2024-09-21T08:24:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: 4.14 deep-scrub starts 2024-09-21T08:24:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:24:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: 4.14 deep-scrub ok 2024-09-21T08:24:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:24:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:58 smithi165 ceph-mon[32893]: osdmap e105: 8 total, 8 up, 5 in 2024-09-21T08:24:59.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:24:59.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:24:59.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:59.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:59.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:24:59.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T08:24:59.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:59.258 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:59.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:59.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:59.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:59.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:59.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:59.383 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:59.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:155: done (1 left) 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:156: read oid 23 snap -1 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:156: expect deleted 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:157: snap_create 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:154: finishing write tid 1 to smithi13779221-46 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:154: finishing write tid 2 to smithi13779221-46 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:154: finishing write tid 3 to smithi13779221-46 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:154: finishing write tid 4 to smithi13779221-46 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:154: finishing write tid 5 to smithi13779221-46 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:154: oid 46 updating version 0 to 66 2024-09-21T08:24:59.492 INFO:tasks.rados.rados.0.smithi137.stdout:154: oid 46 updating version 66 to 68 2024-09-21T08:24:59.493 INFO:tasks.rados.rados.0.smithi137.stdout:154: oid 46 version 68 is already newer than 67 2024-09-21T08:24:59.493 INFO:tasks.rados.rados.0.smithi137.stdout:154: oid 46 version 68 is already newer than 65 2024-09-21T08:24:59.493 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 68 (ObjNum 84 snap 10 seq_num 84) dirty exists 2024-09-21T08:24:59.493 INFO:tasks.rados.rados.0.smithi137.stdout:154: left oid 46 (ObjNum 84 snap 10 seq_num 84) 2024-09-21T08:24:59.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:59.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:59.508 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:24:59.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:24:59.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:24:59.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:59.630 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:24:59.633 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:59.678 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:24:59.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:59.752 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:59.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:59.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:59.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[29634]: 3.c scrub starts 2024-09-21T08:24:59.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[29634]: 3.c scrub ok 2024-09-21T08:24:59.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:24:59.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:24:59.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[29634]: 5.10 scrub starts 2024-09-21T08:24:59.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[29634]: 5.10 scrub ok 2024-09-21T08:24:59.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[29634]: osdmap e106: 8 total, 8 up, 5 in 2024-09-21T08:24:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[41044]: 3.c scrub starts 2024-09-21T08:24:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[41044]: 3.c scrub ok 2024-09-21T08:24:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:24:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:24:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[41044]: 5.10 scrub starts 2024-09-21T08:24:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[41044]: 5.10 scrub ok 2024-09-21T08:24:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:24:59 smithi137 ceph-mon[41044]: osdmap e106: 8 total, 8 up, 5 in 2024-09-21T08:24:59.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:24:59.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:24:59.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:24:59.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:24:59.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:24:59.981 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:59 smithi165 ceph-mon[32893]: 3.c scrub starts 2024-09-21T08:24:59.981 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:59 smithi165 ceph-mon[32893]: 3.c scrub ok 2024-09-21T08:24:59.981 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:59 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:24:59.981 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:59 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:24:59.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:59 smithi165 ceph-mon[32893]: 5.10 scrub starts 2024-09-21T08:24:59.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:59 smithi165 ceph-mon[32893]: 5.10 scrub ok 2024-09-21T08:24:59.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:24:59 smithi165 ceph-mon[32893]: osdmap e106: 8 total, 8 up, 5 in 2024-09-21T08:25:00.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:00.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:00.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:00.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:00.123 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:00.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:00.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:00.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:00.243 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:00.243 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:00.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:00.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:00.247 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:00.260 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:00.260 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:00.276 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:00.276 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:00.292 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:00.292 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:00.309 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:00.309 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:00.327 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:00.327 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:00.344 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:00.344 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:00.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:00.362 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:00.362 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:00.370 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:00.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:00.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:00.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:00.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:00.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:00.495 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:00.502 INFO:tasks.rados.rados.0.smithi137.stdout:154: done (2 left) 2024-09-21T08:25:00.502 INFO:tasks.rados.rados.0.smithi137.stdout:156: done (1 left) 2024-09-21T08:25:00.502 INFO:tasks.rados.rados.0.smithi137.stdout:157: done (0 left) 2024-09-21T08:25:00.502 INFO:tasks.rados.rados.0.smithi137.stdout:158: rollback oid 38 current snap is 11 2024-09-21T08:25:00.502 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 38 to 10 2024-09-21T08:25:00.502 INFO:tasks.rados.rados.0.smithi137.stdout:159: write oid 33 current snap is 11 2024-09-21T08:25:00.502 INFO:tasks.rados.rados.0.smithi137.stdout:159: seq_num 85 ranges {628595=476048,1807440=632344,2934719=452474} 2024-09-21T08:25:00.507 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:25:00.509 INFO:tasks.rados.rados.0.smithi137.stdout:159: writing smithi13779221-33 from 628595 to 1104643 tid 1 2024-09-21T08:25:00.517 INFO:tasks.rados.rados.0.smithi137.stdout:159: writing smithi13779221-33 from 1807440 to 2439784 tid 2 2024-09-21T08:25:00.524 INFO:tasks.rados.rados.0.smithi137.stdout:159: writing smithi13779221-33 from 2934719 to 3387193 tid 3 2024-09-21T08:25:00.524 INFO:tasks.rados.rados.0.smithi137.stdout:160: read oid 34 snap 10 2024-09-21T08:25:00.524 INFO:tasks.rados.rados.0.smithi137.stdout:160: expect deleted 2024-09-21T08:25:00.524 INFO:tasks.rados.rados.0.smithi137.stdout:161: write (excl) oid 22 current snap is 11 2024-09-21T08:25:00.524 INFO:tasks.rados.rados.0.smithi137.stdout:158: finishing rollback tid 0 to smithi13779221-38 2024-09-21T08:25:00.524 INFO:tasks.rados.rados.0.smithi137.stdout:161: seq_num 86 ranges {645402=551297,1749569=492355,2845789=599613} 2024-09-21T08:25:00.531 INFO:tasks.rados.rados.0.smithi137.stdout:161: writing smithi13779221-22 from 645402 to 1196699 tid 1 2024-09-21T08:25:00.538 INFO:tasks.rados.rados.0.smithi137.stdout:161: writing smithi13779221-22 from 1749569 to 2241924 tid 2 2024-09-21T08:25:00.546 INFO:tasks.rados.rados.0.smithi137.stdout:161: writing smithi13779221-22 from 2845789 to 3445402 tid 3 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:162: read oid 19 snap -1 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:162: expect deleted 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:163: delete oid 31 current snap is 11 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:158: finishing rollback tid 1 to smithi13779221-38 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:158: finishing rollback tid 2 to smithi13779221-38 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 74 (ObjNum 37 snap 0 seq_num 37) dirty exists 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:159: finishing write tid 1 to smithi13779221-33 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:159: finishing write tid 2 to smithi13779221-33 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:159: finishing write tid 3 to smithi13779221-33 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:159: finishing write tid 4 to smithi13779221-33 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:159: finishing write tid 5 to smithi13779221-33 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:159: oid 33 updating version 0 to 16 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:159: oid 33 version 16 is already newer than 14 2024-09-21T08:25:00.547 INFO:tasks.rados.rados.0.smithi137.stdout:159: oid 33 version 16 is already newer than 13 2024-09-21T08:25:00.548 INFO:tasks.rados.rados.0.smithi137.stdout:159: oid 33 version 16 is already newer than 15 2024-09-21T08:25:00.548 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 16 (ObjNum 85 snap 11 seq_num 85) dirty exists 2024-09-21T08:25:00.548 INFO:tasks.rados.rados.0.smithi137.stdout:159: left oid 33 (ObjNum 85 snap 11 seq_num 85) 2024-09-21T08:25:00.548 INFO:tasks.rados.rados.0.smithi137.stdout:161: finishing write tid 1 to smithi13779221-22 2024-09-21T08:25:00.548 INFO:tasks.rados.rados.0.smithi137.stdout:161: finishing write tid 2 to smithi13779221-22 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:161: finishing write tid 3 to smithi13779221-22 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:161: finishing write tid 4 to smithi13779221-22 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:161: finishing write tid 5 to smithi13779221-22 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:161: oid 22 updating version 0 to 31 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:161: oid 22 updating version 31 to 32 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:161: oid 22 updating version 32 to 34 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:161: oid 22 version 34 is already newer than 33 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 34 (ObjNum 86 snap 11 seq_num 86) dirty exists 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:161: left oid 22 (ObjNum 86 snap 11 seq_num 86) 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:158: done (5 left) 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:159: done (4 left) 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:160: done (3 left) 2024-09-21T08:25:00.550 INFO:tasks.rados.rados.0.smithi137.stdout:161: done (2 left) 2024-09-21T08:25:00.551 INFO:tasks.rados.rados.0.smithi137.stdout:162: done (1 left) 2024-09-21T08:25:00.551 INFO:tasks.rados.rados.0.smithi137.stdout:163: done (0 left) 2024-09-21T08:25:00.551 INFO:tasks.rados.rados.0.smithi137.stdout:164: write (excl) oid 2 current snap is 11 2024-09-21T08:25:00.551 INFO:tasks.rados.rados.0.smithi137.stdout:164: seq_num 88 ranges {679016=701795,1866533=496024,2900635=402643} 2024-09-21T08:25:00.557 INFO:tasks.rados.rados.0.smithi137.stdout:164: writing smithi13779221-2 from 679016 to 1380811 tid 1 2024-09-21T08:25:00.563 INFO:tasks.rados.rados.0.smithi137.stdout:164: writing smithi13779221-2 from 1866533 to 2362557 tid 2 2024-09-21T08:25:00.568 INFO:tasks.rados.rados.0.smithi137.stdout:164: writing smithi13779221-2 from 2900635 to 3303278 tid 3 2024-09-21T08:25:00.568 INFO:tasks.rados.rados.0.smithi137.stdout:165: snap_create 2024-09-21T08:25:00.568 INFO:tasks.rados.rados.0.smithi137.stdout:164: finishing write tid 1 to smithi13779221-2 2024-09-21T08:25:00.568 INFO:tasks.rados.rados.0.smithi137.stdout:164: finishing write tid 2 to smithi13779221-2 2024-09-21T08:25:00.571 INFO:tasks.rados.rados.0.smithi137.stdout:164: finishing write tid 3 to smithi13779221-2 2024-09-21T08:25:00.571 INFO:tasks.rados.rados.0.smithi137.stdout:164: finishing write tid 4 to smithi13779221-2 2024-09-21T08:25:00.571 INFO:tasks.rados.rados.0.smithi137.stdout:164: finishing write tid 5 to smithi13779221-2 2024-09-21T08:25:00.571 INFO:tasks.rados.rados.0.smithi137.stdout:164: oid 2 updating version 0 to 22 2024-09-21T08:25:00.571 INFO:tasks.rados.rados.0.smithi137.stdout:164: oid 2 version 22 is already newer than 19 2024-09-21T08:25:00.571 INFO:tasks.rados.rados.0.smithi137.stdout:164: oid 2 version 22 is already newer than 21 2024-09-21T08:25:00.571 INFO:tasks.rados.rados.0.smithi137.stdout:164: oid 2 version 22 is already newer than 20 2024-09-21T08:25:00.571 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 22 (ObjNum 88 snap 11 seq_num 88) dirty exists 2024-09-21T08:25:00.571 INFO:tasks.rados.rados.0.smithi137.stdout:164: left oid 2 (ObjNum 88 snap 11 seq_num 88) 2024-09-21T08:25:00.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:00.597 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:25:00.597 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:25:00.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:00.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:00.619 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:00.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:00.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:00.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:00.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:00.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:00.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:00.865 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:00.865 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:00.921 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:00 smithi165 ceph-mon[32893]: pgmap v64: 148 pgs: 71 peering, 3 active+clean+snaptrim, 74 active+clean; 119 MiB data, 353 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:00.921 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:00 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/135912547' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:00.921 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:00 smithi165 ceph-mon[32893]: from='client.44789 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:00.921 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:00.921 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:00.921 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:00.921 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:00 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1642515527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:00.922 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:00 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:00.922 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:00 smithi165 ceph-mon[32893]: from='client.44789 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:25:00.922 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:00 smithi165 ceph-mon[32893]: osdmap e107: 8 total, 8 up, 5 in 2024-09-21T08:25:00.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:00.987 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:00.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:00.988 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:01.017 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:25:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:25:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[29634]: pgmap v64: 148 pgs: 71 peering, 3 active+clean+snaptrim, 74 active+clean; 119 MiB data, 353 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/135912547' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[29634]: from='client.44789 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1642515527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[29634]: from='client.44789 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:25:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[29634]: osdmap e107: 8 total, 8 up, 5 in 2024-09-21T08:25:01.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[41044]: pgmap v64: 148 pgs: 71 peering, 3 active+clean+snaptrim, 74 active+clean; 119 MiB data, 353 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:01.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/135912547' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:01.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[41044]: from='client.44789 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:01.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:01.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:01.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:01.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1642515527' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:01.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:01.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[41044]: from='client.44789 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:25:01.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:00 smithi137 ceph-mon[41044]: osdmap e107: 8 total, 8 up, 5 in 2024-09-21T08:25:01.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:01.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:01.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:01.111 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:01.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:01.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:01.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:01.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:01.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:01.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:01.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:01.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:01.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:01.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:01.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:01.480 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:01.491 INFO:tasks.rados.rados.0.smithi137.stdout:164: done (1 left) 2024-09-21T08:25:01.491 INFO:tasks.rados.rados.0.smithi137.stdout:165: done (0 left) 2024-09-21T08:25:01.491 INFO:tasks.rados.rados.0.smithi137.stdout:166: snap_create 2024-09-21T08:25:01.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T08:25:01.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:01.602 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:01.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:01.603 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:01.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:01.724 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:01.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:01.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:01.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:01.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:01.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:01.848 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:01.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:01.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:01.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:01.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:02.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:25:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:25:02.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:02.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:02.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:02.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:02.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:02.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:02.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:02.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:02.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:02.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:02.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:02.344 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:02.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:02.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:02.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:02.467 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:02.489 INFO:tasks.rados.rados.0.smithi137.stdout:166: done (0 left) 2024-09-21T08:25:02.489 INFO:tasks.rados.rados.0.smithi137.stdout:167: write (excl) oid 5 current snap is 13 2024-09-21T08:25:02.489 INFO:tasks.rados.rados.0.smithi137.stdout:167: seq_num 89 ranges {695823=777044,1925015=639682,3328058=167765} 2024-09-21T08:25:02.497 INFO:tasks.rados.rados.0.smithi137.stdout:167: writing smithi13779221-5 from 695823 to 1472867 tid 1 2024-09-21T08:25:02.504 INFO:tasks.rados.rados.0.smithi137.stdout:167: writing smithi13779221-5 from 1925015 to 2564697 tid 2 2024-09-21T08:25:02.509 INFO:tasks.rados.rados.0.smithi137.stdout:167: writing smithi13779221-5 from 3328058 to 3495823 tid 3 2024-09-21T08:25:02.509 INFO:tasks.rados.rados.0.smithi137.stdout:168: read oid 22 snap -1 2024-09-21T08:25:02.509 INFO:tasks.rados.rados.0.smithi137.stdout:168: expect (ObjNum 86 snap 11 seq_num 86) 2024-09-21T08:25:02.509 INFO:tasks.rados.rados.0.smithi137.stdout:169: snap_remove snap 10 2024-09-21T08:25:02.517 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:02 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:02.517 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:02 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:02.517 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:02 smithi137 ceph-mon[29634]: osdmap e108: 8 total, 8 up, 5 in 2024-09-21T08:25:02.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:02 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:02.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:02 smithi137 ceph-mon[29634]: pgmap v67: 148 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 144 active+clean; 129 MiB data, 405 MiB used, 447 GiB / 447 GiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 29 op/s; 6.8 MiB/s, 34 objects/s recovering 2024-09-21T08:25:02.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:02 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:02.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:02 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:02.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:02 smithi137 ceph-mon[41044]: osdmap e108: 8 total, 8 up, 5 in 2024-09-21T08:25:02.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:02 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:02.518 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:02 smithi137 ceph-mon[41044]: pgmap v67: 148 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 144 active+clean; 129 MiB data, 405 MiB used, 447 GiB / 447 GiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 29 op/s; 6.8 MiB/s, 34 objects/s recovering 2024-09-21T08:25:02.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:02.588 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:02.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:02.589 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:02.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:02.710 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:02.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:02.711 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:02 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:02 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:02 smithi165 ceph-mon[32893]: osdmap e108: 8 total, 8 up, 5 in 2024-09-21T08:25:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:02 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:02 smithi165 ceph-mon[32893]: pgmap v67: 148 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 144 active+clean; 129 MiB data, 405 MiB used, 447 GiB / 447 GiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 29 op/s; 6.8 MiB/s, 34 objects/s recovering 2024-09-21T08:25:02.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:02.835 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:02.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:02.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:02.920 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 7] out_osds: [4, 3, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:25:02.920 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:25:02.920 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-21T08:25:02.920 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T08:25:02.920 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T08:25:02.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-21T08:25:02.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:02.987 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:02.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:02.988 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:03.020 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:03.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:03.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:03.113 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:03.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:03.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:03.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:03.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:03.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:03.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:03.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:03.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:03.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:03.488 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:03.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:03.489 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:03.499 INFO:tasks.rados.rados.0.smithi137.stdout:169: done (2 left) 2024-09-21T08:25:03.499 INFO:tasks.rados.rados.0.smithi137.stdout:170: copy_from oid 47 from oid 43 current snap is 13 2024-09-21T08:25:03.499 INFO:tasks.rados.rados.0.smithi137.stdout:167: finishing write tid 1 to smithi13779221-5 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:167: finishing write tid 2 to smithi13779221-5 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:167: finishing write tid 3 to smithi13779221-5 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:167: finishing write tid 4 to smithi13779221-5 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:167: finishing write tid 5 to smithi13779221-5 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:167: oid 5 updating version 0 to 64 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:167: oid 5 version 64 is already newer than 63 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:167: oid 5 version 64 is already newer than 59 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:167: oid 5 version 64 is already newer than 62 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 64 (ObjNum 89 snap 13 seq_num 89) dirty exists 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:167: left oid 5 (ObjNum 89 snap 13 seq_num 89) 2024-09-21T08:25:03.500 INFO:tasks.rados.rados.0.smithi137.stdout:168: expect (ObjNum 86 snap 11 seq_num 86) 2024-09-21T08:25:03.590 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:03.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:03.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:03.615 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:03.657 INFO:tasks.rados.rados.0.smithi137.stdout:170: finishing copy_from racing read to smithi13779221-47 2024-09-21T08:25:03.657 INFO:tasks.rados.rados.0.smithi137.stdout:170: finishing copy_from to smithi13779221-47 2024-09-21T08:25:03.658 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 13 (ObjNum 62 snap 4 seq_num 62) dirty exists 2024-09-21T08:25:03.658 INFO:tasks.rados.rados.0.smithi137.stdout:167: done (2 left) 2024-09-21T08:25:03.658 INFO:tasks.rados.rados.0.smithi137.stdout:168: done (1 left) 2024-09-21T08:25:03.658 INFO:tasks.rados.rados.0.smithi137.stdout:170: done (0 left) 2024-09-21T08:25:03.658 INFO:tasks.rados.rados.0.smithi137.stdout:171: snap_create 2024-09-21T08:25:03.715 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:03.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:03.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:03.738 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:03 smithi165 ceph-mon[32893]: osdmap e109: 8 total, 8 up, 5 in 2024-09-21T08:25:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:03 smithi165 ceph-mon[32893]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 13 pgs peering) 2024-09-21T08:25:03.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:03 smithi137 ceph-mon[29634]: osdmap e109: 8 total, 8 up, 5 in 2024-09-21T08:25:03.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:03 smithi137 ceph-mon[29634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 13 pgs peering) 2024-09-21T08:25:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:03 smithi137 ceph-mon[41044]: osdmap e109: 8 total, 8 up, 5 in 2024-09-21T08:25:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:03 smithi137 ceph-mon[41044]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 6 pgs inactive, 13 pgs peering) 2024-09-21T08:25:03.785 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:25:03.785 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:25:03.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:03.861 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:03.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:03.861 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:03.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:03.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:03.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:03.984 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:04.003 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:04.071 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:04 smithi137 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service: Scheduled restart job, restart counter is at 1. 2024-09-21T08:25:04.071 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:04 smithi137 systemd[1]: Stopped Ceph node-exporter.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:25:04.071 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:04 smithi137 systemd[1]: Starting Ceph node-exporter.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:25:04.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:04.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:04.108 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:04.109 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:04.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:25:04.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:04.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:04.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:04.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:04.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T08:25:04.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:04.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:04.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:04.423 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:04.502 INFO:tasks.rados.rados.0.smithi137.stdout:171: done (0 left) 2024-09-21T08:25:04.503 INFO:tasks.rados.rados.0.smithi137.stdout:172: write (excl) oid 42 current snap is 14 2024-09-21T08:25:04.503 INFO:tasks.rados.rados.0.smithi137.stdout:172: seq_num 90 ranges {712630=452293,1583497=499693,2555481=425187} 2024-09-21T08:25:04.509 INFO:tasks.rados.rados.0.smithi137.stdout:172: writing smithi13779221-42 from 712630 to 1164923 tid 1 2024-09-21T08:25:04.514 INFO:tasks.rados.rados.0.smithi137.stdout:172: writing smithi13779221-42 from 1583497 to 2083190 tid 2 2024-09-21T08:25:04.520 INFO:tasks.rados.rados.0.smithi137.stdout:172: writing smithi13779221-42 from 2555481 to 2980668 tid 3 2024-09-21T08:25:04.520 INFO:tasks.rados.rados.0.smithi137.stdout:173: copy_from oid 6 from oid 31 current snap is 14 2024-09-21T08:25:04.520 INFO:tasks.rados.rados.0.smithi137.stdout:174: snap_remove snap 12 2024-09-21T08:25:04.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:04.546 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:04.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:04.547 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:04.586 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:04 smithi137 ceph-mon[41044]: osdmap e110: 8 total, 8 up, 5 in 2024-09-21T08:25:04.586 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:04 smithi137 ceph-mon[41044]: pgmap v70: 148 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 144 active+clean; 129 MiB data, 405 MiB used, 447 GiB / 447 GiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 29 op/s; 6.8 MiB/s, 34 objects/s recovering 2024-09-21T08:25:04.587 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:04 smithi137 ceph-mon[29634]: osdmap e110: 8 total, 8 up, 5 in 2024-09-21T08:25:04.587 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:04 smithi137 ceph-mon[29634]: pgmap v70: 148 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 144 active+clean; 129 MiB data, 405 MiB used, 447 GiB / 447 GiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 29 op/s; 6.8 MiB/s, 34 objects/s recovering 2024-09-21T08:25:04.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:04.669 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:04.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:04.670 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:04.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:25:04.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:04 smithi165 ceph-mon[32893]: osdmap e110: 8 total, 8 up, 5 in 2024-09-21T08:25:04.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:04 smithi165 ceph-mon[32893]: pgmap v70: 148 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 144 active+clean; 129 MiB data, 405 MiB used, 447 GiB / 447 GiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 29 op/s; 6.8 MiB/s, 34 objects/s recovering 2024-09-21T08:25:04.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:04.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:04.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:04.795 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:04.846 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:25:04.846 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T08:25:04.846 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:25:04.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:04.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:04.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:04.919 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:05.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:05.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:05.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:05.047 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:05.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:05.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:05.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:05.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:05.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:05.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:05.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:05.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:05.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:05.427 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:05.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:05.428 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:05.495 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:05.496 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:174: done (2 left) 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:175: copy_from oid 15 from oid 7 current snap is 14 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:176: delete oid 46 current snap is 14 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:172: finishing write tid 1 to smithi13779221-42 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:172: finishing write tid 2 to smithi13779221-42 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:172: finishing write tid 3 to smithi13779221-42 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:172: finishing write tid 4 to smithi13779221-42 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:172: finishing write tid 5 to smithi13779221-42 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:172: oid 42 updating version 0 to 60 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:172: oid 42 version 60 is already newer than 59 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:172: oid 42 updating version 60 to 62 2024-09-21T08:25:05.514 INFO:tasks.rados.rados.0.smithi137.stdout:172: oid 42 version 62 is already newer than 61 2024-09-21T08:25:05.515 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 62 (ObjNum 90 snap 14 seq_num 90) dirty exists 2024-09-21T08:25:05.515 INFO:tasks.rados.rados.0.smithi137.stdout:172: left oid 42 (ObjNum 90 snap 14 seq_num 90) 2024-09-21T08:25:05.515 INFO:tasks.rados.rados.0.smithi137.stdout:173: finishing copy_from to smithi13779221-6 2024-09-21T08:25:05.515 INFO:tasks.rados.rados.0.smithi137.stdout:173: got expected ENOENT (src dne) 2024-09-21T08:25:05.515 INFO:tasks.rados.rados.0.smithi137.stdout:173: finishing copy_from racing read to smithi13779221-6 2024-09-21T08:25:05.524 INFO:tasks.rados.rados.0.smithi137.stdout:172: done (3 left) 2024-09-21T08:25:05.524 INFO:tasks.rados.rados.0.smithi137.stdout:173: done (2 left) 2024-09-21T08:25:05.524 INFO:tasks.rados.rados.0.smithi137.stdout:176: done (1 left) 2024-09-21T08:25:05.524 INFO:tasks.rados.rados.0.smithi137.stdout:177: rollback oid 19 current snap is 14 2024-09-21T08:25:05.524 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 19 to 11 2024-09-21T08:25:05.524 INFO:tasks.rados.rados.0.smithi137.stdout:178: delete oid 11 current snap is 14 2024-09-21T08:25:05.524 INFO:tasks.rados.rados.0.smithi137.stdout:177: finishing rollback tid 1 to smithi13779221-19 2024-09-21T08:25:05.524 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 0 (ObjNum 129 snap 0 seq_num 3760922448) dirty dne 2024-09-21T08:25:05.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:05.530 INFO:tasks.rados.rados.0.smithi137.stdout:175: finishing copy_from racing read to smithi13779221-15 2024-09-21T08:25:05.536 INFO:tasks.rados.rados.0.smithi137.stdout:177: done (2 left) 2024-09-21T08:25:05.536 INFO:tasks.rados.rados.0.smithi137.stdout:178: done (1 left) 2024-09-21T08:25:05.536 INFO:tasks.rados.rados.0.smithi137.stdout:179: snap_create 2024-09-21T08:25:05.539 INFO:tasks.rados.rados.0.smithi137.stdout:175: finishing copy_from to smithi13779221-15 2024-09-21T08:25:05.539 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 50 (ObjNum 64 snap 5 seq_num 64) dirty exists 2024-09-21T08:25:05.553 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:05.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:05.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:05.559 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:05.659 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:05.684 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:05.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:05.685 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:05.721 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:05 smithi137 ceph-mon[29634]: osdmap e111: 8 total, 8 up, 5 in 2024-09-21T08:25:05.721 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:05 smithi137 ceph-mon[41044]: osdmap e111: 8 total, 8 up, 5 in 2024-09-21T08:25:05.721 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:05 smithi137 bash[84025]: Trying to pull quay.io/prometheus/node-exporter:v1.5.0... 2024-09-21T08:25:05.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:05 smithi165 ceph-mon[32893]: osdmap e111: 8 total, 8 up, 5 in 2024-09-21T08:25:05.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:05.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:05.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:05.809 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:05.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:05.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:05.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:05.931 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:05.987 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:05 smithi137 bash[84025]: Getting image source signatures 2024-09-21T08:25:06.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:06.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:06.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:06.055 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:06.138 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:06.139 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:06.155 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:06.155 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:06.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:06.171 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:06.172 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:06.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:06.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:06.178 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:06.188 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:06.188 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:06.205 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:06.205 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:06.222 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:06.222 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:06.239 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:06.240 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:06.240 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:05 smithi137 bash[84025]: Copying blob sha256:c0e87333d3805a7b7e1c25783dd25f27da0fe70a8e262eee7ed190c8dc843fa7 2024-09-21T08:25:06.240 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:05 smithi137 bash[84025]: Copying blob sha256:5c12815fee558b157dee7f7509dedbaba0a8379098858a65ec869e1f1526ea0c 2024-09-21T08:25:06.240 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:05 smithi137 bash[84025]: Copying blob sha256:22b70bddd3acadc892fca4c2af4260629bfda5dfd11ebc106a93ce24e752b5ed 2024-09-21T08:25:06.259 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:06.260 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:06.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:06.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:06.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:06.305 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:06.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:06.427 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:06.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:06.428 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:06.519 INFO:tasks.rados.rados.0.smithi137.stdout:175: done (1 left) 2024-09-21T08:25:06.520 INFO:tasks.rados.rados.0.smithi137.stdout:179: done (0 left) 2024-09-21T08:25:06.520 INFO:tasks.rados.rados.0.smithi137.stdout:180: write (excl) oid 12 current snap is 15 2024-09-21T08:25:06.520 INFO:tasks.rados.rados.0.smithi137.stdout:180: seq_num 93 ranges {763051=794393,2158943=647020,3437750=125301} 2024-09-21T08:25:06.528 INFO:tasks.rados.rados.0.smithi137.stdout:180: writing smithi13779221-12 from 763051 to 1557444 tid 1 2024-09-21T08:25:06.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:06.536 INFO:tasks.rados.rados.0.smithi137.stdout:180: writing smithi13779221-12 from 2158943 to 2805963 tid 2 2024-09-21T08:25:06.540 INFO:tasks.rados.rados.0.smithi137.stdout:180: writing smithi13779221-12 from 3437750 to 3563051 tid 3 2024-09-21T08:25:06.540 INFO:tasks.rados.rados.0.smithi137.stdout:181: snap_create 2024-09-21T08:25:06.540 INFO:tasks.rados.rados.0.smithi137.stdout:180: finishing write tid 1 to smithi13779221-12 2024-09-21T08:25:06.541 INFO:tasks.rados.rados.0.smithi137.stdout:180: finishing write tid 2 to smithi13779221-12 2024-09-21T08:25:06.542 INFO:tasks.rados.rados.0.smithi137.stdout:180: finishing write tid 3 to smithi13779221-12 2024-09-21T08:25:06.542 INFO:tasks.rados.rados.0.smithi137.stdout:180: finishing write tid 4 to smithi13779221-12 2024-09-21T08:25:06.542 INFO:tasks.rados.rados.0.smithi137.stdout:180: finishing write tid 5 to smithi13779221-12 2024-09-21T08:25:06.542 INFO:tasks.rados.rados.0.smithi137.stdout:180: oid 12 updating version 0 to 72 2024-09-21T08:25:06.542 INFO:tasks.rados.rados.0.smithi137.stdout:180: oid 12 version 72 is already newer than 71 2024-09-21T08:25:06.542 INFO:tasks.rados.rados.0.smithi137.stdout:180: oid 12 updating version 72 to 73 2024-09-21T08:25:06.542 INFO:tasks.rados.rados.0.smithi137.stdout:180: oid 12 version 73 is already newer than 70 2024-09-21T08:25:06.542 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 73 (ObjNum 93 snap 15 seq_num 93) dirty exists 2024-09-21T08:25:06.542 INFO:tasks.rados.rados.0.smithi137.stdout:180: left oid 12 (ObjNum 93 snap 15 seq_num 93) 2024-09-21T08:25:06.554 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:06.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:06.555 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:06.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:06.680 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:06.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:06.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:06.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:06 smithi137 ceph-mon[29634]: osdmap e112: 8 total, 8 up, 5 in 2024-09-21T08:25:06.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:06 smithi137 ceph-mon[29634]: pgmap v73: 148 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 144 active+clean; 129 MiB data, 405 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:06.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:06 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3262694693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:06.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:06 smithi137 ceph-mon[41044]: osdmap e112: 8 total, 8 up, 5 in 2024-09-21T08:25:06.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:06 smithi137 ceph-mon[41044]: pgmap v73: 148 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 144 active+clean; 129 MiB data, 405 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:06.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:06 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3262694693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:06.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:06.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:06.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:06.805 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:06.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:06.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:06.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:06.932 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:06.938 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:25:06.938 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:25:06.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:25:06.938 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:25:06.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.938 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44417.0:49 5.0 5:05bf5b68:::notify.1:head [watch watch cookie 94074441549248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:25:06.938 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:34.575056+0000", 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 92.308944928000003, 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020510799999999998, 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44417", 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 49 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575056+0000", 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575056+0000", 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575057+0000", 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0869999999999999e-06 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575060+0000", 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3189999999999999e-06 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575072+0000", 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1497e-05 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575076+0000", 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.437e-06 2024-09-21T08:25:06.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575414+0000", 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033762100000000003 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575490+0000", 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6458000000000006e-05 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.576135+0000", 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064504000000000004 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.576216+0000", 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1156999999999996e-05 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.576330+0000", 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113461 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.576666+0000", 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033670100000000001 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.576964+0000", 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029736500000000001 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.577005+0000", 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1458999999999997e-05 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.577049+0000", 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.401e-05 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.577107+0000", 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7467999999999999e-05 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44417.0:59 5.14 5:2b04a3e9:::notify.6:head [watch watch cookie 94074441582864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:25:06.945 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:34.582635+0000", 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 92.301365325000006, 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001918236, 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44417", 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 59 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582635+0000", 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582635+0000", 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582637+0000", 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.393e-06 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582641+0000", 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5170000000000001e-06 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582649+0000", 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1200000000000002e-06 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582656+0000", 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.917e-06 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582973+0000", 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031646400000000002 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.583053+0000", 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0791999999999995e-05 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.583691+0000", 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00063707800000000001 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.583750+0000", 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9629000000000001e-05 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.583840+0000", 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0261999999999997e-05 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.584190+0000", 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035001200000000002 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.584384+0000", 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000193637 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.584483+0000", 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9339999999999994e-05 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.584515+0000", 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1498000000000002e-05 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.584553+0000", 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8577e-05 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:359 4.19 4:9ed7f5b7:::obj_delete_at_hint.0000000004:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:37.648931+0000", 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 89.235069445999997, 2024-09-21T08:25:06.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025273050000000001, 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 359 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648931+0000", 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648931+0000", 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648932+0000", 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.164e-06 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648937+0000", 2024-09-21T08:25:06.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9529999999999998e-06 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648948+0000", 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0589e-05 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648955+0000", 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0230000000000004e-06 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.649000+0000", 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4855999999999999e-05 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.649071+0000", 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1392999999999999e-05 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.649224+0000", 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015272800000000001 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:06.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.649703+0000", 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047908 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.650827+0000", 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011241090000000001 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.651380+0000", 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00055312099999999995 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.651424+0000", 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3677999999999998e-05 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.651458+0000", 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4610999999999998e-05 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:993 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 94837804043520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-21T08:25:06.958 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:55.596150+0000", 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 71.287850215000006, 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034626729999999999, 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 993 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596150+0000", 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596150+0000", 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596152+0000", 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.186e-06 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596156+0000", 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8359999999999999e-06 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596164+0000", 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4460000000000004e-06 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596172+0000", 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9610000000000005e-06 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596241+0000", 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9238000000000004e-05 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596247+0000", 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6210000000000001e-06 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599458+0000", 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003211546 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599525+0000", 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6511000000000001e-05 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599613+0000", 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8327999999999999e-05 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.963 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:994 5.14 5:2b04a3e9:::notify.6:head [watch ping cookie 94837804033152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:55.596282+0000", 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 71.287718695999999, 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029709879999999999, 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 994 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596282+0000", 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596282+0000", 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596283+0000", 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9599999999999998e-07 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596287+0000", 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7799999999999998e-06 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596296+0000", 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1360000000000003e-06 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596302+0000", 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1890000000000004e-06 2024-09-21T08:25:06.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596377+0000", 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5512999999999993e-05 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596384+0000", 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1619999999999996e-06 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599087+0000", 2024-09-21T08:25:06.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002703488 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599154+0000", 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7207999999999998e-05 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599253+0000", 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8616000000000006e-05 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:1082 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94837804043520 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 28.724475177999999, 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.39751146399999998, 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1082 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.969 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159528+0000", 2024-09-21T08:25:06.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2749999999999998e-06 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159533+0000", 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2480000000000004e-06 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159547+0000", 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4516e-05 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159556+0000", 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9460000000000004e-06 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159617+0000", 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0999000000000003e-05 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159625+0000", 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240000000000007e-06 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544062+0000", 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.38443652099999998 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544359+0000", 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000297378 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548773+0000", 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044136440000000004 2024-09-21T08:25:06.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548858+0000", 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5638999999999998e-05 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.557037+0000", 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081783740000000004 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(osd.0.0:103 7.6 e91/80 7:611b1c4b:::smithi13779221-3:3 v 91'13, mlcod=89'10)", 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:44.494057+0000", 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 22.389943337999998, 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043783829999999996, 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494057+0000", 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494057+0000", 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494058+0000", 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6600000000000005e-07 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494070+0000", 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2245e-05 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494080+0000", 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4299999999999995e-06 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494089+0000", 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.662e-06 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494146+0000", 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6526000000000001e-05 2024-09-21T08:25:06.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494167+0000", 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0690999999999999e-05 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.498394+0000", 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042278339999999998 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.498436+0000", 2024-09-21T08:25:06.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1128999999999999e-05 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:255 7.2 7:4031afd1:::smithi13779221-48:head [copy-from ver 3] snapc 4=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 21.376164784, 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015259864, 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:06.979 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 255 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507839+0000", 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0089999999999999e-06 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507852+0000", 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3617e-05 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507872+0000", 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9482000000000001e-05 2024-09-21T08:25:06.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507883+0000", 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1036000000000001e-05 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507941+0000", 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7819999999999999e-05 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507946+0000", 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9080000000000003e-06 2024-09-21T08:25:06.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510282+0000", 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023368170000000001 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510376+0000", 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3846999999999998e-05 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.516270+0000", 2024-09-21T08:25:06.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005893934 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522776+0000", 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065057179999999997 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522902+0000", 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012641 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522990+0000", 2024-09-21T08:25:06.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7491000000000006e-05 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523050+0000", 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0075999999999999e-05 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523096+0000", 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5698999999999997e-05 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:256 7.2 7:4031afd1:::smithi13779221-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 21.376097485999999, 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013379356, 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 256 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.986 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507904+0000", 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8000000000000005e-07 2024-09-21T08:25:06.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507907+0000", 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.258e-06 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507915+0000", 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.571e-06 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507920+0000", 2024-09-21T08:25:06.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1189999999999996e-06 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507998+0000", 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8181000000000002e-05 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.508004+0000", 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3960000000000003e-06 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510529+0000", 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025244989999999999 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511053+0000", 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052380199999999997 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521102+0000", 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010049361999999999 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521183+0000", 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0495999999999996e-05 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521282+0000", 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9891999999999998e-05 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(osd.3.25:1 7.3 7:c851e2c3:::smithi13779221-25:head [assert-version v3,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:25:06.991 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.511125+0000", 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 21.372875710999999, 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032274180000000001, 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511125+0000", 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511125+0000", 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511127+0000", 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.689e-06 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511133+0000", 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0599999999999996e-06 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511143+0000", 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.039e-05 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511151+0000", 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3569999999999998e-06 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511209+0000", 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7954000000000001e-05 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511273+0000", 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3498999999999996e-05 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.514352+0000", 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003079469 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.995 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:262 7.a 7:53537bd3:::smithi13779221-39:5 [sparse-read 0~1161879,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554307+0000", 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.329693375000002, 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030460629999999999, 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 262 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554307+0000", 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554307+0000", 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554310+0000", 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4269999999999998e-06 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554319+0000", 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0149999999999992e-06 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554332+0000", 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3229e-05 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554343+0000", 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0669999999999999e-05 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554401+0000", 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8100000000000003e-05 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554575+0000", 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000174461 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557353+0000", 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027781609999999999 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:06.999 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:263 7.a 7:53537bd3:::smithi13779221-39:5 [sparse-read 0~1161879] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554362+0000", 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.329638286000002, 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032289990000000002, 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:07.000 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 263 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554362+0000", 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554362+0000", 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:07.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554363+0000", 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3200000000000004e-07 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554367+0000", 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0980000000000004e-06 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554374+0000", 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6699999999999997e-06 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554380+0000", 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6339999999999998e-06 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557390+0000", 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030107010000000002 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557479+0000", 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8479000000000004e-05 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557591+0000", 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011258499999999999 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:264 7.a 7:53537bd3:::smithi13779221-39:5 [sparse-read 0~1161879] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554387+0000", 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.329613309999999, 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045927440000000002, 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:07.004 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 264 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554387+0000", 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554387+0000", 2024-09-21T08:25:07.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554388+0000", 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.37e-07 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554392+0000", 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7120000000000002e-06 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:07.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554397+0000", 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6810000000000003e-06 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554402+0000", 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6990000000000002e-06 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558684+0000", 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042816920000000001 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558800+0000", 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116122 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558980+0000", 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180101 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:260 7.8 e93/84 7:1fd29771:::smithi13779221-22:head v 93'20, mlcod=86'17)", 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:25:07.008 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.315051152999999, 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078947310000000003, 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568951+0000", 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1510000000000001e-06 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571221+0000", 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022708279999999999 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571244+0000", 2024-09-21T08:25:07.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2759999999999999e-05 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571257+0000", 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2669e-05 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571601+0000", 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034448900000000001 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:07.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571645+0000", 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3498999999999998e-05 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576787+0000", 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051417010000000003 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576844+0000", 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7633999999999997e-05 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:283 7.2 7:43ab705f:::smithi13779221-10:head [read 0~2151263,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762012+0000", 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.121988414, 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044470589999999997, 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 283 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.013 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762012+0000", 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762012+0000", 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762013+0000", 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9599999999999998e-07 2024-09-21T08:25:07.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762028+0000", 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5381000000000001e-05 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762036+0000", 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5290000000000003e-06 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762042+0000", 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0399999999999998e-06 2024-09-21T08:25:07.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.763797+0000", 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001754898 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.763899+0000", 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102248 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766459+0000", 2024-09-21T08:25:07.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025601669999999999 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:284 7.2 7:43ab705f:::smithi13779221-10:head [sparse-read 0~1051225] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762051+0000", 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.121949280999999, 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047848099999999996, 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:07.017 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 284 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762051+0000", 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762051+0000", 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762052+0000", 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.85e-07 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762055+0000", 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.723e-06 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762060+0000", 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4890000000000003e-06 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762065+0000", 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0019999999999997e-06 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766502+0000", 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044364219999999998 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766636+0000", 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133831 2024-09-21T08:25:07.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766836+0000", 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020055800000000001 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:294 7.2 7:4d793d31:::smithi13779221-9:head [copy-from ver 3] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.121334356999999, 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017337353, 2024-09-21T08:25:07.021 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 294 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762668+0000", 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.353e-06 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:07.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762673+0000", 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1569999999999999e-06 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762683+0000", 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0331e-05 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762688+0000", 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1410000000000004e-06 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:07.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766904+0000", 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042154899999999997 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767004+0000", 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100135 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.771465+0000", 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044613170000000002 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779379+0000", 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079137649999999997 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779883+0000", 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050370000000000005 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779896+0000", 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3458999999999999e-05 2024-09-21T08:25:07.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779950+0000", 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3851000000000003e-05 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.780004+0000", 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3653999999999998e-05 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:295 7.2 7:4d793d31:::smithi13779221-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.121303639000001, 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012332286, 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 295 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4400000000000001e-07 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762699+0000", 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0099999999999998e-06 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762705+0000", 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0279999999999997e-06 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762710+0000", 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8210000000000001e-06 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767340+0000", 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046298770000000001 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.768012+0000", 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067169199999999999 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774900+0000", 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068879680000000004 2024-09-21T08:25:07.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774961+0000", 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0980999999999998e-05 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.775029+0000", 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8465000000000003e-05 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(osd.3.25:2 7.2 7:48ea7305:::smithi13779221-5:head [assert-version v3,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.767305+0000", 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.11669573, 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002516744, 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767305+0000", 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767305+0000", 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767311+0000", 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2809999999999997e-06 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767331+0000", 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9417000000000001e-05 2024-09-21T08:25:07.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767352+0000", 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1790999999999999e-05 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767366+0000", 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.376e-05 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:07.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.768057+0000", 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069047800000000001 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.768156+0000", 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9221000000000004e-05 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.769822+0000", 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001665796 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:321 7.c e93/80 7:3cae00e7:::smithi13779221-28:head v 93'11, mlcod=86'8)", 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.815621+0000", 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.068379182000001, 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054018499999999997, 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.815621+0000", 2024-09-21T08:25:07.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.815621+0000", 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.815622+0000", 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9900000000000001e-07 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.818864+0000", 2024-09-21T08:25:07.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032423980000000001 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.818878+0000", 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3699e-05 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.818886+0000", 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0960000000000001e-06 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.818961+0000", 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5345000000000005e-05 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.818989+0000", 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.811e-05 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.820957+0000", 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001967856 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:07.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.821023+0000", 2024-09-21T08:25:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5946999999999997e-05 2024-09-21T08:25:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:07.041 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:07.041 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:25:07.041 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:07.042 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:06 smithi165 ceph-mon[32893]: osdmap e112: 8 total, 8 up, 5 in 2024-09-21T08:25:07.042 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:06 smithi165 ceph-mon[32893]: pgmap v73: 148 pgs: 1 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 144 active+clean; 129 MiB data, 405 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:07.042 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:06 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3262694693' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:07.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:07.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:07.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:07.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:07.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:07.192 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:07.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:07.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:07.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:07.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:07.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:07.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:07.441 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:07.441 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:07.522 INFO:tasks.rados.rados.0.smithi137.stdout:180: done (1 left) 2024-09-21T08:25:07.523 INFO:tasks.rados.rados.0.smithi137.stdout:181: done (0 left) 2024-09-21T08:25:07.523 INFO:tasks.rados.rados.0.smithi137.stdout:182: delete oid 7 current snap is 16 2024-09-21T08:25:07.527 INFO:tasks.rados.rados.0.smithi137.stdout:182: done (0 left) 2024-09-21T08:25:07.527 INFO:tasks.rados.rados.0.smithi137.stdout:183: snap_create 2024-09-21T08:25:07.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:07.563 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:07.564 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:07.564 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:07.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:07.691 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:07.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:07.693 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:07.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:07 smithi137 ceph-mon[29634]: osdmap e113: 8 total, 8 up, 5 in 2024-09-21T08:25:07.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:07 smithi137 bash[84025]: Copying config sha256:0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 2024-09-21T08:25:07.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:07 smithi137 ceph-mon[41044]: osdmap e113: 8 total, 8 up, 5 in 2024-09-21T08:25:07.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:07.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:07.819 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:07.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:07.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:07.944 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:07.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:07.945 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:07 smithi165 ceph-mon[32893]: osdmap e113: 8 total, 8 up, 5 in 2024-09-21T08:25:08.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:08.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:08.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:08.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:08.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:08.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:08.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:08.190 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:08.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:08.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:08.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:08.318 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:08.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:08.443 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:08.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:08.444 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:08.517 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:08 smithi137 bash[84025]: Writing manifest to image destination 2024-09-21T08:25:08.527 INFO:tasks.rados.rados.0.smithi137.stdout:183: done (0 left) 2024-09-21T08:25:08.527 INFO:tasks.rados.rados.0.smithi137.stdout:184: read oid 32 snap 13 2024-09-21T08:25:08.527 INFO:tasks.rados.rados.0.smithi137.stdout:184: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-21T08:25:08.528 INFO:tasks.rados.rados.0.smithi137.stdout:185: snap_create 2024-09-21T08:25:08.534 INFO:tasks.rados.rados.0.smithi137.stdout:184: expect (ObjNum 31 snap 0 seq_num 31) 2024-09-21T08:25:08.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:08.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:08.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:08.571 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:08.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:08.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:08.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:08.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:08.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:08.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:08 smithi137 ceph-mon[29634]: osdmap e114: 8 total, 8 up, 5 in 2024-09-21T08:25:08.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:08 smithi137 ceph-mon[29634]: pgmap v76: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 130 MiB data, 413 MiB used, 447 GiB / 447 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-21T08:25:08.812 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:08 smithi137 ceph-mon[41044]: osdmap e114: 8 total, 8 up, 5 in 2024-09-21T08:25:08.812 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:08 smithi137 ceph-mon[41044]: pgmap v76: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 130 MiB data, 413 MiB used, 447 GiB / 447 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-21T08:25:08.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:08.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:08.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:08.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:08.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:08.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:08.946 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:08 smithi165 ceph-mon[32893]: osdmap e114: 8 total, 8 up, 5 in 2024-09-21T08:25:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:08 smithi165 ceph-mon[32893]: pgmap v76: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 130 MiB data, 413 MiB used, 447 GiB / 447 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-21T08:25:09.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:09.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:09.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:09.069 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:09.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:09.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:09.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:09.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:09.267 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:08 smithi137 podman[84025]: 2024-09-21 08:25:08.763705179 +0000 UTC m=+3.907609544 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T08:25:09.268 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:08 smithi137 podman[84025]: 2024-09-21 08:25:08.938895936 +0000 UTC m=+4.082800287 container create 3a02837850704e13e22adccfb498f3c7f1f618c743f0abfc726946d3fc9c6b38 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T08:25:09.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:09.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:09.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:09.319 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:09.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:09.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:09.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:09.443 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:09.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T08:25:09.529 INFO:tasks.rados.rados.0.smithi137.stdout:184: done (1 left) 2024-09-21T08:25:09.529 INFO:tasks.rados.rados.0.smithi137.stdout:185: done (0 left) 2024-09-21T08:25:09.529 INFO:tasks.rados.rados.0.smithi137.stdout:186: copy_from oid 19 from oid 22 current snap is 18 2024-09-21T08:25:09.529 INFO:tasks.rados.rados.0.smithi137.stdout:187: read oid 21 snap 14 2024-09-21T08:25:09.529 INFO:tasks.rados.rados.0.smithi137.stdout:187: expect (ObjNum 82 snap 10 seq_num 82) 2024-09-21T08:25:09.529 INFO:tasks.rados.rados.0.smithi137.stdout:188: write oid 49 current snap is 18 2024-09-21T08:25:09.529 INFO:tasks.rados.rados.0.smithi137.stdout:188: seq_num 95 ranges {796665=544891,1759554=650689,2976243=620422} 2024-09-21T08:25:09.536 INFO:tasks.rados.rados.0.smithi137.stdout:188: writing smithi13779221-49 from 796665 to 1341556 tid 1 2024-09-21T08:25:09.543 INFO:tasks.rados.rados.0.smithi137.stdout:188: writing smithi13779221-49 from 1759554 to 2410243 tid 2 2024-09-21T08:25:09.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:09.551 INFO:tasks.rados.rados.0.smithi137.stdout:188: writing smithi13779221-49 from 2976243 to 3596665 tid 3 2024-09-21T08:25:09.551 INFO:tasks.rados.rados.0.smithi137.stdout:189: read oid 24 snap -1 2024-09-21T08:25:09.551 INFO:tasks.rados.rados.0.smithi137.stdout:189: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-21T08:25:09.551 INFO:tasks.rados.rados.0.smithi137.stdout:190: delete oid 43 current snap is 18 2024-09-21T08:25:09.551 INFO:tasks.rados.rados.0.smithi137.stdout:187: expect (ObjNum 82 snap 10 seq_num 82) 2024-09-21T08:25:09.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:09.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:09.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:09.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:09.688 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:09.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:09.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:09.741 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:09.744 INFO:tasks.rados.rados.0.smithi137.stdout:186: finishing copy_from racing read to smithi13779221-19 2024-09-21T08:25:09.744 INFO:tasks.rados.rados.0.smithi137.stdout:188: finishing write tid 1 to smithi13779221-49 2024-09-21T08:25:09.744 INFO:tasks.rados.rados.0.smithi137.stdout:186: finishing copy_from to smithi13779221-19 2024-09-21T08:25:09.744 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 34 (ObjNum 86 snap 11 seq_num 86) dirty exists 2024-09-21T08:25:09.744 INFO:tasks.rados.rados.0.smithi137.stdout:188: finishing write tid 2 to smithi13779221-49 2024-09-21T08:25:09.744 INFO:tasks.rados.rados.0.smithi137.stdout:189: expect (ObjNum 0 snap 0 seq_num 0) 2024-09-21T08:25:09.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:09 smithi165 ceph-mon[32893]: osdmap e115: 8 total, 8 up, 5 in 2024-09-21T08:25:09.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-mon[29634]: osdmap e115: 8 total, 8 up, 5 in 2024-09-21T08:25:09.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-mon[41044]: osdmap e115: 8 total, 8 up, 5 in 2024-09-21T08:25:09.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 podman[84025]: 2024-09-21 08:25:09.56885684 +0000 UTC m=+4.712761195 container init 3a02837850704e13e22adccfb498f3c7f1f618c743f0abfc726946d3fc9c6b38 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T08:25:09.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 podman[84025]: 2024-09-21 08:25:09.575779815 +0000 UTC m=+4.719684176 container start 3a02837850704e13e22adccfb498f3c7f1f618c743f0abfc726946d3fc9c6b38 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T08:25:09.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.579Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T08:25:09.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.579Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T08:25:09.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T08:25:09.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T08:25:09.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T08:25:09.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T08:25:09.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T08:25:09.769 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T08:25:09.769 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T08:25:09.769 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T08:25:09.769 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T08:25:09.769 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T08:25:09.769 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T08:25:09.769 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.580Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T08:25:09.770 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T08:25:09.771 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[84664]: ts=2024-09-21T08:25:09.581Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 bash[84025]: 3a02837850704e13e22adccfb498f3c7f1f618c743f0abfc726946d3fc9c6b38 2024-09-21T08:25:09.772 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:09 smithi137 systemd[1]: Started Ceph node-exporter.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:25:09.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:09.810 INFO:tasks.rados.rados.0.smithi137.stdout:188: finishing write tid 3 to smithi13779221-49 2024-09-21T08:25:09.810 INFO:tasks.rados.rados.0.smithi137.stdout:188: finishing write tid 4 to smithi13779221-49 2024-09-21T08:25:09.810 INFO:tasks.rados.rados.0.smithi137.stdout:188: finishing write tid 5 to smithi13779221-49 2024-09-21T08:25:09.811 INFO:tasks.rados.rados.0.smithi137.stdout:188: oid 49 updating version 0 to 70 2024-09-21T08:25:09.811 INFO:tasks.rados.rados.0.smithi137.stdout:188: oid 49 updating version 70 to 71 2024-09-21T08:25:09.811 INFO:tasks.rados.rados.0.smithi137.stdout:188: oid 49 version 71 is already newer than 69 2024-09-21T08:25:09.811 INFO:tasks.rados.rados.0.smithi137.stdout:188: oid 49 version 71 is already newer than 68 2024-09-21T08:25:09.811 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 71 (ObjNum 95 snap 18 seq_num 95) dirty exists 2024-09-21T08:25:09.811 INFO:tasks.rados.rados.0.smithi137.stdout:188: left oid 49 (ObjNum 95 snap 18 seq_num 95) 2024-09-21T08:25:09.812 INFO:tasks.rados.rados.0.smithi137.stdout:186: done (4 left) 2024-09-21T08:25:09.813 INFO:tasks.rados.rados.0.smithi137.stdout:187: done (3 left) 2024-09-21T08:25:09.813 INFO:tasks.rados.rados.0.smithi137.stdout:188: done (2 left) 2024-09-21T08:25:09.813 INFO:tasks.rados.rados.0.smithi137.stdout:189: done (1 left) 2024-09-21T08:25:09.813 INFO:tasks.rados.rados.0.smithi137.stdout:190: done (0 left) 2024-09-21T08:25:09.813 INFO:tasks.rados.rados.0.smithi137.stdout:191: copy_from oid 7 from oid 44 current snap is 18 2024-09-21T08:25:09.813 INFO:tasks.rados.rados.0.smithi137.stdout:192: snap_create 2024-09-21T08:25:09.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:09.814 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:09.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:09.819 INFO:tasks.rados.rados.0.smithi137.stdout:191: finishing copy_from racing read to smithi13779221-7 2024-09-21T08:25:09.822 INFO:tasks.rados.rados.0.smithi137.stdout:191: finishing copy_from to smithi13779221-7 2024-09-21T08:25:09.822 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 40 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-21T08:25:09.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:09.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:09.939 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:09.939 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:10.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:10.062 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:10.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:10.063 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:10.164 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:10.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:10.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:10.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:10.225 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:25:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:25:10.225 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:25:10.225 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:25:10.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:10.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:10.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:10.312 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:10.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:10.434 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:10.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:10.435 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:10.437 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 7] out_osds: [4, 3, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:25:10.437 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:25:10.438 INFO:tasks.thrashosds.thrasher:fixing pg num pool .mgr 2024-09-21T08:25:10.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:25:10.535 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:25:10.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:10.546 INFO:tasks.rados.rados.0.smithi137.stdout:191: done (1 left) 2024-09-21T08:25:10.547 INFO:tasks.rados.rados.0.smithi137.stdout:192: done (0 left) 2024-09-21T08:25:10.547 INFO:tasks.rados.rados.0.smithi137.stdout:193: copy_from oid 41 from oid 40 current snap is 19 2024-09-21T08:25:10.547 INFO:tasks.rados.rados.0.smithi137.stdout:194: copy_from oid 6 from oid 47 current snap is 19 2024-09-21T08:25:10.547 INFO:tasks.rados.rados.0.smithi137.stdout:195: read oid 3 snap -1 2024-09-21T08:25:10.547 INFO:tasks.rados.rados.0.smithi137.stdout:195: expect (ObjNum 61 snap 4 seq_num 61) 2024-09-21T08:25:10.547 INFO:tasks.rados.rados.0.smithi137.stdout:196: rollback oid 48 current snap is 19 2024-09-21T08:25:10.548 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 48 to 18 2024-09-21T08:25:10.548 INFO:tasks.rados.rados.0.smithi137.stdout:197: write (excl) oid 45 current snap is 19 2024-09-21T08:25:10.548 INFO:tasks.rados.rados.0.smithi137.stdout:197: seq_num 97 ranges {430279=695389,1876518=654358,3031089=536797} 2024-09-21T08:25:10.555 INFO:tasks.rados.rados.0.smithi137.stdout:197: writing smithi13779221-45 from 430279 to 1125668 tid 1 2024-09-21T08:25:10.564 INFO:tasks.rados.rados.0.smithi137.stdout:197: writing smithi13779221-45 from 1876518 to 2530876 tid 2 2024-09-21T08:25:10.574 INFO:tasks.rados.rados.0.smithi137.stdout:197: writing smithi13779221-45 from 3031089 to 3567886 tid 3 2024-09-21T08:25:10.574 INFO:tasks.rados.rados.0.smithi137.stdout:198: read oid 50 snap -1 2024-09-21T08:25:10.575 INFO:tasks.rados.rados.0.smithi137.stdout:198: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-21T08:25:10.575 INFO:tasks.rados.rados.0.smithi137.stdout:199: write (excl) oid 44 current snap is 19 2024-09-21T08:25:10.575 INFO:tasks.rados.rados.0.smithi137.stdout:199: seq_num 98 ranges {447086=770638,1935000=798016,3458512=188574} 2024-09-21T08:25:10.583 INFO:tasks.rados.rados.0.smithi137.stdout:199: writing smithi13779221-44 from 447086 to 1217724 tid 1 2024-09-21T08:25:10.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:10.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:10.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:10.594 INFO:tasks.rados.rados.0.smithi137.stdout:199: writing smithi13779221-44 from 1935000 to 2733016 tid 2 2024-09-21T08:25:10.599 INFO:tasks.rados.rados.0.smithi137.stdout:199: writing smithi13779221-44 from 3458512 to 3647086 tid 3 2024-09-21T08:25:10.599 INFO:tasks.rados.rados.0.smithi137.stdout:195: expect (ObjNum 61 snap 4 seq_num 61) 2024-09-21T08:25:10.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:10.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:10.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:10.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:197: finishing write tid 1 to smithi13779221-45 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:193: finishing copy_from racing read to smithi13779221-41 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:196: finishing rollback tid 0 to smithi13779221-48 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:195: done (6 left) 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:200: copy_from oid 20 from oid 7 current snap is 19 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:196: finishing rollback tid 1 to smithi13779221-48 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:196: finishing rollback tid 2 to smithi13779221-48 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 77 (ObjNum 81 snap 9 seq_num 81) dirty exists 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:197: finishing write tid 2 to smithi13779221-45 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:193: finishing copy_from to smithi13779221-41 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 7 (ObjNum 53 snap 2 seq_num 53) dirty exists 2024-09-21T08:25:10.739 INFO:tasks.rados.rados.0.smithi137.stdout:194: finishing copy_from racing read to smithi13779221-6 2024-09-21T08:25:10.740 INFO:tasks.rados.rados.0.smithi137.stdout:197: finishing write tid 3 to smithi13779221-45 2024-09-21T08:25:10.740 INFO:tasks.rados.rados.0.smithi137.stdout:197: finishing write tid 4 to smithi13779221-45 2024-09-21T08:25:10.740 INFO:tasks.rados.rados.0.smithi137.stdout:198: expect (ObjNum 49 snap 0 seq_num 49) 2024-09-21T08:25:10.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:10 smithi165 ceph-mon[32893]: osdmap e116: 8 total, 8 up, 5 in 2024-09-21T08:25:10.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:10 smithi165 ceph-mon[32893]: pgmap v79: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 130 MiB data, 413 MiB used, 447 GiB / 447 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-21T08:25:10.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:10 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4103416215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:10.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:10 smithi137 ceph-mon[29634]: osdmap e116: 8 total, 8 up, 5 in 2024-09-21T08:25:10.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:10 smithi137 ceph-mon[29634]: pgmap v79: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 130 MiB data, 413 MiB used, 447 GiB / 447 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-21T08:25:10.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:10 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4103416215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:10.770 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:25:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:25:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:25:10.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:10 smithi137 ceph-mon[41044]: osdmap e116: 8 total, 8 up, 5 in 2024-09-21T08:25:10.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:10 smithi137 ceph-mon[41044]: pgmap v79: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 130 MiB data, 413 MiB used, 447 GiB / 447 GiB avail; 3.3 MiB/s rd, 2.2 MiB/s wr, 8 op/s 2024-09-21T08:25:10.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:10 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4103416215' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:10.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:10.828 INFO:tasks.rados.rados.0.smithi137.stdout:197: finishing write tid 5 to smithi13779221-45 2024-09-21T08:25:10.828 INFO:tasks.rados.rados.0.smithi137.stdout:197: oid 45 updating version 0 to 44 2024-09-21T08:25:10.828 INFO:tasks.rados.rados.0.smithi137.stdout:197: oid 45 updating version 44 to 45 2024-09-21T08:25:10.828 INFO:tasks.rados.rados.0.smithi137.stdout:197: oid 45 version 45 is already newer than 42 2024-09-21T08:25:10.828 INFO:tasks.rados.rados.0.smithi137.stdout:197: oid 45 version 45 is already newer than 43 2024-09-21T08:25:10.828 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 45 (ObjNum 97 snap 19 seq_num 97) dirty exists 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:197: left oid 45 (ObjNum 97 snap 19 seq_num 97) 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:194: finishing copy_from to smithi13779221-6 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 13 (ObjNum 62 snap 4 seq_num 62) dirty exists 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:199: finishing write tid 1 to smithi13779221-44 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:199: finishing write tid 2 to smithi13779221-44 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:199: finishing write tid 3 to smithi13779221-44 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:193: done (6 left) 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:194: done (5 left) 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:196: done (4 left) 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:197: done (3 left) 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:198: done (2 left) 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:201: write (excl) oid 34 current snap is 19 2024-09-21T08:25:10.829 INFO:tasks.rados.rados.0.smithi137.stdout:199: finishing write tid 4 to smithi13779221-44 2024-09-21T08:25:10.830 INFO:tasks.rados.rados.0.smithi137.stdout:201: seq_num 99 ranges {463893=562240,1593482=658027,2685935=559341} 2024-09-21T08:25:10.833 INFO:tasks.rados.rados.0.smithi137.stdout:201: writing smithi13779221-34 from 463893 to 1026133 tid 1 2024-09-21T08:25:10.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:10.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:10.840 INFO:tasks.rados.rados.0.smithi137.stdout:201: writing smithi13779221-34 from 1593482 to 2251509 tid 2 2024-09-21T08:25:10.846 INFO:tasks.rados.rados.0.smithi137.stdout:201: writing smithi13779221-34 from 2685935 to 3245276 tid 3 2024-09-21T08:25:10.846 INFO:tasks.rados.rados.0.smithi137.stdout:202: snap_create 2024-09-21T08:25:10.846 INFO:tasks.rados.rados.0.smithi137.stdout:199: finishing write tid 5 to smithi13779221-44 2024-09-21T08:25:10.846 INFO:tasks.rados.rados.0.smithi137.stdout:199: oid 44 updating version 0 to 45 2024-09-21T08:25:10.846 INFO:tasks.rados.rados.0.smithi137.stdout:199: oid 44 version 45 is already newer than 44 2024-09-21T08:25:10.846 INFO:tasks.rados.rados.0.smithi137.stdout:199: oid 44 updating version 45 to 46 2024-09-21T08:25:10.846 INFO:tasks.rados.rados.0.smithi137.stdout:199: oid 44 version 46 is already newer than 43 2024-09-21T08:25:10.846 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 46 (ObjNum 98 snap 19 seq_num 98) dirty exists 2024-09-21T08:25:10.846 INFO:tasks.rados.rados.0.smithi137.stdout:199: left oid 44 (ObjNum 98 snap 19 seq_num 98) 2024-09-21T08:25:10.846 INFO:tasks.rados.rados.0.smithi137.stdout:200: finishing copy_from racing read to smithi13779221-20 2024-09-21T08:25:10.847 INFO:tasks.rados.rados.0.smithi137.stdout:200: finishing copy_from to smithi13779221-20 2024-09-21T08:25:10.847 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 40 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-21T08:25:10.847 INFO:tasks.rados.rados.0.smithi137.stdout:201: finishing write tid 1 to smithi13779221-34 2024-09-21T08:25:10.847 INFO:tasks.rados.rados.0.smithi137.stdout:201: finishing write tid 2 to smithi13779221-34 2024-09-21T08:25:10.849 INFO:tasks.rados.rados.0.smithi137.stdout:201: finishing write tid 3 to smithi13779221-34 2024-09-21T08:25:10.849 INFO:tasks.rados.rados.0.smithi137.stdout:201: finishing write tid 4 to smithi13779221-34 2024-09-21T08:25:10.849 INFO:tasks.rados.rados.0.smithi137.stdout:201: finishing write tid 5 to smithi13779221-34 2024-09-21T08:25:10.849 INFO:tasks.rados.rados.0.smithi137.stdout:201: oid 34 updating version 0 to 76 2024-09-21T08:25:10.849 INFO:tasks.rados.rados.0.smithi137.stdout:201: oid 34 version 76 is already newer than 75 2024-09-21T08:25:10.849 INFO:tasks.rados.rados.0.smithi137.stdout:201: oid 34 updating version 76 to 77 2024-09-21T08:25:10.849 INFO:tasks.rados.rados.0.smithi137.stdout:201: oid 34 version 77 is already newer than 74 2024-09-21T08:25:10.849 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 77 (ObjNum 99 snap 19 seq_num 99) dirty exists 2024-09-21T08:25:10.849 INFO:tasks.rados.rados.0.smithi137.stdout:201: left oid 34 (ObjNum 99 snap 19 seq_num 99) 2024-09-21T08:25:10.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:25:10.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:10.956 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T08:25:10.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:10.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:10.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:11.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:11.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:11.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:11.081 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:11.182 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:11.205 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:11.206 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:11.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:11.207 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:11.308 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:11.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:11.331 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:11.331 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:11.432 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:11.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:11.454 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:11.490 INFO:tasks.rados.rados.0.smithi137.stdout:199: done (3 left) 2024-09-21T08:25:11.491 INFO:tasks.rados.rados.0.smithi137.stdout:200: done (2 left) 2024-09-21T08:25:11.491 INFO:tasks.rados.rados.0.smithi137.stdout:201: done (1 left) 2024-09-21T08:25:11.491 INFO:tasks.rados.rados.0.smithi137.stdout:202: done (0 left) 2024-09-21T08:25:11.491 INFO:tasks.rados.rados.0.smithi137.stdout:203: write oid 27 current snap is 20 2024-09-21T08:25:11.492 INFO:tasks.rados.rados.0.smithi137.stdout:203: seq_num 100 ranges {480700=637489,1651964=401685,2713358=770613} 2024-09-21T08:25:11.496 INFO:tasks.rados.rados.0.smithi137.stdout:203: writing smithi13779221-27 from 480700 to 1118189 tid 1 2024-09-21T08:25:11.503 INFO:tasks.rados.rados.0.smithi137.stdout:203: writing smithi13779221-27 from 1651964 to 2053649 tid 2 2024-09-21T08:25:11.511 INFO:tasks.rados.rados.0.smithi137.stdout:203: writing smithi13779221-27 from 2713358 to 3483971 tid 3 2024-09-21T08:25:11.511 INFO:tasks.rados.rados.0.smithi137.stdout:204: write (excl) oid 20 current snap is 20 2024-09-21T08:25:11.511 INFO:tasks.rados.rados.0.smithi137.stdout:204: seq_num 101 ranges {497507=712738,1710446=661696,3140781=556726} 2024-09-21T08:25:11.518 INFO:tasks.rados.rados.0.smithi137.stdout:204: writing smithi13779221-20 from 497507 to 1210245 tid 1 2024-09-21T08:25:11.525 INFO:tasks.rados.rados.0.smithi137.stdout:204: writing smithi13779221-20 from 1710446 to 2372142 tid 2 2024-09-21T08:25:11.532 INFO:tasks.rados.rados.0.smithi137.stdout:204: writing smithi13779221-20 from 3140781 to 3697507 tid 3 2024-09-21T08:25:11.532 INFO:tasks.rados.rados.0.smithi137.stdout:205: write (excl) oid 5 current snap is 20 2024-09-21T08:25:11.532 INFO:tasks.rados.rados.0.smithi137.stdout:203: finishing write tid 1 to smithi13779221-27 2024-09-21T08:25:11.532 INFO:tasks.rados.rados.0.smithi137.stdout:203: finishing write tid 2 to smithi13779221-27 2024-09-21T08:25:11.533 INFO:tasks.rados.rados.0.smithi137.stdout:205: seq_num 102 ranges {514314=787987,1768928=405354,2768204=509510} 2024-09-21T08:25:11.540 INFO:tasks.rados.rados.0.smithi137.stdout:205: writing smithi13779221-5 from 514314 to 1302301 tid 1 2024-09-21T08:25:11.546 INFO:tasks.rados.rados.0.smithi137.stdout:205: writing smithi13779221-5 from 1768928 to 2174282 tid 2 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:205: writing smithi13779221-5 from 2768204 to 3277714 tid 3 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:206: delete oid 49 current snap is 20 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:203: finishing write tid 3 to smithi13779221-27 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:203: finishing write tid 4 to smithi13779221-27 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:203: finishing write tid 5 to smithi13779221-27 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:203: oid 27 updating version 0 to 50 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:203: oid 27 version 50 is already newer than 49 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:203: oid 27 updating version 50 to 51 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:203: oid 27 version 51 is already newer than 48 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 51 (ObjNum 100 snap 20 seq_num 100) dirty exists 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:203: left oid 27 (ObjNum 100 snap 20 seq_num 100) 2024-09-21T08:25:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:204: finishing write tid 1 to smithi13779221-20 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:204: finishing write tid 2 to smithi13779221-20 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:204: finishing write tid 3 to smithi13779221-20 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:204: finishing write tid 4 to smithi13779221-20 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:204: finishing write tid 5 to smithi13779221-20 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:204: oid 20 updating version 0 to 82 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:204: oid 20 version 82 is already newer than 79 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:204: oid 20 version 82 is already newer than 81 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:204: oid 20 version 82 is already newer than 80 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 82 (ObjNum 101 snap 20 seq_num 101) dirty exists 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:204: left oid 20 (ObjNum 101 snap 20 seq_num 101) 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:205: finishing write tid 1 to smithi13779221-5 2024-09-21T08:25:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:205: finishing write tid 2 to smithi13779221-5 2024-09-21T08:25:11.554 INFO:tasks.rados.rados.0.smithi137.stdout:203: done (3 left) 2024-09-21T08:25:11.554 INFO:tasks.rados.rados.0.smithi137.stdout:204: done (2 left) 2024-09-21T08:25:11.554 INFO:tasks.rados.rados.0.smithi137.stdout:206: done (1 left) 2024-09-21T08:25:11.554 INFO:tasks.rados.rados.0.smithi137.stdout:207: snap_remove snap 13 2024-09-21T08:25:11.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:11.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:11.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:11.577 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:11.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:11 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:11.600 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:11 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4103416215' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:11.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:11 smithi165 ceph-mon[32893]: osdmap e117: 8 total, 8 up, 5 in 2024-09-21T08:25:11.601 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:11 smithi165 ceph-mon[32893]: osdmap e118: 8 total, 8 up, 5 in 2024-09-21T08:25:11.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:11.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:11.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:11.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:11.724 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:25:11.724 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:25:11.724 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:25:11.724 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:25:11.724 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:25:11.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:11 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:11.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:11 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4103416215' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:11.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:11 smithi137 ceph-mon[29634]: osdmap e117: 8 total, 8 up, 5 in 2024-09-21T08:25:11.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:11 smithi137 ceph-mon[29634]: osdmap e118: 8 total, 8 up, 5 in 2024-09-21T08:25:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:11 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:11 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4103416215' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:11 smithi137 ceph-mon[41044]: osdmap e117: 8 total, 8 up, 5 in 2024-09-21T08:25:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:11 smithi137 ceph-mon[41044]: osdmap e118: 8 total, 8 up, 5 in 2024-09-21T08:25:11.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:11.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:11.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:11.825 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:11.896 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:11.912 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:11.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:11.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:11.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:11.948 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:12.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:25:11 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:25:12.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:12.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:12.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:12.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:12.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:12.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:12.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:12.194 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:12.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:12.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:12.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:12.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:12.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T08:25:12.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:12.441 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:12.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:12.442 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:12.475 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:25:12.477 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:25:12.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:12.546 INFO:tasks.rados.rados.0.smithi137.stdout:207: done (1 left) 2024-09-21T08:25:12.546 INFO:tasks.rados.rados.0.smithi137.stdout:208: snap_remove snap 18 2024-09-21T08:25:12.567 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:12.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:12.568 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:12.621 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:12.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:12.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:12.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:12.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:12.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:12 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:12.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:12 smithi165 ceph-mon[32893]: pgmap v82: 148 pgs: 148 active+clean; 150 MiB data, 451 MiB used, 447 GiB / 447 GiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 22 op/s 2024-09-21T08:25:12.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:12 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:12.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:12 smithi137 ceph-mon[29634]: pgmap v82: 148 pgs: 148 active+clean; 150 MiB data, 451 MiB used, 447 GiB / 447 GiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 22 op/s 2024-09-21T08:25:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:12 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:12 smithi137 ceph-mon[41044]: pgmap v82: 148 pgs: 148 active+clean; 150 MiB data, 451 MiB used, 447 GiB / 447 GiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 22 op/s 2024-09-21T08:25:12.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:12.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:12.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:12.814 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:12.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:12.936 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:12.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:12.937 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:13.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:13.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:13.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:13.063 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:13.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:13.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:13.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:13.186 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:13.199 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:25:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:25:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:25:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:25:13.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44417.0:46 5.1f e70/69 5:f95f44c2:::notify.0:head v 70'1, mlcod=0'0)", 2024-09-21T08:25:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.569323+0000", 2024-09-21T08:25:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 98.624768046, 2024-09-21T08:25:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021424550000000001, 2024-09-21T08:25:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.200 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569323+0000", 2024-09-21T08:25:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569323+0000", 2024-09-21T08:25:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569324+0000", 2024-09-21T08:25:13.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.009e-06 2024-09-21T08:25:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569331+0000", 2024-09-21T08:25:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.528e-06 2024-09-21T08:25:13.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569343+0000", 2024-09-21T08:25:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2483999999999999e-05 2024-09-21T08:25:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569351+0000", 2024-09-21T08:25:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5079999999999999e-06 2024-09-21T08:25:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569398+0000", 2024-09-21T08:25:13.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6708000000000001e-05 2024-09-21T08:25:13.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569414+0000", 2024-09-21T08:25:13.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6829000000000001e-05 2024-09-21T08:25:13.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571435+0000", 2024-09-21T08:25:13.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020206299999999998 2024-09-21T08:25:13.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571466+0000", 2024-09-21T08:25:13.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0759000000000001e-05 2024-09-21T08:25:13.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.210 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.210 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.210 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44387.0:46 5.1f e70/69 5:f95f44c2:::notify.0:head v 70'2, mlcod=0'0)", 2024-09-21T08:25:13.210 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.569550+0000", 2024-09-21T08:25:13.211 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 98.624540999000004, 2024-09-21T08:25:13.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019394410000000001, 2024-09-21T08:25:13.211 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.211 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.211 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.211 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569550+0000", 2024-09-21T08:25:13.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569550+0000", 2024-09-21T08:25:13.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569551+0000", 2024-09-21T08:25:13.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2500000000000005e-07 2024-09-21T08:25:13.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569557+0000", 2024-09-21T08:25:13.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.412e-06 2024-09-21T08:25:13.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569564+0000", 2024-09-21T08:25:13.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1420000000000004e-06 2024-09-21T08:25:13.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569570+0000", 2024-09-21T08:25:13.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6250000000000001e-06 2024-09-21T08:25:13.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.570055+0000", 2024-09-21T08:25:13.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048423599999999998 2024-09-21T08:25:13.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.570072+0000", 2024-09-21T08:25:13.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7007000000000002e-05 2024-09-21T08:25:13.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571468+0000", 2024-09-21T08:25:13.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013962219999999999 2024-09-21T08:25:13.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571490+0000", 2024-09-21T08:25:13.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2072000000000001e-05 2024-09-21T08:25:13.220 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.220 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.220 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.220 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:52 5.e 5:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:25:13.221 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.576342+0000", 2024-09-21T08:25:13.221 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 98.617749504000003, 2024-09-21T08:25:13.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011620549999999999, 2024-09-21T08:25:13.221 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.221 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.221 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.221 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.222 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.222 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 52 2024-09-21T08:25:13.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.222 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576342+0000", 2024-09-21T08:25:13.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576342+0000", 2024-09-21T08:25:13.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576344+0000", 2024-09-21T08:25:13.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3659999999999999e-06 2024-09-21T08:25:13.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576351+0000", 2024-09-21T08:25:13.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7129999999999999e-06 2024-09-21T08:25:13.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576362+0000", 2024-09-21T08:25:13.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0740999999999999e-05 2024-09-21T08:25:13.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576367+0000", 2024-09-21T08:25:13.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5670000000000002e-06 2024-09-21T08:25:13.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576412+0000", 2024-09-21T08:25:13.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5079e-05 2024-09-21T08:25:13.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576503+0000", 2024-09-21T08:25:13.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0703000000000002e-05 2024-09-21T08:25:13.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:25:13.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576585+0000", 2024-09-21T08:25:13.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1724999999999994e-05 2024-09-21T08:25:13.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577152+0000", 2024-09-21T08:25:13.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000567054 2024-09-21T08:25:13.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577187+0000", 2024-09-21T08:25:13.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5212000000000001e-05 2024-09-21T08:25:13.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577426+0000", 2024-09-21T08:25:13.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023901099999999999 2024-09-21T08:25:13.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577463+0000", 2024-09-21T08:25:13.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6820000000000003e-05 2024-09-21T08:25:13.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577504+0000", 2024-09-21T08:25:13.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1063999999999999e-05 2024-09-21T08:25:13.234 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.234 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.234 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.235 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:52 5.e 5:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:25:13.235 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.576451+0000", 2024-09-21T08:25:13.235 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 98.617640268000002, 2024-09-21T08:25:13.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001253133, 2024-09-21T08:25:13.235 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.235 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.236 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.236 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.236 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:25:13.236 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:25:13.236 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 52 2024-09-21T08:25:13.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576451+0000", 2024-09-21T08:25:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576451+0000", 2024-09-21T08:25:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576453+0000", 2024-09-21T08:25:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8050000000000001e-06 2024-09-21T08:25:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576458+0000", 2024-09-21T08:25:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3299999999999998e-06 2024-09-21T08:25:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576468+0000", 2024-09-21T08:25:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8700000000000004e-06 2024-09-21T08:25:13.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576476+0000", 2024-09-21T08:25:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6729999999999992e-06 2024-09-21T08:25:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576767+0000", 2024-09-21T08:25:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029066200000000002 2024-09-21T08:25:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576845+0000", 2024-09-21T08:25:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8517999999999996e-05 2024-09-21T08:25:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:25:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576961+0000", 2024-09-21T08:25:13.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011566900000000001 2024-09-21T08:25:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577297+0000", 2024-09-21T08:25:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033612900000000001 2024-09-21T08:25:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577569+0000", 2024-09-21T08:25:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027198899999999999 2024-09-21T08:25:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.245 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577623+0000", 2024-09-21T08:25:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3841000000000002e-05 2024-09-21T08:25:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577667+0000", 2024-09-21T08:25:13.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4152e-05 2024-09-21T08:25:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577704+0000", 2024-09-21T08:25:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7494999999999999e-05 2024-09-21T08:25:13.247 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:53 5.e 5:7759931f:::notify.3:head [watch watch cookie 94074441556624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:25:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.577867+0000", 2024-09-21T08:25:13.248 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 98.616224822999996, 2024-09-21T08:25:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001803189, 2024-09-21T08:25:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:25:13.249 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:25:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 53 2024-09-21T08:25:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577867+0000", 2024-09-21T08:25:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577867+0000", 2024-09-21T08:25:13.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577868+0000", 2024-09-21T08:25:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1289999999999999e-06 2024-09-21T08:25:13.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577872+0000", 2024-09-21T08:25:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4050000000000002e-06 2024-09-21T08:25:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577880+0000", 2024-09-21T08:25:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1130000000000001e-06 2024-09-21T08:25:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577885+0000", 2024-09-21T08:25:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8080000000000004e-06 2024-09-21T08:25:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578144+0000", 2024-09-21T08:25:13.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025872800000000001 2024-09-21T08:25:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:25:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578215+0000", 2024-09-21T08:25:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1330999999999999e-05 2024-09-21T08:25:13.256 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578696+0000", 2024-09-21T08:25:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048043800000000002 2024-09-21T08:25:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.257 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578769+0000", 2024-09-21T08:25:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3302999999999999e-05 2024-09-21T08:25:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:25:13.258 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578911+0000", 2024-09-21T08:25:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014194000000000001 2024-09-21T08:25:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579286+0000", 2024-09-21T08:25:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037537600000000002 2024-09-21T08:25:13.259 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579511+0000", 2024-09-21T08:25:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022471700000000001 2024-09-21T08:25:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.260 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579547+0000", 2024-09-21T08:25:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5947e-05 2024-09-21T08:25:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.261 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579599+0000", 2024-09-21T08:25:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1929999999999999e-05 2024-09-21T08:25:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579670+0000", 2024-09-21T08:25:13.262 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1024000000000003e-05 2024-09-21T08:25:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.263 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:206 4.18 4:1e85421f:gc::gc.1:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T08:25:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.590771+0000", 2024-09-21T08:25:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.603320674000003, 2024-09-21T08:25:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023079490000000001, 2024-09-21T08:25:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.264 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 206 2024-09-21T08:25:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.265 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590771+0000", 2024-09-21T08:25:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.266 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590771+0000", 2024-09-21T08:25:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590773+0000", 2024-09-21T08:25:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4739999999999999e-06 2024-09-21T08:25:13.267 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590782+0000", 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1290000000000002e-06 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590795+0000", 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2456000000000001e-05 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590805+0000", 2024-09-21T08:25:13.268 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0111e-05 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590876+0000", 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1111000000000004e-05 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590965+0000", 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9061999999999995e-05 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:25:13.269 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.591134+0000", 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000169044 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.591516+0000", 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000381362 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.591767+0000", 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025140299999999998 2024-09-21T08:25:13.270 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.593025+0000", 2024-09-21T08:25:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012583200000000001 2024-09-21T08:25:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.271 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.593054+0000", 2024-09-21T08:25:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8863000000000001e-05 2024-09-21T08:25:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.593079+0000", 2024-09-21T08:25:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4613999999999998e-05 2024-09-21T08:25:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.272 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:210 4.9 4:954be8d2:gc::gc.5:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.600055+0000", 2024-09-21T08:25:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.594036277000001, 2024-09-21T08:25:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012045479999999999, 2024-09-21T08:25:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.273 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 210 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600055+0000", 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.274 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600055+0000", 2024-09-21T08:25:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600057+0000", 2024-09-21T08:25:13.275 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9149999999999999e-06 2024-09-21T08:25:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600063+0000", 2024-09-21T08:25:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6230000000000002e-06 2024-09-21T08:25:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.276 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600076+0000", 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3128e-05 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600083+0000", 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2439999999999996e-06 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.277 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600148+0000", 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5118999999999998e-05 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600224+0000", 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6283000000000001e-05 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600400+0000", 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000175794 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.278 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600700+0000", 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029931899999999998 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601121+0000", 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042133399999999999 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601205+0000", 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4358999999999996e-05 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.279 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601232+0000", 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6902000000000001e-05 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601260+0000", 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7528000000000001e-05 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.280 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:238 4.18 4:1a734c59:::obj_delete_at_hint.0000000000:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.633924+0000", 2024-09-21T08:25:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.560167406000005, 2024-09-21T08:25:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001234052, 2024-09-21T08:25:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.282 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 238 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633924+0000", 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633924+0000", 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.283 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633925+0000", 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1600000000000001e-07 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633928+0000", 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1810000000000001e-06 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633933+0000", 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4399999999999996e-06 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.284 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633937+0000", 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4249999999999998e-06 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633973+0000", 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6247e-05 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634046+0000", 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2575000000000002e-05 2024-09-21T08:25:13.285 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634151+0000", 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010584199999999999 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634527+0000", 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037577699999999999 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634630+0000", 2024-09-21T08:25:13.286 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102846 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.635070+0000", 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000439796 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.635121+0000", 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0847999999999998e-05 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.287 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.635158+0000", 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7358999999999999e-05 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:255 4.9 4:921ecdb1:gc::gc.25:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.636496+0000", 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.557595753000001, 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015126289999999999, 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.288 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 255 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636496+0000", 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.289 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636496+0000", 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636497+0000", 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2099999999999995e-07 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636507+0000", 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9620000000000005e-06 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.290 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636514+0000", 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5270000000000001e-06 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636519+0000", 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7589999999999997e-06 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636564+0000", 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4919999999999997e-05 2024-09-21T08:25:13.291 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636637+0000", 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3219999999999994e-05 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636792+0000", 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015530599999999999 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637091+0000", 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029869500000000003 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.292 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637625+0000", 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053441199999999995 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637927+0000", 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030120199999999998 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637974+0000", 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6999999999999997e-05 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.293 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.638008+0000", 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4705000000000001e-05 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:287 4.e 4:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.640319+0000", 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.553772866000003, 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013081729999999999, 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.294 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 287 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640319+0000", 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.295 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640319+0000", 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640320+0000", 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4300000000000001e-07 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640323+0000", 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.715e-06 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.296 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640331+0000", 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5619999999999998e-06 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640337+0000", 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8579999999999996e-06 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640379+0000", 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2290000000000003e-05 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.297 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640452+0000", 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2948999999999996e-05 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640582+0000", 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013043000000000001 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640890+0000", 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000307614 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.298 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641132+0000", 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000241728 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641472+0000", 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034040600000000002 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641530+0000", 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8112999999999997e-05 2024-09-21T08:25:13.299 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641627+0000", 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6564999999999994e-05 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:300 4.1 4:856b9b26:reshard::reshard.0000000002:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.642024+0000", 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.552067499000003, 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011698909999999999, 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.300 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 300 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642024+0000", 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.301 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642024+0000", 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642025+0000", 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642030+0000", 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0370000000000002e-06 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.302 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642040+0000", 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3929999999999997e-06 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642045+0000", 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0470000000000001e-06 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642083+0000", 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8195000000000003e-05 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.303 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642167+0000", 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3720000000000005e-05 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642315+0000", 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014880500000000001 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642777+0000", 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046160999999999999 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.304 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643089+0000", 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031199199999999998 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643132+0000", 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3343999999999997e-05 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643171+0000", 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9088999999999998e-05 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.305 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643194+0000", 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2489000000000001e-05 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:302 4.e 4:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.642059+0000", 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.552032452000006, 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012335429999999999, 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.306 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 302 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642059+0000", 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.307 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642059+0000", 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642060+0000", 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2900000000000004e-07 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642062+0000", 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5160000000000001e-06 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.308 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642066+0000", 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7979999999999999e-06 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642069+0000", 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0010000000000002e-06 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642105+0000", 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6066999999999999e-05 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.309 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642153+0000", 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7642000000000002e-05 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642252+0000", 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9345000000000005e-05 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642627+0000", 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000374745 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.310 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642847+0000", 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022007999999999999 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643187+0000", 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034015599999999999 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643250+0000", 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2829000000000004e-05 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.311 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643293+0000", 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2834999999999999e-05 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:316 4.1 4:856b9b26:reshard::reshard.0000000002:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.643653+0000", 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.550438021999994, 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011504950000000001, 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.312 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 316 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643653+0000", 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643653+0000", 2024-09-21T08:25:13.313 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643655+0000", 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2309999999999999e-06 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643659+0000", 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3669999999999999e-06 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643669+0000", 2024-09-21T08:25:13.314 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0081999999999999e-05 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643675+0000", 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8039999999999998e-06 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643727+0000", 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2144000000000001e-05 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.315 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643822+0000", 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4899000000000001e-05 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643982+0000", 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159887 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644499+0000", 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051742300000000004 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.316 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644668+0000", 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016911000000000001 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644716+0000", 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7833000000000001e-05 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644768+0000", 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1634000000000001e-05 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.317 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644804+0000", 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6081000000000002e-05 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:329 4.1a 4:5909e671:gc::gc.27:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.644844+0000", 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.549247579999999, 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012130229999999999, 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.318 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 329 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644844+0000", 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644844+0000", 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.319 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644845+0000", 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3030000000000001e-06 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644850+0000", 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9200000000000003e-06 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644859+0000", 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0529999999999996e-06 2024-09-21T08:25:13.320 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644864+0000", 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6949999999999999e-06 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644905+0000", 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1433999999999997e-05 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644949+0000", 2024-09-21T08:25:13.321 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4024999999999999e-05 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645045+0000", 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5297999999999995e-05 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645301+0000", 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025601699999999999 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.322 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645800+0000", 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000499834 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645965+0000", 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016433600000000001 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.646021+0000", 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6687999999999997e-05 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.323 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.646057+0000", 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5420000000000003e-05 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:453 4.1 4:80b6fe23:reshard::reshard.0000000011:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.670680+0000", 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.523411664999998, 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011835179999999999, 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.324 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 453 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670680+0000", 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.325 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670680+0000", 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670681+0000", 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2049999999999999e-06 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670690+0000", 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3230000000000004e-06 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.326 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670698+0000", 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7810000000000005e-06 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670703+0000", 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6879999999999998e-06 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670757+0000", 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4660999999999997e-05 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.327 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670860+0000", 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102471 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671030+0000", 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016985599999999999 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671457+0000", 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042770500000000001 2024-09-21T08:25:13.328 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671715+0000", 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025777299999999999 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671761+0000", 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5836e-05 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671812+0000", 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0719e-05 2024-09-21T08:25:13.329 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671863+0000", 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1499999999999998e-05 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:462 4.1 4:80b6fe23:reshard::reshard.0000000011:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.672625+0000", 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.521466837000006, 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014703749999999999, 2024-09-21T08:25:13.330 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 462 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672625+0000", 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.331 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672625+0000", 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672626+0000", 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.139e-06 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672630+0000", 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7190000000000001e-06 2024-09-21T08:25:13.332 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672640+0000", 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0459999999999994e-06 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672644+0000", 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7609999999999998e-06 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672684+0000", 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9431000000000002e-05 2024-09-21T08:25:13.333 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672739+0000", 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4994999999999998e-05 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672855+0000", 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115975 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.673270+0000", 2024-09-21T08:25:13.334 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041560499999999999 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.673495+0000", 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022474600000000001 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.674004+0000", 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050865299999999997 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.335 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.674061+0000", 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6925999999999997e-05 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.674095+0000", 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4378999999999997e-05 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:501 4.1a 4:5c7e8a6d:reshard::reshard.0000000015:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.684553+0000", 2024-09-21T08:25:13.336 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.509538684000006, 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015740999999999999, 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 501 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684553+0000", 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.337 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684553+0000", 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684554+0000", 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6000000000000002e-07 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684563+0000", 2024-09-21T08:25:13.338 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0790000000000007e-06 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684570+0000", 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8379999999999996e-06 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684575+0000", 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5910000000000003e-06 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.339 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684605+0000", 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0256999999999998e-05 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684651+0000", 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5160999999999997e-05 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684747+0000", 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6220999999999999e-05 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.340 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.685024+0000", 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027716299999999998 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.685367+0000", 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034277100000000002 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.686053+0000", 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00068657400000000002 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.341 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.686090+0000", 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6801000000000002e-05 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.686127+0000", 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6783999999999999e-05 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:697 4.1b 4:db9d130d:gc::gc.24:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.342 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.750453+0000", 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.443638683000003, 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012349640000000001, 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 697 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.343 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750453+0000", 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750453+0000", 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750454+0000", 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3999999999999999e-06 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.344 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750462+0000", 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4279999999999999e-06 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750473+0000", 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1362999999999999e-05 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750480+0000", 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8360000000000003e-06 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.345 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750531+0000", 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1008999999999999e-05 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750610+0000", 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9122000000000006e-05 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750784+0000", 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174329 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.346 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751214+0000", 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042964799999999998 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751607+0000", 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039261899999999998 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751636+0000", 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9655999999999999e-05 2024-09-21T08:25:13.347 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751663+0000", 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7254e-05 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751688+0000", 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4300000000000001e-05 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.348 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:780 4.1b 4:dc490aaa:::obj_delete_at_hint.0000000067:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.810765+0000", 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 95.383326834000002, 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023807759999999998, 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 780 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.349 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810765+0000", 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810765+0000", 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810766+0000", 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.497e-06 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.350 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810773+0000", 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8689999999999998e-06 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810784+0000", 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0675e-05 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810789+0000", 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7490000000000004e-06 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.351 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810832+0000", 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2174e-05 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810916+0000", 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4040000000000002e-05 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.811045+0000", 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012987299999999999 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.352 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.811286+0000", 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000240743 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.811623+0000", 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033676300000000001 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.813096+0000", 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014725059999999999 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.353 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.813122+0000", 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6744e-05 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.813145+0000", 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3142999999999998e-05 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.354 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDScrubReserve(3.f,QREQUEST,e:79,from:7,reservation_nonce:3)", 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.451759707000001, 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003249558, 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:13.355 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742334+0000", 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6809999999999998e-06 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742342+0000", 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6930000000000007e-06 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:13.356 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742355+0000", 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2556e-05 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742366+0000", 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0994e-05 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742426+0000", 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0509e-05 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:25:13.357 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742436+0000", 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8060000000000006e-06 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745540+0000", 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031038630000000001 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745553+0000", 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2669e-05 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:13.358 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:13.359 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745581+0000", 2024-09-21T08:25:13.359 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8787000000000001e-05 2024-09-21T08:25:13.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.359 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.359 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:13.359 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:13.359 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:25:13.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:13.360 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":82,"stamp":"2024-09-21T08:25:11.616871+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":157610253,"num_objects":260,"num_object_clones":15,"num_object_copies":780,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":260,"num_whiteouts":3,"num_read":1407,"num_read_kb":280165,"num_write":1032,"num_write_kb":156078,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":69697317,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1123,"ondisk_log_size":1123,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":461392,"kb_used_data":305444,"kb_used_omap":7,"kb_used_meta":155832,"kb_avail":468264368,"statfs":{"total":479975178240,"available":479502712832,"internally_reserved":0,"allocated":312774656,"data_stored":321577572,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7970,"internal_metadata":159572190},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":20889738,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":61,"num_read_kb":58990,"num_write":28,"num_write_kb":18688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001081"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.545246+0000","last_change":"2024-09-21T08:23:30.549696+0000","last_active":"2024-09-21T08:25:10.545246+0000","last_peered":"2024-09-21T08:25:10.545246+0000","last_clean":"2024-09-21T08:25:10.545246+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:25:10.545246+0000","last_undegraded":"2024-09-21T08:25:10.545246+0000","last_fullsized":"2024-09-21T08:25:10.545246+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215038+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":85,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553505+0000","last_change":"2024-09-21T08:24:56.570163+0000","last_active":"2024-09-21T08:25:10.553505+0000","last_peered":"2024-09-21T08:25:10.553505+0000","last_clean":"2024-09-21T08:25:10.553505+0000","last_became_active":"2024-09-21T08:24:56.524489+0000","last_became_peered":"2024-09-21T08:24:56.524489+0000","last_unstale":"2024-09-21T08:25:10.553505+0000","last_undegraded":"2024-09-21T08:25:10.553505+0000","last_fullsized":"2024-09-21T08:25:10.553505+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:24:50.572709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.572709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:06.339471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553316+0000","last_change":"2024-09-21T08:23:34.569082+0000","last_active":"2024-09-21T08:25:10.553316+0000","last_peered":"2024-09-21T08:25:10.553316+0000","last_clean":"2024-09-21T08:25:10.553316+0000","last_became_active":"2024-09-21T08:23:34.568762+0000","last_became_peered":"2024-09-21T08:23:34.568762+0000","last_unstale":"2024-09-21T08:25:10.553316+0000","last_undegraded":"2024-09-21T08:25:10.553316+0000","last_fullsized":"2024-09-21T08:25:10.553316+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582284+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":77,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560043+0000","last_change":"2024-09-21T08:24:38.552223+0000","last_active":"2024-09-21T08:25:10.560043+0000","last_peered":"2024-09-21T08:25:10.560043+0000","last_clean":"2024-09-21T08:25:10.560043+0000","last_became_active":"2024-09-21T08:24:38.551819+0000","last_became_peered":"2024-09-21T08:24:38.551819+0000","last_unstale":"2024-09-21T08:25:10.560043+0000","last_undegraded":"2024-09-21T08:25:10.560043+0000","last_fullsized":"2024-09-21T08:25:10.560043+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770136+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":61,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537736+0000","last_change":"2024-09-21T08:24:56.514713+0000","last_active":"2024-09-21T08:25:10.537736+0000","last_peered":"2024-09-21T08:25:10.537736+0000","last_clean":"2024-09-21T08:25:10.537736+0000","last_became_active":"2024-09-21T08:24:56.514170+0000","last_became_peered":"2024-09-21T08:24:56.514170+0000","last_unstale":"2024-09-21T08:25:10.537736+0000","last_undegraded":"2024-09-21T08:25:10.537736+0000","last_fullsized":"2024-09-21T08:25:10.537736+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":43,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.556281+0000","last_change":"2024-09-21T08:24:56.517255+0000","last_active":"2024-09-21T08:25:10.556281+0000","last_peered":"2024-09-21T08:25:10.556281+0000","last_clean":"2024-09-21T08:25:10.556281+0000","last_became_active":"2024-09-21T08:24:56.516951+0000","last_became_peered":"2024-09-21T08:24:56.516951+0000","last_unstale":"2024-09-21T08:25:10.556281+0000","last_undegraded":"2024-09-21T08:25:10.556281+0000","last_fullsized":"2024-09-21T08:25:10.556281+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.555330+0000","last_change":"2024-09-21T08:24:48.649038+0000","last_active":"2024-09-21T08:25:10.555330+0000","last_peered":"2024-09-21T08:25:10.555330+0000","last_clean":"2024-09-21T08:25:10.555330+0000","last_became_active":"2024-09-21T08:24:48.594855+0000","last_became_peered":"2024-09-21T08:24:48.594855+0000","last_unstale":"2024-09-21T08:25:10.555330+0000","last_undegraded":"2024-09-21T08:25:10.555330+0000","last_fullsized":"2024-09-21T08:25:10.555330+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":42,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553366+0000","last_change":"2024-09-21T08:24:56.519439+0000","last_active":"2024-09-21T08:25:10.553366+0000","last_peered":"2024-09-21T08:25:10.553366+0000","last_clean":"2024-09-21T08:25:10.553366+0000","last_became_active":"2024-09-21T08:24:56.517228+0000","last_became_peered":"2024-09-21T08:24:56.517228+0000","last_unstale":"2024-09-21T08:25:10.553366+0000","last_undegraded":"2024-09-21T08:25:10.553366+0000","last_fullsized":"2024-09-21T08:25:10.553366+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.640831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.640831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:16.137464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":125,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561171+0000","last_change":"2024-09-21T08:24:56.510104+0000","last_active":"2024-09-21T08:25:10.561171+0000","last_peered":"2024-09-21T08:25:10.561171+0000","last_clean":"2024-09-21T08:25:10.561171+0000","last_became_active":"2024-09-21T08:24:56.509786+0000","last_became_peered":"2024-09-21T08:24:56.509786+0000","last_unstale":"2024-09-21T08:25:10.561171+0000","last_undegraded":"2024-09-21T08:25:10.561171+0000","last_fullsized":"2024-09-21T08:25:10.561171+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543101+0000","last_change":"2024-09-21T08:24:56.523774+0000","last_active":"2024-09-21T08:25:10.543101+0000","last_peered":"2024-09-21T08:25:10.543101+0000","last_clean":"2024-09-21T08:25:10.543101+0000","last_became_active":"2024-09-21T08:24:56.500171+0000","last_became_peered":"2024-09-21T08:24:56.500171+0000","last_unstale":"2024-09-21T08:25:10.543101+0000","last_undegraded":"2024-09-21T08:25:10.543101+0000","last_fullsized":"2024-09-21T08:25:10.543101+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215313+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560808+0000","last_change":"2024-09-21T08:23:34.567872+0000","last_active":"2024-09-21T08:25:10.560808+0000","last_peered":"2024-09-21T08:25:10.560808+0000","last_clean":"2024-09-21T08:25:10.560808+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:25:10.560808+0000","last_undegraded":"2024-09-21T08:25:10.560808+0000","last_fullsized":"2024-09-21T08:25:10.560808+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.769902+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":100,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.542453+0000","last_change":"2024-09-21T08:23:36.582491+0000","last_active":"2024-09-21T08:25:10.542453+0000","last_peered":"2024-09-21T08:25:10.542453+0000","last_clean":"2024-09-21T08:25:10.542453+0000","last_became_active":"2024-09-21T08:23:36.581720+0000","last_became_peered":"2024-09-21T08:23:36.581720+0000","last_unstale":"2024-09-21T08:25:10.542453+0000","last_undegraded":"2024-09-21T08:25:10.542453+0000","last_fullsized":"2024-09-21T08:25:10.542453+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":133,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560929+0000","last_change":"2024-09-21T08:24:56.698348+0000","last_active":"2024-09-21T08:25:10.560929+0000","last_peered":"2024-09-21T08:25:10.560929+0000","last_clean":"2024-09-21T08:25:10.560929+0000","last_became_active":"2024-09-21T08:24:56.517077+0000","last_became_peered":"2024-09-21T08:24:56.517077+0000","last_unstale":"2024-09-21T08:25:10.560929+0000","last_undegraded":"2024-09-21T08:25:10.560929+0000","last_fullsized":"2024-09-21T08:25:10.560929+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.466677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.562316+0000","last_change":"2024-09-21T08:24:48.602351+0000","last_active":"2024-09-21T08:25:10.562316+0000","last_peered":"2024-09-21T08:25:10.562316+0000","last_clean":"2024-09-21T08:25:10.562316+0000","last_became_active":"2024-09-21T08:24:48.587366+0000","last_became_peered":"2024-09-21T08:24:48.587366+0000","last_unstale":"2024-09-21T08:25:10.562316+0000","last_undegraded":"2024-09-21T08:25:10.562316+0000","last_fullsized":"2024-09-21T08:25:10.562316+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:24:32.569846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:32.569846+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:42.983663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":109,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537235+0000","last_change":"2024-09-21T08:24:38.539036+0000","last_active":"2024-09-21T08:25:10.537235+0000","last_peered":"2024-09-21T08:25:10.537235+0000","last_clean":"2024-09-21T08:25:10.537235+0000","last_became_active":"2024-09-21T08:24:38.538724+0000","last_became_peered":"2024-09-21T08:24:38.538724+0000","last_unstale":"2024-09-21T08:25:10.537235+0000","last_undegraded":"2024-09-21T08:25:10.537235+0000","last_fullsized":"2024-09-21T08:25:10.537235+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672290+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560540+0000","last_change":"2024-09-21T08:24:48.605875+0000","last_active":"2024-09-21T08:25:10.560540+0000","last_peered":"2024-09-21T08:25:10.560540+0000","last_clean":"2024-09-21T08:25:10.560540+0000","last_became_active":"2024-09-21T08:24:48.586120+0000","last_became_peered":"2024-09-21T08:24:48.586120+0000","last_unstale":"2024-09-21T08:25:10.560540+0000","last_undegraded":"2024-09-21T08:25:10.560540+0000","last_fullsized":"2024-09-21T08:25:10.560540+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.769981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":70,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561152+0000","last_change":"2024-09-21T08:24:56.500353+0000","last_active":"2024-09-21T08:25:10.561152+0000","last_peered":"2024-09-21T08:25:10.561152+0000","last_clean":"2024-09-21T08:25:10.561152+0000","last_became_active":"2024-09-21T08:24:56.499884+0000","last_became_peered":"2024-09-21T08:24:56.499884+0000","last_unstale":"2024-09-21T08:25:10.561152+0000","last_undegraded":"2024-09-21T08:25:10.561152+0000","last_fullsized":"2024-09-21T08:25:10.561152+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.769933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":126,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.544047+0000","last_change":"2024-09-21T08:24:54.870568+0000","last_active":"2024-09-21T08:25:10.544047+0000","last_peered":"2024-09-21T08:25:10.544047+0000","last_clean":"2024-09-21T08:25:10.544047+0000","last_became_active":"2024-09-21T08:24:38.546590+0000","last_became_peered":"2024-09-21T08:24:38.546590+0000","last_unstale":"2024-09-21T08:25:10.544047+0000","last_undegraded":"2024-09-21T08:25:10.544047+0000","last_fullsized":"2024-09-21T08:25:10.544047+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.354297+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":155,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.555831+0000","last_change":"2024-09-21T08:24:48.668150+0000","last_active":"2024-09-21T08:25:10.555831+0000","last_peered":"2024-09-21T08:25:10.555831+0000","last_clean":"2024-09-21T08:25:10.555831+0000","last_became_active":"2024-09-21T08:24:48.601541+0000","last_became_peered":"2024-09-21T08:24:48.601541+0000","last_unstale":"2024-09-21T08:25:10.555831+0000","last_undegraded":"2024-09-21T08:25:10.555831+0000","last_fullsized":"2024-09-21T08:25:10.555831+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.554711+0000","last_change":"2024-09-21T08:24:38.551744+0000","last_active":"2024-09-21T08:25:10.554711+0000","last_peered":"2024-09-21T08:25:10.554711+0000","last_clean":"2024-09-21T08:25:10.554711+0000","last_became_active":"2024-09-21T08:24:38.551452+0000","last_became_peered":"2024-09-21T08:24:38.551452+0000","last_unstale":"2024-09-21T08:25:10.554711+0000","last_undegraded":"2024-09-21T08:25:10.554711+0000","last_fullsized":"2024-09-21T08:25:10.554711+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":42,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561291+0000","last_change":"2024-09-21T08:24:56.518401+0000","last_active":"2024-09-21T08:25:10.561291+0000","last_peered":"2024-09-21T08:25:10.561291+0000","last_clean":"2024-09-21T08:25:10.561291+0000","last_became_active":"2024-09-21T08:24:56.518102+0000","last_became_peered":"2024-09-21T08:24:56.518102+0000","last_unstale":"2024-09-21T08:25:10.561291+0000","last_undegraded":"2024-09-21T08:25:10.561291+0000","last_fullsized":"2024-09-21T08:25:10.561291+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":76,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561213+0000","last_change":"2024-09-21T08:24:38.554008+0000","last_active":"2024-09-21T08:25:10.561213+0000","last_peered":"2024-09-21T08:25:10.561213+0000","last_clean":"2024-09-21T08:25:10.561213+0000","last_became_active":"2024-09-21T08:24:38.551991+0000","last_became_peered":"2024-09-21T08:24:38.551991+0000","last_unstale":"2024-09-21T08:25:10.561213+0000","last_undegraded":"2024-09-21T08:25:10.561213+0000","last_fullsized":"2024-09-21T08:25:10.561213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582184+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":128,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.554794+0000","last_change":"2024-09-21T08:24:56.555008+0000","last_active":"2024-09-21T08:25:10.554794+0000","last_peered":"2024-09-21T08:25:10.554794+0000","last_clean":"2024-09-21T08:25:10.554794+0000","last_became_active":"2024-09-21T08:24:56.519036+0000","last_became_peered":"2024-09-21T08:24:56.519036+0000","last_unstale":"2024-09-21T08:25:10.554794+0000","last_undegraded":"2024-09-21T08:25:10.554794+0000","last_fullsized":"2024-09-21T08:25:10.554794+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":66,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.555463+0000","last_change":"2024-09-21T08:24:55.797165+0000","last_active":"2024-09-21T08:25:10.555463+0000","last_peered":"2024-09-21T08:25:10.555463+0000","last_clean":"2024-09-21T08:25:10.555463+0000","last_became_active":"2024-09-21T08:24:48.598748+0000","last_became_peered":"2024-09-21T08:24:48.598748+0000","last_unstale":"2024-09-21T08:25:10.555463+0000","last_undegraded":"2024-09-21T08:25:10.555463+0000","last_fullsized":"2024-09-21T08:25:10.555463+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.797084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:06.836794+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":100,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537613+0000","last_change":"2024-09-21T08:23:36.584003+0000","last_active":"2024-09-21T08:25:10.537613+0000","last_peered":"2024-09-21T08:25:10.537613+0000","last_clean":"2024-09-21T08:25:10.537613+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:25:10.537613+0000","last_undegraded":"2024-09-21T08:25:10.537613+0000","last_fullsized":"2024-09-21T08:25:10.537613+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672460+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":143,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553490+0000","last_change":"2024-09-21T08:24:56.543188+0000","last_active":"2024-09-21T08:25:10.553490+0000","last_peered":"2024-09-21T08:25:10.553490+0000","last_clean":"2024-09-21T08:25:10.553490+0000","last_became_active":"2024-09-21T08:24:56.517078+0000","last_became_peered":"2024-09-21T08:24:56.517078+0000","last_unstale":"2024-09-21T08:25:10.553490+0000","last_undegraded":"2024-09-21T08:25:10.553490+0000","last_fullsized":"2024-09-21T08:25:10.553490+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":141,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543722+0000","last_change":"2024-09-21T08:24:48.659647+0000","last_active":"2024-09-21T08:25:10.543722+0000","last_peered":"2024-09-21T08:25:10.543722+0000","last_clean":"2024-09-21T08:25:10.543722+0000","last_became_active":"2024-09-21T08:24:48.602672+0000","last_became_peered":"2024-09-21T08:24:48.602672+0000","last_unstale":"2024-09-21T08:25:10.543722+0000","last_undegraded":"2024-09-21T08:25:10.543722+0000","last_fullsized":"2024-09-21T08:25:10.543722+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"103'11","reported_seq":159,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.538344+0000","last_change":"2024-09-21T08:24:56.555287+0000","last_active":"2024-09-21T08:25:10.538344+0000","last_peered":"2024-09-21T08:25:10.538344+0000","last_clean":"2024-09-21T08:25:10.538344+0000","last_became_active":"2024-09-21T08:24:56.507316+0000","last_became_peered":"2024-09-21T08:24:56.507316+0000","last_unstale":"2024-09-21T08:25:10.538344+0000","last_undegraded":"2024-09-21T08:25:10.538344+0000","last_fullsized":"2024-09-21T08:25:10.538344+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.594212+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:52.368992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"112'16","reported_seq":188,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537545+0000","last_change":"2024-09-21T08:25:05.517760+0000","last_active":"2024-09-21T08:25:10.537545+0000","last_peered":"2024-09-21T08:25:10.537545+0000","last_clean":"2024-09-21T08:25:10.537545+0000","last_became_active":"2024-09-21T08:24:56.503681+0000","last_became_peered":"2024-09-21T08:24:56.503681+0000","last_unstale":"2024-09-21T08:25:10.537545+0000","last_undegraded":"2024-09-21T08:25:10.537545+0000","last_fullsized":"2024-09-21T08:25:10.537545+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:43.957885+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014162720000000001,"stat_sum":{"num_bytes":10873997,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":25,"num_read_kb":22446,"num_write":15,"num_write_kb":11823,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":12634927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.f","version":"73'15","reported_seq":177,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.554110+0000","last_change":"2024-09-21T08:24:48.611807+0000","last_active":"2024-09-21T08:25:10.554110+0000","last_peered":"2024-09-21T08:25:10.554110+0000","last_clean":"2024-09-21T08:25:10.554110+0000","last_became_active":"2024-09-21T08:24:48.581857+0000","last_became_peered":"2024-09-21T08:24:48.581857+0000","last_unstale":"2024-09-21T08:25:10.554110+0000","last_undegraded":"2024-09-21T08:25:10.554110+0000","last_fullsized":"2024-09-21T08:25:10.554110+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":57,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553591+0000","last_change":"2024-09-21T08:24:48.580125+0000","last_active":"2024-09-21T08:25:10.553591+0000","last_peered":"2024-09-21T08:25:10.553591+0000","last_clean":"2024-09-21T08:25:10.553591+0000","last_became_active":"2024-09-21T08:24:48.579498+0000","last_became_peered":"2024-09-21T08:24:48.579498+0000","last_unstale":"2024-09-21T08:25:10.553591+0000","last_undegraded":"2024-09-21T08:25:10.553591+0000","last_fullsized":"2024-09-21T08:25:10.553591+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582123+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"85'9","reported_seq":113,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560133+0000","last_change":"2024-09-21T08:24:38.559661+0000","last_active":"2024-09-21T08:25:10.560133+0000","last_peered":"2024-09-21T08:25:10.560133+0000","last_clean":"2024-09-21T08:25:10.560133+0000","last_became_active":"2024-09-21T08:24:38.548789+0000","last_became_peered":"2024-09-21T08:24:38.548789+0000","last_unstale":"2024-09-21T08:25:10.560133+0000","last_undegraded":"2024-09-21T08:25:10.560133+0000","last_fullsized":"2024-09-21T08:25:10.560133+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.769880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":101,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560887+0000","last_change":"2024-09-21T08:23:36.578919+0000","last_active":"2024-09-21T08:25:10.560887+0000","last_peered":"2024-09-21T08:25:10.560887+0000","last_clean":"2024-09-21T08:25:10.560887+0000","last_became_active":"2024-09-21T08:23:36.578648+0000","last_became_peered":"2024-09-21T08:23:36.578648+0000","last_unstale":"2024-09-21T08:25:10.560887+0000","last_undegraded":"2024-09-21T08:25:10.560887+0000","last_fullsized":"2024-09-21T08:25:10.560887+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.769963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"86'3","reported_seq":157,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.558477+0000","last_change":"2024-09-21T08:25:05.520516+0000","last_active":"2024-09-21T08:25:10.558477+0000","last_peered":"2024-09-21T08:25:10.558477+0000","last_clean":"2024-09-21T08:25:10.558477+0000","last_became_active":"2024-09-21T08:24:56.521044+0000","last_became_peered":"2024-09-21T08:24:56.521044+0000","last_unstale":"2024-09-21T08:25:10.558477+0000","last_undegraded":"2024-09-21T08:25:10.558477+0000","last_fullsized":"2024-09-21T08:25:10.558477+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.001101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00052607200000000004,"stat_sum":{"num_bytes":2403368,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10649,"num_write":4,"num_write_kb":1254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2565420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.0","version":"73'18","reported_seq":124,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.536855+0000","last_change":"2024-09-21T08:24:48.650167+0000","last_active":"2024-09-21T08:25:10.536855+0000","last_peered":"2024-09-21T08:25:10.536855+0000","last_clean":"2024-09-21T08:25:10.536855+0000","last_became_active":"2024-09-21T08:24:48.585831+0000","last_became_peered":"2024-09-21T08:24:48.585831+0000","last_unstale":"2024-09-21T08:25:10.536855+0000","last_undegraded":"2024-09-21T08:25:10.536855+0000","last_fullsized":"2024-09-21T08:25:10.536855+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672353+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":64,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.538392+0000","last_change":"2024-09-21T08:24:50.648706+0000","last_active":"2024-09-21T08:25:10.538392+0000","last_peered":"2024-09-21T08:25:10.538392+0000","last_clean":"2024-09-21T08:25:10.538392+0000","last_became_active":"2024-09-21T08:24:48.603740+0000","last_became_peered":"2024-09-21T08:24:48.603740+0000","last_unstale":"2024-09-21T08:25:10.538392+0000","last_undegraded":"2024-09-21T08:25:10.538392+0000","last_fullsized":"2024-09-21T08:25:10.538392+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.532278+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":57,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537991+0000","last_change":"2024-09-21T08:24:48.604125+0000","last_active":"2024-09-21T08:25:10.537991+0000","last_peered":"2024-09-21T08:25:10.537991+0000","last_clean":"2024-09-21T08:25:10.537991+0000","last_became_active":"2024-09-21T08:24:48.603825+0000","last_became_peered":"2024-09-21T08:24:48.603825+0000","last_unstale":"2024-09-21T08:25:10.537991+0000","last_undegraded":"2024-09-21T08:25:10.537991+0000","last_fullsized":"2024-09-21T08:25:10.537991+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":65,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.557258+0000","last_change":"2024-09-21T08:24:56.501652+0000","last_active":"2024-09-21T08:25:10.557258+0000","last_peered":"2024-09-21T08:25:10.557258+0000","last_clean":"2024-09-21T08:25:10.557258+0000","last_became_active":"2024-09-21T08:24:56.501237+0000","last_became_peered":"2024-09-21T08:24:56.501237+0000","last_unstale":"2024-09-21T08:25:10.557258+0000","last_undegraded":"2024-09-21T08:25:10.557258+0000","last_fullsized":"2024-09-21T08:25:10.557258+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"117'77","reported_seq":278,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.582558+0000","last_change":"2024-09-21T08:25:08.507833+0000","last_active":"2024-09-21T08:25:10.582558+0000","last_peered":"2024-09-21T08:25:10.582558+0000","last_clean":"2024-09-21T08:25:10.582558+0000","last_became_active":"2024-09-21T08:24:48.595901+0000","last_became_peered":"2024-09-21T08:24:48.595901+0000","last_unstale":"2024-09-21T08:25:10.582558+0000","last_undegraded":"2024-09-21T08:25:10.582558+0000","last_fullsized":"2024-09-21T08:25:10.582558+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:39.255813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.98192790900000004,"stat_sum":{"num_bytes":21100277,"num_objects":9,"num_object_clones":3,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":42,"num_read_kb":27399,"num_write":64,"num_write_kb":22015,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10580183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.1","version":"73'14","reported_seq":154,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560463+0000","last_change":"2024-09-21T08:24:56.561414+0000","last_active":"2024-09-21T08:25:10.560463+0000","last_peered":"2024-09-21T08:25:10.560463+0000","last_clean":"2024-09-21T08:25:10.560463+0000","last_became_active":"2024-09-21T08:24:56.509169+0000","last_became_peered":"2024-09-21T08:24:56.509169+0000","last_unstale":"2024-09-21T08:25:10.560463+0000","last_undegraded":"2024-09-21T08:25:10.560463+0000","last_fullsized":"2024-09-21T08:25:10.560463+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":131,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.544055+0000","last_change":"2024-09-21T08:24:55.889139+0000","last_active":"2024-09-21T08:25:10.544055+0000","last_peered":"2024-09-21T08:25:10.544055+0000","last_clean":"2024-09-21T08:25:10.544055+0000","last_became_active":"2024-09-21T08:24:38.547546+0000","last_became_peered":"2024-09-21T08:24:38.547546+0000","last_unstale":"2024-09-21T08:25:10.544055+0000","last_undegraded":"2024-09-21T08:25:10.544055+0000","last_fullsized":"2024-09-21T08:25:10.544055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:20.158135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"96'10","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553951+0000","last_change":"2024-09-21T08:24:48.622146+0000","last_active":"2024-09-21T08:25:10.553951+0000","last_peered":"2024-09-21T08:25:10.553951+0000","last_clean":"2024-09-21T08:25:10.553951+0000","last_became_active":"2024-09-21T08:24:48.585809+0000","last_became_peered":"2024-09-21T08:24:48.585809+0000","last_unstale":"2024-09-21T08:25:10.553951+0000","last_undegraded":"2024-09-21T08:25:10.553951+0000","last_fullsized":"2024-09-21T08:25:10.553951+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537483+0000","last_change":"2024-09-21T08:24:56.514295+0000","last_active":"2024-09-21T08:25:10.537483+0000","last_peered":"2024-09-21T08:25:10.537483+0000","last_clean":"2024-09-21T08:25:10.537483+0000","last_became_active":"2024-09-21T08:24:56.513734+0000","last_became_peered":"2024-09-21T08:24:56.513734+0000","last_unstale":"2024-09-21T08:25:10.537483+0000","last_undegraded":"2024-09-21T08:25:10.537483+0000","last_fullsized":"2024-09-21T08:25:10.537483+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672196+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"117'8","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.579287+0000","last_change":"2024-09-21T08:25:05.525919+0000","last_active":"2024-09-21T08:25:10.579287+0000","last_peered":"2024-09-21T08:25:10.579287+0000","last_clean":"2024-09-21T08:25:10.579287+0000","last_became_active":"2024-09-21T08:24:56.527516+0000","last_became_peered":"2024-09-21T08:24:56.527516+0000","last_unstale":"2024-09-21T08:25:10.579287+0000","last_undegraded":"2024-09-21T08:25:10.579287+0000","last_fullsized":"2024-09-21T08:25:10.579287+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:48.533846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013060845,"stat_sum":{"num_bytes":7344560,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":14,"num_read_kb":9554,"num_write":9,"num_write_kb":4636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1702450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.2","version":"73'10","reported_seq":141,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561484+0000","last_change":"2024-09-21T08:24:38.567048+0000","last_active":"2024-09-21T08:25:10.561484+0000","last_peered":"2024-09-21T08:25:10.561484+0000","last_clean":"2024-09-21T08:25:10.561484+0000","last_became_active":"2024-09-21T08:24:38.551531+0000","last_became_peered":"2024-09-21T08:24:38.551531+0000","last_unstale":"2024-09-21T08:25:10.561484+0000","last_undegraded":"2024-09-21T08:25:10.561484+0000","last_fullsized":"2024-09-21T08:25:10.561484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":139,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561299+0000","last_change":"2024-09-21T08:24:56.519880+0000","last_active":"2024-09-21T08:25:10.561299+0000","last_peered":"2024-09-21T08:25:10.561299+0000","last_clean":"2024-09-21T08:25:10.561299+0000","last_became_active":"2024-09-21T08:24:56.500882+0000","last_became_peered":"2024-09-21T08:24:56.500882+0000","last_unstale":"2024-09-21T08:25:10.561299+0000","last_undegraded":"2024-09-21T08:25:10.561299+0000","last_fullsized":"2024-09-21T08:25:10.561299+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770077+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"103'9","reported_seq":143,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543737+0000","last_change":"2024-09-21T08:24:56.541475+0000","last_active":"2024-09-21T08:25:10.543737+0000","last_peered":"2024-09-21T08:25:10.543737+0000","last_clean":"2024-09-21T08:25:10.543737+0000","last_became_active":"2024-09-21T08:24:56.518379+0000","last_became_peered":"2024-09-21T08:24:56.518379+0000","last_unstale":"2024-09-21T08:25:10.543737+0000","last_undegraded":"2024-09-21T08:25:10.543737+0000","last_fullsized":"2024-09-21T08:25:10.543737+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543316+0000","last_change":"2024-09-21T08:24:48.597514+0000","last_active":"2024-09-21T08:25:10.543316+0000","last_peered":"2024-09-21T08:25:10.543316+0000","last_clean":"2024-09-21T08:25:10.543316+0000","last_became_active":"2024-09-21T08:24:48.597048+0000","last_became_peered":"2024-09-21T08:24:48.597048+0000","last_unstale":"2024-09-21T08:25:10.543316+0000","last_undegraded":"2024-09-21T08:25:10.543316+0000","last_fullsized":"2024-09-21T08:25:10.543316+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215717+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"107'22","reported_seq":208,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560336+0000","last_change":"2024-09-21T08:25:05.519778+0000","last_active":"2024-09-21T08:25:10.560336+0000","last_peered":"2024-09-21T08:25:10.560336+0000","last_clean":"2024-09-21T08:25:10.560336+0000","last_became_active":"2024-09-21T08:24:48.586911+0000","last_became_peered":"2024-09-21T08:24:48.586911+0000","last_unstale":"2024-09-21T08:25:10.560336+0000","last_undegraded":"2024-09-21T08:25:10.560336+0000","last_fullsized":"2024-09-21T08:25:10.560336+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:48.446803+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00062885999999999997,"stat_sum":{"num_bytes":8571471,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":19,"num_read_kb":6165,"num_write":24,"num_write_kb":8102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3259822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.3","version":"73'19","reported_seq":144,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543880+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:25:10.543880+0000","last_peered":"2024-09-21T08:25:10.543880+0000","last_clean":"2024-09-21T08:25:10.543880+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:25:10.543880+0000","last_undegraded":"2024-09-21T08:25:10.543880+0000","last_fullsized":"2024-09-21T08:25:10.543880+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":120,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.552574+0000","last_change":"2024-09-21T08:24:52.529765+0000","last_active":"2024-09-21T08:25:10.552574+0000","last_peered":"2024-09-21T08:25:10.552574+0000","last_clean":"2024-09-21T08:25:10.552574+0000","last_became_active":"2024-09-21T08:23:30.547290+0000","last_became_peered":"2024-09-21T08:23:30.547290+0000","last_unstale":"2024-09-21T08:25:10.552574+0000","last_undegraded":"2024-09-21T08:25:10.552574+0000","last_fullsized":"2024-09-21T08:25:10.552574+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.529672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.529672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.610202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":39,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543014+0000","last_change":"2024-09-21T08:24:56.503913+0000","last_active":"2024-09-21T08:25:10.543014+0000","last_peered":"2024-09-21T08:25:10.543014+0000","last_clean":"2024-09-21T08:25:10.543014+0000","last_became_active":"2024-09-21T08:24:56.499628+0000","last_became_peered":"2024-09-21T08:24:56.499628+0000","last_unstale":"2024-09-21T08:25:10.543014+0000","last_undegraded":"2024-09-21T08:25:10.543014+0000","last_fullsized":"2024-09-21T08:25:10.543014+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215065+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553235+0000","last_change":"2024-09-21T08:24:56.520460+0000","last_active":"2024-09-21T08:25:10.553235+0000","last_peered":"2024-09-21T08:25:10.553235+0000","last_clean":"2024-09-21T08:25:10.553235+0000","last_became_active":"2024-09-21T08:24:56.517498+0000","last_became_peered":"2024-09-21T08:24:56.517498+0000","last_unstale":"2024-09-21T08:25:10.553235+0000","last_undegraded":"2024-09-21T08:25:10.553235+0000","last_fullsized":"2024-09-21T08:25:10.553235+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"112'14","reported_seq":197,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.544664+0000","last_change":"2024-09-21T08:25:05.529892+0000","last_active":"2024-09-21T08:25:10.544664+0000","last_peered":"2024-09-21T08:25:10.544664+0000","last_clean":"2024-09-21T08:25:10.544664+0000","last_became_active":"2024-09-21T08:24:56.513790+0000","last_became_peered":"2024-09-21T08:24:56.513790+0000","last_unstale":"2024-09-21T08:25:10.544664+0000","last_undegraded":"2024-09-21T08:25:10.544664+0000","last_fullsized":"2024-09-21T08:25:10.544664+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:43.300144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020476579999999999,"stat_sum":{"num_bytes":4689087,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":14,"num_read_kb":11105,"num_write":11,"num_write_kb":2312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4730540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.4","version":"103'30","reported_seq":233,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.552528+0000","last_change":"2024-09-21T08:24:56.561857+0000","last_active":"2024-09-21T08:25:10.552528+0000","last_peered":"2024-09-21T08:25:10.552528+0000","last_clean":"2024-09-21T08:25:10.552528+0000","last_became_active":"2024-09-21T08:24:56.519586+0000","last_became_peered":"2024-09-21T08:24:56.519586+0000","last_unstale":"2024-09-21T08:25:10.552528+0000","last_undegraded":"2024-09-21T08:25:10.552528+0000","last_fullsized":"2024-09-21T08:25:10.552528+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":81,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.544045+0000","last_change":"2024-09-21T08:24:56.519301+0000","last_active":"2024-09-21T08:25:10.544045+0000","last_peered":"2024-09-21T08:25:10.544045+0000","last_clean":"2024-09-21T08:25:10.544045+0000","last_became_active":"2024-09-21T08:24:56.518876+0000","last_became_peered":"2024-09-21T08:24:56.518876+0000","last_unstale":"2024-09-21T08:25:10.544045+0000","last_undegraded":"2024-09-21T08:25:10.544045+0000","last_fullsized":"2024-09-21T08:25:10.544045+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":133,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560662+0000","last_change":"2024-09-21T08:24:56.556151+0000","last_active":"2024-09-21T08:25:10.560662+0000","last_peered":"2024-09-21T08:25:10.560662+0000","last_clean":"2024-09-21T08:25:10.560662+0000","last_became_active":"2024-09-21T08:24:56.508710+0000","last_became_peered":"2024-09-21T08:24:56.508710+0000","last_unstale":"2024-09-21T08:25:10.560662+0000","last_undegraded":"2024-09-21T08:25:10.560662+0000","last_fullsized":"2024-09-21T08:25:10.560662+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.769998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.546043+0000","last_change":"2024-09-21T08:24:56.516897+0000","last_active":"2024-09-21T08:25:10.546043+0000","last_peered":"2024-09-21T08:25:10.546043+0000","last_clean":"2024-09-21T08:25:10.546043+0000","last_became_active":"2024-09-21T08:24:56.516563+0000","last_became_peered":"2024-09-21T08:24:56.516563+0000","last_unstale":"2024-09-21T08:25:10.546043+0000","last_undegraded":"2024-09-21T08:25:10.546043+0000","last_fullsized":"2024-09-21T08:25:10.546043+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215347+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":95,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.538115+0000","last_change":"2024-09-21T08:24:56.506298+0000","last_active":"2024-09-21T08:25:10.538115+0000","last_peered":"2024-09-21T08:25:10.538115+0000","last_clean":"2024-09-21T08:25:10.538115+0000","last_became_active":"2024-09-21T08:24:56.505969+0000","last_became_peered":"2024-09-21T08:24:56.505969+0000","last_unstale":"2024-09-21T08:25:10.538115+0000","last_undegraded":"2024-09-21T08:25:10.538115+0000","last_fullsized":"2024-09-21T08:25:10.538115+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"110'18","reported_seq":201,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.544507+0000","last_change":"2024-09-21T08:25:05.526296+0000","last_active":"2024-09-21T08:25:10.544507+0000","last_peered":"2024-09-21T08:25:10.544507+0000","last_clean":"2024-09-21T08:25:10.544507+0000","last_became_active":"2024-09-21T08:24:56.517718+0000","last_became_peered":"2024-09-21T08:24:56.517718+0000","last_unstale":"2024-09-21T08:25:10.544507+0000","last_undegraded":"2024-09-21T08:25:10.544507+0000","last_fullsized":"2024-09-21T08:25:10.544507+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:00.454266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00060703199999999997,"stat_sum":{"num_bytes":3428595,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":12585,"num_write":13,"num_write_kb":3668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1312313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.7","version":"73'13","reported_seq":135,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.554022+0000","last_change":"2024-09-21T08:23:32.560593+0000","last_active":"2024-09-21T08:25:10.554022+0000","last_peered":"2024-09-21T08:25:10.554022+0000","last_clean":"2024-09-21T08:25:10.554022+0000","last_became_active":"2024-09-21T08:23:32.560323+0000","last_became_peered":"2024-09-21T08:23:32.560323+0000","last_unstale":"2024-09-21T08:25:10.554022+0000","last_undegraded":"2024-09-21T08:25:10.554022+0000","last_fullsized":"2024-09-21T08:25:10.554022+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582391+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553287+0000","last_change":"2024-09-21T08:24:56.525402+0000","last_active":"2024-09-21T08:25:10.553287+0000","last_peered":"2024-09-21T08:25:10.553287+0000","last_clean":"2024-09-21T08:25:10.553287+0000","last_became_active":"2024-09-21T08:24:56.524958+0000","last_became_peered":"2024-09-21T08:24:56.524958+0000","last_unstale":"2024-09-21T08:25:10.553287+0000","last_undegraded":"2024-09-21T08:25:10.553287+0000","last_fullsized":"2024-09-21T08:25:10.553287+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582299+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":139,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.555414+0000","last_change":"2024-09-21T08:24:50.675752+0000","last_active":"2024-09-21T08:25:10.555414+0000","last_peered":"2024-09-21T08:25:10.555414+0000","last_clean":"2024-09-21T08:25:10.555414+0000","last_became_active":"2024-09-21T08:24:48.604993+0000","last_became_peered":"2024-09-21T08:24:48.604993+0000","last_unstale":"2024-09-21T08:25:10.555414+0000","last_undegraded":"2024-09-21T08:25:10.555414+0000","last_fullsized":"2024-09-21T08:25:10.555414+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:50.675673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.675673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.957118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.556161+0000","last_change":"2024-09-21T08:23:34.567541+0000","last_active":"2024-09-21T08:25:10.556161+0000","last_peered":"2024-09-21T08:25:10.556161+0000","last_clean":"2024-09-21T08:25:10.556161+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:25:10.556161+0000","last_undegraded":"2024-09-21T08:25:10.556161+0000","last_fullsized":"2024-09-21T08:25:10.556161+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":111,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537792+0000","last_change":"2024-09-21T08:24:56.501148+0000","last_active":"2024-09-21T08:25:10.537792+0000","last_peered":"2024-09-21T08:25:10.537792+0000","last_clean":"2024-09-21T08:25:10.537792+0000","last_became_active":"2024-09-21T08:24:56.498476+0000","last_became_peered":"2024-09-21T08:24:56.498476+0000","last_unstale":"2024-09-21T08:25:10.537792+0000","last_undegraded":"2024-09-21T08:25:10.537792+0000","last_fullsized":"2024-09-21T08:25:10.537792+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"116'71","reported_seq":245,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537723+0000","last_change":"2024-09-21T08:25:05.528869+0000","last_active":"2024-09-21T08:25:10.537723+0000","last_peered":"2024-09-21T08:25:10.537723+0000","last_clean":"2024-09-21T08:25:10.537723+0000","last_became_active":"2024-09-21T08:24:34.507710+0000","last_became_peered":"2024-09-21T08:24:34.507710+0000","last_unstale":"2024-09-21T08:25:10.537723+0000","last_undegraded":"2024-09-21T08:25:10.537723+0000","last_fullsized":"2024-09-21T08:25:10.537723+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:02.723680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012186123,"stat_sum":{"num_bytes":18903980,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":46,"num_read_kb":31006,"num_write":54,"num_write_kb":20040,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.6","version":"73'12","reported_seq":126,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543859+0000","last_change":"2024-09-21T08:23:32.561943+0000","last_active":"2024-09-21T08:25:10.543859+0000","last_peered":"2024-09-21T08:25:10.543859+0000","last_clean":"2024-09-21T08:25:10.543859+0000","last_became_active":"2024-09-21T08:23:32.561617+0000","last_became_peered":"2024-09-21T08:23:32.561617+0000","last_unstale":"2024-09-21T08:25:10.543859+0000","last_undegraded":"2024-09-21T08:25:10.543859+0000","last_fullsized":"2024-09-21T08:25:10.543859+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215125+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":124,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543791+0000","last_change":"2024-09-21T08:24:48.601343+0000","last_active":"2024-09-21T08:25:10.543791+0000","last_peered":"2024-09-21T08:25:10.543791+0000","last_clean":"2024-09-21T08:25:10.543791+0000","last_became_active":"2024-09-21T08:24:48.601044+0000","last_became_peered":"2024-09-21T08:24:48.601044+0000","last_unstale":"2024-09-21T08:25:10.543791+0000","last_undegraded":"2024-09-21T08:25:10.543791+0000","last_fullsized":"2024-09-21T08:25:10.543791+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":225,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537035+0000","last_change":"2024-09-21T08:24:47.740562+0000","last_active":"2024-09-21T08:25:10.537035+0000","last_peered":"2024-09-21T08:25:10.537035+0000","last_clean":"2024-09-21T08:25:10.537035+0000","last_became_active":"2024-09-21T08:23:29.532520+0000","last_became_peered":"2024-09-21T08:23:29.532520+0000","last_unstale":"2024-09-21T08:25:10.537035+0000","last_undegraded":"2024-09-21T08:25:10.537035+0000","last_fullsized":"2024-09-21T08:25:10.537035+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:04.383800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":95,"num_read_kb":90,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":105,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561496+0000","last_change":"2024-09-21T08:23:34.570449+0000","last_active":"2024-09-21T08:25:10.561496+0000","last_peered":"2024-09-21T08:25:10.561496+0000","last_clean":"2024-09-21T08:25:10.561496+0000","last_became_active":"2024-09-21T08:23:34.569970+0000","last_became_peered":"2024-09-21T08:23:34.569970+0000","last_unstale":"2024-09-21T08:25:10.561496+0000","last_undegraded":"2024-09-21T08:25:10.561496+0000","last_fullsized":"2024-09-21T08:25:10.561496+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.769917+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.552261+0000","last_change":"2024-09-21T08:24:56.520951+0000","last_active":"2024-09-21T08:25:10.552261+0000","last_peered":"2024-09-21T08:25:10.552261+0000","last_clean":"2024-09-21T08:25:10.552261+0000","last_became_active":"2024-09-21T08:24:56.520320+0000","last_became_peered":"2024-09-21T08:24:56.520320+0000","last_unstale":"2024-09-21T08:25:10.552261+0000","last_undegraded":"2024-09-21T08:25:10.552261+0000","last_fullsized":"2024-09-21T08:25:10.552261+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:11.591309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"106'31","reported_seq":214,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553063+0000","last_change":"2024-09-21T08:25:05.532058+0000","last_active":"2024-09-21T08:25:10.553063+0000","last_peered":"2024-09-21T08:25:10.553063+0000","last_clean":"2024-09-21T08:25:10.553063+0000","last_became_active":"2024-09-21T08:24:48.593246+0000","last_became_peered":"2024-09-21T08:24:48.593246+0000","last_unstale":"2024-09-21T08:25:10.553063+0000","last_undegraded":"2024-09-21T08:25:10.553063+0000","last_fullsized":"2024-09-21T08:25:10.553063+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:45.521510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0041341329999999999,"stat_sum":{"num_bytes":3025227,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":24699,"num_write":17,"num_write_kb":5815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4966638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.5","version":"103'18","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.538355+0000","last_change":"2024-09-21T08:24:56.564202+0000","last_active":"2024-09-21T08:25:10.538355+0000","last_peered":"2024-09-21T08:25:10.538355+0000","last_clean":"2024-09-21T08:25:10.538355+0000","last_became_active":"2024-09-21T08:24:56.503984+0000","last_became_peered":"2024-09-21T08:24:56.503984+0000","last_unstale":"2024-09-21T08:25:10.538355+0000","last_undegraded":"2024-09-21T08:25:10.538355+0000","last_fullsized":"2024-09-21T08:25:10.538355+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-09-21T08:24:48.583137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.583137+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.538958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":63,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560579+0000","last_change":"2024-09-21T08:24:56.509057+0000","last_active":"2024-09-21T08:25:10.560579+0000","last_peered":"2024-09-21T08:25:10.560579+0000","last_clean":"2024-09-21T08:25:10.560579+0000","last_became_active":"2024-09-21T08:24:56.508272+0000","last_became_peered":"2024-09-21T08:24:56.508272+0000","last_unstale":"2024-09-21T08:25:10.560579+0000","last_undegraded":"2024-09-21T08:25:10.560579+0000","last_fullsized":"2024-09-21T08:25:10.560579+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770045+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":381,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537278+0000","last_change":"2024-09-21T08:24:59.498580+0000","last_active":"2024-09-21T08:25:10.537278+0000","last_peered":"2024-09-21T08:25:10.537278+0000","last_clean":"2024-09-21T08:25:10.537278+0000","last_became_active":"2024-09-21T08:24:59.498199+0000","last_became_peered":"2024-09-21T08:24:59.498199+0000","last_unstale":"2024-09-21T08:25:10.537278+0000","last_undegraded":"2024-09-21T08:25:10.537278+0000","last_fullsized":"2024-09-21T08:25:10.537278+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:24:05.765804+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672477+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":103,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537709+0000","last_change":"2024-09-21T08:23:34.568359+0000","last_active":"2024-09-21T08:25:10.537709+0000","last_peered":"2024-09-21T08:25:10.537709+0000","last_clean":"2024-09-21T08:25:10.537709+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:25:10.537709+0000","last_undegraded":"2024-09-21T08:25:10.537709+0000","last_fullsized":"2024-09-21T08:25:10.537709+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":121,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561396+0000","last_change":"2024-09-21T08:24:48.726197+0000","last_active":"2024-09-21T08:25:10.561396+0000","last_peered":"2024-09-21T08:25:10.561396+0000","last_clean":"2024-09-21T08:25:10.561396+0000","last_became_active":"2024-09-21T08:24:48.575960+0000","last_became_peered":"2024-09-21T08:24:48.575960+0000","last_unstale":"2024-09-21T08:25:10.561396+0000","last_undegraded":"2024-09-21T08:25:10.561396+0000","last_fullsized":"2024-09-21T08:25:10.561396+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:02.092267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"116'26","reported_seq":130,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.554384+0000","last_change":"2024-09-21T08:25:05.525640+0000","last_active":"2024-09-21T08:25:10.554384+0000","last_peered":"2024-09-21T08:25:10.554384+0000","last_clean":"2024-09-21T08:25:10.554384+0000","last_became_active":"2024-09-21T08:24:56.524759+0000","last_became_peered":"2024-09-21T08:24:56.524759+0000","last_unstale":"2024-09-21T08:25:10.554384+0000","last_undegraded":"2024-09-21T08:25:10.554384+0000","last_fullsized":"2024-09-21T08:25:10.554384+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:49.957763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.013111496,"stat_sum":{"num_bytes":6823576,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":11023,"num_write":20,"num_write_kb":8505,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2039342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.e","version":"73'11","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.552223+0000","last_change":"2024-09-21T08:24:56.538842+0000","last_active":"2024-09-21T08:25:10.552223+0000","last_peered":"2024-09-21T08:25:10.552223+0000","last_clean":"2024-09-21T08:25:10.552223+0000","last_became_active":"2024-09-21T08:24:56.510706+0000","last_became_peered":"2024-09-21T08:24:56.510706+0000","last_unstale":"2024-09-21T08:25:10.552223+0000","last_undegraded":"2024-09-21T08:25:10.552223+0000","last_fullsized":"2024-09-21T08:25:10.552223+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":84,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.554765+0000","last_change":"2024-09-21T08:24:48.756972+0000","last_active":"2024-09-21T08:25:10.554765+0000","last_peered":"2024-09-21T08:25:10.554765+0000","last_clean":"2024-09-21T08:25:10.554765+0000","last_became_active":"2024-09-21T08:24:38.548474+0000","last_became_peered":"2024-09-21T08:24:38.548474+0000","last_unstale":"2024-09-21T08:25:10.554765+0000","last_undegraded":"2024-09-21T08:25:10.554765+0000","last_fullsized":"2024-09-21T08:25:10.554765+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.756905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.756905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.943320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561395+0000","last_change":"2024-09-21T08:24:56.508737+0000","last_active":"2024-09-21T08:25:10.561395+0000","last_peered":"2024-09-21T08:25:10.561395+0000","last_clean":"2024-09-21T08:25:10.561395+0000","last_became_active":"2024-09-21T08:24:56.508405+0000","last_became_peered":"2024-09-21T08:24:56.508405+0000","last_unstale":"2024-09-21T08:25:10.561395+0000","last_undegraded":"2024-09-21T08:25:10.561395+0000","last_fullsized":"2024-09-21T08:25:10.561395+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":42,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560752+0000","last_change":"2024-09-21T08:24:56.517612+0000","last_active":"2024-09-21T08:25:10.560752+0000","last_peered":"2024-09-21T08:25:10.560752+0000","last_clean":"2024-09-21T08:25:10.560752+0000","last_became_active":"2024-09-21T08:24:56.516670+0000","last_became_peered":"2024-09-21T08:24:56.516670+0000","last_unstale":"2024-09-21T08:25:10.560752+0000","last_undegraded":"2024-09-21T08:25:10.560752+0000","last_fullsized":"2024-09-21T08:25:10.560752+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"117'23","reported_seq":193,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.571528+0000","last_change":"2024-09-21T08:25:05.522973+0000","last_active":"2024-09-21T08:25:10.571528+0000","last_peered":"2024-09-21T08:25:10.571528+0000","last_clean":"2024-09-21T08:25:10.571528+0000","last_became_active":"2024-09-21T08:24:34.508974+0000","last_became_peered":"2024-09-21T08:24:34.508974+0000","last_unstale":"2024-09-21T08:25:10.571528+0000","last_undegraded":"2024-09-21T08:25:10.571528+0000","last_fullsized":"2024-09-21T08:25:10.571528+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:01.498209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0012739979999999999,"stat_sum":{"num_bytes":8605085,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":16,"num_read_kb":7780,"num_write":19,"num_write_kb":7668,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.d","version":"73'17","reported_seq":165,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.559640+0000","last_change":"2024-09-21T08:24:56.553183+0000","last_active":"2024-09-21T08:25:10.559640+0000","last_peered":"2024-09-21T08:25:10.559640+0000","last_clean":"2024-09-21T08:25:10.559640+0000","last_became_active":"2024-09-21T08:24:56.514487+0000","last_became_peered":"2024-09-21T08:24:56.514487+0000","last_unstale":"2024-09-21T08:25:10.559640+0000","last_undegraded":"2024-09-21T08:25:10.559640+0000","last_fullsized":"2024-09-21T08:25:10.559640+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":112,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561310+0000","last_change":"2024-09-21T08:24:56.516700+0000","last_active":"2024-09-21T08:25:10.561310+0000","last_peered":"2024-09-21T08:25:10.561310+0000","last_clean":"2024-09-21T08:25:10.561310+0000","last_became_active":"2024-09-21T08:24:56.516326+0000","last_became_peered":"2024-09-21T08:24:56.516326+0000","last_unstale":"2024-09-21T08:25:10.561310+0000","last_undegraded":"2024-09-21T08:25:10.561310+0000","last_fullsized":"2024-09-21T08:25:10.561310+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.552323+0000","last_change":"2024-09-21T08:24:56.526336+0000","last_active":"2024-09-21T08:25:10.552323+0000","last_peered":"2024-09-21T08:25:10.552323+0000","last_clean":"2024-09-21T08:25:10.552323+0000","last_became_active":"2024-09-21T08:24:56.524399+0000","last_became_peered":"2024-09-21T08:24:56.524399+0000","last_unstale":"2024-09-21T08:25:10.552323+0000","last_undegraded":"2024-09-21T08:25:10.552323+0000","last_fullsized":"2024-09-21T08:25:10.552323+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582137+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":126,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.552454+0000","last_change":"2024-09-21T08:24:56.520526+0000","last_active":"2024-09-21T08:25:10.552454+0000","last_peered":"2024-09-21T08:25:10.552454+0000","last_clean":"2024-09-21T08:25:10.552454+0000","last_became_active":"2024-09-21T08:24:56.517208+0000","last_became_peered":"2024-09-21T08:24:56.517208+0000","last_unstale":"2024-09-21T08:25:10.552454+0000","last_undegraded":"2024-09-21T08:25:10.552454+0000","last_fullsized":"2024-09-21T08:25:10.552454+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.766815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:01.409327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"100'7","reported_seq":179,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.544832+0000","last_change":"2024-09-21T08:25:05.531836+0000","last_active":"2024-09-21T08:25:10.544832+0000","last_peered":"2024-09-21T08:25:10.544832+0000","last_clean":"2024-09-21T08:25:10.544832+0000","last_became_active":"2024-09-21T08:24:48.586187+0000","last_became_peered":"2024-09-21T08:24:48.586187+0000","last_unstale":"2024-09-21T08:25:10.544832+0000","last_undegraded":"2024-09-21T08:25:10.544832+0000","last_fullsized":"2024-09-21T08:25:10.544832+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:50.280532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038822209999999999,"stat_sum":{"num_bytes":2369754,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":3244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":949927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.c","version":"73'10","reported_seq":88,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.552089+0000","last_change":"2024-09-21T08:24:56.567156+0000","last_active":"2024-09-21T08:25:10.552089+0000","last_peered":"2024-09-21T08:25:10.552089+0000","last_clean":"2024-09-21T08:25:10.552089+0000","last_became_active":"2024-09-21T08:24:56.521403+0000","last_became_peered":"2024-09-21T08:24:56.521403+0000","last_unstale":"2024-09-21T08:25:10.552089+0000","last_undegraded":"2024-09-21T08:25:10.552089+0000","last_fullsized":"2024-09-21T08:25:10.552089+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":101,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543970+0000","last_change":"2024-09-21T08:24:48.596461+0000","last_active":"2024-09-21T08:25:10.543970+0000","last_peered":"2024-09-21T08:25:10.543970+0000","last_clean":"2024-09-21T08:25:10.543970+0000","last_became_active":"2024-09-21T08:24:48.595725+0000","last_became_peered":"2024-09-21T08:24:48.595725+0000","last_unstale":"2024-09-21T08:25:10.543970+0000","last_undegraded":"2024-09-21T08:25:10.543970+0000","last_fullsized":"2024-09-21T08:25:10.543970+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":141,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.559512+0000","last_change":"2024-09-21T08:24:51.689976+0000","last_active":"2024-09-21T08:25:10.559512+0000","last_peered":"2024-09-21T08:25:10.559512+0000","last_clean":"2024-09-21T08:25:10.559512+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:25:10.559512+0000","last_undegraded":"2024-09-21T08:25:10.559512+0000","last_fullsized":"2024-09-21T08:25:10.559512+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:24:51.689901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.689901+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.850967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":77,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.552454+0000","last_change":"2024-09-21T08:24:38.550749+0000","last_active":"2024-09-21T08:25:10.552454+0000","last_peered":"2024-09-21T08:25:10.552454+0000","last_clean":"2024-09-21T08:25:10.552454+0000","last_became_active":"2024-09-21T08:24:38.550434+0000","last_became_peered":"2024-09-21T08:24:38.550434+0000","last_unstale":"2024-09-21T08:25:10.552454+0000","last_undegraded":"2024-09-21T08:25:10.552454+0000","last_fullsized":"2024-09-21T08:25:10.552454+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582423+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"117'45","reported_seq":244,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.584290+0000","last_change":"2024-09-21T08:25:05.519600+0000","last_active":"2024-09-21T08:25:10.584290+0000","last_peered":"2024-09-21T08:25:10.584290+0000","last_clean":"2024-09-21T08:25:10.584290+0000","last_became_active":"2024-09-21T08:24:48.590826+0000","last_became_peered":"2024-09-21T08:24:48.590826+0000","last_unstale":"2024-09-21T08:25:10.584290+0000","last_undegraded":"2024-09-21T08:25:10.584290+0000","last_fullsized":"2024-09-21T08:25:10.584290+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.813636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061659699999999998,"stat_sum":{"num_bytes":12593498,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":26,"num_read_kb":13397,"num_write":41,"num_write_kb":13786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8754687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.b","version":"73'9","reported_seq":156,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543785+0000","last_change":"2024-09-21T08:24:48.620574+0000","last_active":"2024-09-21T08:25:10.543785+0000","last_peered":"2024-09-21T08:25:10.543785+0000","last_clean":"2024-09-21T08:25:10.543785+0000","last_became_active":"2024-09-21T08:24:48.590921+0000","last_became_peered":"2024-09-21T08:24:48.590921+0000","last_unstale":"2024-09-21T08:25:10.543785+0000","last_undegraded":"2024-09-21T08:25:10.543785+0000","last_fullsized":"2024-09-21T08:25:10.543785+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.214973+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":133,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560993+0000","last_change":"2024-09-21T08:24:58.665514+0000","last_active":"2024-09-21T08:25:10.560993+0000","last_peered":"2024-09-21T08:25:10.560993+0000","last_clean":"2024-09-21T08:25:10.560993+0000","last_became_active":"2024-09-21T08:24:56.517512+0000","last_became_peered":"2024-09-21T08:24:56.517512+0000","last_unstale":"2024-09-21T08:25:10.560993+0000","last_undegraded":"2024-09-21T08:25:10.560993+0000","last_fullsized":"2024-09-21T08:25:10.560993+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.665444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.665444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.770807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":116,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.557336+0000","last_change":"2024-09-21T08:24:48.598120+0000","last_active":"2024-09-21T08:25:10.557336+0000","last_peered":"2024-09-21T08:25:10.557336+0000","last_clean":"2024-09-21T08:25:10.557336+0000","last_became_active":"2024-09-21T08:24:48.597800+0000","last_became_peered":"2024-09-21T08:24:48.597800+0000","last_unstale":"2024-09-21T08:25:10.557336+0000","last_undegraded":"2024-09-21T08:25:10.557336+0000","last_fullsized":"2024-09-21T08:25:10.557336+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":100,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543687+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:25:10.543687+0000","last_peered":"2024-09-21T08:25:10.543687+0000","last_clean":"2024-09-21T08:25:10.543687+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:25:10.543687+0000","last_undegraded":"2024-09-21T08:25:10.543687+0000","last_fullsized":"2024-09-21T08:25:10.543687+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"117'46","reported_seq":234,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.601695+0000","last_change":"2024-09-21T08:25:05.531014+0000","last_active":"2024-09-21T08:25:10.601695+0000","last_peered":"2024-09-21T08:25:10.601695+0000","last_clean":"2024-09-21T08:25:10.601695+0000","last_became_active":"2024-09-21T08:24:38.546433+0000","last_became_peered":"2024-09-21T08:24:38.546433+0000","last_unstale":"2024-09-21T08:25:10.601695+0000","last_undegraded":"2024-09-21T08:25:10.601695+0000","last_fullsized":"2024-09-21T08:25:10.601695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:40.902647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0049869839999999999,"stat_sum":{"num_bytes":16314720,"num_objects":6,"num_object_clones":1,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":58,"num_read_kb":41538,"num_write":40,"num_write_kb":17801,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.a","version":"73'19","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561574+0000","last_change":"2024-09-21T08:24:56.553824+0000","last_active":"2024-09-21T08:25:10.561574+0000","last_peered":"2024-09-21T08:25:10.561574+0000","last_clean":"2024-09-21T08:25:10.561574+0000","last_became_active":"2024-09-21T08:24:56.511681+0000","last_became_peered":"2024-09-21T08:24:56.511681+0000","last_unstale":"2024-09-21T08:25:10.561574+0000","last_undegraded":"2024-09-21T08:25:10.561574+0000","last_fullsized":"2024-09-21T08:25:10.561574+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":125,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.538319+0000","last_change":"2024-09-21T08:24:56.507593+0000","last_active":"2024-09-21T08:25:10.538319+0000","last_peered":"2024-09-21T08:25:10.538319+0000","last_clean":"2024-09-21T08:25:10.538319+0000","last_became_active":"2024-09-21T08:24:56.505054+0000","last_became_peered":"2024-09-21T08:24:56.505054+0000","last_unstale":"2024-09-21T08:25:10.538319+0000","last_undegraded":"2024-09-21T08:25:10.538319+0000","last_fullsized":"2024-09-21T08:25:10.538319+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.740674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.740674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.640221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.558702+0000","last_change":"2024-09-21T08:23:34.564923+0000","last_active":"2024-09-21T08:25:10.558702+0000","last_peered":"2024-09-21T08:25:10.558702+0000","last_clean":"2024-09-21T08:25:10.558702+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:25:10.558702+0000","last_undegraded":"2024-09-21T08:25:10.558702+0000","last_fullsized":"2024-09-21T08:25:10.558702+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":119,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537096+0000","last_change":"2024-09-21T08:24:56.504692+0000","last_active":"2024-09-21T08:25:10.537096+0000","last_peered":"2024-09-21T08:25:10.537096+0000","last_clean":"2024-09-21T08:25:10.537096+0000","last_became_active":"2024-09-21T08:24:56.503564+0000","last_became_peered":"2024-09-21T08:24:56.503564+0000","last_unstale":"2024-09-21T08:25:10.537096+0000","last_undegraded":"2024-09-21T08:25:10.537096+0000","last_fullsized":"2024-09-21T08:25:10.537096+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:54.693321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"101'13","reported_seq":151,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.556907+0000","last_change":"2024-09-21T08:25:05.520226+0000","last_active":"2024-09-21T08:25:10.556907+0000","last_peered":"2024-09-21T08:25:10.556907+0000","last_clean":"2024-09-21T08:25:10.556907+0000","last_became_active":"2024-09-21T08:24:48.598068+0000","last_became_peered":"2024-09-21T08:24:48.598068+0000","last_unstale":"2024-09-21T08:25:10.556907+0000","last_undegraded":"2024-09-21T08:25:10.556907+0000","last_fullsized":"2024-09-21T08:25:10.556907+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:02.963454+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00087521400000000003,"stat_sum":{"num_bytes":3159683,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":10,"num_write_kb":2495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1161879,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.9","version":"73'12","reported_seq":166,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553398+0000","last_change":"2024-09-21T08:24:56.577372+0000","last_active":"2024-09-21T08:25:10.553398+0000","last_peered":"2024-09-21T08:25:10.553398+0000","last_clean":"2024-09-21T08:25:10.553398+0000","last_became_active":"2024-09-21T08:24:56.526686+0000","last_became_peered":"2024-09-21T08:24:56.526686+0000","last_unstale":"2024-09-21T08:25:10.553398+0000","last_undegraded":"2024-09-21T08:25:10.553398+0000","last_fullsized":"2024-09-21T08:25:10.553398+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537563+0000","last_change":"2024-09-21T08:24:38.544579+0000","last_active":"2024-09-21T08:25:10.537563+0000","last_peered":"2024-09-21T08:25:10.537563+0000","last_clean":"2024-09-21T08:25:10.537563+0000","last_became_active":"2024-09-21T08:24:38.544215+0000","last_became_peered":"2024-09-21T08:24:38.544215+0000","last_unstale":"2024-09-21T08:25:10.537563+0000","last_undegraded":"2024-09-21T08:25:10.537563+0000","last_fullsized":"2024-09-21T08:25:10.537563+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672305+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.558485+0000","last_change":"2024-09-21T08:23:34.566472+0000","last_active":"2024-09-21T08:25:10.558485+0000","last_peered":"2024-09-21T08:25:10.558485+0000","last_clean":"2024-09-21T08:25:10.558485+0000","last_became_active":"2024-09-21T08:23:34.566193+0000","last_became_peered":"2024-09-21T08:23:34.566193+0000","last_unstale":"2024-09-21T08:25:10.558485+0000","last_undegraded":"2024-09-21T08:25:10.558485+0000","last_fullsized":"2024-09-21T08:25:10.558485+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":61,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537307+0000","last_change":"2024-09-21T08:24:56.514470+0000","last_active":"2024-09-21T08:25:10.537307+0000","last_peered":"2024-09-21T08:25:10.537307+0000","last_clean":"2024-09-21T08:25:10.537307+0000","last_became_active":"2024-09-21T08:24:56.513965+0000","last_became_peered":"2024-09-21T08:24:56.513965+0000","last_unstale":"2024-09-21T08:25:10.537307+0000","last_undegraded":"2024-09-21T08:25:10.537307+0000","last_fullsized":"2024-09-21T08:25:10.537307+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672228+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"116'81","reported_seq":299,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.555244+0000","last_change":"2024-09-21T08:25:05.526197+0000","last_active":"2024-09-21T08:25:10.555244+0000","last_peered":"2024-09-21T08:25:10.555244+0000","last_clean":"2024-09-21T08:25:10.555244+0000","last_became_active":"2024-09-21T08:24:56.524762+0000","last_became_peered":"2024-09-21T08:24:56.524762+0000","last_unstale":"2024-09-21T08:25:10.555244+0000","last_undegraded":"2024-09-21T08:25:10.555244+0000","last_fullsized":"2024-09-21T08:25:10.555244+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":81,"log_dups_size":0,"ondisk_log_size":81,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:50.970670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0094472789999999994,"stat_sum":{"num_bytes":26806868,"num_objects":10,"num_object_clones":2,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":69,"num_read_kb":49961,"num_write":71,"num_write_kb":20453,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":12669831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"7","length":"1"},{"start":"9","length":"4"},{"start":"e","length":"1"}]},{"pgid":"4.8","version":"73'15","reported_seq":164,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560377+0000","last_change":"2024-09-21T08:24:56.538033+0000","last_active":"2024-09-21T08:25:10.560377+0000","last_peered":"2024-09-21T08:25:10.560377+0000","last_clean":"2024-09-21T08:25:10.560377+0000","last_became_active":"2024-09-21T08:24:56.506378+0000","last_became_peered":"2024-09-21T08:24:56.506378+0000","last_unstale":"2024-09-21T08:25:10.560377+0000","last_undegraded":"2024-09-21T08:25:10.560377+0000","last_fullsized":"2024-09-21T08:25:10.560377+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.521582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":143,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.538397+0000","last_change":"2024-09-21T08:24:38.583223+0000","last_active":"2024-09-21T08:25:10.538397+0000","last_peered":"2024-09-21T08:25:10.538397+0000","last_clean":"2024-09-21T08:25:10.538397+0000","last_became_active":"2024-09-21T08:24:38.551180+0000","last_became_peered":"2024-09-21T08:24:38.551180+0000","last_unstale":"2024-09-21T08:25:10.538397+0000","last_undegraded":"2024-09-21T08:25:10.538397+0000","last_fullsized":"2024-09-21T08:25:10.538397+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:32.960573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"103'10","reported_seq":153,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537245+0000","last_change":"2024-09-21T08:24:56.562671+0000","last_active":"2024-09-21T08:25:10.537245+0000","last_peered":"2024-09-21T08:25:10.537245+0000","last_clean":"2024-09-21T08:25:10.537245+0000","last_became_active":"2024-09-21T08:24:56.511525+0000","last_became_peered":"2024-09-21T08:24:56.511525+0000","last_unstale":"2024-09-21T08:25:10.537245+0000","last_undegraded":"2024-09-21T08:25:10.537245+0000","last_fullsized":"2024-09-21T08:25:10.537245+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":107,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.560406+0000","last_change":"2024-09-21T08:24:56.500567+0000","last_active":"2024-09-21T08:25:10.560406+0000","last_peered":"2024-09-21T08:25:10.560406+0000","last_clean":"2024-09-21T08:25:10.560406+0000","last_became_active":"2024-09-21T08:24:56.500287+0000","last_became_peered":"2024-09-21T08:24:56.500287+0000","last_unstale":"2024-09-21T08:25:10.560406+0000","last_undegraded":"2024-09-21T08:25:10.560406+0000","last_fullsized":"2024-09-21T08:25:10.560406+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":39,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.542198+0000","last_change":"2024-09-21T08:24:56.500240+0000","last_active":"2024-09-21T08:25:10.542198+0000","last_peered":"2024-09-21T08:25:10.542198+0000","last_clean":"2024-09-21T08:25:10.542198+0000","last_became_active":"2024-09-21T08:24:56.499114+0000","last_became_peered":"2024-09-21T08:24:56.499114+0000","last_unstale":"2024-09-21T08:25:10.542198+0000","last_undegraded":"2024-09-21T08:25:10.542198+0000","last_fullsized":"2024-09-21T08:25:10.542198+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":91,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.538330+0000","last_change":"2024-09-21T08:24:48.621753+0000","last_active":"2024-09-21T08:25:10.538330+0000","last_peered":"2024-09-21T08:25:10.538330+0000","last_clean":"2024-09-21T08:25:10.538330+0000","last_became_active":"2024-09-21T08:24:48.590639+0000","last_became_peered":"2024-09-21T08:24:48.590639+0000","last_unstale":"2024-09-21T08:25:10.538330+0000","last_undegraded":"2024-09-21T08:25:10.538330+0000","last_fullsized":"2024-09-21T08:25:10.538330+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672384+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":117,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561283+0000","last_change":"2024-09-21T08:24:56.508740+0000","last_active":"2024-09-21T08:25:10.561283+0000","last_peered":"2024-09-21T08:25:10.561283+0000","last_clean":"2024-09-21T08:25:10.561283+0000","last_became_active":"2024-09-21T08:24:56.508415+0000","last_became_peered":"2024-09-21T08:24:56.508415+0000","last_unstale":"2024-09-21T08:25:10.561283+0000","last_undegraded":"2024-09-21T08:25:10.561283+0000","last_fullsized":"2024-09-21T08:25:10.561283+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.769948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":111,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537637+0000","last_change":"2024-09-21T08:24:56.506870+0000","last_active":"2024-09-21T08:25:10.537637+0000","last_peered":"2024-09-21T08:25:10.537637+0000","last_clean":"2024-09-21T08:25:10.537637+0000","last_became_active":"2024-09-21T08:24:56.502937+0000","last_became_peered":"2024-09-21T08:24:56.502937+0000","last_unstale":"2024-09-21T08:25:10.537637+0000","last_undegraded":"2024-09-21T08:25:10.537637+0000","last_fullsized":"2024-09-21T08:25:10.537637+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":140,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543961+0000","last_change":"2024-09-21T08:24:48.629077+0000","last_active":"2024-09-21T08:25:10.543961+0000","last_peered":"2024-09-21T08:25:10.543961+0000","last_clean":"2024-09-21T08:25:10.543961+0000","last_became_active":"2024-09-21T08:24:48.602176+0000","last_became_peered":"2024-09-21T08:24:48.602176+0000","last_unstale":"2024-09-21T08:25:10.543961+0000","last_undegraded":"2024-09-21T08:25:10.543961+0000","last_fullsized":"2024-09-21T08:25:10.543961+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215534+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":123,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537703+0000","last_change":"2024-09-21T08:24:56.505855+0000","last_active":"2024-09-21T08:25:10.537703+0000","last_peered":"2024-09-21T08:25:10.537703+0000","last_clean":"2024-09-21T08:25:10.537703+0000","last_became_active":"2024-09-21T08:24:56.505134+0000","last_became_peered":"2024-09-21T08:24:56.505134+0000","last_unstale":"2024-09-21T08:25:10.537703+0000","last_undegraded":"2024-09-21T08:25:10.537703+0000","last_fullsized":"2024-09-21T08:25:10.537703+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672275+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":56,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.554399+0000","last_change":"2024-09-21T08:24:48.579742+0000","last_active":"2024-09-21T08:25:10.554399+0000","last_peered":"2024-09-21T08:25:10.554399+0000","last_clean":"2024-09-21T08:25:10.554399+0000","last_became_active":"2024-09-21T08:24:48.579439+0000","last_became_peered":"2024-09-21T08:24:48.579439+0000","last_unstale":"2024-09-21T08:25:10.554399+0000","last_undegraded":"2024-09-21T08:25:10.554399+0000","last_fullsized":"2024-09-21T08:25:10.554399+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":114,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.555030+0000","last_change":"2024-09-21T08:24:53.746385+0000","last_active":"2024-09-21T08:25:10.555030+0000","last_peered":"2024-09-21T08:25:10.555030+0000","last_clean":"2024-09-21T08:25:10.555030+0000","last_became_active":"2024-09-21T08:24:38.539845+0000","last_became_peered":"2024-09-21T08:24:38.539845+0000","last_unstale":"2024-09-21T08:25:10.555030+0000","last_undegraded":"2024-09-21T08:25:10.555030+0000","last_fullsized":"2024-09-21T08:25:10.555030+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.746295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.746295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.585736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":157,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561167+0000","last_change":"2024-09-21T08:24:56.529301+0000","last_active":"2024-09-21T08:25:10.561167+0000","last_peered":"2024-09-21T08:25:10.561167+0000","last_clean":"2024-09-21T08:25:10.561167+0000","last_became_active":"2024-09-21T08:24:56.499937+0000","last_became_peered":"2024-09-21T08:24:56.499937+0000","last_unstale":"2024-09-21T08:25:10.561167+0000","last_undegraded":"2024-09-21T08:25:10.561167+0000","last_fullsized":"2024-09-21T08:25:10.561167+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":118,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543196+0000","last_change":"2024-09-21T08:24:48.587676+0000","last_active":"2024-09-21T08:25:10.543196+0000","last_peered":"2024-09-21T08:25:10.543196+0000","last_clean":"2024-09-21T08:25:10.543196+0000","last_became_active":"2024-09-21T08:24:48.587393+0000","last_became_peered":"2024-09-21T08:24:48.587393+0000","last_unstale":"2024-09-21T08:25:10.543196+0000","last_undegraded":"2024-09-21T08:25:10.543196+0000","last_fullsized":"2024-09-21T08:25:10.543196+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"101'10","reported_seq":98,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537691+0000","last_change":"2024-09-21T08:24:54.690664+0000","last_active":"2024-09-21T08:25:10.537691+0000","last_peered":"2024-09-21T08:25:10.537691+0000","last_clean":"2024-09-21T08:25:10.537691+0000","last_became_active":"2024-09-21T08:24:48.590539+0000","last_became_peered":"2024-09-21T08:24:48.590539+0000","last_unstale":"2024-09-21T08:25:10.537691+0000","last_undegraded":"2024-09-21T08:25:10.537691+0000","last_fullsized":"2024-09-21T08:25:10.537691+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:02.692436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":77,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.552086+0000","last_change":"2024-09-21T08:24:38.543166+0000","last_active":"2024-09-21T08:25:10.552086+0000","last_peered":"2024-09-21T08:25:10.552086+0000","last_clean":"2024-09-21T08:25:10.552086+0000","last_became_active":"2024-09-21T08:24:38.542854+0000","last_became_peered":"2024-09-21T08:24:38.542854+0000","last_unstale":"2024-09-21T08:25:10.552086+0000","last_undegraded":"2024-09-21T08:25:10.552086+0000","last_fullsized":"2024-09-21T08:25:10.552086+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"104'15","reported_seq":115,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.552412+0000","last_change":"2024-09-21T08:24:57.624866+0000","last_active":"2024-09-21T08:25:10.552412+0000","last_peered":"2024-09-21T08:25:10.552412+0000","last_clean":"2024-09-21T08:25:10.552412+0000","last_became_active":"2024-09-21T08:24:48.598970+0000","last_became_peered":"2024-09-21T08:24:48.598970+0000","last_unstale":"2024-09-21T08:25:10.552412+0000","last_undegraded":"2024-09-21T08:25:10.552412+0000","last_fullsized":"2024-09-21T08:25:10.552412+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:23.937573+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":145,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537944+0000","last_change":"2024-09-21T08:24:56.709528+0000","last_active":"2024-09-21T08:25:10.537944+0000","last_peered":"2024-09-21T08:25:10.537944+0000","last_clean":"2024-09-21T08:25:10.537944+0000","last_became_active":"2024-09-21T08:24:56.504665+0000","last_became_peered":"2024-09-21T08:24:56.504665+0000","last_unstale":"2024-09-21T08:25:10.537944+0000","last_undegraded":"2024-09-21T08:25:10.537944+0000","last_fullsized":"2024-09-21T08:25:10.537944+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:56.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.709437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.598671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":134,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561041+0000","last_change":"2024-09-21T08:23:34.567042+0000","last_active":"2024-09-21T08:25:10.561041+0000","last_peered":"2024-09-21T08:25:10.561041+0000","last_clean":"2024-09-21T08:25:10.561041+0000","last_became_active":"2024-09-21T08:23:34.566736+0000","last_became_peered":"2024-09-21T08:23:34.566736+0000","last_unstale":"2024-09-21T08:25:10.561041+0000","last_undegraded":"2024-09-21T08:25:10.561041+0000","last_fullsized":"2024-09-21T08:25:10.561041+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.769845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543114+0000","last_change":"2024-09-21T08:24:48.592895+0000","last_active":"2024-09-21T08:25:10.543114+0000","last_peered":"2024-09-21T08:25:10.543114+0000","last_clean":"2024-09-21T08:25:10.543114+0000","last_became_active":"2024-09-21T08:24:48.592280+0000","last_became_peered":"2024-09-21T08:24:48.592280+0000","last_unstale":"2024-09-21T08:25:10.543114+0000","last_undegraded":"2024-09-21T08:25:10.543114+0000","last_fullsized":"2024-09-21T08:25:10.543114+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.562130+0000","last_change":"2024-09-21T08:24:56.539935+0000","last_active":"2024-09-21T08:25:10.562130+0000","last_peered":"2024-09-21T08:25:10.562130+0000","last_clean":"2024-09-21T08:25:10.562130+0000","last_became_active":"2024-09-21T08:24:56.517397+0000","last_became_peered":"2024-09-21T08:24:56.517397+0000","last_unstale":"2024-09-21T08:25:10.562130+0000","last_undegraded":"2024-09-21T08:25:10.562130+0000","last_fullsized":"2024-09-21T08:25:10.562130+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":81,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537814+0000","last_change":"2024-09-21T08:24:56.504037+0000","last_active":"2024-09-21T08:25:10.537814+0000","last_peered":"2024-09-21T08:25:10.537814+0000","last_clean":"2024-09-21T08:25:10.537814+0000","last_became_active":"2024-09-21T08:24:56.503232+0000","last_became_peered":"2024-09-21T08:24:56.503232+0000","last_unstale":"2024-09-21T08:25:10.537814+0000","last_undegraded":"2024-09-21T08:25:10.537814+0000","last_fullsized":"2024-09-21T08:25:10.537814+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672415+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":110,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561660+0000","last_change":"2024-09-21T08:24:48.591631+0000","last_active":"2024-09-21T08:25:10.561660+0000","last_peered":"2024-09-21T08:25:10.561660+0000","last_clean":"2024-09-21T08:25:10.561660+0000","last_became_active":"2024-09-21T08:24:48.591342+0000","last_became_peered":"2024-09-21T08:24:48.591342+0000","last_unstale":"2024-09-21T08:25:10.561660+0000","last_undegraded":"2024-09-21T08:25:10.561660+0000","last_fullsized":"2024-09-21T08:25:10.561660+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":55,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543238+0000","last_change":"2024-09-21T08:24:48.600966+0000","last_active":"2024-09-21T08:25:10.543238+0000","last_peered":"2024-09-21T08:25:10.543238+0000","last_clean":"2024-09-21T08:25:10.543238+0000","last_became_active":"2024-09-21T08:24:48.600045+0000","last_became_peered":"2024-09-21T08:24:48.600045+0000","last_unstale":"2024-09-21T08:25:10.543238+0000","last_undegraded":"2024-09-21T08:25:10.543238+0000","last_fullsized":"2024-09-21T08:25:10.543238+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":140,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561394+0000","last_change":"2024-09-21T08:24:56.573418+0000","last_active":"2024-09-21T08:25:10.561394+0000","last_peered":"2024-09-21T08:25:10.561394+0000","last_clean":"2024-09-21T08:25:10.561394+0000","last_became_active":"2024-09-21T08:24:56.524898+0000","last_became_peered":"2024-09-21T08:24:56.524898+0000","last_unstale":"2024-09-21T08:25:10.561394+0000","last_undegraded":"2024-09-21T08:25:10.561394+0000","last_fullsized":"2024-09-21T08:25:10.561394+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582408+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":129,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.538196+0000","last_change":"2024-09-21T08:24:56.505458+0000","last_active":"2024-09-21T08:25:10.538196+0000","last_peered":"2024-09-21T08:25:10.538196+0000","last_clean":"2024-09-21T08:25:10.538196+0000","last_became_active":"2024-09-21T08:24:56.504584+0000","last_became_peered":"2024-09-21T08:24:56.504584+0000","last_unstale":"2024-09-21T08:25:10.538196+0000","last_undegraded":"2024-09-21T08:25:10.538196+0000","last_fullsized":"2024-09-21T08:25:10.538196+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":55,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543565+0000","last_change":"2024-09-21T08:24:48.591350+0000","last_active":"2024-09-21T08:25:10.543565+0000","last_peered":"2024-09-21T08:25:10.543565+0000","last_clean":"2024-09-21T08:25:10.543565+0000","last_became_active":"2024-09-21T08:24:48.590426+0000","last_became_peered":"2024-09-21T08:24:48.590426+0000","last_unstale":"2024-09-21T08:25:10.543565+0000","last_undegraded":"2024-09-21T08:25:10.543565+0000","last_fullsized":"2024-09-21T08:25:10.543565+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":100,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.542589+0000","last_change":"2024-09-21T08:23:36.580460+0000","last_active":"2024-09-21T08:25:10.542589+0000","last_peered":"2024-09-21T08:25:10.542589+0000","last_clean":"2024-09-21T08:25:10.542589+0000","last_became_active":"2024-09-21T08:23:36.579949+0000","last_became_peered":"2024-09-21T08:23:36.579949+0000","last_unstale":"2024-09-21T08:25:10.542589+0000","last_undegraded":"2024-09-21T08:25:10.542589+0000","last_fullsized":"2024-09-21T08:25:10.542589+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":131,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537091+0000","last_change":"2024-09-21T08:24:56.531655+0000","last_active":"2024-09-21T08:25:10.537091+0000","last_peered":"2024-09-21T08:25:10.537091+0000","last_clean":"2024-09-21T08:25:10.537091+0000","last_became_active":"2024-09-21T08:24:56.503223+0000","last_became_peered":"2024-09-21T08:24:56.503223+0000","last_unstale":"2024-09-21T08:25:10.537091+0000","last_undegraded":"2024-09-21T08:25:10.537091+0000","last_fullsized":"2024-09-21T08:25:10.537091+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.478113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":113,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561368+0000","last_change":"2024-09-21T08:23:30.544860+0000","last_active":"2024-09-21T08:25:10.561368+0000","last_peered":"2024-09-21T08:25:10.561368+0000","last_clean":"2024-09-21T08:25:10.561368+0000","last_became_active":"2024-09-21T08:23:30.544486+0000","last_became_peered":"2024-09-21T08:23:30.544486+0000","last_unstale":"2024-09-21T08:25:10.561368+0000","last_undegraded":"2024-09-21T08:25:10.561368+0000","last_fullsized":"2024-09-21T08:25:10.561368+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.770181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":123,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.536724+0000","last_change":"2024-09-21T08:24:58.688115+0000","last_active":"2024-09-21T08:25:10.536724+0000","last_peered":"2024-09-21T08:25:10.536724+0000","last_clean":"2024-09-21T08:25:10.536724+0000","last_became_active":"2024-09-21T08:24:56.503123+0000","last_became_peered":"2024-09-21T08:24:56.503123+0000","last_unstale":"2024-09-21T08:25:10.536724+0000","last_undegraded":"2024-09-21T08:25:10.536724+0000","last_fullsized":"2024-09-21T08:25:10.536724+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.688046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.688046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.364343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":61,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543913+0000","last_change":"2024-09-21T08:24:56.508632+0000","last_active":"2024-09-21T08:25:10.543913+0000","last_peered":"2024-09-21T08:25:10.543913+0000","last_clean":"2024-09-21T08:25:10.543913+0000","last_became_active":"2024-09-21T08:24:56.508323+0000","last_became_peered":"2024-09-21T08:24:56.508323+0000","last_unstale":"2024-09-21T08:25:10.543913+0000","last_undegraded":"2024-09-21T08:25:10.543913+0000","last_fullsized":"2024-09-21T08:25:10.543913+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":99,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.553729+0000","last_change":"2024-09-21T08:24:56.546097+0000","last_active":"2024-09-21T08:25:10.553729+0000","last_peered":"2024-09-21T08:25:10.553729+0000","last_clean":"2024-09-21T08:25:10.553729+0000","last_became_active":"2024-09-21T08:24:56.517807+0000","last_became_peered":"2024-09-21T08:24:56.517807+0000","last_unstale":"2024-09-21T08:25:10.553729+0000","last_undegraded":"2024-09-21T08:25:10.553729+0000","last_fullsized":"2024-09-21T08:25:10.553729+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582453+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":124,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.545327+0000","last_change":"2024-09-21T08:24:56.517657+0000","last_active":"2024-09-21T08:25:10.545327+0000","last_peered":"2024-09-21T08:25:10.545327+0000","last_clean":"2024-09-21T08:25:10.545327+0000","last_became_active":"2024-09-21T08:24:56.517349+0000","last_became_peered":"2024-09-21T08:24:56.517349+0000","last_unstale":"2024-09-21T08:25:10.545327+0000","last_undegraded":"2024-09-21T08:25:10.545327+0000","last_fullsized":"2024-09-21T08:25:10.545327+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":77,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537640+0000","last_change":"2024-09-21T08:24:56.507802+0000","last_active":"2024-09-21T08:25:10.537640+0000","last_peered":"2024-09-21T08:25:10.537640+0000","last_clean":"2024-09-21T08:25:10.537640+0000","last_became_active":"2024-09-21T08:24:56.506356+0000","last_became_peered":"2024-09-21T08:24:56.506356+0000","last_unstale":"2024-09-21T08:25:10.537640+0000","last_undegraded":"2024-09-21T08:25:10.537640+0000","last_fullsized":"2024-09-21T08:25:10.537640+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:59.651421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":111,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.537127+0000","last_change":"2024-09-21T08:24:48.590701+0000","last_active":"2024-09-21T08:25:10.537127+0000","last_peered":"2024-09-21T08:25:10.537127+0000","last_clean":"2024-09-21T08:25:10.537127+0000","last_became_active":"2024-09-21T08:24:48.590283+0000","last_became_peered":"2024-09-21T08:24:48.590283+0000","last_unstale":"2024-09-21T08:25:10.537127+0000","last_undegraded":"2024-09-21T08:25:10.537127+0000","last_fullsized":"2024-09-21T08:25:10.537127+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:14.672443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":106,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.554481+0000","last_change":"2024-09-21T08:24:38.549206+0000","last_active":"2024-09-21T08:25:10.554481+0000","last_peered":"2024-09-21T08:25:10.554481+0000","last_clean":"2024-09-21T08:25:10.554481+0000","last_became_active":"2024-09-21T08:24:38.547039+0000","last_became_peered":"2024-09-21T08:24:38.547039+0000","last_unstale":"2024-09-21T08:25:10.554481+0000","last_undegraded":"2024-09-21T08:25:10.554481+0000","last_fullsized":"2024-09-21T08:25:10.554481+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:13.582153+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":64,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.542144+0000","last_change":"2024-09-21T08:24:51.848588+0000","last_active":"2024-09-21T08:25:10.542144+0000","last_peered":"2024-09-21T08:25:10.542144+0000","last_clean":"2024-09-21T08:25:10.542144+0000","last_became_active":"2024-09-21T08:24:48.598312+0000","last_became_peered":"2024-09-21T08:24:48.598312+0000","last_unstale":"2024-09-21T08:25:10.542144+0000","last_undegraded":"2024-09-21T08:25:10.542144+0000","last_fullsized":"2024-09-21T08:25:10.542144+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:53.281030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":90,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.561450+0000","last_change":"2024-09-21T08:24:56.568191+0000","last_active":"2024-09-21T08:25:10.561450+0000","last_peered":"2024-09-21T08:25:10.561450+0000","last_clean":"2024-09-21T08:25:10.561450+0000","last_became_active":"2024-09-21T08:24:56.516832+0000","last_became_peered":"2024-09-21T08:24:56.516832+0000","last_unstale":"2024-09-21T08:25:10.561450+0000","last_undegraded":"2024-09-21T08:25:10.561450+0000","last_fullsized":"2024-09-21T08:25:10.561450+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.301088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":104,"reported_epoch":117,"state":"active+clean","last_fresh":"2024-09-21T08:25:10.543595+0000","last_change":"2024-09-21T08:23:34.573167+0000","last_active":"2024-09-21T08:25:10.543595+0000","last_peered":"2024-09-21T08:25:10.543595+0000","last_clean":"2024-09-21T08:25:10.543595+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:25:10.543595+0000","last_undegraded":"2024-09-21T08:25:10.543595+0000","last_fullsized":"2024-09-21T08:25:10.543595+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:15.215007+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":157013746,"num_objects":61,"num_object_clones":15,"num_object_copies":183,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":61,"num_whiteouts":3,"num_read":397,"num_read_kb":279309,"num_write":417,"num_write_kb":153617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":67327959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":390344704,"data_stored":409642777,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":501,"ondisk_log_size":501,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":679,"num_read_kb":436,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":4720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":16970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":402,"ondisk_log_size":402,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":7973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":95,"num_read_kb":90,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070489,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110848,"kb_used_data":79660,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93634304,"statfs":{"total":95995035648,"available":95881527296,"internally_reserved":0,"allocated":81571840,"data_stored":87743022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397535,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659170,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":67792,"kb_used_data":36596,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93677360,"statfs":{"total":95995035648,"available":95925616640,"internally_reserved":0,"allocated":37474304,"data_stored":36549482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986216,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182444,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411378,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102932,"kb_used_data":71744,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93642220,"statfs":{"total":95995035648,"available":95889633280,"internally_reserved":0,"allocated":73465856,"data_stored":75722490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574902,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":93148,"kb_used_data":61960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93652004,"statfs":{"total":95995035648,"available":95899652096,"internally_reserved":0,"allocated":63447040,"data_stored":63707951,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738427,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86672,"kb_used_data":55484,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93658480,"statfs":{"total":95995035648,"available":95906283520,"internally_reserved":0,"allocated":56815616,"data_stored":57854627,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":62734336,"data_stored":64634402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":66457600,"data_stored":70959052,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77090816,"data_stored":83608974,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32387072,"data_stored":32188068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":43806720,"data_stored":43797644,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":79773696,"data_stored":86584222,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:25:13.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pgp_num 1 2024-09-21T08:25:13.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:13.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:13.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:13.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:13.545 INFO:tasks.rados.rados.0.smithi137.stdout:208: done (1 left) 2024-09-21T08:25:13.545 INFO:tasks.rados.rados.0.smithi137.stdout:209: rollback oid 19 current snap is 20 2024-09-21T08:25:13.545 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 19 to 16 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:205: finishing write tid 3 to smithi13779221-5 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:205: finishing write tid 4 to smithi13779221-5 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:205: finishing write tid 5 to smithi13779221-5 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:205: oid 5 updating version 0 to 80 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:205: oid 5 updating version 80 to 82 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:205: oid 5 version 82 is already newer than 81 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:205: oid 5 version 82 is already newer than 79 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 82 (ObjNum 102 snap 20 seq_num 102) dirty exists 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:205: left oid 5 (ObjNum 102 snap 20 seq_num 102) 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:205: done (1 left) 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:210: read oid 40 snap -1 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:210: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-21T08:25:13.546 INFO:tasks.rados.rados.0.smithi137.stdout:211: write (excl) oid 4 current snap is 20 2024-09-21T08:25:13.547 INFO:tasks.rados.rados.0.smithi137.stdout:211: seq_num 104 ranges {547928=538485,1769539=409023,2706697=532054} 2024-09-21T08:25:13.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:13.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:13.548 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:13.552 INFO:tasks.rados.rados.0.smithi137.stdout:211: writing smithi13779221-4 from 547928 to 1086413 tid 1 2024-09-21T08:25:13.559 INFO:tasks.rados.rados.0.smithi137.stdout:211: writing smithi13779221-4 from 1769539 to 2178562 tid 2 2024-09-21T08:25:13.565 INFO:tasks.rados.rados.0.smithi137.stdout:211: writing smithi13779221-4 from 2706697 to 3238751 tid 3 2024-09-21T08:25:13.565 INFO:tasks.rados.rados.0.smithi137.stdout:212: copy_from oid 36 from oid 23 current snap is 20 2024-09-21T08:25:13.565 INFO:tasks.rados.rados.0.smithi137.stdout:209: finishing rollback tid 0 to smithi13779221-19 2024-09-21T08:25:13.565 INFO:tasks.rados.rados.0.smithi137.stdout:209: finishing rollback tid 1 to smithi13779221-19 2024-09-21T08:25:13.565 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 36 (ObjNum 129 snap 0 seq_num 3760534640) dirty dne 2024-09-21T08:25:13.566 INFO:tasks.rados.rados.0.smithi137.stdout:211: finishing write tid 1 to smithi13779221-4 2024-09-21T08:25:13.566 INFO:tasks.rados.rados.0.smithi137.stdout:210: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-21T08:25:13.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:13.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:13.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:13.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:13.675 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:13.719 INFO:tasks.rados.rados.0.smithi137.stdout:211: finishing write tid 2 to smithi13779221-4 2024-09-21T08:25:13.719 INFO:tasks.rados.rados.0.smithi137.stdout:209: done (3 left) 2024-09-21T08:25:13.719 INFO:tasks.rados.rados.0.smithi137.stdout:210: done (2 left) 2024-09-21T08:25:13.719 INFO:tasks.rados.rados.0.smithi137.stdout:213: copy_from oid 38 from oid 28 current snap is 20 2024-09-21T08:25:13.719 INFO:tasks.rados.rados.0.smithi137.stdout:212: finishing copy_from to smithi13779221-36 2024-09-21T08:25:13.719 INFO:tasks.rados.rados.0.smithi137.stdout:212: got expected ENOENT (src dne) 2024-09-21T08:25:13.719 INFO:tasks.rados.rados.0.smithi137.stdout:212: finishing copy_from racing read to smithi13779221-36 2024-09-21T08:25:13.719 INFO:tasks.rados.rados.0.smithi137.stdout:211: finishing write tid 3 to smithi13779221-4 2024-09-21T08:25:13.719 INFO:tasks.rados.rados.0.smithi137.stdout:211: finishing write tid 4 to smithi13779221-4 2024-09-21T08:25:13.720 INFO:tasks.rados.rados.0.smithi137.stdout:211: finishing write tid 5 to smithi13779221-4 2024-09-21T08:25:13.720 INFO:tasks.rados.rados.0.smithi137.stdout:211: oid 4 updating version 0 to 52 2024-09-21T08:25:13.720 INFO:tasks.rados.rados.0.smithi137.stdout:211: oid 4 version 52 is already newer than 49 2024-09-21T08:25:13.720 INFO:tasks.rados.rados.0.smithi137.stdout:211: oid 4 version 52 is already newer than 50 2024-09-21T08:25:13.720 INFO:tasks.rados.rados.0.smithi137.stdout:211: oid 4 version 52 is already newer than 51 2024-09-21T08:25:13.720 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 52 (ObjNum 104 snap 20 seq_num 104) dirty exists 2024-09-21T08:25:13.720 INFO:tasks.rados.rados.0.smithi137.stdout:211: left oid 4 (ObjNum 104 snap 20 seq_num 104) 2024-09-21T08:25:13.720 INFO:tasks.rados.rados.0.smithi137.stdout:211: done (2 left) 2024-09-21T08:25:13.720 INFO:tasks.rados.rados.0.smithi137.stdout:212: done (1 left) 2024-09-21T08:25:13.720 INFO:tasks.rados.rados.0.smithi137.stdout:214: snap_create 2024-09-21T08:25:13.729 INFO:tasks.rados.rados.0.smithi137.stdout:213: finishing copy_from racing read to smithi13779221-38 2024-09-21T08:25:13.731 INFO:tasks.rados.rados.0.smithi137.stdout:213: finishing copy_from to smithi13779221-38 2024-09-21T08:25:13.731 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 6 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-21T08:25:13.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:13.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:13.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:13.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:13.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:25:13.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:13 smithi137 ceph-mon[29634]: from='client.44813 v1:172.21.15.137:0/2327372500' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:25:13.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:13 smithi137 ceph-mon[29634]: osdmap e119: 8 total, 8 up, 5 in 2024-09-21T08:25:13.866 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:13 smithi137 ceph-mon[41044]: from='client.44813 v1:172.21.15.137:0/2327372500' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:25:13.866 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:13 smithi137 ceph-mon[41044]: osdmap e119: 8 total, 8 up, 5 in 2024-09-21T08:25:13.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:13.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:13.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:13.918 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:13.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:13 smithi165 ceph-mon[32893]: from='client.44813 v1:172.21.15.137:0/2327372500' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:25:13.989 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:13 smithi165 ceph-mon[32893]: osdmap e119: 8 total, 8 up, 5 in 2024-09-21T08:25:14.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T08:25:14.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:14.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:14.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:14.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:14.153 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:14.154 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:14.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:14.186 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:14.186 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:14.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:14.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:14.205 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:14.211 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:14.212 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:14.228 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:14.229 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:14.246 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:14.247 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:14.263 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:14.263 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:14.266 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:14.280 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:14.280 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:14.297 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:14.298 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:14.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:14.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:14.329 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:14.329 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:14.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:14.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:14.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:14.453 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:14.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:14.575 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:14.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:14.576 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:14.627 INFO:tasks.rados.rados.0.smithi137.stdout:213: done (1 left) 2024-09-21T08:25:14.627 INFO:tasks.rados.rados.0.smithi137.stdout:214: done (0 left) 2024-09-21T08:25:14.627 INFO:tasks.rados.rados.0.smithi137.stdout:215: write oid 24 current snap is 21 2024-09-21T08:25:14.627 INFO:tasks.rados.rados.0.smithi137.stdout:215: seq_num 105 ranges {564735=613734,1828021=669034,3134120=630615} 2024-09-21T08:25:14.634 INFO:tasks.rados.rados.0.smithi137.stdout:215: writing smithi13779221-24 from 564735 to 1178469 tid 1 2024-09-21T08:25:14.642 INFO:tasks.rados.rados.0.smithi137.stdout:215: writing smithi13779221-24 from 1828021 to 2497055 tid 2 2024-09-21T08:25:14.650 INFO:tasks.rados.rados.0.smithi137.stdout:215: writing smithi13779221-24 from 3134120 to 3764735 tid 3 2024-09-21T08:25:14.650 INFO:tasks.rados.rados.0.smithi137.stdout:216: read oid 34 snap -1 2024-09-21T08:25:14.650 INFO:tasks.rados.rados.0.smithi137.stdout:216: expect (ObjNum 99 snap 19 seq_num 99) 2024-09-21T08:25:14.650 INFO:tasks.rados.rados.0.smithi137.stdout:217: rollback oid 36 current snap is 21 2024-09-21T08:25:14.650 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 36 to 14 2024-09-21T08:25:14.650 INFO:tasks.rados.rados.0.smithi137.stdout:218: snap_create 2024-09-21T08:25:14.650 INFO:tasks.rados.rados.0.smithi137.stdout:215: finishing write tid 1 to smithi13779221-24 2024-09-21T08:25:14.650 INFO:tasks.rados.rados.0.smithi137.stdout:215: finishing write tid 2 to smithi13779221-24 2024-09-21T08:25:14.653 INFO:tasks.rados.rados.0.smithi137.stdout:215: finishing write tid 3 to smithi13779221-24 2024-09-21T08:25:14.653 INFO:tasks.rados.rados.0.smithi137.stdout:215: finishing write tid 4 to smithi13779221-24 2024-09-21T08:25:14.654 INFO:tasks.rados.rados.0.smithi137.stdout:215: finishing write tid 5 to smithi13779221-24 2024-09-21T08:25:14.654 INFO:tasks.rados.rados.0.smithi137.stdout:215: oid 24 updating version 0 to 64 2024-09-21T08:25:14.654 INFO:tasks.rados.rados.0.smithi137.stdout:215: oid 24 version 64 is already newer than 62 2024-09-21T08:25:14.654 INFO:tasks.rados.rados.0.smithi137.stdout:215: oid 24 version 64 is already newer than 63 2024-09-21T08:25:14.654 INFO:tasks.rados.rados.0.smithi137.stdout:215: oid 24 version 64 is already newer than 61 2024-09-21T08:25:14.654 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 64 (ObjNum 105 snap 21 seq_num 105) dirty exists 2024-09-21T08:25:14.654 INFO:tasks.rados.rados.0.smithi137.stdout:215: left oid 24 (ObjNum 105 snap 21 seq_num 105) 2024-09-21T08:25:14.655 INFO:tasks.rados.rados.0.smithi137.stdout:217: finishing rollback tid 0 to smithi13779221-36 2024-09-21T08:25:14.655 INFO:tasks.rados.rados.0.smithi137.stdout:217: finishing rollback tid 1 to smithi13779221-36 2024-09-21T08:25:14.655 INFO:tasks.rados.rados.0.smithi137.stdout:217: finishing rollback tid 2 to smithi13779221-36 2024-09-21T08:25:14.655 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 91 (ObjNum 70 snap 7 seq_num 70) dirty exists 2024-09-21T08:25:14.658 INFO:tasks.rados.rados.0.smithi137.stdout:216: expect (ObjNum 99 snap 19 seq_num 99) 2024-09-21T08:25:14.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:14.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:14.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:14.699 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:14.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:14 smithi165 ceph-mon[32893]: osdmap e120: 8 total, 8 up, 5 in 2024-09-21T08:25:14.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:14 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:14.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:14 smithi165 ceph-mon[32893]: pgmap v85: 148 pgs: 148 active+clean; 150 MiB data, 451 MiB used, 447 GiB / 447 GiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 22 op/s 2024-09-21T08:25:14.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:14 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:14.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:14 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:25:14.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:14 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:25:14.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:14 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:14.733 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:14 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/138531150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:14.736 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:25:14.736 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:25:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[29634]: osdmap e120: 8 total, 8 up, 5 in 2024-09-21T08:25:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[29634]: pgmap v85: 148 pgs: 148 active+clean; 150 MiB data, 451 MiB used, 447 GiB / 447 GiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 22 op/s 2024-09-21T08:25:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:25:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:25:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/138531150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[41044]: osdmap e120: 8 total, 8 up, 5 in 2024-09-21T08:25:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[41044]: pgmap v85: 148 pgs: 148 active+clean; 150 MiB data, 451 MiB used, 447 GiB / 447 GiB avail; 14 MiB/s rd, 4.6 MiB/s wr, 22 op/s 2024-09-21T08:25:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:25:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:25:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:14 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/138531150' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:14.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:14.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:14.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:14.825 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:14.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:14.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:14.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:14.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:15.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:15.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:15.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:15.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:15.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:15.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:15.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:15.195 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:15.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:15.319 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:15.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:15.320 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:15.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T08:25:15.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:15.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:15.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:15.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:15.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:15.570 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:15.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:15.571 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:15.625 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 1 pgp_num to 1 2024-09-21T08:25:15.631 INFO:tasks.rados.rados.0.smithi137.stdout:215: done (3 left) 2024-09-21T08:25:15.631 INFO:tasks.rados.rados.0.smithi137.stdout:216: done (2 left) 2024-09-21T08:25:15.631 INFO:tasks.rados.rados.0.smithi137.stdout:217: done (1 left) 2024-09-21T08:25:15.631 INFO:tasks.rados.rados.0.smithi137.stdout:218: done (0 left) 2024-09-21T08:25:15.631 INFO:tasks.rados.rados.0.smithi137.stdout:219: snap_remove snap 14 2024-09-21T08:25:15.643 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:15.645 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:15.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:15.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:15.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:15.694 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:15 smithi165 ceph-mon[32893]: pgmap v86: 148 pgs: 148 active+clean; 150 MiB data, 451 MiB used, 447 GiB / 447 GiB avail; 16 MiB/s rd, 5.2 MiB/s wr, 25 op/s 2024-09-21T08:25:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:15 smithi165 ceph-mon[32893]: Deploying daemon grafana.a on smithi165 2024-09-21T08:25:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:15 smithi165 ceph-mon[32893]: Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) 2024-09-21T08:25:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:15 smithi165 ceph-mon[32893]: osdmap e121: 8 total, 8 up, 5 in 2024-09-21T08:25:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:15 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:15.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:15 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:15.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:15 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1874545772' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-21T08:25:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[41044]: pgmap v86: 148 pgs: 148 active+clean; 150 MiB data, 451 MiB used, 447 GiB / 447 GiB avail; 16 MiB/s rd, 5.2 MiB/s wr, 25 op/s 2024-09-21T08:25:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[41044]: Deploying daemon grafana.a on smithi165 2024-09-21T08:25:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[41044]: Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) 2024-09-21T08:25:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[41044]: osdmap e121: 8 total, 8 up, 5 in 2024-09-21T08:25:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:15.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:15.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1874545772' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-21T08:25:15.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[29634]: pgmap v86: 148 pgs: 148 active+clean; 150 MiB data, 451 MiB used, 447 GiB / 447 GiB avail; 16 MiB/s rd, 5.2 MiB/s wr, 25 op/s 2024-09-21T08:25:15.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[29634]: Deploying daemon grafana.a on smithi165 2024-09-21T08:25:15.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[29634]: Health check cleared: CEPHADM_DAEMON_PLACE_FAIL (was: Failed to place 1 daemon(s)) 2024-09-21T08:25:15.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[29634]: osdmap e121: 8 total, 8 up, 5 in 2024-09-21T08:25:15.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:15.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:15.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:15 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1874545772' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]: dispatch 2024-09-21T08:25:15.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:15.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:15.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:15.816 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:15.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:15.938 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:15.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:15.939 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:16.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:16.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:16.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:16.062 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:16.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:16.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:16.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:16.185 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:16.206 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:25:16.206 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:25:16.286 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:16.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:16.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:16.308 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:16.409 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:16.430 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:16.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:16.431 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:16.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:16.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:16.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:16.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:16.637 INFO:tasks.rados.rados.0.smithi137.stdout:219: done (0 left) 2024-09-21T08:25:16.637 INFO:tasks.rados.rados.0.smithi137.stdout:220: write oid 20 current snap is 22 2024-09-21T08:25:16.637 INFO:tasks.rados.rados.0.smithi137.stdout:220: seq_num 106 ranges {581542=688983,1886503=412692,2761543=670951} 2024-09-21T08:25:16.643 INFO:tasks.rados.rados.0.smithi137.stdout:220: writing smithi13779221-20 from 581542 to 1270525 tid 1 2024-09-21T08:25:16.649 INFO:tasks.rados.rados.0.smithi137.stdout:220: writing smithi13779221-20 from 1886503 to 2299195 tid 2 2024-09-21T08:25:16.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:16.658 INFO:tasks.rados.rados.0.smithi137.stdout:220: writing smithi13779221-20 from 2761543 to 3432494 tid 3 2024-09-21T08:25:16.658 INFO:tasks.rados.rados.0.smithi137.stdout:221: write oid 32 current snap is 22 2024-09-21T08:25:16.658 INFO:tasks.rados.rados.0.smithi137.stdout:220: finishing write tid 1 to smithi13779221-20 2024-09-21T08:25:16.658 INFO:tasks.rados.rados.0.smithi137.stdout:221: seq_num 107 ranges {598349=480585,1544985=672703,2905319=482223} 2024-09-21T08:25:16.663 INFO:tasks.rados.rados.0.smithi137.stdout:221: writing smithi13779221-32 from 598349 to 1078934 tid 1 2024-09-21T08:25:16.669 INFO:tasks.rados.rados.0.smithi137.stdout:221: writing smithi13779221-32 from 1544985 to 2217688 tid 2 2024-09-21T08:25:16.676 INFO:tasks.rados.rados.0.smithi137.stdout:221: writing smithi13779221-32 from 2905319 to 3387542 tid 3 2024-09-21T08:25:16.676 INFO:tasks.rados.rados.0.smithi137.stdout:222: delete oid 17 current snap is 22 2024-09-21T08:25:16.676 INFO:tasks.rados.rados.0.smithi137.stdout:220: finishing write tid 2 to smithi13779221-20 2024-09-21T08:25:16.676 INFO:tasks.rados.rados.0.smithi137.stdout:220: finishing write tid 3 to smithi13779221-20 2024-09-21T08:25:16.676 INFO:tasks.rados.rados.0.smithi137.stdout:220: finishing write tid 4 to smithi13779221-20 2024-09-21T08:25:16.677 INFO:tasks.rados.rados.0.smithi137.stdout:220: finishing write tid 5 to smithi13779221-20 2024-09-21T08:25:16.677 INFO:tasks.rados.rados.0.smithi137.stdout:220: oid 20 updating version 0 to 101 2024-09-21T08:25:16.677 INFO:tasks.rados.rados.0.smithi137.stdout:220: oid 20 updating version 101 to 102 2024-09-21T08:25:16.677 INFO:tasks.rados.rados.0.smithi137.stdout:220: oid 20 updating version 102 to 103 2024-09-21T08:25:16.677 INFO:tasks.rados.rados.0.smithi137.stdout:220: oid 20 version 103 is already newer than 96 2024-09-21T08:25:16.677 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 103 (ObjNum 106 snap 22 seq_num 106) dirty exists 2024-09-21T08:25:16.677 INFO:tasks.rados.rados.0.smithi137.stdout:220: left oid 20 (ObjNum 106 snap 22 seq_num 106) 2024-09-21T08:25:16.677 INFO:tasks.rados.rados.0.smithi137.stdout:221: finishing write tid 1 to smithi13779221-32 2024-09-21T08:25:16.677 INFO:tasks.rados.rados.0.smithi137.stdout:221: finishing write tid 2 to smithi13779221-32 2024-09-21T08:25:16.678 INFO:tasks.rados.rados.0.smithi137.stdout:220: done (2 left) 2024-09-21T08:25:16.679 INFO:tasks.rados.rados.0.smithi137.stdout:222: done (1 left) 2024-09-21T08:25:16.679 INFO:tasks.rados.rados.0.smithi137.stdout:223: write (excl) oid 29 current snap is 22 2024-09-21T08:25:16.679 INFO:tasks.rados.rados.0.smithi137.stdout:221: finishing write tid 3 to smithi13779221-32 2024-09-21T08:25:16.679 INFO:tasks.rados.rados.0.smithi137.stdout:223: seq_num 109 ranges {631963=631083,2061949=560019,3243812=504767} 2024-09-21T08:25:16.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:16.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:16.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:16.685 INFO:tasks.rados.rados.0.smithi137.stdout:223: writing smithi13779221-29 from 631963 to 1263046 tid 1 2024-09-21T08:25:16.693 INFO:tasks.rados.rados.0.smithi137.stdout:223: writing smithi13779221-29 from 2061949 to 2621968 tid 2 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:223: writing smithi13779221-29 from 3243812 to 3748579 tid 3 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:224: copy_from oid 34 from oid 7 current snap is 22 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:221: finishing write tid 4 to smithi13779221-32 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:225: delete oid 3 current snap is 22 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:221: finishing write tid 5 to smithi13779221-32 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:221: oid 32 updating version 0 to 18 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:221: oid 32 updating version 18 to 20 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:221: oid 32 version 20 is already newer than 19 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:221: oid 32 updating version 20 to 21 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 21 (ObjNum 107 snap 22 seq_num 107) dirty exists 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:221: left oid 32 (ObjNum 107 snap 22 seq_num 107) 2024-09-21T08:25:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:223: finishing write tid 1 to smithi13779221-29 2024-09-21T08:25:16.701 INFO:tasks.rados.rados.0.smithi137.stdout:223: finishing write tid 2 to smithi13779221-29 2024-09-21T08:25:16.701 INFO:tasks.rados.rados.0.smithi137.stdout:221: done (3 left) 2024-09-21T08:25:16.701 INFO:tasks.rados.rados.0.smithi137.stdout:225: done (2 left) 2024-09-21T08:25:16.701 INFO:tasks.rados.rados.0.smithi137.stdout:226: snap_remove snap 16 2024-09-21T08:25:16.709 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:16 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1874545772' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T08:25:16.709 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:16 smithi137 ceph-mon[29634]: osdmap e122: 8 total, 8 up, 5 in 2024-09-21T08:25:16.709 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:16 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:25:16.709 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:16 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:25:16.709 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:16 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1874545772' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T08:25:16.709 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:16 smithi137 ceph-mon[41044]: osdmap e122: 8 total, 8 up, 5 in 2024-09-21T08:25:16.709 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:16 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:25:16.709 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:16 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:25:16.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:16.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:16.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:16.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:16.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:16.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:16.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:16.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:16 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1874545772' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num", "val": "1"}]': finished 2024-09-21T08:25:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:16 smithi165 ceph-mon[32893]: osdmap e122: 8 total, 8 up, 5 in 2024-09-21T08:25:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:16 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:25:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:16 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:25:17.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:17.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:17.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:17.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:17.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:17.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T08:25:17.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:17.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:17.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:17.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:17.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:17.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:17.299 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:17.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:17.425 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:17.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:17.426 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:17.498 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:17.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:17.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:17.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:17.556 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:17.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:25:17.635 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:226: done (2 left) 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:227: rollback oid 38 current snap is 22 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 38 to 20 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:223: finishing write tid 3 to smithi13779221-29 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:223: finishing write tid 4 to smithi13779221-29 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:223: finishing write tid 5 to smithi13779221-29 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:223: oid 29 updating version 0 to 96 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:223: oid 29 version 96 is already newer than 95 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:223: oid 29 version 96 is already newer than 94 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:223: oid 29 version 96 is already newer than 93 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 96 (ObjNum 109 snap 22 seq_num 109) dirty exists 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:223: left oid 29 (ObjNum 109 snap 22 seq_num 109) 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:224: finishing copy_from racing read to smithi13779221-34 2024-09-21T08:25:17.645 INFO:tasks.rados.rados.0.smithi137.stdout:224: finishing copy_from to smithi13779221-34 2024-09-21T08:25:17.646 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 40 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-21T08:25:17.646 INFO:tasks.rados.rados.0.smithi137.stdout:223: done (2 left) 2024-09-21T08:25:17.646 INFO:tasks.rados.rados.0.smithi137.stdout:224: done (1 left) 2024-09-21T08:25:17.646 INFO:tasks.rados.rados.0.smithi137.stdout:228: snap_remove snap 17 2024-09-21T08:25:17.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:17.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:17.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:17.681 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:17.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:17.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:17.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:17.803 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:17.883 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:17 smithi165 ceph-mon[32893]: pgmap v89: 148 pgs: 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 141 active+clean; 170 MiB data, 521 MiB used, 447 GiB / 447 GiB avail; 6.1 MiB/s rd, 4.4 MiB/s wr, 20 op/s 2024-09-21T08:25:17.883 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:17 smithi165 ceph-mon[32893]: osdmap e123: 8 total, 8 up, 5 in 2024-09-21T08:25:17.883 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:17 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2705041547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:25:17.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:17.931 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:17.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:17.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:17.964 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:17 smithi137 ceph-mon[29634]: pgmap v89: 148 pgs: 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 141 active+clean; 170 MiB data, 521 MiB used, 447 GiB / 447 GiB avail; 6.1 MiB/s rd, 4.4 MiB/s wr, 20 op/s 2024-09-21T08:25:17.964 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:17 smithi137 ceph-mon[29634]: osdmap e123: 8 total, 8 up, 5 in 2024-09-21T08:25:17.964 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:17 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2705041547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:25:17.964 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:17 smithi137 ceph-mon[41044]: pgmap v89: 148 pgs: 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 141 active+clean; 170 MiB data, 521 MiB used, 447 GiB / 447 GiB avail; 6.1 MiB/s rd, 4.4 MiB/s wr, 20 op/s 2024-09-21T08:25:17.964 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:17 smithi137 ceph-mon[41044]: osdmap e123: 8 total, 8 up, 5 in 2024-09-21T08:25:17.964 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:17 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2705041547' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:25:18.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:18.057 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:18.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:18.058 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:18.159 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:18.180 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:18.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:18.181 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:18.224 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:25:18.224 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:25:18.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:18.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:18.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:18.305 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:18.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:18.428 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:18.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:18.429 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:18.445 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:18.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:18.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:18.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:18.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:18.646 INFO:tasks.rados.rados.0.smithi137.stdout:228: done (1 left) 2024-09-21T08:25:18.646 INFO:tasks.rados.rados.0.smithi137.stdout:229: write oid 12 current snap is 22 2024-09-21T08:25:18.646 INFO:tasks.rados.rados.0.smithi137.stdout:227: finishing rollback tid 0 to smithi13779221-38 2024-09-21T08:25:18.646 INFO:tasks.rados.rados.0.smithi137.stdout:229: seq_num 111 ranges {665577=781581,2062560=563688,3182305=643664} 2024-09-21T08:25:18.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:18.653 INFO:tasks.rados.rados.0.smithi137.stdout:229: writing smithi13779221-12 from 665577 to 1447158 tid 1 2024-09-21T08:25:18.661 INFO:tasks.rados.rados.0.smithi137.stdout:229: writing smithi13779221-12 from 2062560 to 2626248 tid 2 2024-09-21T08:25:18.668 INFO:tasks.rados.rados.0.smithi137.stdout:229: writing smithi13779221-12 from 3182305 to 3825969 tid 3 2024-09-21T08:25:18.668 INFO:tasks.rados.rados.0.smithi137.stdout:230: write (excl) oid 33 current snap is 22 2024-09-21T08:25:18.668 INFO:tasks.rados.rados.0.smithi137.stdout:227: finishing rollback tid 1 to smithi13779221-38 2024-09-21T08:25:18.668 INFO:tasks.rados.rados.0.smithi137.stdout:230: seq_num 112 ranges {682384=456830,1721042=423699,2809728=454936,3731357=151027} 2024-09-21T08:25:18.674 INFO:tasks.rados.rados.0.smithi137.stdout:230: writing smithi13779221-33 from 682384 to 1139214 tid 1 2024-09-21T08:25:18.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:18.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:18.679 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:18.679 INFO:tasks.rados.rados.0.smithi137.stdout:230: writing smithi13779221-33 from 1721042 to 2144741 tid 2 2024-09-21T08:25:18.686 INFO:tasks.rados.rados.0.smithi137.stdout:230: writing smithi13779221-33 from 2809728 to 3264664 tid 3 2024-09-21T08:25:18.689 INFO:tasks.rados.rados.0.smithi137.stdout:230: writing smithi13779221-33 from 3731357 to 3882384 tid 4 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:231: snap_create 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:227: finishing rollback tid 2 to smithi13779221-38 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 100 (ObjNum 40 snap 0 seq_num 40) dirty exists 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:229: finishing write tid 1 to smithi13779221-12 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:229: finishing write tid 2 to smithi13779221-12 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:229: finishing write tid 3 to smithi13779221-12 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:229: finishing write tid 4 to smithi13779221-12 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:229: finishing write tid 5 to smithi13779221-12 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:229: oid 12 updating version 0 to 95 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:229: oid 12 updating version 95 to 96 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:229: oid 12 version 96 is already newer than 94 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:229: oid 12 updating version 96 to 97 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 97 (ObjNum 111 snap 22 seq_num 111) dirty exists 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:229: left oid 12 (ObjNum 111 snap 22 seq_num 111) 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:230: finishing write tid 1 to smithi13779221-33 2024-09-21T08:25:18.690 INFO:tasks.rados.rados.0.smithi137.stdout:230: finishing write tid 2 to smithi13779221-33 2024-09-21T08:25:18.691 INFO:tasks.rados.rados.0.smithi137.stdout:230: finishing write tid 3 to smithi13779221-33 2024-09-21T08:25:18.691 INFO:tasks.rados.rados.0.smithi137.stdout:230: finishing write tid 4 to smithi13779221-33 2024-09-21T08:25:18.691 INFO:tasks.rados.rados.0.smithi137.stdout:230: finishing write tid 5 to smithi13779221-33 2024-09-21T08:25:18.691 INFO:tasks.rados.rados.0.smithi137.stdout:230: finishing write tid 6 to smithi13779221-33 2024-09-21T08:25:18.691 INFO:tasks.rados.rados.0.smithi137.stdout:230: oid 33 updating version 0 to 23 2024-09-21T08:25:18.691 INFO:tasks.rados.rados.0.smithi137.stdout:230: oid 33 version 23 is already newer than 20 2024-09-21T08:25:18.692 INFO:tasks.rados.rados.0.smithi137.stdout:230: oid 33 version 23 is already newer than 21 2024-09-21T08:25:18.692 INFO:tasks.rados.rados.0.smithi137.stdout:230: oid 33 version 23 is already newer than 22 2024-09-21T08:25:18.692 INFO:tasks.rados.rados.0.smithi137.stdout:230: oid 33 updating version 23 to 24 2024-09-21T08:25:18.692 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 24 (ObjNum 112 snap 22 seq_num 112) dirty exists 2024-09-21T08:25:18.692 INFO:tasks.rados.rados.0.smithi137.stdout:230: left oid 33 (ObjNum 112 snap 22 seq_num 112) 2024-09-21T08:25:18.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:18.800 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:18.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:18.801 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:18.868 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:18 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:18.868 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:18 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2705041547' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:25:18.869 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:18 smithi137 ceph-mon[29634]: osdmap e124: 8 total, 8 up, 5 in 2024-09-21T08:25:18.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:18 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:18.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:18 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2705041547' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:25:18.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:18 smithi137 ceph-mon[41044]: osdmap e124: 8 total, 8 up, 5 in 2024-09-21T08:25:18.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:18.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:18.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:18.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:18.968 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:18.969 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:18.984 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:18.985 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:19.000 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:19.000 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:19.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:18 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:19.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:18 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2705041547' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:25:19.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:18 smithi165 ceph-mon[32893]: osdmap e124: 8 total, 8 up, 5 in 2024-09-21T08:25:19.016 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:19.016 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:19.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:19.032 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:19.033 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:19.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:19.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:19.047 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:19.049 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:19.049 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:19.065 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:19.066 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:19.082 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:19.082 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:19.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T08:25:19.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:19.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:19.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:19.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:19.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:19.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:19.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:19.309 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:19.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:19.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:19.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:19.432 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:19.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:19.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:19.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:19.557 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:19.654 INFO:tasks.rados.rados.0.smithi137.stdout:227: done (3 left) 2024-09-21T08:25:19.654 INFO:tasks.rados.rados.0.smithi137.stdout:229: done (2 left) 2024-09-21T08:25:19.654 INFO:tasks.rados.rados.0.smithi137.stdout:230: done (1 left) 2024-09-21T08:25:19.655 INFO:tasks.rados.rados.0.smithi137.stdout:231: done (0 left) 2024-09-21T08:25:19.655 INFO:tasks.rados.rados.0.smithi137.stdout:232: write oid 18 current snap is 23 2024-09-21T08:25:19.655 INFO:tasks.rados.rados.0.smithi137.stdout:232: seq_num 113 ranges {699191=532079,1779524=567357,2837151=666208} 2024-09-21T08:25:19.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:19.661 INFO:tasks.rados.rados.0.smithi137.stdout:232: writing smithi13779221-18 from 699191 to 1231270 tid 1 2024-09-21T08:25:19.668 INFO:tasks.rados.rados.0.smithi137.stdout:232: writing smithi13779221-18 from 1779524 to 2346881 tid 2 2024-09-21T08:25:19.675 INFO:tasks.rados.rados.0.smithi137.stdout:232: writing smithi13779221-18 from 2837151 to 3503359 tid 3 2024-09-21T08:25:19.675 INFO:tasks.rados.rados.0.smithi137.stdout:233: read oid 27 snap -1 2024-09-21T08:25:19.675 INFO:tasks.rados.rados.0.smithi137.stdout:233: expect (ObjNum 100 snap 20 seq_num 100) 2024-09-21T08:25:19.676 INFO:tasks.rados.rados.0.smithi137.stdout:234: snap_remove snap 22 2024-09-21T08:25:19.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:19.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:19.682 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:19.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:19 smithi137 ceph-mon[29634]: pgmap v92: 148 pgs: 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 141 active+clean; 170 MiB data, 521 MiB used, 447 GiB / 447 GiB avail; 5.4 MiB/s rd, 3.8 MiB/s wr, 17 op/s 2024-09-21T08:25:19.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:19 smithi137 ceph-mon[29634]: 5.15 scrub starts 2024-09-21T08:25:19.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:19 smithi137 ceph-mon[29634]: 5.15 scrub ok 2024-09-21T08:25:19.766 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:19 smithi137 ceph-mon[29634]: osdmap e125: 8 total, 8 up, 5 in 2024-09-21T08:25:19.766 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:19 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/452297200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:19.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:19 smithi137 ceph-mon[41044]: pgmap v92: 148 pgs: 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 141 active+clean; 170 MiB data, 521 MiB used, 447 GiB / 447 GiB avail; 5.4 MiB/s rd, 3.8 MiB/s wr, 17 op/s 2024-09-21T08:25:19.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:19 smithi137 ceph-mon[41044]: 5.15 scrub starts 2024-09-21T08:25:19.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:19 smithi137 ceph-mon[41044]: 5.15 scrub ok 2024-09-21T08:25:19.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:19 smithi137 ceph-mon[41044]: osdmap e125: 8 total, 8 up, 5 in 2024-09-21T08:25:19.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:19 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/452297200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:19.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:19.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:19.807 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:19.807 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:19.908 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:19.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:19.930 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:19.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:19 smithi165 ceph-mon[32893]: pgmap v92: 148 pgs: 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 141 active+clean; 170 MiB data, 521 MiB used, 447 GiB / 447 GiB avail; 5.4 MiB/s rd, 3.8 MiB/s wr, 17 op/s 2024-09-21T08:25:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:19 smithi165 ceph-mon[32893]: 5.15 scrub starts 2024-09-21T08:25:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:19 smithi165 ceph-mon[32893]: 5.15 scrub ok 2024-09-21T08:25:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:19 smithi165 ceph-mon[32893]: osdmap e125: 8 total, 8 up, 5 in 2024-09-21T08:25:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:19 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/452297200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:20.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:20.057 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:20.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:20.058 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:20.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:20.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:20.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:20.186 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:20.287 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:20.312 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:20.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:20.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:20.357 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:20.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:25:20.414 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:20.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:20.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:20.437 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:20.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:20.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:20.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:20.562 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:20.651 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:20.659 INFO:tasks.rados.rados.0.smithi137.stdout:234: done (2 left) 2024-09-21T08:25:20.659 INFO:tasks.rados.rados.0.smithi137.stdout:235: write (excl) oid 29 current snap is 23 2024-09-21T08:25:20.659 INFO:tasks.rados.rados.0.smithi137.stdout:232: finishing write tid 1 to smithi13779221-18 2024-09-21T08:25:20.659 INFO:tasks.rados.rados.0.smithi137.stdout:235: seq_num 114 ranges {715998=607328,1838006=427368,2864574=477480,3863835=52163} 2024-09-21T08:25:20.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:20.666 INFO:tasks.rados.rados.0.smithi137.stdout:235: writing smithi13779221-29 from 715998 to 1323326 tid 1 2024-09-21T08:25:20.672 INFO:tasks.rados.rados.0.smithi137.stdout:235: writing smithi13779221-29 from 1838006 to 2265374 tid 2 2024-09-21T08:25:20.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:20 smithi137 ceph-mon[29634]: osdmap e126: 8 total, 8 up, 5 in 2024-09-21T08:25:20.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:20 smithi137 ceph-mon[29634]: pgmap v95: 148 pgs: 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 141 active+clean; 170 MiB data, 521 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:20.679 INFO:tasks.rados.rados.0.smithi137.stdout:235: writing smithi13779221-29 from 2864574 to 3342054 tid 3 2024-09-21T08:25:20.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:20 smithi137 ceph-mon[41044]: osdmap e126: 8 total, 8 up, 5 in 2024-09-21T08:25:20.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:20 smithi137 ceph-mon[41044]: pgmap v95: 148 pgs: 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 141 active+clean; 170 MiB data, 521 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:235: writing smithi13779221-29 from 3863835 to 3915998 tid 4 2024-09-21T08:25:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:236: write (excl) oid 41 current snap is 23 2024-09-21T08:25:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:232: finishing write tid 2 to smithi13779221-18 2024-09-21T08:25:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:232: finishing write tid 3 to smithi13779221-18 2024-09-21T08:25:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:232: finishing write tid 4 to smithi13779221-18 2024-09-21T08:25:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:232: finishing write tid 5 to smithi13779221-18 2024-09-21T08:25:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:232: oid 18 updating version 0 to 24 2024-09-21T08:25:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:232: oid 18 updating version 24 to 27 2024-09-21T08:25:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:232: oid 18 version 27 is already newer than 25 2024-09-21T08:25:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:232: oid 18 version 27 is already newer than 26 2024-09-21T08:25:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 27 (ObjNum 113 snap 23 seq_num 113) dirty exists 2024-09-21T08:25:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:232: left oid 18 (ObjNum 113 snap 23 seq_num 113) 2024-09-21T08:25:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:236: seq_num 115 ranges {732805=798930,2296488=571026,3291997=405105} 2024-09-21T08:25:20.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:20.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:20.691 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:20.691 INFO:tasks.rados.rados.0.smithi137.stdout:236: writing smithi13779221-41 from 732805 to 1531735 tid 1 2024-09-21T08:25:20.699 INFO:tasks.rados.rados.0.smithi137.stdout:236: writing smithi13779221-41 from 2296488 to 2867514 tid 2 2024-09-21T08:25:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:236: writing smithi13779221-41 from 3291997 to 3697102 tid 3 2024-09-21T08:25:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:232: done (3 left) 2024-09-21T08:25:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:237: read oid 13 snap -1 2024-09-21T08:25:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:237: expect (ObjNum 63 snap 4 seq_num 63) 2024-09-21T08:25:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:238: read oid 5 snap -1 2024-09-21T08:25:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:238: expect (ObjNum 102 snap 20 seq_num 102) 2024-09-21T08:25:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:239: snap_remove snap 15 2024-09-21T08:25:20.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:20.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:20.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:20.818 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:20.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:20.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:20.940 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:25:20 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:25:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:25:20.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:20.941 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:20 smithi165 ceph-mon[32893]: osdmap e126: 8 total, 8 up, 5 in 2024-09-21T08:25:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:20 smithi165 ceph-mon[32893]: pgmap v95: 148 pgs: 5 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 141 active+clean; 170 MiB data, 521 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:21.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:21.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:21.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:21.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:21.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:21.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:21.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:21.187 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:21.225 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 7] out_osds: [4, 3, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:25:21.225 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:25:21.226 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T08:25:21.226 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T08:25:21.226 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T08:25:21.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T08:25:21.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:21.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:21.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:21.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:21.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:21.441 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:21.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:21.442 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:21.489 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:25:21.493 INFO:tasks.rados.rados.0.smithi137.stdout:239: done (5 left) 2024-09-21T08:25:21.493 INFO:tasks.rados.rados.0.smithi137.stdout:240: delete oid 28 current snap is 23 2024-09-21T08:25:21.493 INFO:tasks.rados.rados.0.smithi137.stdout:233: expect (ObjNum 100 snap 20 seq_num 100) 2024-09-21T08:25:21.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:21.564 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:21.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:21.565 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:21.666 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:21.687 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:21.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:21.688 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:21.710 INFO:tasks.rados.rados.0.smithi137.stdout:235: finishing write tid 1 to smithi13779221-29 2024-09-21T08:25:21.710 INFO:tasks.rados.rados.0.smithi137.stdout:235: finishing write tid 2 to smithi13779221-29 2024-09-21T08:25:21.710 INFO:tasks.rados.rados.0.smithi137.stdout:235: finishing write tid 3 to smithi13779221-29 2024-09-21T08:25:21.710 INFO:tasks.rados.rados.0.smithi137.stdout:235: finishing write tid 4 to smithi13779221-29 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:235: finishing write tid 5 to smithi13779221-29 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:235: finishing write tid 6 to smithi13779221-29 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:235: oid 29 updating version 0 to 107 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:235: oid 29 updating version 107 to 111 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:235: oid 29 version 111 is already newer than 108 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:235: oid 29 version 111 is already newer than 110 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:235: oid 29 version 111 is already newer than 109 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 111 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:235: left oid 29 (ObjNum 114 snap 23 seq_num 114) 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:236: finishing write tid 1 to smithi13779221-41 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:236: finishing write tid 2 to smithi13779221-41 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:236: finishing write tid 3 to smithi13779221-41 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:236: finishing write tid 4 to smithi13779221-41 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:233: done (5 left) 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:235: done (4 left) 2024-09-21T08:25:21.711 INFO:tasks.rados.rados.0.smithi137.stdout:240: done (3 left) 2024-09-21T08:25:21.712 INFO:tasks.rados.rados.0.smithi137.stdout:241: snap_remove snap 19 2024-09-21T08:25:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[29634]: 3.1e scrub starts 2024-09-21T08:25:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[29634]: 3.1e scrub ok 2024-09-21T08:25:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[29634]: osdmap e127: 8 total, 8 up, 5 in 2024-09-21T08:25:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1739972935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:25:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:25:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1739972935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[29634]: osdmap e128: 8 total, 8 up, 5 in 2024-09-21T08:25:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[41044]: 3.1e scrub starts 2024-09-21T08:25:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[41044]: 3.1e scrub ok 2024-09-21T08:25:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[41044]: osdmap e127: 8 total, 8 up, 5 in 2024-09-21T08:25:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1739972935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:25:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:25:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1739972935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[41044]: osdmap e128: 8 total, 8 up, 5 in 2024-09-21T08:25:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:21 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:21.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:21.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:25:21.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:21.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:21.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:21.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:21.937 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:21.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:21.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:22.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:25:21 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:25:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:21 smithi165 ceph-mon[32893]: 3.1e scrub starts 2024-09-21T08:25:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:21 smithi165 ceph-mon[32893]: 3.1e scrub ok 2024-09-21T08:25:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:21 smithi165 ceph-mon[32893]: osdmap e127: 8 total, 8 up, 5 in 2024-09-21T08:25:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:21 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1739972935' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:21 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:25:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:21 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:25:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:21 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1739972935' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:21 smithi165 ceph-mon[32893]: osdmap e128: 8 total, 8 up, 5 in 2024-09-21T08:25:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:21 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:22.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:22.060 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:22.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:22.061 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:22.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:22.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:22.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:22.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:22.277 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:22.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:22.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:22.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:22.309 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:22.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:22.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:22.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:22.432 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:22.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:22.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:22.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:22.555 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:22.656 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:22.666 INFO:tasks.rados.rados.0.smithi137.stdout:241: done (3 left) 2024-09-21T08:25:22.667 INFO:tasks.rados.rados.0.smithi137.stdout:242: write oid 3 current snap is 23 2024-09-21T08:25:22.667 INFO:tasks.rados.rados.0.smithi137.stdout:242: seq_num 117 ranges {766419=549428,2013452=574695,3346843=427649} 2024-09-21T08:25:22.673 INFO:tasks.rados.rados.0.smithi137.stdout:242: writing smithi13779221-3 from 766419 to 1315847 tid 1 2024-09-21T08:25:22.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:22.681 INFO:tasks.rados.rados.0.smithi137.stdout:242: writing smithi13779221-3 from 2013452 to 2588147 tid 2 2024-09-21T08:25:22.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:22.682 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:22.688 INFO:tasks.rados.rados.0.smithi137.stdout:242: writing smithi13779221-3 from 3346843 to 3774492 tid 3 2024-09-21T08:25:22.688 INFO:tasks.rados.rados.0.smithi137.stdout:243: snap_remove snap 20 2024-09-21T08:25:22.761 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:22.762 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:22.762 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:22.762 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:22.775 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:22.775 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 4.19 scrub starts 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 4.19 scrub ok 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 4.1d scrub starts 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 4.1d scrub ok 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 6.3 scrub starts 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 6.3 scrub ok 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 5.e scrub starts 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 5.e scrub ok 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: pgmap v98: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 186 MiB data, 559 MiB used, 446 GiB / 447 GiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 21 op/s 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 4.b scrub starts 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 4.b scrub ok 2024-09-21T08:25:22.776 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 6.1 deep-scrub starts 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[29634]: 6.1 deep-scrub ok 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 4.19 scrub starts 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 4.19 scrub ok 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 4.1d scrub starts 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 4.1d scrub ok 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 6.3 scrub starts 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 6.3 scrub ok 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 5.e scrub starts 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 5.e scrub ok 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: pgmap v98: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 186 MiB data, 559 MiB used, 446 GiB / 447 GiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 21 op/s 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 4.b scrub starts 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 4.b scrub ok 2024-09-21T08:25:22.777 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 6.1 deep-scrub starts 2024-09-21T08:25:22.778 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:22 smithi137 ceph-mon[41044]: 6.1 deep-scrub ok 2024-09-21T08:25:22.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:22.791 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:22.791 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:22.804 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:22.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:22.805 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:22.809 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:22.809 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:22.825 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:22.825 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:22.841 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:22.841 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:22.859 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:22.859 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:22.896 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 4.19 scrub starts 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 4.19 scrub ok 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 4.1d scrub starts 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 4.1d scrub ok 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 6.3 scrub starts 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 6.3 scrub ok 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 5.e scrub starts 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 5.e scrub ok 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: pgmap v98: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 186 MiB data, 559 MiB used, 446 GiB / 447 GiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 21 op/s 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 4.b scrub starts 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 4.b scrub ok 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 6.1 deep-scrub starts 2024-09-21T08:25:22.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:22 smithi165 ceph-mon[32893]: 6.1 deep-scrub ok 2024-09-21T08:25:22.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:22.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:22.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:23.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:23.045 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:25:23.046 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:25:23.050 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:23.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:23.051 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:23.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:23.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:23.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:23.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:23.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:23.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:23.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:23.296 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:23.397 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:23.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:23.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:23.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:23.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:23.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:23.546 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:23.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:23.672 INFO:tasks.rados.rados.0.smithi137.stdout:243: done (4 left) 2024-09-21T08:25:23.672 INFO:tasks.rados.rados.0.smithi137.stdout:244: delete oid 48 current snap is 23 2024-09-21T08:25:23.672 INFO:tasks.rados.rados.0.smithi137.stdout:236: finishing write tid 5 to smithi13779221-41 2024-09-21T08:25:23.672 INFO:tasks.rados.rados.0.smithi137.stdout:236: oid 41 updating version 0 to 46 2024-09-21T08:25:23.672 INFO:tasks.rados.rados.0.smithi137.stdout:236: oid 41 version 46 is already newer than 44 2024-09-21T08:25:23.672 INFO:tasks.rados.rados.0.smithi137.stdout:236: oid 41 version 46 is already newer than 45 2024-09-21T08:25:23.672 INFO:tasks.rados.rados.0.smithi137.stdout:236: oid 41 version 46 is already newer than 43 2024-09-21T08:25:23.672 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 46 (ObjNum 115 snap 23 seq_num 115) dirty exists 2024-09-21T08:25:23.672 INFO:tasks.rados.rados.0.smithi137.stdout:236: left oid 41 (ObjNum 115 snap 23 seq_num 115) 2024-09-21T08:25:23.672 INFO:tasks.rados.rados.0.smithi137.stdout:237: expect (ObjNum 63 snap 4 seq_num 63) 2024-09-21T08:25:23.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:23.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:23.676 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:23.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:23.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:23.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:23.800 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:23.806 INFO:tasks.rados.rados.0.smithi137.stdout:238: expect (ObjNum 102 snap 20 seq_num 102) 2024-09-21T08:25:23.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:23.922 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:23.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:23.923 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:23.962 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:23 smithi165 ceph-mon[32893]: 5.1b scrub starts 2024-09-21T08:25:23.963 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:23 smithi165 ceph-mon[32893]: 5.1b scrub ok 2024-09-21T08:25:23.963 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:23 smithi165 ceph-mon[32893]: osdmap e129: 8 total, 8 up, 5 in 2024-09-21T08:25:23.963 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:23 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3887490474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:23.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T08:25:24.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:23 smithi137 ceph-mon[29634]: 5.1b scrub starts 2024-09-21T08:25:24.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:23 smithi137 ceph-mon[29634]: 5.1b scrub ok 2024-09-21T08:25:24.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:23 smithi137 ceph-mon[29634]: osdmap e129: 8 total, 8 up, 5 in 2024-09-21T08:25:24.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:23 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3887490474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:24.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:23 smithi137 ceph-mon[41044]: 5.1b scrub starts 2024-09-21T08:25:24.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:23 smithi137 ceph-mon[41044]: 5.1b scrub ok 2024-09-21T08:25:24.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:23 smithi137 ceph-mon[41044]: osdmap e129: 8 total, 8 up, 5 in 2024-09-21T08:25:24.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:23 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3887490474' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:24.021 INFO:tasks.rados.rados.0.smithi137.stdout:242: finishing write tid 1 to smithi13779221-3 2024-09-21T08:25:24.021 INFO:tasks.rados.rados.0.smithi137.stdout:242: finishing write tid 2 to smithi13779221-3 2024-09-21T08:25:24.021 INFO:tasks.rados.rados.0.smithi137.stdout:242: finishing write tid 3 to smithi13779221-3 2024-09-21T08:25:24.021 INFO:tasks.rados.rados.0.smithi137.stdout:242: finishing write tid 4 to smithi13779221-3 2024-09-21T08:25:24.021 INFO:tasks.rados.rados.0.smithi137.stdout:242: finishing write tid 5 to smithi13779221-3 2024-09-21T08:25:24.022 INFO:tasks.rados.rados.0.smithi137.stdout:242: oid 3 updating version 0 to 40 2024-09-21T08:25:24.022 INFO:tasks.rados.rados.0.smithi137.stdout:242: oid 3 updating version 40 to 41 2024-09-21T08:25:24.022 INFO:tasks.rados.rados.0.smithi137.stdout:242: oid 3 version 41 is already newer than 39 2024-09-21T08:25:24.022 INFO:tasks.rados.rados.0.smithi137.stdout:242: oid 3 version 41 is already newer than 38 2024-09-21T08:25:24.023 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 41 (ObjNum 117 snap 23 seq_num 117) dirty exists 2024-09-21T08:25:24.023 INFO:tasks.rados.rados.0.smithi137.stdout:242: left oid 3 (ObjNum 117 snap 23 seq_num 117) 2024-09-21T08:25:24.023 INFO:tasks.rados.rados.0.smithi137.stdout:236: done (4 left) 2024-09-21T08:25:24.023 INFO:tasks.rados.rados.0.smithi137.stdout:237: done (3 left) 2024-09-21T08:25:24.023 INFO:tasks.rados.rados.0.smithi137.stdout:238: done (2 left) 2024-09-21T08:25:24.023 INFO:tasks.rados.rados.0.smithi137.stdout:242: done (1 left) 2024-09-21T08:25:24.023 INFO:tasks.rados.rados.0.smithi137.stdout:244: done (0 left) 2024-09-21T08:25:24.024 INFO:tasks.rados.rados.0.smithi137.stdout:245: delete oid 20 current snap is 23 2024-09-21T08:25:24.024 INFO:tasks.rados.rados.0.smithi137.stdout:245: done (0 left) 2024-09-21T08:25:24.024 INFO:tasks.rados.rados.0.smithi137.stdout:246: copy_from oid 48 from oid 47 current snap is 23 2024-09-21T08:25:24.024 INFO:tasks.rados.rados.0.smithi137.stdout:247: copy_from oid 34 from oid 25 current snap is 23 2024-09-21T08:25:24.024 INFO:tasks.rados.rados.0.smithi137.stdout:248: copy_from oid 19 from oid 10 current snap is 23 2024-09-21T08:25:24.025 INFO:tasks.rados.rados.0.smithi137.stdout:249: read oid 9 snap -1 2024-09-21T08:25:24.025 INFO:tasks.rados.rados.0.smithi137.stdout:249: expect (ObjNum 80 snap 9 seq_num 80) 2024-09-21T08:25:24.025 INFO:tasks.rados.rados.0.smithi137.stdout:250: read oid 46 snap -1 2024-09-21T08:25:24.025 INFO:tasks.rados.rados.0.smithi137.stdout:250: expect deleted 2024-09-21T08:25:24.026 INFO:tasks.rados.rados.0.smithi137.stdout:251: snap_remove snap 11 2024-09-21T08:25:24.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:24.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:24.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:24.051 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:24.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:24.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:24.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:24.173 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:24.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:25:24.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:24.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:24.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:24.296 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:24.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:24.411 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:24.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:24.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:24.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:24.456 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:24.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:24.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:24.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:24.542 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:24.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:24.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:24.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:24.666 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:24.677 INFO:tasks.rados.rados.0.smithi137.stdout:251: done (5 left) 2024-09-21T08:25:24.677 INFO:tasks.rados.rados.0.smithi137.stdout:252: rollback oid 26 current snap is 23 2024-09-21T08:25:24.677 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 26 to 21 2024-09-21T08:25:24.677 INFO:tasks.rados.rados.0.smithi137.stdout:247: finishing copy_from racing read to smithi13779221-34 2024-09-21T08:25:24.677 INFO:tasks.rados.rados.0.smithi137.stdout:249: expect (ObjNum 80 snap 9 seq_num 80) 2024-09-21T08:25:24.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:24.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:24.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:24.789 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:24 smithi165 ceph-mon[32893]: 6.6 deep-scrub starts 2024-09-21T08:25:24.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:24 smithi165 ceph-mon[32893]: 6.6 deep-scrub ok 2024-09-21T08:25:24.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:24 smithi165 ceph-mon[32893]: osdmap e130: 8 total, 8 up, 5 in 2024-09-21T08:25:24.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:24 smithi165 ceph-mon[32893]: pgmap v101: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 186 MiB data, 559 MiB used, 446 GiB / 447 GiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 21 op/s 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:246: finishing copy_from racing read to smithi13779221-48 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:246: finishing copy_from to smithi13779221-48 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 13 (ObjNum 62 snap 4 seq_num 62) dirty exists 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:247: finishing copy_from to smithi13779221-34 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 3 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:248: finishing copy_from racing read to smithi13779221-19 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:248: finishing copy_from to smithi13779221-19 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 44 (ObjNum 9 snap 0 seq_num 9) dirty exists 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:246: done (5 left) 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:247: done (4 left) 2024-09-21T08:25:24.839 INFO:tasks.rados.rados.0.smithi137.stdout:248: done (3 left) 2024-09-21T08:25:24.840 INFO:tasks.rados.rados.0.smithi137.stdout:249: done (2 left) 2024-09-21T08:25:24.840 INFO:tasks.rados.rados.0.smithi137.stdout:250: done (1 left) 2024-09-21T08:25:24.840 INFO:tasks.rados.rados.0.smithi137.stdout:253: read oid 47 snap 21 2024-09-21T08:25:24.840 INFO:tasks.rados.rados.0.smithi137.stdout:253: expect (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:25:24.840 INFO:tasks.rados.rados.0.smithi137.stdout:254: copy_from oid 12 from oid 40 current snap is 23 2024-09-21T08:25:24.840 INFO:tasks.rados.rados.0.smithi137.stdout:252: finishing rollback tid 1 to smithi13779221-26 2024-09-21T08:25:24.840 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 0 (ObjNum 177 snap 0 seq_num 3760890640) dirty dne 2024-09-21T08:25:24.840 INFO:tasks.rados.rados.0.smithi137.stdout:252: done (2 left) 2024-09-21T08:25:24.840 INFO:tasks.rados.rados.0.smithi137.stdout:255: delete oid 30 current snap is 23 2024-09-21T08:25:24.841 INFO:tasks.rados.rados.0.smithi137.stdout:255: done (2 left) 2024-09-21T08:25:24.841 INFO:tasks.rados.rados.0.smithi137.stdout:256: write oid 36 current snap is 23 2024-09-21T08:25:24.841 INFO:tasks.rados.rados.0.smithi137.stdout:256: seq_num 121 ranges {433647=450424,1331027=582033} 2024-09-21T08:25:24.845 INFO:tasks.rados.rados.0.smithi137.stdout:256: writing smithi13779221-36 from 433647 to 884071 tid 1 2024-09-21T08:25:24.851 INFO:tasks.rados.rados.0.smithi137.stdout:256: writing smithi13779221-36 from 1331027 to 1913060 tid 2 2024-09-21T08:25:24.851 INFO:tasks.rados.rados.0.smithi137.stdout:257: write oid 38 current snap is 23 2024-09-21T08:25:24.851 INFO:tasks.rados.rados.0.smithi137.stdout:257: seq_num 122 ranges {450454=642026,1505862=544592} 2024-09-21T08:25:24.857 INFO:tasks.rados.rados.0.smithi137.stdout:257: writing smithi13779221-38 from 450454 to 1092480 tid 1 2024-09-21T08:25:24.863 INFO:tasks.rados.rados.0.smithi137.stdout:257: writing smithi13779221-38 from 1505862 to 2050454 tid 2 2024-09-21T08:25:24.863 INFO:tasks.rados.rados.0.smithi137.stdout:258: read oid 44 snap -1 2024-09-21T08:25:24.863 INFO:tasks.rados.rados.0.smithi137.stdout:258: expect (ObjNum 98 snap 19 seq_num 98) 2024-09-21T08:25:24.863 INFO:tasks.rados.rados.0.smithi137.stdout:259: delete oid 42 current snap is 23 2024-09-21T08:25:24.863 INFO:tasks.rados.rados.0.smithi137.stdout:256: finishing write tid 1 to smithi13779221-36 2024-09-21T08:25:24.863 INFO:tasks.rados.rados.0.smithi137.stdout:254: finishing copy_from racing read to smithi13779221-12 2024-09-21T08:25:24.863 INFO:tasks.rados.rados.0.smithi137.stdout:253: expect (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:25:24.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:24.895 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:24 smithi137 ceph-mon[41044]: 6.6 deep-scrub starts 2024-09-21T08:25:24.895 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:24 smithi137 ceph-mon[41044]: 6.6 deep-scrub ok 2024-09-21T08:25:24.895 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:24 smithi137 ceph-mon[41044]: osdmap e130: 8 total, 8 up, 5 in 2024-09-21T08:25:24.896 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:24 smithi137 ceph-mon[41044]: pgmap v101: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 186 MiB data, 559 MiB used, 446 GiB / 447 GiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 21 op/s 2024-09-21T08:25:24.896 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:24 smithi137 ceph-mon[29634]: 6.6 deep-scrub starts 2024-09-21T08:25:24.896 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:24 smithi137 ceph-mon[29634]: 6.6 deep-scrub ok 2024-09-21T08:25:24.896 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:24 smithi137 ceph-mon[29634]: osdmap e130: 8 total, 8 up, 5 in 2024-09-21T08:25:24.896 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:24 smithi137 ceph-mon[29634]: pgmap v101: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 186 MiB data, 559 MiB used, 446 GiB / 447 GiB avail; 11 MiB/s rd, 3.8 MiB/s wr, 21 op/s 2024-09-21T08:25:24.910 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:25:24.910 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T08:25:24.910 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:25:24.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:24.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:24.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:25.012 INFO:tasks.rados.rados.0.smithi137.stdout:254: finishing copy_from to smithi13779221-12 2024-09-21T08:25:25.012 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 7 (ObjNum 53 snap 2 seq_num 53) dirty exists 2024-09-21T08:25:25.012 INFO:tasks.rados.rados.0.smithi137.stdout:253: done (5 left) 2024-09-21T08:25:25.012 INFO:tasks.rados.rados.0.smithi137.stdout:254: done (4 left) 2024-09-21T08:25:25.012 INFO:tasks.rados.rados.0.smithi137.stdout:259: done (3 left) 2024-09-21T08:25:25.012 INFO:tasks.rados.rados.0.smithi137.stdout:260: write (excl) oid 27 current snap is 23 2024-09-21T08:25:25.012 INFO:tasks.rados.rados.0.smithi137.stdout:256: finishing write tid 2 to smithi13779221-36 2024-09-21T08:25:25.012 INFO:tasks.rados.rados.0.smithi137.stdout:256: finishing write tid 3 to smithi13779221-36 2024-09-21T08:25:25.012 INFO:tasks.rados.rados.0.smithi137.stdout:260: seq_num 124 ranges {484068=792524,1906473=177595} 2024-09-21T08:25:25.015 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:25.018 INFO:tasks.rados.rados.0.smithi137.stdout:260: writing smithi13779221-27 from 484068 to 1276592 tid 1 2024-09-21T08:25:25.023 INFO:tasks.rados.rados.0.smithi137.stdout:260: writing smithi13779221-27 from 1906473 to 2084068 tid 2 2024-09-21T08:25:25.023 INFO:tasks.rados.rados.0.smithi137.stdout:261: rollback oid 46 current snap is 23 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 46 to 21 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:256: finishing write tid 4 to smithi13779221-36 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:256: oid 36 updating version 0 to 121 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:256: oid 36 version 121 is already newer than 120 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:256: oid 36 version 121 is already newer than 119 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 121 (ObjNum 121 snap 23 seq_num 121) dirty exists 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:256: left oid 36 (ObjNum 121 snap 23 seq_num 121) 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:257: finishing write tid 1 to smithi13779221-38 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:257: finishing write tid 2 to smithi13779221-38 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:257: finishing write tid 3 to smithi13779221-38 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:257: finishing write tid 4 to smithi13779221-38 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:257: oid 38 updating version 0 to 122 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:257: oid 38 updating version 122 to 124 2024-09-21T08:25:25.024 INFO:tasks.rados.rados.0.smithi137.stdout:257: oid 38 version 124 is already newer than 123 2024-09-21T08:25:25.025 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 124 (ObjNum 122 snap 23 seq_num 122) dirty exists 2024-09-21T08:25:25.025 INFO:tasks.rados.rados.0.smithi137.stdout:257: left oid 38 (ObjNum 122 snap 23 seq_num 122) 2024-09-21T08:25:25.025 INFO:tasks.rados.rados.0.smithi137.stdout:258: expect (ObjNum 98 snap 19 seq_num 98) 2024-09-21T08:25:25.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:25.037 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:25.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:25.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:25.159 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:25.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:25.160 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:25.184 INFO:tasks.rados.rados.0.smithi137.stdout:260: finishing write tid 1 to smithi13779221-27 2024-09-21T08:25:25.184 INFO:tasks.rados.rados.0.smithi137.stdout:261: finishing rollback tid 1 to smithi13779221-46 2024-09-21T08:25:25.184 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 78 (ObjNum 129 snap 0 seq_num 3761033888) dirty dne 2024-09-21T08:25:25.184 INFO:tasks.rados.rados.0.smithi137.stdout:260: finishing write tid 2 to smithi13779221-27 2024-09-21T08:25:25.184 INFO:tasks.rados.rados.0.smithi137.stdout:260: finishing write tid 3 to smithi13779221-27 2024-09-21T08:25:25.184 INFO:tasks.rados.rados.0.smithi137.stdout:260: finishing write tid 4 to smithi13779221-27 2024-09-21T08:25:25.184 INFO:tasks.rados.rados.0.smithi137.stdout:260: oid 27 updating version 0 to 83 2024-09-21T08:25:25.184 INFO:tasks.rados.rados.0.smithi137.stdout:260: oid 27 updating version 83 to 84 2024-09-21T08:25:25.185 INFO:tasks.rados.rados.0.smithi137.stdout:260: oid 27 version 84 is already newer than 82 2024-09-21T08:25:25.185 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 84 (ObjNum 124 snap 23 seq_num 124) dirty exists 2024-09-21T08:25:25.185 INFO:tasks.rados.rados.0.smithi137.stdout:260: left oid 27 (ObjNum 124 snap 23 seq_num 124) 2024-09-21T08:25:25.185 INFO:tasks.rados.rados.0.smithi137.stdout:256: done (4 left) 2024-09-21T08:25:25.185 INFO:tasks.rados.rados.0.smithi137.stdout:257: done (3 left) 2024-09-21T08:25:25.185 INFO:tasks.rados.rados.0.smithi137.stdout:258: done (2 left) 2024-09-21T08:25:25.185 INFO:tasks.rados.rados.0.smithi137.stdout:260: done (1 left) 2024-09-21T08:25:25.185 INFO:tasks.rados.rados.0.smithi137.stdout:261: done (0 left) 2024-09-21T08:25:25.185 INFO:tasks.rados.rados.0.smithi137.stdout:262: read oid 40 snap -1 2024-09-21T08:25:25.185 INFO:tasks.rados.rados.0.smithi137.stdout:262: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-21T08:25:25.186 INFO:tasks.rados.rados.0.smithi137.stdout:263: write oid 32 current snap is 23 2024-09-21T08:25:25.186 INFO:tasks.rados.rados.0.smithi137.stdout:263: seq_num 125 ranges {500875=467773,1564955=535920} 2024-09-21T08:25:25.190 INFO:tasks.rados.rados.0.smithi137.stdout:263: writing smithi13779221-32 from 500875 to 968648 tid 1 2024-09-21T08:25:25.197 INFO:tasks.rados.rados.0.smithi137.stdout:263: writing smithi13779221-32 from 1564955 to 2100875 tid 2 2024-09-21T08:25:25.197 INFO:tasks.rados.rados.0.smithi137.stdout:264: read oid 46 snap -1 2024-09-21T08:25:25.197 INFO:tasks.rados.rados.0.smithi137.stdout:264: expect deleted 2024-09-21T08:25:25.197 INFO:tasks.rados.rados.0.smithi137.stdout:265: read oid 37 snap -1 2024-09-21T08:25:25.197 INFO:tasks.rados.rados.0.smithi137.stdout:265: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-21T08:25:25.198 INFO:tasks.rados.rados.0.smithi137.stdout:266: write (excl) oid 42 current snap is 23 2024-09-21T08:25:25.198 INFO:tasks.rados.rados.0.smithi137.stdout:266: seq_num 126 ranges {517682=543022,1623437=494245} 2024-09-21T08:25:25.203 INFO:tasks.rados.rados.0.smithi137.stdout:266: writing smithi13779221-42 from 517682 to 1060704 tid 1 2024-09-21T08:25:25.209 INFO:tasks.rados.rados.0.smithi137.stdout:266: writing smithi13779221-42 from 1623437 to 2117682 tid 2 2024-09-21T08:25:25.209 INFO:tasks.rados.rados.0.smithi137.stdout:267: write oid 31 current snap is 23 2024-09-21T08:25:25.209 INFO:tasks.rados.rados.0.smithi137.stdout:267: seq_num 127 ranges {534489=618271,1565566=568923} 2024-09-21T08:25:25.215 INFO:tasks.rados.rados.0.smithi137.stdout:267: writing smithi13779221-31 from 534489 to 1152760 tid 1 2024-09-21T08:25:25.221 INFO:tasks.rados.rados.0.smithi137.stdout:267: writing smithi13779221-31 from 1565566 to 2134489 tid 2 2024-09-21T08:25:25.221 INFO:tasks.rados.rados.0.smithi137.stdout:268: write (excl) oid 18 current snap is 23 2024-09-21T08:25:25.221 INFO:tasks.rados.rados.0.smithi137.stdout:268: seq_num 128 ranges {551296=693520,2024048=127248} 2024-09-21T08:25:25.228 INFO:tasks.rados.rados.0.smithi137.stdout:268: writing smithi13779221-18 from 551296 to 1244816 tid 1 2024-09-21T08:25:25.233 INFO:tasks.rados.rados.0.smithi137.stdout:268: writing smithi13779221-18 from 2024048 to 2151296 tid 2 2024-09-21T08:25:25.233 INFO:tasks.rados.rados.0.smithi137.stdout:269: write (excl) oid 15 current snap is 23 2024-09-21T08:25:25.233 INFO:tasks.rados.rados.0.smithi137.stdout:269: seq_num 129 ranges {568103=768769,2082530=85573} 2024-09-21T08:25:25.240 INFO:tasks.rados.rados.0.smithi137.stdout:269: writing smithi13779221-15 from 568103 to 1336872 tid 1 2024-09-21T08:25:25.245 INFO:tasks.rados.rados.0.smithi137.stdout:269: writing smithi13779221-15 from 2082530 to 2168103 tid 2 2024-09-21T08:25:25.245 INFO:tasks.rados.rados.0.smithi137.stdout:270: write (excl) oid 21 current snap is 23 2024-09-21T08:25:25.245 INFO:tasks.rados.rados.0.smithi137.stdout:262: expect (ObjNum 53 snap 2 seq_num 53) 2024-09-21T08:25:25.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:25.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:25.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:25.282 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:25.379 INFO:tasks.rados.rados.0.smithi137.stdout:263: finishing write tid 1 to smithi13779221-32 2024-09-21T08:25:25.379 INFO:tasks.rados.rados.0.smithi137.stdout:270: seq_num 130 ranges {584910=560371,1857365=327545} 2024-09-21T08:25:25.383 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:25.384 INFO:tasks.rados.rados.0.smithi137.stdout:270: writing smithi13779221-21 from 584910 to 1145281 tid 1 2024-09-21T08:25:25.391 INFO:tasks.rados.rados.0.smithi137.stdout:270: writing smithi13779221-21 from 1857365 to 2184910 tid 2 2024-09-21T08:25:25.391 INFO:tasks.rados.rados.0.smithi137.stdout:262: done (8 left) 2024-09-21T08:25:25.391 INFO:tasks.rados.rados.0.smithi137.stdout:271: read oid 47 snap -1 2024-09-21T08:25:25.391 INFO:tasks.rados.rados.0.smithi137.stdout:271: expect (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:25:25.391 INFO:tasks.rados.rados.0.smithi137.stdout:272: snap_create 2024-09-21T08:25:25.391 INFO:tasks.rados.rados.0.smithi137.stdout:263: finishing write tid 2 to smithi13779221-32 2024-09-21T08:25:25.391 INFO:tasks.rados.rados.0.smithi137.stdout:263: finishing write tid 3 to smithi13779221-32 2024-09-21T08:25:25.391 INFO:tasks.rados.rados.0.smithi137.stdout:263: finishing write tid 4 to smithi13779221-32 2024-09-21T08:25:25.392 INFO:tasks.rados.rados.0.smithi137.stdout:263: oid 32 updating version 0 to 28 2024-09-21T08:25:25.392 INFO:tasks.rados.rados.0.smithi137.stdout:263: oid 32 version 28 is already newer than 26 2024-09-21T08:25:25.392 INFO:tasks.rados.rados.0.smithi137.stdout:263: oid 32 version 28 is already newer than 27 2024-09-21T08:25:25.392 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 28 (ObjNum 125 snap 23 seq_num 125) dirty exists 2024-09-21T08:25:25.392 INFO:tasks.rados.rados.0.smithi137.stdout:263: left oid 32 (ObjNum 125 snap 23 seq_num 125) 2024-09-21T08:25:25.392 INFO:tasks.rados.rados.0.smithi137.stdout:265: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-21T08:25:25.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:25.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:25.406 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:25.472 INFO:tasks.rados.rados.0.smithi137.stdout:266: finishing write tid 1 to smithi13779221-42 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:266: finishing write tid 2 to smithi13779221-42 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:266: finishing write tid 3 to smithi13779221-42 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:266: finishing write tid 4 to smithi13779221-42 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:266: oid 42 updating version 0 to 142 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:266: oid 42 version 142 is already newer than 141 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:266: oid 42 version 142 is already newer than 140 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 142 (ObjNum 126 snap 23 seq_num 126) dirty exists 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:266: left oid 42 (ObjNum 126 snap 23 seq_num 126) 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:267: finishing write tid 1 to smithi13779221-31 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:267: finishing write tid 2 to smithi13779221-31 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:267: finishing write tid 3 to smithi13779221-31 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:267: finishing write tid 4 to smithi13779221-31 2024-09-21T08:25:25.473 INFO:tasks.rados.rados.0.smithi137.stdout:267: oid 31 updating version 0 to 75 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:267: oid 31 version 75 is already newer than 73 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:267: oid 31 version 75 is already newer than 74 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 75 (ObjNum 127 snap 23 seq_num 127) dirty exists 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:267: left oid 31 (ObjNum 127 snap 23 seq_num 127) 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:268: finishing write tid 1 to smithi13779221-18 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:268: finishing write tid 2 to smithi13779221-18 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:268: finishing write tid 3 to smithi13779221-18 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:268: finishing write tid 4 to smithi13779221-18 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:268: oid 18 updating version 0 to 34 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:268: oid 18 updating version 34 to 35 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:268: oid 18 updating version 35 to 36 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 36 (ObjNum 128 snap 23 seq_num 128) dirty exists 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:268: left oid 18 (ObjNum 128 snap 23 seq_num 128) 2024-09-21T08:25:25.474 INFO:tasks.rados.rados.0.smithi137.stdout:269: finishing write tid 1 to smithi13779221-15 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:269: finishing write tid 2 to smithi13779221-15 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:269: finishing write tid 3 to smithi13779221-15 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:269: finishing write tid 4 to smithi13779221-15 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:269: oid 15 updating version 0 to 52 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:269: oid 15 updating version 52 to 53 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:269: oid 15 updating version 53 to 54 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 54 (ObjNum 129 snap 23 seq_num 129) dirty exists 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:269: left oid 15 (ObjNum 129 snap 23 seq_num 129) 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:270: finishing write tid 1 to smithi13779221-21 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:270: finishing write tid 2 to smithi13779221-21 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:270: finishing write tid 3 to smithi13779221-21 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:270: finishing write tid 4 to smithi13779221-21 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:270: oid 21 updating version 0 to 46 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:270: oid 21 version 46 is already newer than 45 2024-09-21T08:25:25.475 INFO:tasks.rados.rados.0.smithi137.stdout:270: oid 21 updating version 46 to 47 2024-09-21T08:25:25.476 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 47 (ObjNum 130 snap 23 seq_num 130) dirty exists 2024-09-21T08:25:25.476 INFO:tasks.rados.rados.0.smithi137.stdout:270: left oid 21 (ObjNum 130 snap 23 seq_num 130) 2024-09-21T08:25:25.476 INFO:tasks.rados.rados.0.smithi137.stdout:271: expect (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:25:25.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:25.528 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:25.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:25.529 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:25.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:25:25.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:25.652 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:25.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:25.652 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:25.678 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:25:25.682 INFO:tasks.rados.rados.0.smithi137.stdout:263: done (9 left) 2024-09-21T08:25:25.682 INFO:tasks.rados.rados.0.smithi137.stdout:264: done (8 left) 2024-09-21T08:25:25.682 INFO:tasks.rados.rados.0.smithi137.stdout:265: done (7 left) 2024-09-21T08:25:25.682 INFO:tasks.rados.rados.0.smithi137.stdout:266: done (6 left) 2024-09-21T08:25:25.683 INFO:tasks.rados.rados.0.smithi137.stdout:267: done (5 left) 2024-09-21T08:25:25.683 INFO:tasks.rados.rados.0.smithi137.stdout:268: done (4 left) 2024-09-21T08:25:25.683 INFO:tasks.rados.rados.0.smithi137.stdout:269: done (3 left) 2024-09-21T08:25:25.683 INFO:tasks.rados.rados.0.smithi137.stdout:270: done (2 left) 2024-09-21T08:25:25.683 INFO:tasks.rados.rados.0.smithi137.stdout:271: done (1 left) 2024-09-21T08:25:25.683 INFO:tasks.rados.rados.0.smithi137.stdout:272: done (0 left) 2024-09-21T08:25:25.683 INFO:tasks.rados.rados.0.smithi137.stdout:273: snap_remove snap 23 2024-09-21T08:25:25.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:25.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[41044]: 5.a scrub starts 2024-09-21T08:25:25.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[41044]: 5.a scrub ok 2024-09-21T08:25:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[41044]: osdmap e131: 8 total, 8 up, 5 in 2024-09-21T08:25:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/524021799' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[41044]: 5.1e scrub starts 2024-09-21T08:25:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[41044]: 5.1e scrub ok 2024-09-21T08:25:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[29634]: 5.a scrub starts 2024-09-21T08:25:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[29634]: 5.a scrub ok 2024-09-21T08:25:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[29634]: osdmap e131: 8 total, 8 up, 5 in 2024-09-21T08:25:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/524021799' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[29634]: 5.1e scrub starts 2024-09-21T08:25:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:25 smithi137 ceph-mon[29634]: 5.1e scrub ok 2024-09-21T08:25:25.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:25.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:25.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:25.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:25.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:25.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:25.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:26.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:25 smithi165 ceph-mon[32893]: 5.a scrub starts 2024-09-21T08:25:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:25 smithi165 ceph-mon[32893]: 5.a scrub ok 2024-09-21T08:25:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:25 smithi165 ceph-mon[32893]: osdmap e131: 8 total, 8 up, 5 in 2024-09-21T08:25:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:25 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/524021799' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:25 smithi165 ceph-mon[32893]: 5.1e scrub starts 2024-09-21T08:25:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:25 smithi165 ceph-mon[32893]: 5.1e scrub ok 2024-09-21T08:25:26.023 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:26.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:26.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:26.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:26.148 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:26.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:26.149 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:26.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:26.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:26.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:26.272 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:26.373 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:26.398 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:26.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:26.399 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:26.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:26.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:26.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:26.523 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:26.553 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:26.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:26.644 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:26.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:26.645 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:26.692 INFO:tasks.rados.rados.0.smithi137.stdout:273: done (0 left) 2024-09-21T08:25:26.692 INFO:tasks.rados.rados.0.smithi137.stdout:274: copy_from oid 12 from oid 49 current snap is 24 2024-09-21T08:25:26.692 INFO:tasks.rados.rados.0.smithi137.stdout:275: delete oid 18 current snap is 24 2024-09-21T08:25:26.697 INFO:tasks.rados.rados.0.smithi137.stdout:275: done (1 left) 2024-09-21T08:25:26.697 INFO:tasks.rados.rados.0.smithi137.stdout:276: delete oid 45 current snap is 24 2024-09-21T08:25:26.697 INFO:tasks.rados.rados.0.smithi137.stdout:274: finishing copy_from to smithi13779221-12 2024-09-21T08:25:26.698 INFO:tasks.rados.rados.0.smithi137.stdout:274: got expected ENOENT (src dne) 2024-09-21T08:25:26.698 INFO:tasks.rados.rados.0.smithi137.stdout:274: finishing copy_from racing read to smithi13779221-12 2024-09-21T08:25:26.699 INFO:tasks.rados.rados.0.smithi137.stdout:274: done (1 left) 2024-09-21T08:25:26.699 INFO:tasks.rados.rados.0.smithi137.stdout:276: done (0 left) 2024-09-21T08:25:26.699 INFO:tasks.rados.rados.0.smithi137.stdout:277: rollback oid 42 current snap is 24 2024-09-21T08:25:26.699 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 42 to 21 2024-09-21T08:25:26.699 INFO:tasks.rados.rados.0.smithi137.stdout:278: write (excl) oid 40 current snap is 24 2024-09-21T08:25:26.699 INFO:tasks.rados.rados.0.smithi137.stdout:278: seq_num 133 ranges {635331=786118,1916458=318873} 2024-09-21T08:25:26.708 INFO:tasks.rados.rados.0.smithi137.stdout:278: writing smithi13779221-40 from 635331 to 1421449 tid 1 2024-09-21T08:25:26.713 INFO:tasks.rados.rados.0.smithi137.stdout:278: writing smithi13779221-40 from 1916458 to 2235331 tid 2 2024-09-21T08:25:26.713 INFO:tasks.rados.rados.0.smithi137.stdout:279: read oid 26 snap 21 2024-09-21T08:25:26.713 INFO:tasks.rados.rados.0.smithi137.stdout:279: expect deleted 2024-09-21T08:25:26.713 INFO:tasks.rados.rados.0.smithi137.stdout:280: write oid 23 current snap is 24 2024-09-21T08:25:26.713 INFO:tasks.rados.rados.0.smithi137.stdout:277: finishing rollback tid 0 to smithi13779221-42 2024-09-21T08:25:26.713 INFO:tasks.rados.rados.0.smithi137.stdout:277: finishing rollback tid 1 to smithi13779221-42 2024-09-21T08:25:26.713 INFO:tasks.rados.rados.0.smithi137.stdout:277: finishing rollback tid 2 to smithi13779221-42 2024-09-21T08:25:26.714 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 145 (ObjNum 90 snap 14 seq_num 90) dirty exists 2024-09-21T08:25:26.714 INFO:tasks.rados.rados.0.smithi137.stdout:278: finishing write tid 1 to smithi13779221-40 2024-09-21T08:25:26.714 INFO:tasks.rados.rados.0.smithi137.stdout:280: seq_num 134 ranges {652138=461367,1574940=677198} 2024-09-21T08:25:26.719 INFO:tasks.rados.rados.0.smithi137.stdout:280: writing smithi13779221-23 from 652138 to 1113505 tid 1 2024-09-21T08:25:26.725 INFO:tasks.rados.rados.0.smithi137.stdout:280: writing smithi13779221-23 from 1574940 to 2252138 tid 2 2024-09-21T08:25:26.725 INFO:tasks.rados.rados.0.smithi137.stdout:277: done (3 left) 2024-09-21T08:25:26.725 INFO:tasks.rados.rados.0.smithi137.stdout:281: delete oid 12 current snap is 24 2024-09-21T08:25:26.726 INFO:tasks.rados.rados.0.smithi137.stdout:278: finishing write tid 2 to smithi13779221-40 2024-09-21T08:25:26.726 INFO:tasks.rados.rados.0.smithi137.stdout:278: finishing write tid 3 to smithi13779221-40 2024-09-21T08:25:26.726 INFO:tasks.rados.rados.0.smithi137.stdout:278: finishing write tid 4 to smithi13779221-40 2024-09-21T08:25:26.726 INFO:tasks.rados.rados.0.smithi137.stdout:278: oid 40 updating version 0 to 55 2024-09-21T08:25:26.726 INFO:tasks.rados.rados.0.smithi137.stdout:278: oid 40 updating version 55 to 56 2024-09-21T08:25:26.726 INFO:tasks.rados.rados.0.smithi137.stdout:278: oid 40 updating version 56 to 57 2024-09-21T08:25:26.726 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 57 (ObjNum 133 snap 24 seq_num 133) dirty exists 2024-09-21T08:25:26.726 INFO:tasks.rados.rados.0.smithi137.stdout:278: left oid 40 (ObjNum 133 snap 24 seq_num 133) 2024-09-21T08:25:26.726 INFO:tasks.rados.rados.0.smithi137.stdout:280: finishing write tid 1 to smithi13779221-23 2024-09-21T08:25:26.727 INFO:tasks.rados.rados.0.smithi137.stdout:278: done (3 left) 2024-09-21T08:25:26.727 INFO:tasks.rados.rados.0.smithi137.stdout:279: done (2 left) 2024-09-21T08:25:26.727 INFO:tasks.rados.rados.0.smithi137.stdout:281: done (1 left) 2024-09-21T08:25:26.727 INFO:tasks.rados.rados.0.smithi137.stdout:282: copy_from oid 45 from oid 37 current snap is 24 2024-09-21T08:25:26.727 INFO:tasks.rados.rados.0.smithi137.stdout:283: snap_create 2024-09-21T08:25:26.728 INFO:tasks.rados.rados.0.smithi137.stdout:280: finishing write tid 2 to smithi13779221-23 2024-09-21T08:25:26.729 INFO:tasks.rados.rados.0.smithi137.stdout:280: finishing write tid 3 to smithi13779221-23 2024-09-21T08:25:26.729 INFO:tasks.rados.rados.0.smithi137.stdout:280: finishing write tid 4 to smithi13779221-23 2024-09-21T08:25:26.729 INFO:tasks.rados.rados.0.smithi137.stdout:280: oid 23 updating version 0 to 45 2024-09-21T08:25:26.729 INFO:tasks.rados.rados.0.smithi137.stdout:280: oid 23 updating version 45 to 46 2024-09-21T08:25:26.729 INFO:tasks.rados.rados.0.smithi137.stdout:280: oid 23 version 46 is already newer than 44 2024-09-21T08:25:26.729 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 46 (ObjNum 134 snap 24 seq_num 134) dirty exists 2024-09-21T08:25:26.729 INFO:tasks.rados.rados.0.smithi137.stdout:280: left oid 23 (ObjNum 134 snap 24 seq_num 134) 2024-09-21T08:25:26.733 INFO:tasks.rados.rados.0.smithi137.stdout:282: finishing copy_from racing read to smithi13779221-45 2024-09-21T08:25:26.740 INFO:tasks.rados.rados.0.smithi137.stdout:282: finishing copy_from to smithi13779221-45 2024-09-21T08:25:26.740 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 18 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-21T08:25:26.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:26.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:26.770 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:26.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:26.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:26.893 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:26.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:26.894 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:26.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[29634]: 4.1c scrub starts 2024-09-21T08:25:26.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[29634]: 4.1c scrub ok 2024-09-21T08:25:26.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[29634]: 6.b deep-scrub starts 2024-09-21T08:25:26.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[29634]: 6.b deep-scrub ok 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/524021799' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[29634]: osdmap e132: 8 total, 8 up, 5 in 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[29634]: pgmap v104: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 184 MiB data, 572 MiB used, 446 GiB / 447 GiB avail; 19 MiB/s rd, 5.1 MiB/s wr, 32 op/s 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[41044]: 4.1c scrub starts 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[41044]: 4.1c scrub ok 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[41044]: 6.b deep-scrub starts 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[41044]: 6.b deep-scrub ok 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:26.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/524021799' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:25:26.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[41044]: osdmap e132: 8 total, 8 up, 5 in 2024-09-21T08:25:26.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:25:26.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:25:26.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:26 smithi137 ceph-mon[41044]: pgmap v104: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 184 MiB data, 572 MiB used, 446 GiB / 447 GiB avail; 19 MiB/s rd, 5.1 MiB/s wr, 32 op/s 2024-09-21T08:25:26.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:26 smithi165 ceph-mon[32893]: 4.1c scrub starts 2024-09-21T08:25:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:26 smithi165 ceph-mon[32893]: 4.1c scrub ok 2024-09-21T08:25:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:26 smithi165 ceph-mon[32893]: 6.b deep-scrub starts 2024-09-21T08:25:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:26 smithi165 ceph-mon[32893]: 6.b deep-scrub ok 2024-09-21T08:25:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:26 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:26 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/524021799' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:25:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:26 smithi165 ceph-mon[32893]: osdmap e132: 8 total, 8 up, 5 in 2024-09-21T08:25:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:26 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:25:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:26 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:25:27.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:26 smithi165 ceph-mon[32893]: pgmap v104: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 184 MiB data, 572 MiB used, 446 GiB / 447 GiB avail; 19 MiB/s rd, 5.1 MiB/s wr, 32 op/s 2024-09-21T08:25:27.020 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:27.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:27.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:27.037 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:27.038 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:27.053 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:27.054 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:27.070 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:27.070 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:27.085 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:27.085 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:27.086 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:27.102 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:27.102 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:27.119 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:27.119 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:27.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:27.136 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:27.136 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:27.144 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:27.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:27.144 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:27.153 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:27.153 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:27.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:27.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:27.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:27.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:27.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:27.392 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:27.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:27.393 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:27.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:27.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:27.516 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:27.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:27.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:27.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:27.640 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:27.694 INFO:tasks.rados.rados.0.smithi137.stdout:280: done (2 left) 2024-09-21T08:25:27.694 INFO:tasks.rados.rados.0.smithi137.stdout:282: done (1 left) 2024-09-21T08:25:27.694 INFO:tasks.rados.rados.0.smithi137.stdout:283: done (0 left) 2024-09-21T08:25:27.694 INFO:tasks.rados.rados.0.smithi137.stdout:284: delete oid 50 current snap is 25 2024-09-21T08:25:27.698 INFO:tasks.rados.rados.0.smithi137.stdout:284: done (0 left) 2024-09-21T08:25:27.698 INFO:tasks.rados.rados.0.smithi137.stdout:285: write (excl) oid 28 current snap is 25 2024-09-21T08:25:27.698 INFO:tasks.rados.rados.0.smithi137.stdout:285: seq_num 137 ranges {702559=403467,1750386=495032} 2024-09-21T08:25:27.704 INFO:tasks.rados.rados.0.smithi137.stdout:285: writing smithi13779221-28 from 702559 to 1106026 tid 1 2024-09-21T08:25:27.711 INFO:tasks.rados.rados.0.smithi137.stdout:285: writing smithi13779221-28 from 1750386 to 2245418 tid 2 2024-09-21T08:25:27.711 INFO:tasks.rados.rados.0.smithi137.stdout:286: delete oid 26 current snap is 25 2024-09-21T08:25:27.711 INFO:tasks.rados.rados.0.smithi137.stdout:285: finishing write tid 1 to smithi13779221-28 2024-09-21T08:25:27.712 INFO:tasks.rados.rados.0.smithi137.stdout:286: done (1 left) 2024-09-21T08:25:27.712 INFO:tasks.rados.rados.0.smithi137.stdout:287: snap_remove snap 21 2024-09-21T08:25:27.741 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:27.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:27.764 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:27.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:27 smithi137 ceph-mon[29634]: osdmap e133: 8 total, 8 up, 5 in 2024-09-21T08:25:27.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:27 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3838051868' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:27 smithi137 ceph-mon[41044]: osdmap e133: 8 total, 8 up, 5 in 2024-09-21T08:25:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:27 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3838051868' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:27.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:27.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:27.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:27.888 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:27.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:28.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:28.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:28.013 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:27 smithi165 ceph-mon[32893]: osdmap e133: 8 total, 8 up, 5 in 2024-09-21T08:25:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:27 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3838051868' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:28.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:28.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:28.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:28.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:28.188 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:25:28.188 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:25:28.188 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:25:28.188 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:25:28.188 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:25:28.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:28.259 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:28.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:28.260 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:28.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:28.381 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:28.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:28.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:28.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:25:28.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:28.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:28.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:28.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:28.608 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:28.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:28.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:28.631 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:28.691 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:28.706 INFO:tasks.rados.rados.0.smithi137.stdout:287: done (1 left) 2024-09-21T08:25:28.706 INFO:tasks.rados.rados.0.smithi137.stdout:288: read oid 20 snap 24 2024-09-21T08:25:28.706 INFO:tasks.rados.rados.0.smithi137.stdout:288: expect deleted 2024-09-21T08:25:28.707 INFO:tasks.rados.rados.0.smithi137.stdout:289: snap_create 2024-09-21T08:25:28.707 INFO:tasks.rados.rados.0.smithi137.stdout:285: finishing write tid 2 to smithi13779221-28 2024-09-21T08:25:28.707 INFO:tasks.rados.rados.0.smithi137.stdout:285: finishing write tid 3 to smithi13779221-28 2024-09-21T08:25:28.707 INFO:tasks.rados.rados.0.smithi137.stdout:285: finishing write tid 4 to smithi13779221-28 2024-09-21T08:25:28.707 INFO:tasks.rados.rados.0.smithi137.stdout:285: oid 28 updating version 0 to 58 2024-09-21T08:25:28.707 INFO:tasks.rados.rados.0.smithi137.stdout:285: oid 28 updating version 58 to 60 2024-09-21T08:25:28.707 INFO:tasks.rados.rados.0.smithi137.stdout:285: oid 28 version 60 is already newer than 59 2024-09-21T08:25:28.707 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 60 (ObjNum 137 snap 25 seq_num 137) dirty exists 2024-09-21T08:25:28.709 INFO:tasks.rados.rados.0.smithi137.stdout:285: left oid 28 (ObjNum 137 snap 25 seq_num 137) 2024-09-21T08:25:28.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:28.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:28.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:28.754 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:28.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:28 smithi137 ceph-mon[29634]: osdmap e134: 8 total, 8 up, 5 in 2024-09-21T08:25:28.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:28 smithi137 ceph-mon[29634]: pgmap v107: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 184 MiB data, 572 MiB used, 446 GiB / 447 GiB avail; 19 MiB/s rd, 5.1 MiB/s wr, 32 op/s 2024-09-21T08:25:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:28 smithi137 ceph-mon[41044]: osdmap e134: 8 total, 8 up, 5 in 2024-09-21T08:25:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:28 smithi137 ceph-mon[41044]: pgmap v107: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 184 MiB data, 572 MiB used, 446 GiB / 447 GiB avail; 19 MiB/s rd, 5.1 MiB/s wr, 32 op/s 2024-09-21T08:25:28.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:28.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:28 smithi165 ceph-mon[32893]: osdmap e134: 8 total, 8 up, 5 in 2024-09-21T08:25:28.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:28 smithi165 ceph-mon[32893]: pgmap v107: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 184 MiB data, 572 MiB used, 446 GiB / 447 GiB avail; 19 MiB/s rd, 5.1 MiB/s wr, 32 op/s 2024-09-21T08:25:28.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:28.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:28.997 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:29.098 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:29.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:29.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:29.122 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:29.223 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:29.244 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:29.245 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:29.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:29.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:29.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:29.368 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:29.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:29.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T08:25:29.490 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:29.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:29.491 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:29.592 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:29.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:29.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:29.614 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:29.705 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:25:29.706 INFO:tasks.rados.rados.0.smithi137.stdout:285: done (2 left) 2024-09-21T08:25:29.706 INFO:tasks.rados.rados.0.smithi137.stdout:288: done (1 left) 2024-09-21T08:25:29.706 INFO:tasks.rados.rados.0.smithi137.stdout:289: done (0 left) 2024-09-21T08:25:29.707 INFO:tasks.rados.rados.0.smithi137.stdout:290: read oid 39 snap 25 2024-09-21T08:25:29.707 INFO:tasks.rados.rados.0.smithi137.stdout:290: expect (ObjNum 69 snap 7 seq_num 69) 2024-09-21T08:25:29.707 INFO:tasks.rados.rados.0.smithi137.stdout:291: read oid 38 snap -1 2024-09-21T08:25:29.707 INFO:tasks.rados.rados.0.smithi137.stdout:291: expect (ObjNum 122 snap 23 seq_num 122) 2024-09-21T08:25:29.707 INFO:tasks.rados.rados.0.smithi137.stdout:292: copy_from oid 3 from oid 30 current snap is 26 2024-09-21T08:25:29.707 INFO:tasks.rados.rados.0.smithi137.stdout:293: write (excl) oid 35 current snap is 26 2024-09-21T08:25:29.707 INFO:tasks.rados.rados.0.smithi137.stdout:293: seq_num 139 ranges {736173=553965,1867350=468823} 2024-09-21T08:25:29.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:29.716 INFO:tasks.rados.rados.0.smithi137.stdout:293: writing smithi13779221-35 from 736173 to 1290138 tid 1 2024-09-21T08:25:29.723 INFO:tasks.rados.rados.0.smithi137.stdout:293: writing smithi13779221-35 from 1867350 to 2336173 tid 2 2024-09-21T08:25:29.723 INFO:tasks.rados.rados.0.smithi137.stdout:294: read oid 4 snap -1 2024-09-21T08:25:29.723 INFO:tasks.rados.rados.0.smithi137.stdout:294: expect (ObjNum 104 snap 20 seq_num 104) 2024-09-21T08:25:29.723 INFO:tasks.rados.rados.0.smithi137.stdout:295: rollback oid 42 current snap is 26 2024-09-21T08:25:29.723 INFO:tasks.rados.rados.0.smithi137.stdout:292: finishing copy_from to smithi13779221-3 2024-09-21T08:25:29.723 INFO:tasks.rados.rados.0.smithi137.stdout:292: got expected ENOENT (src dne) 2024-09-21T08:25:29.723 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 42 to 25 2024-09-21T08:25:29.723 INFO:tasks.rados.rados.0.smithi137.stdout:292: finishing copy_from racing read to smithi13779221-3 2024-09-21T08:25:29.723 INFO:tasks.rados.rados.0.smithi137.stdout:290: expect (ObjNum 69 snap 7 seq_num 69) 2024-09-21T08:25:29.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:29.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:29.739 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:29.828 INFO:tasks.rados.rados.0.smithi137.stdout:293: finishing write tid 1 to smithi13779221-35 2024-09-21T08:25:29.829 INFO:tasks.rados.rados.0.smithi137.stdout:293: finishing write tid 2 to smithi13779221-35 2024-09-21T08:25:29.829 INFO:tasks.rados.rados.0.smithi137.stdout:293: finishing write tid 3 to smithi13779221-35 2024-09-21T08:25:29.829 INFO:tasks.rados.rados.0.smithi137.stdout:293: finishing write tid 4 to smithi13779221-35 2024-09-21T08:25:29.829 INFO:tasks.rados.rados.0.smithi137.stdout:293: oid 35 updating version 0 to 90 2024-09-21T08:25:29.829 INFO:tasks.rados.rados.0.smithi137.stdout:293: oid 35 updating version 90 to 91 2024-09-21T08:25:29.829 INFO:tasks.rados.rados.0.smithi137.stdout:293: oid 35 updating version 91 to 92 2024-09-21T08:25:29.829 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 92 (ObjNum 139 snap 26 seq_num 139) dirty exists 2024-09-21T08:25:29.829 INFO:tasks.rados.rados.0.smithi137.stdout:293: left oid 35 (ObjNum 139 snap 26 seq_num 139) 2024-09-21T08:25:29.829 INFO:tasks.rados.rados.0.smithi137.stdout:291: expect (ObjNum 122 snap 23 seq_num 122) 2024-09-21T08:25:29.841 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:29.861 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:29.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:29.862 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:29.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:25:29.916 INFO:tasks.rados.rados.0.smithi137.stdout:295: finishing rollback tid 0 to smithi13779221-42 2024-09-21T08:25:29.916 INFO:tasks.rados.rados.0.smithi137.stdout:295: finishing rollback tid 1 to smithi13779221-42 2024-09-21T08:25:29.916 INFO:tasks.rados.rados.0.smithi137.stdout:295: finishing rollback tid 2 to smithi13779221-42 2024-09-21T08:25:29.916 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 155 (ObjNum 90 snap 14 seq_num 90) dirty exists 2024-09-21T08:25:29.916 INFO:tasks.rados.rados.0.smithi137.stdout:294: expect (ObjNum 104 snap 20 seq_num 104) 2024-09-21T08:25:29.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:29.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:29.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:29.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:29.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:29 smithi165 ceph-mon[32893]: osdmap e135: 8 total, 8 up, 5 in 2024-09-21T08:25:29.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:29 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3588831176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:29.985 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:29 smithi165 ceph-mon[32893]: from='client.44867 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:29.985 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:29 smithi165 podman[70661]: 2024-09-21 08:25:29.984498629 +0000 UTC m=+0.375505502 container init 580bd27f7bc0d8ee0094b375581e10bf10e7f8377a5cd3f74586e965c6cf91ed (image=quay.io/ceph/grafana:9.4.12, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T08:25:30.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:29 smithi137 ceph-mon[29634]: osdmap e135: 8 total, 8 up, 5 in 2024-09-21T08:25:30.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:29 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3588831176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:30.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:29 smithi137 ceph-mon[29634]: from='client.44867 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:30.011 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:29 smithi137 ceph-mon[41044]: osdmap e135: 8 total, 8 up, 5 in 2024-09-21T08:25:30.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:29 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3588831176' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:30.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:29 smithi137 ceph-mon[41044]: from='client.44867 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:30.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:30.095 INFO:tasks.rados.rados.0.smithi137.stdout:290: done (5 left) 2024-09-21T08:25:30.096 INFO:tasks.rados.rados.0.smithi137.stdout:291: done (4 left) 2024-09-21T08:25:30.096 INFO:tasks.rados.rados.0.smithi137.stdout:292: done (3 left) 2024-09-21T08:25:30.096 INFO:tasks.rados.rados.0.smithi137.stdout:293: done (2 left) 2024-09-21T08:25:30.096 INFO:tasks.rados.rados.0.smithi137.stdout:294: done (1 left) 2024-09-21T08:25:30.096 INFO:tasks.rados.rados.0.smithi137.stdout:295: done (0 left) 2024-09-21T08:25:30.096 INFO:tasks.rados.rados.0.smithi137.stdout:296: copy_from oid 13 from oid 49 current snap is 26 2024-09-21T08:25:30.096 INFO:tasks.rados.rados.0.smithi137.stdout:297: delete oid 38 current snap is 26 2024-09-21T08:25:30.097 INFO:tasks.rados.rados.0.smithi137.stdout:296: finishing copy_from to smithi13779221-13 2024-09-21T08:25:30.097 INFO:tasks.rados.rados.0.smithi137.stdout:296: got expected ENOENT (src dne) 2024-09-21T08:25:30.097 INFO:tasks.rados.rados.0.smithi137.stdout:296: finishing copy_from racing read to smithi13779221-13 2024-09-21T08:25:30.097 INFO:tasks.rados.rados.0.smithi137.stdout:296: done (1 left) 2024-09-21T08:25:30.098 INFO:tasks.rados.rados.0.smithi137.stdout:297: done (0 left) 2024-09-21T08:25:30.098 INFO:tasks.rados.rados.0.smithi137.stdout:298: snap_remove snap 25 2024-09-21T08:25:30.107 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:30.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:30.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:30.114 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:30.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:30.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:30.230 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:30.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:29 smithi165 podman[70661]: 2024-09-21 08:25:29.989527571 +0000 UTC m=+0.380534434 container start 580bd27f7bc0d8ee0094b375581e10bf10e7f8377a5cd3f74586e965c6cf91ed (image=quay.io/ceph/grafana:9.4.12, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a, maintainer=Grafana Labs ) 2024-09-21T08:25:30.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 bash[70661]: 580bd27f7bc0d8ee0094b375581e10bf10e7f8377a5cd3f74586e965c6cf91ed 2024-09-21T08:25:30.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 systemd[1]: Started Ceph grafana.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:25:30.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144316767Z level=info msg="Starting Grafana" version=9.4.12 commit=37ee25ba46 branch=HEAD compiled=2023-05-22T15:58:05Z 2024-09-21T08:25:30.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144556838Z level=warn msg="\"sentry\" frontend logging provider is deprecated and will be removed in the next major version. Use \"grafana\" provider instead." 2024-09-21T08:25:30.236 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144573046Z level=info msg="Config loaded from" file=/usr/share/grafana/conf/defaults.ini 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144578158Z level=info msg="Config loaded from" file=/etc/grafana/grafana.ini 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144582656Z level=info msg="Config overridden from command line" arg="default.paths.data=/var/lib/grafana" 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144586965Z level=info msg="Config overridden from command line" arg="default.paths.logs=/var/log/grafana" 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144591242Z level=info msg="Config overridden from command line" arg="default.paths.plugins=/var/lib/grafana/plugins" 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144595664Z level=info msg="Config overridden from command line" arg="default.paths.provisioning=/etc/grafana/provisioning" 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144599909Z level=info msg="Config overridden from command line" arg="default.log.mode=console" 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144604418Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_DATA=/var/lib/grafana" 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144609126Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_LOGS=/var/log/grafana" 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144613134Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PLUGINS=/var/lib/grafana/plugins" 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144617382Z level=info msg="Config overridden from Environment variable" var="GF_PATHS_PROVISIONING=/etc/grafana/provisioning" 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144621783Z level=info msg="Path Home" path=/usr/share/grafana 2024-09-21T08:25:30.237 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144626504Z level=info msg="Path Data" path=/var/lib/grafana 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144630451Z level=info msg="Path Logs" path=/var/log/grafana 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144634251Z level=info msg="Path Plugins" path=/var/lib/grafana/plugins 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144638487Z level=info msg="Path Provisioning" path=/etc/grafana/provisioning 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=settings t=2024-09-21T08:25:30.144642925Z level=info msg="App mode production" 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=sqlstore t=2024-09-21T08:25:30.14470236Z level=info msg="Connecting to DB" dbtype=sqlite3 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=sqlstore t=2024-09-21T08:25:30.144719014Z level=warn msg="SQLite database file has broader permissions than it should" path=/var/lib/grafana/grafana.db mode=-rw-r--r-- expected=-rw-r----- 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.145850361Z level=info msg="Starting DB migrations" 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.145911805Z level=info msg="Executing migration" id="create migration_log table" 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.146617984Z level=info msg="Executing migration" id="create user table" 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.147286316Z level=info msg="Executing migration" id="add unique index user.login" 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.14778662Z level=info msg="Executing migration" id="add unique index user.email" 2024-09-21T08:25:30.239 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.148319894Z level=info msg="Executing migration" id="drop index UQE_user_login - v1" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.148910984Z level=info msg="Executing migration" id="drop index UQE_user_email - v1" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.149368164Z level=info msg="Executing migration" id="Rename table user to user_v1 - v1" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.1501363Z level=info msg="Executing migration" id="create user table v2" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.15054347Z level=info msg="Executing migration" id="create index UQE_user_login - v2" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.150927862Z level=info msg="Executing migration" id="create index UQE_user_email - v2" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.151446846Z level=info msg="Executing migration" id="copy data_source v1 to v2" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.151932219Z level=info msg="Executing migration" id="Drop old table user_v1" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.152559945Z level=info msg="Executing migration" id="Add column help_flags1 to user table" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.153053577Z level=info msg="Executing migration" id="Update user table charset" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.153368163Z level=info msg="Executing migration" id="Add last_seen_at column to user" 2024-09-21T08:25:30.240 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.153755486Z level=info msg="Executing migration" id="Add missing user data" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.15407679Z level=info msg="Executing migration" id="Add is_disabled column to user" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.154498784Z level=info msg="Executing migration" id="Add index user.login/user.email" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.154869246Z level=info msg="Executing migration" id="Add is_service_account column to user" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.155289279Z level=info msg="Executing migration" id="Update is_service_account column to nullable" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.157266862Z level=info msg="Executing migration" id="create temp user table v1-7" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.15772114Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v1-7" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.158199462Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v1-7" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.158626207Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v1-7" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.159085299Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v1-7" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.159640549Z level=info msg="Executing migration" id="Update temp_user table charset" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.159927528Z level=info msg="Executing migration" id="drop index IDX_temp_user_email - v1" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.161868671Z level=info msg="Executing migration" id="drop index IDX_temp_user_org_id - v1" 2024-09-21T08:25:30.241 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.16228872Z level=info msg="Executing migration" id="drop index IDX_temp_user_code - v1" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.162695146Z level=info msg="Executing migration" id="drop index IDX_temp_user_status - v1" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.163070611Z level=info msg="Executing migration" id="Rename table temp_user to temp_user_tmp_qwerty - v1" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.163865959Z level=info msg="Executing migration" id="create temp_user v2" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.164286003Z level=info msg="Executing migration" id="create index IDX_temp_user_email - v2" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.164698546Z level=info msg="Executing migration" id="create index IDX_temp_user_org_id - v2" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.165086262Z level=info msg="Executing migration" id="create index IDX_temp_user_code - v2" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.165493814Z level=info msg="Executing migration" id="create index IDX_temp_user_status - v2" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.165914833Z level=info msg="Executing migration" id="copy temp_user v1 to v2" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.166272726Z level=info msg="Executing migration" id="drop temp_user_tmp_qwerty" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.16662819Z level=info msg="Executing migration" id="Set created for temp users that will otherwise prematurely expire" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.166949778Z level=info msg="Executing migration" id="create star table" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.167349288Z level=info msg="Executing migration" id="add unique index star.user_id_dashboard_id" 2024-09-21T08:25:30.242 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.167787085Z level=info msg="Executing migration" id="create org table v1" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.168211448Z level=info msg="Executing migration" id="create index UQE_org_name - v1" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.168631989Z level=info msg="Executing migration" id="create org_user table v1" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.169023106Z level=info msg="Executing migration" id="create index IDX_org_user_org_id - v1" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.169442158Z level=info msg="Executing migration" id="create index UQE_org_user_org_id_user_id - v1" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.169861875Z level=info msg="Executing migration" id="create index IDX_org_user_user_id - v1" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.170275424Z level=info msg="Executing migration" id="Update org table charset" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.170578011Z level=info msg="Executing migration" id="Update org_user table charset" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.170856203Z level=info msg="Executing migration" id="Migrate all Read Only Viewers to Viewers" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.171350164Z level=info msg="Executing migration" id="create dashboard table" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.171766493Z level=info msg="Executing migration" id="add index dashboard.account_id" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.172194392Z level=info msg="Executing migration" id="add unique index dashboard_account_id_slug" 2024-09-21T08:25:30.243 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.172607663Z level=info msg="Executing migration" id="create dashboard_tag table" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.173030357Z level=info msg="Executing migration" id="add unique index dashboard_tag.dasboard_id_term" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.175777985Z level=info msg="Executing migration" id="drop index UQE_dashboard_tag_dashboard_id_term - v1" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.178608416Z level=info msg="Executing migration" id="Rename table dashboard to dashboard_v1 - v1" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.181352333Z level=info msg="Executing migration" id="create dashboard v2" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.184083728Z level=info msg="Executing migration" id="create index IDX_dashboard_org_id - v2" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.187035631Z level=info msg="Executing migration" id="create index UQE_dashboard_org_id_slug - v2" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.189644862Z level=info msg="Executing migration" id="copy dashboard v1 to v2" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.192350004Z level=info msg="Executing migration" id="drop table dashboard_v1" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.192842397Z level=info msg="Executing migration" id="alter dashboard.data to mediumtext v1" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.193163554Z level=info msg="Executing migration" id="Add column updated_by in dashboard - v2" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.195534037Z level=info msg="Executing migration" id="Add column created_by in dashboard - v2" 2024-09-21T08:25:30.244 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.198242846Z level=info msg="Executing migration" id="Add column gnetId in dashboard" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.200894678Z level=info msg="Executing migration" id="Add index for gnetId in dashboard" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.203579573Z level=info msg="Executing migration" id="Add column plugin_id in dashboard" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.206160304Z level=info msg="Executing migration" id="Add index for plugin_id in dashboard" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.208913826Z level=info msg="Executing migration" id="Add index for dashboard_id in dashboard_tag" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.209764439Z level=info msg="Executing migration" id="Update dashboard table charset" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.210106787Z level=info msg="Executing migration" id="Update dashboard_tag table charset" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.210474332Z level=info msg="Executing migration" id="Add column folder_id in dashboard" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.211112884Z level=info msg="Executing migration" id="Add column isFolder in dashboard" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.21197638Z level=info msg="Executing migration" id="Add column has_acl in dashboard" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.212822099Z level=info msg="Executing migration" id="Add column uid in dashboard" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.213704794Z level=info msg="Executing migration" id="Update uid column values in dashboard" 2024-09-21T08:25:30.245 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.214623695Z level=info msg="Executing migration" id="Add unique index dashboard_org_id_uid" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.215456801Z level=info msg="Executing migration" id="Remove unique index org_id_slug" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.216227563Z level=info msg="Executing migration" id="Update dashboard title length" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.217080119Z level=info msg="Executing migration" id="Add unique index for dashboard_org_id_title_folder_id" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.217929268Z level=info msg="Executing migration" id="create dashboard_provisioning" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.218831426Z level=info msg="Executing migration" id="Rename table dashboard_provisioning to dashboard_provisioning_tmp_qwerty - v1" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.219946773Z level=info msg="Executing migration" id="create dashboard_provisioning v2" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.220773883Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id - v2" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.221648534Z level=info msg="Executing migration" id="create index IDX_dashboard_provisioning_dashboard_id_name - v2" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.222464791Z level=info msg="Executing migration" id="copy dashboard_provisioning v1 to v2" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.223299751Z level=info msg="Executing migration" id="drop dashboard_provisioning_tmp_qwerty" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.224187507Z level=info msg="Executing migration" id="Add check_sum column" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.225050351Z level=info msg="Executing migration" id="Add index for dashboard_title" 2024-09-21T08:25:30.246 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.225931773Z level=info msg="Executing migration" id="delete tags for deleted dashboards" 2024-09-21T08:25:30.247 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.226843716Z level=info msg="Executing migration" id="delete stars for deleted dashboards" 2024-09-21T08:25:30.247 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.227695786Z level=info msg="Executing migration" id="Add index for dashboard_is_folder" 2024-09-21T08:25:30.247 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.22859863Z level=info msg="Executing migration" id="Add isPublic for dashboard" 2024-09-21T08:25:30.247 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.229494457Z level=info msg="Executing migration" id="create data_source table" 2024-09-21T08:25:30.247 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.230337125Z level=info msg="Executing migration" id="add index data_source.account_id" 2024-09-21T08:25:30.247 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.232864681Z level=info msg="Executing migration" id="add unique index data_source.account_id_name" 2024-09-21T08:25:30.247 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.235603694Z level=info msg="Executing migration" id="drop index IDX_data_source_account_id - v1" 2024-09-21T08:25:30.247 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.238433371Z level=info msg="Executing migration" id="drop index UQE_data_source_account_id_name - v1" 2024-09-21T08:25:30.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:30.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:30.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:30.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:30.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:30.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:30.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:30.480 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:30.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.241483002Z level=info msg="Executing migration" id="Rename table data_source to data_source_v1 - v1" 2024-09-21T08:25:30.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.244402828Z level=info msg="Executing migration" id="create data_source table v2" 2024-09-21T08:25:30.488 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.247104579Z level=info msg="Executing migration" id="create index IDX_data_source_org_id - v2" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.249756711Z level=info msg="Executing migration" id="create index UQE_data_source_org_id_name - v2" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.252442279Z level=info msg="Executing migration" id="Drop old table data_source_v1 #2" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.255411766Z level=info msg="Executing migration" id="Add column with_credentials" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.258134987Z level=info msg="Executing migration" id="Add secure json data column" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.2610181Z level=info msg="Executing migration" id="Update data_source table charset" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.263841416Z level=info msg="Executing migration" id="Update initial version to 1" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.266690937Z level=info msg="Executing migration" id="Add read_only data column" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.269481149Z level=info msg="Executing migration" id="Migrate logging ds to loki ds" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.273035764Z level=info msg="Executing migration" id="Update json_data with nulls" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.275907772Z level=info msg="Executing migration" id="Add uid column" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.278678093Z level=info msg="Executing migration" id="Update uid value" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.281356894Z level=info msg="Executing migration" id="Add unique index datasource_org_id_uid" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.283926608Z level=info msg="Executing migration" id="add unique index datasource_org_id_is_default" 2024-09-21T08:25:30.489 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.286751038Z level=info msg="Executing migration" id="create api_key table" 2024-09-21T08:25:30.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.289523908Z level=info msg="Executing migration" id="add index api_key.account_id" 2024-09-21T08:25:30.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.292181931Z level=info msg="Executing migration" id="add index api_key.key" 2024-09-21T08:25:30.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.294968913Z level=info msg="Executing migration" id="add index api_key.account_id_name" 2024-09-21T08:25:30.490 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.297733645Z level=info msg="Executing migration" id="drop index IDX_api_key_account_id - v1" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.300396219Z level=info msg="Executing migration" id="drop index UQE_api_key_key - v1" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.303829425Z level=info msg="Executing migration" id="drop index UQE_api_key_account_id_name - v1" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.305928769Z level=info msg="Executing migration" id="Rename table api_key to api_key_v1 - v1" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.307247517Z level=info msg="Executing migration" id="create api_key table v2" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.309953335Z level=info msg="Executing migration" id="create index IDX_api_key_org_id - v2" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.31251259Z level=info msg="Executing migration" id="create index UQE_api_key_key - v2" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.315147273Z level=info msg="Executing migration" id="create index UQE_api_key_org_id_name - v2" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.315627911Z level=info msg="Executing migration" id="copy api_key v1 to v2" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.318272872Z level=info msg="Executing migration" id="Drop old table api_key_v1" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.318697576Z level=info msg="Executing migration" id="Update api_key table charset" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.319118037Z level=info msg="Executing migration" id="Add expires to api_key table" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.319706596Z level=info msg="Executing migration" id="Add service account foreign key" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.32029953Z level=info msg="Executing migration" id="set service account foreign key to nil if 0" 2024-09-21T08:25:30.491 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.320611564Z level=info msg="Executing migration" id="Add last_used_at to api_key table" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.321464806Z level=info msg="Executing migration" id="Add is_revoked column to api_key table" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.32231352Z level=info msg="Executing migration" id="create dashboard_snapshot table v4" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.323161032Z level=info msg="Executing migration" id="drop table dashboard_snapshot_v4 #1" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.323543941Z level=info msg="Executing migration" id="create dashboard_snapshot table v5 #2" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.324283125Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_key - v5" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.32472491Z level=info msg="Executing migration" id="create index UQE_dashboard_snapshot_delete_key - v5" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.32528408Z level=info msg="Executing migration" id="create index IDX_dashboard_snapshot_user_id - v5" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.32614974Z level=info msg="Executing migration" id="alter dashboard_snapshot to mediumtext v2" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.326446813Z level=info msg="Executing migration" id="Update dashboard_snapshot table charset" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.327311493Z level=info msg="Executing migration" id="Add column external_delete_url to dashboard_snapshots table" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.327920773Z level=info msg="Executing migration" id="Add encrypted dashboard json column" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.32855123Z level=info msg="Executing migration" id="Change dashboard_encrypted column to MEDIUMBLOB" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.329433391Z level=info msg="Executing migration" id="create quota table v1" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.329863262Z level=info msg="Executing migration" id="create index UQE_quota_org_id_user_id_target - v1" 2024-09-21T08:25:30.492 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.330707441Z level=info msg="Executing migration" id="Update quota table charset" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.331565111Z level=info msg="Executing migration" id="create plugin_setting table" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.332453485Z level=info msg="Executing migration" id="create index UQE_plugin_setting_org_id_plugin_id - v1" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.33289538Z level=info msg="Executing migration" id="Add column plugin_version to plugin_settings" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.333752518Z level=info msg="Executing migration" id="Update plugin_setting table charset" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.334590709Z level=info msg="Executing migration" id="create session table" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.335455625Z level=info msg="Executing migration" id="Drop old table playlist table" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.335751847Z level=info msg="Executing migration" id="Drop old table playlist_item table" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.336622045Z level=info msg="Executing migration" id="create playlist table v2" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.33710568Z level=info msg="Executing migration" id="create playlist item table v2" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.337848528Z level=info msg="Executing migration" id="Update playlist table charset" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.338487813Z level=info msg="Executing migration" id="Update playlist_item table charset" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.338892202Z level=info msg="Executing migration" id="drop preferences table v2" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.339721515Z level=info msg="Executing migration" id="drop preferences table v3" 2024-09-21T08:25:30.493 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.340601006Z level=info msg="Executing migration" id="create preferences table v3" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.341469736Z level=info msg="Executing migration" id="Update preferences table charset" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.341803631Z level=info msg="Executing migration" id="Add column team_id in preferences" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.344620744Z level=info msg="Executing migration" id="Update team_id column values in preferences" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.347447985Z level=info msg="Executing migration" id="Add column week_start in preferences" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.350258472Z level=info msg="Executing migration" id="Add column preferences.json_data" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.350918053Z level=info msg="Executing migration" id="alter preferences.json_data to mediumtext v1" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.354956083Z level=info msg="Executing migration" id="Add preferences index org_id" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.357937745Z level=info msg="Executing migration" id="Add preferences index user_id" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.360661861Z level=info msg="Executing migration" id="create alert table v1" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.361130867Z level=info msg="Executing migration" id="add index alert org_id & id " 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.364047105Z level=info msg="Executing migration" id="add index alert state" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.366911146Z level=info msg="Executing migration" id="add index alert dashboard_id" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.369647376Z level=info msg="Executing migration" id="Create alert_rule_tag table v1" 2024-09-21T08:25:30.494 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.370052345Z level=info msg="Executing migration" id="Add unique index alert_rule_tag.alert_id_tag_id" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.37275526Z level=info msg="Executing migration" id="drop index UQE_alert_rule_tag_alert_id_tag_id - v1" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.375711237Z level=info msg="Executing migration" id="Rename table alert_rule_tag to alert_rule_tag_v1 - v1" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.378450302Z level=info msg="Executing migration" id="Create alert_rule_tag table v2" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.378840804Z level=info msg="Executing migration" id="create index UQE_alert_rule_tag_alert_id_tag_id - Add unique index alert_rule_tag.alert_id_tag_id V2" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.381582491Z level=info msg="Executing migration" id="copy alert_rule_tag v1 to v2" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.384251987Z level=info msg="Executing migration" id="drop table alert_rule_tag_v1" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.387081585Z level=info msg="Executing migration" id="create alert_notification table v1" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.38749683Z level=info msg="Executing migration" id="Add column is_default" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.390245866Z level=info msg="Executing migration" id="Add column frequency" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.393104505Z level=info msg="Executing migration" id="Add column send_reminder" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.395786203Z level=info msg="Executing migration" id="Add column disable_resolve_message" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.3964843Z level=info msg="Executing migration" id="add index alert_notification org_id & name" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.399159084Z level=info msg="Executing migration" id="Update alert table charset" 2024-09-21T08:25:30.495 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.401818865Z level=info msg="Executing migration" id="Update alert_notification table charset" 2024-09-21T08:25:30.496 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.404510075Z level=info msg="Executing migration" id="create notification_journal table v1" 2024-09-21T08:25:30.496 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.404923327Z level=info msg="Executing migration" id="add index notification_journal org_id & alert_id & notifier_id" 2024-09-21T08:25:30.496 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.407686681Z level=info msg="Executing migration" id="drop alert_notification_journal" 2024-09-21T08:25:30.496 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.410538451Z level=info msg="Executing migration" id="create alert_notification_state table v1" 2024-09-21T08:25:30.496 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.413235049Z level=info msg="Executing migration" id="add index alert_notification_state org_id & alert_id & notifier_id" 2024-09-21T08:25:30.496 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.415934724Z level=info msg="Executing migration" id="Add for to alert table" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.418770352Z level=info msg="Executing migration" id="Add column uid in alert_notification" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.421553371Z level=info msg="Executing migration" id="Update uid column values in alert_notification" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.422405436Z level=info msg="Executing migration" id="Add unique index alert_notification_org_id_uid" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.423170627Z level=info msg="Executing migration" id="Remove unique index org_id_name" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.424077658Z level=info msg="Executing migration" id="Add column secure_settings in alert_notification" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.424873254Z level=info msg="Executing migration" id="alter alert.settings to mediumtext" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.425715201Z level=info msg="Executing migration" id="Add non-unique index alert_notification_state_alert_id" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.426568239Z level=info msg="Executing migration" id="Add non-unique index alert_rule_tag_alert_id" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.427423599Z level=info msg="Executing migration" id="Drop old annotation table v4" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.428290415Z level=info msg="Executing migration" id="create annotation table v5" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.429125733Z level=info msg="Executing migration" id="add index annotation 0 v3" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.430042133Z level=info msg="Executing migration" id="add index annotation 1 v3" 2024-09-21T08:25:30.497 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.430897947Z level=info msg="Executing migration" id="add index annotation 2 v3" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.431695104Z level=info msg="Executing migration" id="add index annotation 3 v3" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.432610104Z level=info msg="Executing migration" id="add index annotation 4 v3" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.433400974Z level=info msg="Executing migration" id="Update annotation table charset" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.434230087Z level=info msg="Executing migration" id="Add column region_id to annotation table" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.43515791Z level=info msg="Executing migration" id="Drop category_id index" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.438861343Z level=info msg="Executing migration" id="Add column tags to annotation table" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.439981777Z level=info msg="Executing migration" id="Create annotation_tag table v2" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.440937284Z level=info msg="Executing migration" id="Add unique index annotation_tag.annotation_id_tag_id" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.441718213Z level=info msg="Executing migration" id="drop index UQE_annotation_tag_annotation_id_tag_id - v2" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.4426027Z level=info msg="Executing migration" id="Rename table annotation_tag to annotation_tag_v2 - v2" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.444742038Z level=info msg="Executing migration" id="Create annotation_tag table v3" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.445586197Z level=info msg="Executing migration" id="create index UQE_annotation_tag_annotation_id_tag_id - Add unique index annotation_tag.annotation_id_tag_id V3" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.446445212Z level=info msg="Executing migration" id="copy annotation_tag v2 to v3" 2024-09-21T08:25:30.498 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.447355203Z level=info msg="Executing migration" id="drop table annotation_tag_v2" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.448151153Z level=info msg="Executing migration" id="Update alert annotations and set TEXT to empty" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.44898916Z level=info msg="Executing migration" id="Add created time to annotation table" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.449886117Z level=info msg="Executing migration" id="Add updated time to annotation table" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.450984657Z level=info msg="Executing migration" id="Add index for created in annotation table" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.453672651Z level=info msg="Executing migration" id="Add index for updated in annotation table" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.45657798Z level=info msg="Executing migration" id="Convert existing annotations from seconds to milliseconds" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.460061091Z level=info msg="Executing migration" id="Add epoch_end column" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.462412438Z level=info msg="Executing migration" id="Add index for epoch_end" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.465267417Z level=info msg="Executing migration" id="Make epoch_end the same as epoch" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.468045616Z level=info msg="Executing migration" id="Move region to single row" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.470743577Z level=info msg="Executing migration" id="Remove index org_id_epoch from annotation table" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.473593056Z level=info msg="Executing migration" id="Remove index org_id_dashboard_id_panel_id_epoch from annotation table" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.476532767Z level=info msg="Executing migration" id="Add index for org_id_dashboard_id_epoch_end_epoch on annotation table" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.479319413Z level=info msg="Executing migration" id="Add index for org_id_epoch_end_epoch on annotation table" 2024-09-21T08:25:30.499 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.482221541Z level=info msg="Executing migration" id="Remove index org_id_epoch_epoch_end from annotation table" 2024-09-21T08:25:30.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.484894832Z level=info msg="Executing migration" id="Add index for alert_id on annotation table" 2024-09-21T08:25:30.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.487740611Z level=info msg="Executing migration" id="Increase tags column to length 4096" 2024-09-21T08:25:30.500 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.490471534Z level=info msg="Executing migration" id="create test_data table" 2024-09-21T08:25:30.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:30.586 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:30.593 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 7] out_osds: [4, 3, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:25:30.593 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:25:30.593 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T08:25:30.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T08:25:30.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:30.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:30.605 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:30.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:30.707 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:25:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:25:30.708 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 116.134465969, 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049090319999999998, 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:25:30.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567959+0000", 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.762e-06 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567965+0000", 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567980+0000", 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4742e-05 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567990+0000", 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8800000000000003e-06 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568114+0000", 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012444499999999999 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568250+0000", 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135205 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568390+0000", 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140335 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569250+0000", 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085971599999999995 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572132+0000", 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028820819999999998 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572759+0000", 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062750100000000004 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572810+0000", 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0488000000000002e-05 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572867+0000", 2024-09-21T08:25:30.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6999000000000001e-05 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 116.13399916, 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045637589999999997, 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.715 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568426+0000", 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6470000000000001e-06 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568432+0000", 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.395e-06 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568443+0000", 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1070000000000001e-05 2024-09-21T08:25:30.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568456+0000", 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3862e-05 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568688+0000", 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023171699999999999 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568773+0000", 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5307000000000002e-05 2024-09-21T08:25:30.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568890+0000", 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116082 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569279+0000", 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038986899999999999 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572238+0000", 2024-09-21T08:25:30.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029581009999999999 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572906+0000", 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000668945 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572955+0000", 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8455000000000003e-05 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572988+0000", 2024-09-21T08:25:30.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3309000000000002e-05 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:47 5.1f 5:f95f44c2:::notify.0:head [watch watch cookie 94074441540560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 116.12921808, 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026954729999999999, 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:25:30.721 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 47 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573208+0000", 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9970000000000001e-06 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573213+0000", 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5679999999999999e-06 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573224+0000", 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1270000000000001e-05 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573233+0000", 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8939999999999999e-06 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573770+0000", 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053668100000000001 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573841+0000", 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.135e-05 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.574869+0000", 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010281719999999999 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.574911+0000", 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1631000000000002e-05 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575071+0000", 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015977 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575463+0000", 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039199299999999999 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575737+0000", 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027415300000000002 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575785+0000", 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7970999999999999e-05 2024-09-21T08:25:30.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575832+0000", 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7120999999999998e-05 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575901+0000", 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8901999999999999e-05 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.727 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:192 4.5 4:a2ace8b3:::data_log.0:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 116.08659739399999, 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031080779999999998, 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615827+0000", 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1629999999999999e-06 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615831+0000", 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0790000000000002e-06 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615841+0000", 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8279999999999998e-06 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615847+0000", 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2660000000000001e-06 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615879+0000", 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2512999999999999e-05 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615924+0000", 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4709999999999997e-05 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.618934+0000", 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030105190000000001 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.731 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1084 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94837804042368 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.542806046999999, 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40449157200000002, 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1084 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159622+0000", 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1400000000000002e-06 2024-09-21T08:25:30.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159631+0000", 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9670000000000007e-06 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159655+0000", 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4037999999999999e-05 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159667+0000", 2024-09-21T08:25:30.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2427e-05 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159756+0000", 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9103000000000004e-05 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159772+0000", 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5877000000000001e-05 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.554979+0000", 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39520667999999998 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.555262+0000", 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028320500000000002 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.560260+0000", 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049979630000000002 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.560331+0000", 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1048000000000001e-05 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.564109+0000", 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037781239999999999 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1085 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94837808833664 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:25:30.737 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 52.542666814999997, 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40437364799999997, 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1085 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159758+0000", 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.539e-06 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159764+0000", 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1330000000000004e-06 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159773+0000", 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3250000000000008e-06 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159779+0000", 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5400000000000001e-06 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159844+0000", 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4580999999999997e-05 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159854+0000", 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.893e-06 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552880+0000", 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.393026184 2024-09-21T08:25:30.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.553195+0000", 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031526299999999998 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559454+0000", 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062587809999999997 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559541+0000", 2024-09-21T08:25:30.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7379999999999993e-05 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.564130+0000", 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045890289999999997 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:265 7.b e93/80 7:dd97af7e:::smithi13779221-30:head v 93'26, mlcod=86'23)", 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 44.138671760999998, 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036488330000000002, 2024-09-21T08:25:30.743 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563753+0000", 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4000000000000001e-07 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565430+0000", 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016774439999999999 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565446+0000", 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5540999999999999e-05 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565455+0000", 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9580000000000002e-06 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565513+0000", 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7312000000000003e-05 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565542+0000", 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8856999999999998e-05 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.567363+0000", 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018212300000000001 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.567401+0000", 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7851000000000001e-05 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:321 7.c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 6] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:25:30.747 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.892796396000001, 2024-09-21T08:25:30.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013109218000000001, 2024-09-21T08:25:30.748 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.748 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.748 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:30.748 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.748 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:30.748 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:30.748 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 321 2024-09-21T08:25:30.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.748 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.749 INFO:tasks.rados.rados.0.smithi137.stdout:298: done (0 left) 2024-09-21T08:25:30.749 INFO:tasks.rados.rados.0.smithi137.stdout:299: delete oid 20 current snap is 26 2024-09-21T08:25:30.749 INFO:tasks.rados.rados.0.smithi137.stdout:299: done (0 left) 2024-09-21T08:25:30.749 INFO:tasks.rados.rados.0.smithi137.stdout:300: snap_remove snap 24 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809632+0000", 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9559999999999996e-06 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809648+0000", 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5814000000000001e-05 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809680+0000", 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1732e-05 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809696+0000", 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6293000000000001e-05 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809815+0000", 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119442 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809903+0000", 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7033999999999994e-05 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.815434+0000", 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055318420000000004 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.820591+0000", 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051568229999999996 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.821125+0000", 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053413400000000002 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:30.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822656+0000", 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001531035 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822697+0000", 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0488999999999998e-05 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822736+0000", 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9623999999999999e-05 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:322 7.c 7:3cae00e7:::smithi13779221-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.892698144000001, 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093384190000000006, 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 322 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809727+0000", 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.13e-06 2024-09-21T08:25:30.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809734+0000", 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9569999999999996e-06 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809744+0000", 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0633e-05 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809753+0000", 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7090000000000008e-06 2024-09-21T08:25:30.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.810046+0000", 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029301499999999999 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.810079+0000", 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3293000000000001e-05 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.818920+0000", 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088405299999999992 2024-09-21T08:25:30.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.818982+0000", 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999000000000001e-05 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.819064+0000", 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2152999999999998e-05 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.760 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:329 7.c 7:361ee62b:::smithi13779221-40:head [sparse-read 0~1272916,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:25:30.760 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:25:30.760 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.016245292000001, 2024-09-21T08:25:30.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90863698400000004, 2024-09-21T08:25:30.760 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.760 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.760 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:30.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:30.762 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 329 2024-09-21T08:25:30.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.762 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:25:30.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686182+0000", 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0459999999999999e-06 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686196+0000", 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4097000000000001e-05 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686225+0000", 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8846000000000001e-05 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686240+0000", 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4256e-05 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686362+0000", 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012248200000000001 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686390+0000", 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7546e-05 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.591277+0000", 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90488744700000001 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.591416+0000", 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013935900000000001 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.594815+0000", 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033989049999999998 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:330 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2655473,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.016156353, 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91026091200000003, 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.766 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 330 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686269+0000", 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7060000000000001e-06 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686277+0000", 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4470000000000006e-06 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686294+0000", 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6099999999999998e-05 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686308+0000", 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4516e-05 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686428+0000", 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119834 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686436+0000", 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1450000000000008e-06 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.595025+0000", 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90858932599999997 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.595096+0000", 2024-09-21T08:25:30.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0821999999999994e-05 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596528+0000", 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014320159999999999 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:331 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2655473,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 43.016096861000001, 2024-09-21T08:25:30.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91109795900000001, 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 331 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686328+0000", 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.387e-06 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686336+0000", 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4030000000000001e-06 2024-09-21T08:25:30.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686350+0000", 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4713e-05 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686364+0000", 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3716999999999999e-05 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686455+0000", 2024-09-21T08:25:30.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0737999999999998e-05 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686460+0000", 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1290000000000004e-06 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596614+0000", 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91015405299999996 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596687+0000", 2024-09-21T08:25:30.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2700000000000005e-05 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.597425+0000", 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00073811899999999997 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:338 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.134591921000002, 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038704049999999999, 2024-09-21T08:25:30.776 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 338 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567835+0000", 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0019999999999998e-06 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567843+0000", 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0090000000000007e-06 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567861+0000", 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8748000000000001e-05 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567872+0000", 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0173000000000001e-05 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567975+0000", 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103398 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568085+0000", 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109691 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571702+0000", 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003617384 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:339 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:25:30.780 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.134516822000002, 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054334199999999996, 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 339 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567908+0000", 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-06 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567913+0000", 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.536e-06 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567922+0000", 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0730000000000005e-06 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567928+0000", 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4489999999999996e-06 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571741+0000", 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038125590000000001 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571859+0000", 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118249 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.573340+0000", 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001481472 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:340 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.134318297, 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082399729999999994, 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 340 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568107+0000", 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1850000000000001e-06 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568112+0000", 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8019999999999997e-06 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568122+0000", 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6970000000000005e-06 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568129+0000", 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2119999999999997e-06 2024-09-21T08:25:30.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.575080+0000", 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069511850000000003 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.575168+0000", 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7613000000000003e-05 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.576345+0000", 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001177279 2024-09-21T08:25:30.788 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.7 95/94 [PushReplyOp(7:e1536981:::smithi13779221-32:head)])", 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.045732802000003, 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030062309999999998, 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656692+0000", 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0690000000000001e-06 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656696+0000", 2024-09-21T08:25:30.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3590000000000001e-06 2024-09-21T08:25:30.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656705+0000", 2024-09-21T08:25:30.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5639999999999997e-06 2024-09-21T08:25:30.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:30.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:30.792 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:30.793 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.493235164Z level=info msg="Executing migration" id="create dashboard_version table v1" 2024-09-21T08:25:30.793 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.493660833Z level=info msg="Executing migration" id="add index dashboard_version.dashboard_id" 2024-09-21T08:25:30.793 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.496589718Z level=info msg="Executing migration" id="add unique index dashboard_version.dashboard_id and dashboard_version.version" 2024-09-21T08:25:30.793 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.499383969Z level=info msg="Executing migration" id="Set dashboard version to 1 where 0" 2024-09-21T08:25:30.793 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.502318772Z level=info msg="Executing migration" id="save existing dashboard data in dashboard_version table v1" 2024-09-21T08:25:30.793 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.50499087Z level=info msg="Executing migration" id="alter dashboard_version.data to mediumtext v1" 2024-09-21T08:25:30.793 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.507845606Z level=info msg="Executing migration" id="create team table" 2024-09-21T08:25:30.793 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.511198332Z level=info msg="Executing migration" id="add index team.org_id" 2024-09-21T08:25:30.793 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.513219209Z level=info msg="Executing migration" id="add unique index team_org_id_name" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.51604205Z level=info msg="Executing migration" id="create team member table" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.518704759Z level=info msg="Executing migration" id="add index team_member.org_id" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.52155757Z level=info msg="Executing migration" id="add unique index team_member_org_id_team_id_user_id" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.524189694Z level=info msg="Executing migration" id="add index team_member.team_id" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.527045355Z level=info msg="Executing migration" id="Add column email to team table" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.529697945Z level=info msg="Executing migration" id="Add column external to team_member table" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.532432574Z level=info msg="Executing migration" id="Add column permission to team_member table" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.535190298Z level=info msg="Executing migration" id="create dashboard acl table" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.537681282Z level=info msg="Executing migration" id="add index dashboard_acl_dashboard_id" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.540506749Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_user_id" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.542989267Z level=info msg="Executing migration" id="add unique index dashboard_acl_dashboard_id_team_id" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.543716701Z level=info msg="Executing migration" id="add index dashboard_acl_user_id" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.546449382Z level=info msg="Executing migration" id="add index dashboard_acl_team_id" 2024-09-21T08:25:30.794 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.54728805Z level=info msg="Executing migration" id="add index dashboard_acl_org_id_role" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.548116002Z level=info msg="Executing migration" id="add index dashboard_permission" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.548977059Z level=info msg="Executing migration" id="save default acl rules in dashboard_acl table" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.549410153Z level=info msg="Executing migration" id="delete acl rules for deleted dashboards and folders" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.550253619Z level=info msg="Executing migration" id="create tag table" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.55067598Z level=info msg="Executing migration" id="add index tag.key_value" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.551137965Z level=info msg="Executing migration" id="create login attempt table" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.551924485Z level=info msg="Executing migration" id="add index login_attempt.username" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.55238366Z level=info msg="Executing migration" id="drop index IDX_login_attempt_username - v1" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.553200979Z level=info msg="Executing migration" id="Rename table login_attempt to login_attempt_tmp_qwerty - v1" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.555558902Z level=info msg="Executing migration" id="create login_attempt v2" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.556405733Z level=info msg="Executing migration" id="create index IDX_login_attempt_username - v2" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.556864152Z level=info msg="Executing migration" id="copy login_attempt v1 to v2" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.557666074Z level=info msg="Executing migration" id="drop login_attempt_tmp_qwerty" 2024-09-21T08:25:30.795 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.558692382Z level=info msg="Executing migration" id="create user auth table" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.559232942Z level=info msg="Executing migration" id="create index IDX_user_auth_auth_module_auth_id - v1" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.559688293Z level=info msg="Executing migration" id="alter user_auth.auth_id to length 190" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.560597431Z level=info msg="Executing migration" id="Add OAuth access token to user_auth" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.561609578Z level=info msg="Executing migration" id="Add OAuth refresh token to user_auth" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.56254208Z level=info msg="Executing migration" id="Add OAuth token type to user_auth" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.563429152Z level=info msg="Executing migration" id="Add OAuth expiry to user_auth" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.564508228Z level=info msg="Executing migration" id="Add index to user_id column in user_auth" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.565174847Z level=info msg="Executing migration" id="Add OAuth ID token to user_auth" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.566128992Z level=info msg="Executing migration" id="create server_lock table" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.566911541Z level=info msg="Executing migration" id="add index server_lock.operation_uid" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.567355398Z level=info msg="Executing migration" id="create user auth token table" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.568197294Z level=info msg="Executing migration" id="add unique index user_auth_token.auth_token" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.569047537Z level=info msg="Executing migration" id="add unique index user_auth_token.prev_auth_token" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.569918491Z level=info msg="Executing migration" id="add index user_auth_token.user_id" 2024-09-21T08:25:30.796 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.570347971Z level=info msg="Executing migration" id="Add revoked_at to the user auth token" 2024-09-21T08:25:30.797 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.571246497Z level=info msg="Executing migration" id="create cache_data table" 2024-09-21T08:25:30.797 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.572094618Z level=info msg="Executing migration" id="add unique index cache_data.cache_key" 2024-09-21T08:25:30.797 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.572941795Z level=info msg="Executing migration" id="create short_url table v1" 2024-09-21T08:25:30.797 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.573383591Z level=info msg="Executing migration" id="add index short_url.org_id-uid" 2024-09-21T08:25:30.797 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.574256836Z level=info msg="Executing migration" id="alter table short_url alter column created_by type to bigint" 2024-09-21T08:25:30.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656710+0000", 2024-09-21T08:25:30.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-21T08:25:30.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656749+0000", 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9026999999999998e-05 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.659697+0000", 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029482829999999999 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.c 95/94 [PushReplyOp(7:3cae00e7:::smithi13779221-28:5)])", 2024-09-21T08:25:30.798 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.024298774000002, 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028000960000000002, 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678126+0000", 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.327e-06 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678131+0000", 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4920000000000004e-06 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678141+0000", 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9839999999999996e-06 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678146+0000", 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.446e-06 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678184+0000", 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8259000000000001e-05 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.680925+0000", 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002740588 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.c 95/94 [PushReplyOp(7:3cae00e7:::smithi13779221-28:head)])", 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 42.013995729000001, 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027171439999999999, 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.802 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688429+0000", 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688434+0000", 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999996e-06 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688443+0000", 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4769999999999992e-06 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688448+0000", 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3499999999999996e-06 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688491+0000", 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2216999999999999e-05 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.691145+0000", 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026543550000000002 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.805 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:353 7.c 7:361ee62b:::smithi13779221-40:head [copy-from ver 7] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.800132179000002, 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014212783, 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:30.806 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 353 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902296+0000", 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1509999999999997e-06 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902323+0000", 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7491000000000002e-05 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902342+0000", 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8522999999999999e-05 2024-09-21T08:25:30.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902350+0000", 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8119999999999997e-06 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902448+0000", 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7867000000000003e-05 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902544+0000", 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5612999999999994e-05 2024-09-21T08:25:30.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.907790+0000", 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052465360000000004 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.913037+0000", 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052469129999999998 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:30.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916372+0000", 2024-09-21T08:25:30.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003334914 2024-09-21T08:25:30.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:30.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916404+0000", 2024-09-21T08:25:30.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1327e-05 2024-09-21T08:25:30.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.577100004Z level=info msg="Executing migration" id="delete alert_definition table" 2024-09-21T08:25:30.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.577410387Z level=info msg="Executing migration" id="recreate alert_definition table" 2024-09-21T08:25:30.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.577843766Z level=info msg="Executing migration" id="add index in alert_definition on org_id and title columns" 2024-09-21T08:25:30.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.58071425Z level=info msg="Executing migration" id="add index in alert_definition on org_id and uid columns" 2024-09-21T08:25:30.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.581214284Z level=info msg="Executing migration" id="alter alert_definition table data column to mediumtext in mysql" 2024-09-21T08:25:30.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.581512993Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and title columns" 2024-09-21T08:25:30.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.581930496Z level=info msg="Executing migration" id="drop index in alert_definition on org_id and uid columns" 2024-09-21T08:25:30.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.582465527Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and title columns" 2024-09-21T08:25:30.812 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.582921714Z level=info msg="Executing migration" id="add unique index in alert_definition on org_id and uid columns" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.583376922Z level=info msg="Executing migration" id="Add column paused in alert_definition" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.584485489Z level=info msg="Executing migration" id="drop alert_definition table" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.58492067Z level=info msg="Executing migration" id="delete alert_definition_version table" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.585248032Z level=info msg="Executing migration" id="recreate alert_definition_version table" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.585652872Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_id and version columns" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.586077093Z level=info msg="Executing migration" id="add index in alert_definition_version table on alert_definition_uid and version columns" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.586523593Z level=info msg="Executing migration" id="alter alert_definition_version table data column to mediumtext in mysql" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.586791126Z level=info msg="Executing migration" id="drop alert_definition_version table" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.587272207Z level=info msg="Executing migration" id="create alert_instance table" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.5878009Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, def_uid and current_state columns" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.588290956Z level=info msg="Executing migration" id="add index in alert_instance table on def_org_id, current_state columns" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.588907394Z level=info msg="Executing migration" id="add column current_state_end to alert_instance" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.589857841Z level=info msg="Executing migration" id="remove index def_org_id, def_uid, current_state on alert_instance" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.590306648Z level=info msg="Executing migration" id="remove index def_org_id, current_state on alert_instance" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.590750679Z level=info msg="Executing migration" id="rename def_org_id to rule_org_id in alert_instance" 2024-09-21T08:25:30.813 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.595259788Z level=info msg="Executing migration" id="rename def_uid to rule_uid in alert_instance" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.599774413Z level=info msg="Executing migration" id="add index rule_org_id, rule_uid, current_state on alert_instance" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.600285205Z level=info msg="Executing migration" id="add index rule_org_id, current_state on alert_instance" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.600746106Z level=info msg="Executing migration" id="add current_reason column related to current_state" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.601773253Z level=info msg="Executing migration" id="create alert_rule table" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.602288664Z level=info msg="Executing migration" id="add index in alert_rule on org_id and title columns" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.602771837Z level=info msg="Executing migration" id="add index in alert_rule on org_id and uid columns" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.603271619Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespace_uid, group_uid columns" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.603746501Z level=info msg="Executing migration" id="alter alert_rule table data column to mediumtext in mysql" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.604277288Z level=info msg="Executing migration" id="add column for to alert_rule" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.605423316Z level=info msg="Executing migration" id="add column annotations to alert_rule" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.606781467Z level=info msg="Executing migration" id="add column labels to alert_rule" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.607712686Z level=info msg="Executing migration" id="remove unique index from alert_rule on org_id, title columns" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.608167318Z level=info msg="Executing migration" id="add index in alert_rule on org_id, namespase_uid and title columns" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.608627742Z level=info msg="Executing migration" id="add dashboard_uid column to alert_rule" 2024-09-21T08:25:30.814 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.609596119Z level=info msg="Executing migration" id="add panel_id column to alert_rule" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.610585894Z level=info msg="Executing migration" id="add index in alert_rule on org_id, dashboard_uid and panel_id columns" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.611058918Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.611985363Z level=info msg="Executing migration" id="add is_paused column to alert_rule table" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.612944904Z level=info msg="Executing migration" id="fix is_paused column for alert_rule table" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.613307157Z level=info msg="Executing migration" id="create alert_rule_version table" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.613763603Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_uid and version columns" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.616581033Z level=info msg="Executing migration" id="add index in alert_rule_version table on rule_org_id, rule_namespace_uid and rule_group columns" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.617165354Z level=info msg="Executing migration" id="alter alert_rule_version table data column to mediumtext in mysql" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.617471494Z level=info msg="Executing migration" id="add column for to alert_rule_version" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.61847987Z level=info msg="Executing migration" id="add column annotations to alert_rule_version" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.621357051Z level=info msg="Executing migration" id="add column labels to alert_rule_version" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.622305738Z level=info msg="Executing migration" id="add rule_group_idx column to alert_rule_version" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.623232838Z level=info msg="Executing migration" id="add is_paused column to alert_rule_versions table" 2024-09-21T08:25:30.815 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.624205201Z level=info msg="Executing migration" id="fix is_paused column for alert_rule_version table" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.624501615Z level=info msg="Executing migration" id=create_alert_configuration_table 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.624875748Z level=info msg="Executing migration" id="Add column default in alert_configuration" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.625844968Z level=info msg="Executing migration" id="alert alert_configuration alertmanager_configuration column from TEXT to MEDIUMTEXT if mysql" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.62615242Z level=info msg="Executing migration" id="add column org_id in alert_configuration" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.627121486Z level=info msg="Executing migration" id="add index in alert_configuration table on org_id column" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.630071198Z level=info msg="Executing migration" id="add configuration_hash column to alert_configuration" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.632860833Z level=info msg="Executing migration" id=create_ngalert_configuration_table 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.633277829Z level=info msg="Executing migration" id="add index in ngalert_configuration on org_id column" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.634135255Z level=info msg="Executing migration" id="add column send_alerts_to in ngalert_configuration" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.635085147Z level=info msg="Executing migration" id="create provenance_type table" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.635917451Z level=info msg="Executing migration" id="add index to uniquify (record_key, record_type, org_id) columns" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.636365963Z level=info msg="Executing migration" id="create alert_image table" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.637191553Z level=info msg="Executing migration" id="add unique index on token to alert_image table" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.638076325Z level=info msg="Executing migration" id="support longer URLs in alert_image table" 2024-09-21T08:25:30.816 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.638954202Z level=info msg="Executing migration" id=create_alert_configuration_history_table 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.639456565Z level=info msg="Executing migration" id="drop non-unique orgID index on alert_configuration" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.640290618Z level=info msg="Executing migration" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.640382921Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop unique orgID index on alert_configuration if exists" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.640731822Z level=info msg="Executing migration" id="extract alertmanager configuration history to separate table" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.641162798Z level=info msg="Executing migration" id="add unique index on orgID to alert_configuration" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.64162944Z level=info msg="Executing migration" id="clear migration entry \"remove unified alerting data\"" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.64174855Z level=info msg="Executing migration" id="move dashboard alerts to unified alerting" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.641856305Z level=info msg="alerts found to migrate" alerts=0 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.64227061Z level=info msg="Executing migration" id="create library_element table v1" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.642722713Z level=info msg="Executing migration" id="add index library_element org_id-folder_id-name-kind" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.645483526Z level=info msg="Executing migration" id="create library_element_connection table v1" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.648314066Z level=info msg="Executing migration" id="add index library_element_connection element_id-kind-connection_id" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.651129349Z level=info msg="Executing migration" id="add unique index library_element org_id_uid" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.651624027Z level=info msg="Executing migration" id="increase max description length to 2048" 2024-09-21T08:25:30.817 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.654511478Z level=info msg="Executing migration" id="clone move dashboard alerts to unified alerting" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.654980608Z level=info msg="Executing migration" id="create data_keys table" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.657350523Z level=info msg="Executing migration" id="create secrets table" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.660083447Z level=info msg="Executing migration" id="rename data_keys name column to id" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.665728117Z level=info msg="Executing migration" id="add name column into data_keys" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.668533609Z level=info msg="Executing migration" id="copy data_keys id column values into name" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.671222031Z level=info msg="Executing migration" id="rename data_keys name column to label" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.677044261Z level=info msg="Executing migration" id="rename data_keys id column back to name" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.682909913Z level=info msg="Executing migration" id="create kv_store table v1" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.683367551Z level=info msg="Executing migration" id="add index kv_store.org_id-namespace-key" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.686321772Z level=info msg="Executing migration" id="update dashboard_uid and panel_id from existing annotations" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.68933711Z level=info msg="Executing migration" id="create permission table" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.692440561Z level=info msg="Executing migration" id="add unique index permission.role_id" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.695206353Z level=info msg="Executing migration" id="add unique index role_id_action_scope" 2024-09-21T08:25:30.818 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.697927302Z level=info msg="Executing migration" id="create role table" 2024-09-21T08:25:30.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.700567269Z level=info msg="Executing migration" id="add column display_name" 2024-09-21T08:25:30.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.703408697Z level=info msg="Executing migration" id="add column group_name" 2024-09-21T08:25:30.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.706143018Z level=info msg="Executing migration" id="add index role.org_id" 2024-09-21T08:25:30.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.708608514Z level=info msg="Executing migration" id="add unique index role_org_id_name" 2024-09-21T08:25:30.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.714203631Z level=info msg="Executing migration" id="add index role_org_id_uid" 2024-09-21T08:25:30.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.715085951Z level=info msg="Executing migration" id="create team role table" 2024-09-21T08:25:30.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.715714418Z level=info msg="Executing migration" id="add index team_role.org_id" 2024-09-21T08:25:30.819 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.718347411Z level=info msg="Executing migration" id="add unique index team_role_org_id_team_id_role_id" 2024-09-21T08:25:30.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:30.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916459+0000", 2024-09-21T08:25:30.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4917000000000003e-05 2024-09-21T08:25:30.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916504+0000", 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5719000000000001e-05 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:354 7.c 7:361ee62b:::smithi13779221-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.800051234000001, 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089418770000000009, 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:30.820 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 354 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:30.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902373+0000", 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6499999999999998e-07 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902376+0000", 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.616e-06 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:30.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902402+0000", 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5890000000000001e-05 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902407+0000", 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0019999999999997e-06 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902704+0000", 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029720499999999998 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902740+0000", 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6383e-05 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911116+0000", 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.008375136 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911208+0000", 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2791e-05 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:30.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911314+0000", 2024-09-21T08:25:30.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105989 2024-09-21T08:25:30.825 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.825 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.825 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.825 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:30.825 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:30.825 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:25:30.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:30.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:30.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:30.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-mon[32893]: from='client.44867 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-mon[32893]: osdmap e136: 8 total, 8 up, 5 in 2024-09-21T08:25:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-mon[32893]: pgmap v110: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 184 MiB data, 572 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:25:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:25:31.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.723699697Z level=info msg="Executing migration" id="add index team_role.team_id" 2024-09-21T08:25:31.014 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.724598463Z level=info msg="Executing migration" id="create user role table" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.726660702Z level=info msg="Executing migration" id="add index user_role.org_id" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.729463437Z level=info msg="Executing migration" id="add unique index user_role_org_id_user_id_role_id" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.730345405Z level=info msg="Executing migration" id="add index user_role.user_id" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.733186904Z level=info msg="Executing migration" id="create builtin role table" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.735788713Z level=info msg="Executing migration" id="add index builtin_role.role_id" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.736557693Z level=info msg="Executing migration" id="add index builtin_role.name" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.737073424Z level=info msg="Executing migration" id="Add column org_id to builtin_role table" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.738324558Z level=info msg="Executing migration" id="add index builtin_role.org_id" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.739181056Z level=info msg="Executing migration" id="add unique index builtin_role_org_id_role_id_role" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.739991685Z level=info msg="Executing migration" id="Remove unique index role_org_id_uid" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.740721531Z level=info msg="Executing migration" id="add unique index role.uid" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.741444543Z level=info msg="Executing migration" id="create seed assignment table" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.742263012Z level=info msg="Executing migration" id="add unique index builtin_role_role_name" 2024-09-21T08:25:31.015 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.743107809Z level=info msg="Executing migration" id="add column hidden to role table" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.744329409Z level=info msg="Executing migration" id="create query_history table v1" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.745193555Z level=info msg="Executing migration" id="add index query_history.org_id-created_by-datasource_uid" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.746922722Z level=info msg="Executing migration" id="alter table query_history alter column created_by type to bigint" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.747469165Z level=info msg="Executing migration" id="teams permissions migration" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.748178144Z level=info msg="Executing migration" id="dashboard permissions" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.748883529Z level=info msg="Executing migration" id="dashboard permissions uid scopes" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.74941979Z level=info msg="Executing migration" id="drop managed folder create actions" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.75026637Z level=info msg="Executing migration" id="alerting notification permissions" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.751148765Z level=info msg="Executing migration" id="create query_history_star table v1" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.75158027Z level=info msg="Executing migration" id="add index query_history.user_id-query_uid" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.752532074Z level=info msg="Executing migration" id="add column org_id in query_history_star" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.753778255Z level=info msg="Executing migration" id="alter table query_history_star_mig column user_id type to bigint" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.754659006Z level=info msg="Executing migration" id="create correlation table v1" 2024-09-21T08:25:31.016 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.755133058Z level=info msg="Executing migration" id="add index correlations.uid" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.755962485Z level=info msg="Executing migration" id="add index correlations.source_uid" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.756816129Z level=info msg="Executing migration" id="add correlation config column" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.757970417Z level=info msg="Executing migration" id="create entity_events table" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.758582685Z level=info msg="Executing migration" id="create dashboard public config v1" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.759429053Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.759511466Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index UQE_dashboard_public_config_uid - v1" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.760292941Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.760389938Z level=warn msg="Skipping migration: Already executed, but not recorded in migration log" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.761167583Z level=info msg="Executing migration" id="Drop old dashboard public config table" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.76164318Z level=info msg="Executing migration" id="recreate dashboard public config v1" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.762483572Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v1" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.762962473Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v1" 2024-09-21T08:25:31.017 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.763433531Z level=info msg="Executing migration" id="drop index UQE_dashboard_public_config_uid - v2" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.764316048Z level=info msg="Executing migration" id="drop index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.764764941Z level=info msg="Executing migration" id="Drop public config table" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.767555841Z level=info msg="Executing migration" id="Recreate dashboard public config v2" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.770284001Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_uid - v2" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.773261006Z level=info msg="Executing migration" id="create index IDX_dashboard_public_config_org_id_dashboard_uid - v2" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.773754831Z level=info msg="Executing migration" id="create index UQE_dashboard_public_config_access_token - v2" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.776583433Z level=info msg="Executing migration" id="Rename table dashboard_public_config to dashboard_public - v2" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.780503235Z level=info msg="Executing migration" id="add annotations_enabled column" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.783396299Z level=info msg="Executing migration" id="add time_selection_enabled column" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.785661629Z level=info msg="Executing migration" id="delete orphaned public dashboards" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.788585598Z level=info msg="Executing migration" id="add share column" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.79144417Z level=info msg="Executing migration" id="create default alerting folders" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.794185787Z level=info msg="Executing migration" id="create file table" 2024-09-21T08:25:31.018 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.794628791Z level=info msg="Executing migration" id="file table idx: path natural pk" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.797453244Z level=info msg="Executing migration" id="file table idx: parent_folder_path_hash fast folder retrieval" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.800240211Z level=info msg="Executing migration" id="create file_meta table" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.803152823Z level=info msg="Executing migration" id="file table idx: path key" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.803603935Z level=info msg="Executing migration" id="set path collation in file table" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.806596297Z level=info msg="Executing migration" id="managed permissions migration" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.807100331Z level=info msg="Executing migration" id="managed folder permissions alert actions migration" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.809103964Z level=info msg="Executing migration" id="RBAC action name migrator" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.811803895Z level=info msg="Executing migration" id="Add UID column to playlist" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.812975845Z level=info msg="Executing migration" id="Update uid column values in playlist" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.815634032Z level=info msg="Executing migration" id="Add index for uid in playlist" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.818348485Z level=info msg="Executing migration" id="update group index for alert rules" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.820990161Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated migration" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.821342131Z level=info msg="Executing migration" id="admin only folder/dashboard permission" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.823952612Z level=info msg="Executing migration" id="add action column to seed_assignment" 2024-09-21T08:25:31.019 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.825243935Z level=info msg="Executing migration" id="add scope column to seed_assignment" 2024-09-21T08:25:31.020 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.826800361Z level=info msg="Executing migration" id="remove unique index builtin_role_role_name before nullable update" 2024-09-21T08:25:31.020 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.829219426Z level=info msg="Executing migration" id="update seed_assignment role_name column to nullable" 2024-09-21T08:25:31.020 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.84207748Z level=info msg="Executing migration" id="add unique index builtin_role_name back" 2024-09-21T08:25:31.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:31.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[29634]: from='client.44867 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:31.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[29634]: osdmap e136: 8 total, 8 up, 5 in 2024-09-21T08:25:31.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:31.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[29634]: pgmap v110: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 184 MiB data, 572 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:25:31.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:25:31.021 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.844696007Z level=info msg="Executing migration" id="add unique index builtin_role_action_scope" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.847415336Z level=info msg="Executing migration" id="add primary key to seed_assigment" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.851754205Z level=info msg="Executing migration" id="managed folder permissions alert actions repeated fixed migration" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.852098815Z level=info msg="Executing migration" id="migrate external alertmanagers to datsourcse" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.853112505Z level=info msg="Executing migration" id="create folder table" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.85384265Z level=info msg="Executing migration" id="Add index for parent_uid" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.854693928Z level=info msg="Executing migration" id="Add unique index for folder.uid and folder.org_id" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.857194648Z level=info msg="Executing migration" id="Update folder title length" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.859970177Z level=info msg="Executing migration" id="Add unique index for folder.title and folder.parent_uid" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=migrator t=2024-09-21T08:25:30.862539288Z level=info msg="migrations completed" performed=485 skipped=0 duration=716.646002ms 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=sqlstore t=2024-09-21T08:25:30.862943007Z level=info msg="Created default organization" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=plugin.finder t=2024-09-21T08:25:30.895756114Z level=warn msg="Skipping finding plugins as directory does not exist" path=/usr/share/grafana/plugins-bundled 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=secrets t=2024-09-21T08:25:30.895861194Z level=info msg="Envelope encryption state" enabled=true currentprovider=secretKey.v1 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=query_data t=2024-09-21T08:25:30.897909741Z level=info msg="Query Service initialization" 2024-09-21T08:25:31.022 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:30 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=live.push_http t=2024-09-21T08:25:30.902425096Z level=info msg="Live Push Gateway initialization" 2024-09-21T08:25:31.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[41044]: from='client.44867 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:31.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[41044]: osdmap e136: 8 total, 8 up, 5 in 2024-09-21T08:25:31.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:31.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[41044]: pgmap v110: 148 pgs: 7 active+clean+snaptrim, 7 active+clean+snaptrim_wait, 134 active+clean; 184 MiB data, 572 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:25:31.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:31.023 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:25:31.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:31.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:31.044 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:31.122 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:31.122 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:31.140 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:31.140 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:31.145 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:31.152 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:31.158 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:31.158 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:31.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:31.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:31.169 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:31.176 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:31.177 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:31.193 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:31.194 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:31.209 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:31.210 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:31.227 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:31.227 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:31.244 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:31.244 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:31.270 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:31.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:31.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:31.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:31.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:31.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:31.420 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:31.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:31.496 INFO:tasks.rados.rados.0.smithi137.stdout:300: done (0 left) 2024-09-21T08:25:31.496 INFO:tasks.rados.rados.0.smithi137.stdout:301: rollback oid 31 current snap is 26 2024-09-21T08:25:31.496 INFO:tasks.rados.rados.0.smithi137.stdout:301: done (0 left) 2024-09-21T08:25:31.497 INFO:tasks.rados.rados.0.smithi137.stdout:302: write (excl) oid 3 current snap is 26 2024-09-21T08:25:31.497 INFO:tasks.rados.rados.0.smithi137.stdout:302: seq_num 142 ranges {786594=779712,2326443=60151} 2024-09-21T08:25:31.505 INFO:tasks.rados.rados.0.smithi137.stdout:302: writing smithi13779221-3 from 786594 to 1566306 tid 1 2024-09-21T08:25:31.509 INFO:tasks.rados.rados.0.smithi137.stdout:302: writing smithi13779221-3 from 2326443 to 2386594 tid 2 2024-09-21T08:25:31.509 INFO:tasks.rados.rados.0.smithi137.stdout:303: rollback oid 39 current snap is 26 2024-09-21T08:25:31.510 INFO:tasks.rados.rados.0.smithi137.stdout:303: done (1 left) 2024-09-21T08:25:31.510 INFO:tasks.rados.rados.0.smithi137.stdout:304: write (excl) oid 34 current snap is 26 2024-09-21T08:25:31.510 INFO:tasks.rados.rados.0.smithi137.stdout:304: seq_num 143 ranges {403401=454961,1584925=506039} 2024-09-21T08:25:31.514 INFO:tasks.rados.rados.0.smithi137.stdout:304: writing smithi13779221-34 from 403401 to 858362 tid 1 2024-09-21T08:25:31.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:304: writing smithi13779221-34 from 1584925 to 2090964 tid 2 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:305: read oid 36 snap -1 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:305: expect (ObjNum 121 snap 23 seq_num 121) 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:306: copy_from oid 11 from oid 24 current snap is 26 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:302: finishing write tid 1 to smithi13779221-3 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:302: finishing write tid 2 to smithi13779221-3 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:302: finishing write tid 3 to smithi13779221-3 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:302: finishing write tid 4 to smithi13779221-3 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:302: oid 3 updating version 0 to 49 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:302: oid 3 updating version 49 to 50 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:302: oid 3 updating version 50 to 51 2024-09-21T08:25:31.523 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 51 (ObjNum 142 snap 26 seq_num 142) dirty exists 2024-09-21T08:25:31.524 INFO:tasks.rados.rados.0.smithi137.stdout:302: left oid 3 (ObjNum 142 snap 26 seq_num 142) 2024-09-21T08:25:31.524 INFO:tasks.rados.rados.0.smithi137.stdout:302: done (3 left) 2024-09-21T08:25:31.524 INFO:tasks.rados.rados.0.smithi137.stdout:307: rollback oid 7 current snap is 26 2024-09-21T08:25:31.524 INFO:tasks.rados.rados.0.smithi137.stdout:307: done (3 left) 2024-09-21T08:25:31.524 INFO:tasks.rados.rados.0.smithi137.stdout:308: write (excl) oid 46 current snap is 26 2024-09-21T08:25:31.524 INFO:tasks.rados.rados.0.smithi137.stdout:308: seq_num 144 ranges {420208=530210,1527054=766050} 2024-09-21T08:25:31.527 INFO:tasks.rados.rados.0.smithi137.stdout:308: writing smithi13779221-46 from 420208 to 950418 tid 1 2024-09-21T08:25:31.536 INFO:tasks.rados.rados.0.smithi137.stdout:308: writing smithi13779221-46 from 1527054 to 2293104 tid 2 2024-09-21T08:25:31.536 INFO:tasks.rados.rados.0.smithi137.stdout:309: delete oid 17 current snap is 26 2024-09-21T08:25:31.536 INFO:tasks.rados.rados.0.smithi137.stdout:304: finishing write tid 1 to smithi13779221-34 2024-09-21T08:25:31.536 INFO:tasks.rados.rados.0.smithi137.stdout:304: finishing write tid 2 to smithi13779221-34 2024-09-21T08:25:31.536 INFO:tasks.rados.rados.0.smithi137.stdout:304: finishing write tid 3 to smithi13779221-34 2024-09-21T08:25:31.536 INFO:tasks.rados.rados.0.smithi137.stdout:304: finishing write tid 4 to smithi13779221-34 2024-09-21T08:25:31.536 INFO:tasks.rados.rados.0.smithi137.stdout:304: oid 34 updating version 0 to 161 2024-09-21T08:25:31.536 INFO:tasks.rados.rados.0.smithi137.stdout:304: oid 34 updating version 161 to 162 2024-09-21T08:25:31.536 INFO:tasks.rados.rados.0.smithi137.stdout:304: oid 34 updating version 162 to 163 2024-09-21T08:25:31.536 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 163 (ObjNum 143 snap 26 seq_num 143) dirty exists 2024-09-21T08:25:31.537 INFO:tasks.rados.rados.0.smithi137.stdout:304: left oid 34 (ObjNum 143 snap 26 seq_num 143) 2024-09-21T08:25:31.541 INFO:tasks.rados.rados.0.smithi137.stdout:306: finishing copy_from racing read to smithi13779221-11 2024-09-21T08:25:31.541 INFO:tasks.rados.rados.0.smithi137.stdout:305: expect (ObjNum 121 snap 23 seq_num 121) 2024-09-21T08:25:31.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:31.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:31.545 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:31.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:31.654 INFO:tasks.rados.rados.0.smithi137.stdout:306: finishing copy_from to smithi13779221-11 2024-09-21T08:25:31.654 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 64 (ObjNum 105 snap 21 seq_num 105) dirty exists 2024-09-21T08:25:31.654 INFO:tasks.rados.rados.0.smithi137.stdout:308: finishing write tid 1 to smithi13779221-46 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:308: finishing write tid 2 to smithi13779221-46 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:308: finishing write tid 3 to smithi13779221-46 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:308: finishing write tid 4 to smithi13779221-46 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:308: oid 46 updating version 0 to 140 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:308: oid 46 version 140 is already newer than 138 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:308: oid 46 version 140 is already newer than 139 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 140 (ObjNum 144 snap 26 seq_num 144) dirty exists 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:308: left oid 46 (ObjNum 144 snap 26 seq_num 144) 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:304: done (4 left) 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:305: done (3 left) 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:306: done (2 left) 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:308: done (1 left) 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:309: done (0 left) 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:310: read oid 17 snap -1 2024-09-21T08:25:31.655 INFO:tasks.rados.rados.0.smithi137.stdout:310: expect deleted 2024-09-21T08:25:31.656 INFO:tasks.rados.rados.0.smithi137.stdout:311: rollback oid 13 current snap is 26 2024-09-21T08:25:31.656 INFO:tasks.rados.rados.0.smithi137.stdout:311: done (1 left) 2024-09-21T08:25:31.656 INFO:tasks.rados.rados.0.smithi137.stdout:312: read oid 15 snap -1 2024-09-21T08:25:31.656 INFO:tasks.rados.rados.0.smithi137.stdout:312: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-21T08:25:31.656 INFO:tasks.rados.rados.0.smithi137.stdout:313: write oid 45 current snap is 26 2024-09-21T08:25:31.656 INFO:tasks.rados.rados.0.smithi137.stdout:313: seq_num 146 ranges {453822=797061,1760371=653366} 2024-09-21T08:25:31.661 INFO:tasks.rados.rados.0.smithi137.stdout:313: writing smithi13779221-45 from 453822 to 1250883 tid 1 2024-09-21T08:25:31.668 INFO:tasks.rados.rados.0.smithi137.stdout:313: writing smithi13779221-45 from 1760371 to 2413737 tid 2 2024-09-21T08:25:31.668 INFO:tasks.rados.rados.0.smithi137.stdout:314: read oid 13 snap -1 2024-09-21T08:25:31.668 INFO:tasks.rados.rados.0.smithi137.stdout:314: expect (ObjNum 63 snap 4 seq_num 63) 2024-09-21T08:25:31.668 INFO:tasks.rados.rados.0.smithi137.stdout:315: rollback oid 43 current snap is 26 2024-09-21T08:25:31.668 INFO:tasks.rados.rados.0.smithi137.stdout:315: done (4 left) 2024-09-21T08:25:31.668 INFO:tasks.rados.rados.0.smithi137.stdout:316: read oid 21 snap -1 2024-09-21T08:25:31.668 INFO:tasks.rados.rados.0.smithi137.stdout:316: expect (ObjNum 130 snap 23 seq_num 130) 2024-09-21T08:25:31.669 INFO:tasks.rados.rados.0.smithi137.stdout:317: copy_from oid 42 from oid 29 current snap is 26 2024-09-21T08:25:31.669 INFO:tasks.rados.rados.0.smithi137.stdout:312: expect (ObjNum 129 snap 23 seq_num 129) 2024-09-21T08:25:31.672 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:31.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:31.673 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:31.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T08:25:31.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:31.796 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:31.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:31.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:31.802 INFO:tasks.rados.rados.0.smithi137.stdout:313: finishing write tid 1 to smithi13779221-45 2024-09-21T08:25:31.802 INFO:tasks.rados.rados.0.smithi137.stdout:310: done (5 left) 2024-09-21T08:25:31.802 INFO:tasks.rados.rados.0.smithi137.stdout:312: done (4 left) 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:318: read oid 4 snap -1 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:318: expect (ObjNum 104 snap 20 seq_num 104) 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:319: rollback oid 44 current snap is 26 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:319: done (5 left) 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:320: rollback oid 16 current snap is 26 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:320: done (5 left) 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:321: snap_create 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:313: finishing write tid 2 to smithi13779221-45 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:313: finishing write tid 3 to smithi13779221-45 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:313: finishing write tid 4 to smithi13779221-45 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:313: oid 45 updating version 0 to 86 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:313: oid 45 updating version 86 to 87 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:313: oid 45 version 87 is already newer than 85 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 87 (ObjNum 146 snap 26 seq_num 146) dirty exists 2024-09-21T08:25:31.803 INFO:tasks.rados.rados.0.smithi137.stdout:313: left oid 45 (ObjNum 146 snap 26 seq_num 146) 2024-09-21T08:25:31.804 INFO:tasks.rados.rados.0.smithi137.stdout:316: expect (ObjNum 130 snap 23 seq_num 130) 2024-09-21T08:25:31.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:31.920 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:31.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:31.921 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:31.926 INFO:tasks.rados.rados.0.smithi137.stdout:314: expect (ObjNum 63 snap 4 seq_num 63) 2024-09-21T08:25:31.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:31 smithi137 ceph-mon[29634]: osdmap e137: 8 total, 8 up, 5 in 2024-09-21T08:25:31.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:31 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2211614137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:31.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:31 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:31.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:31 smithi137 ceph-mon[29634]: osdmap e138: 8 total, 8 up, 5 in 2024-09-21T08:25:31.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:31 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:31.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:31 smithi137 ceph-mon[41044]: osdmap e137: 8 total, 8 up, 5 in 2024-09-21T08:25:31.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:31 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2211614137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:31.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:31 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:31.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:31 smithi137 ceph-mon[41044]: osdmap e138: 8 total, 8 up, 5 in 2024-09-21T08:25:31.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:31 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:31.985 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:25:31.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-mon[32893]: osdmap e137: 8 total, 8 up, 5 in 2024-09-21T08:25:31.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2211614137' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:31.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:31.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-mon[32893]: osdmap e138: 8 total, 8 up, 5 in 2024-09-21T08:25:31.986 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:31.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=infra.usagestats.collector t=2024-09-21T08:25:31.558358032Z level=info msg="registering usage stat providers" usageStatsProvidersLen=2 2024-09-21T08:25:31.986 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=provisioning.datasources t=2024-09-21T08:25:31.559428837Z level=info msg="inserting datasource from configuration " name=Dashboard1 uid=P43CA22E17D0F9596 2024-09-21T08:25:31.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=provisioning.datasources t=2024-09-21T08:25:31.570411176Z level=info msg="inserting datasource from configuration " name=Loki uid=P8E80F9AEF21F6940 2024-09-21T08:25:31.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=provisioning.alerting t=2024-09-21T08:25:31.580824879Z level=info msg="starting to provision alerting" 2024-09-21T08:25:31.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=provisioning.alerting t=2024-09-21T08:25:31.580837769Z level=info msg="finished to provision alerting" 2024-09-21T08:25:31.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=ngalert.state.manager t=2024-09-21T08:25:31.580990633Z level=info msg="Warming state cache for startup" 2024-09-21T08:25:31.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=grafanaStorageLogger t=2024-09-21T08:25:31.581083442Z level=info msg="storage starting" 2024-09-21T08:25:31.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=ngalert.state.manager t=2024-09-21T08:25:31.581193432Z level=info msg="State cache has been initialized" states=0 duration=202.432µs 2024-09-21T08:25:31.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=ticker t=2024-09-21T08:25:31.581246329Z level=info msg=starting first_tick=2024-09-21T08:25:40Z 2024-09-21T08:25:31.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=ngalert.multiorg.alertmanager t=2024-09-21T08:25:31.581260991Z level=info msg="starting MultiOrg Alertmanager" 2024-09-21T08:25:31.987 INFO:journalctl@ceph.grafana.a.smithi165.stdout:Sep 21 08:25:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-grafana-a[70707]: logger=http.server t=2024-09-21T08:25:31.582928816Z level=info msg="HTTP Server Listen" address=[::]:3000 protocol=https subUrl= socket= 2024-09-21T08:25:32.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:32.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:32.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:32.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:32.063 INFO:tasks.rados.rados.0.smithi137.stdout:317: finishing copy_from racing read to smithi13779221-42 2024-09-21T08:25:32.063 INFO:tasks.rados.rados.0.smithi137.stdout:317: finishing copy_from to smithi13779221-42 2024-09-21T08:25:32.064 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 111 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-09-21T08:25:32.064 INFO:tasks.rados.rados.0.smithi137.stdout:318: expect (ObjNum 104 snap 20 seq_num 104) 2024-09-21T08:25:32.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:32.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:32.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:32.165 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:32.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:32.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:32.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:32.288 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:32.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:32.399 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:25:32.399 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:25:32.399 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 138, 2024-09-21T08:25:32.399 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "ec8566d6-77f1-11ef-baf6-efdc52797490", 2024-09-21T08:25:32.399 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-21T08:18:21.794737+0000", 2024-09-21T08:25:32.399 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-21T08:25:31.489911+0000", 2024-09-21T08:25:32.399 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-21T08:23:19.473450+0000", 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-21T08:24:55.589261+0000", 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5806080, 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "nodeep-scrub", 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "noscrub", 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T08:25:32.400 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 7, 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "jewel", 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:21:15.037508+0000", 2024-09-21T08:25:32.401 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 1, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 1, 2024-09-21T08:25:32.402 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 1, 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 1, 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 1, 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "122", 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:25:32.403 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:25:32.404 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:25:32.405 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:25:32.406 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 7.8899998664855957, 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 7.8899998664855957, 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3, 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3, 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:28.246186+0000", 2024-09-21T08:25:32.407 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-21T08:25:32.408 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:25:32.409 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 68, 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:25:32.410 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:25:32.411 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:25:32.412 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.6500000953674316, 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.6500000953674316, 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.6699999570846558, 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.6699999570846558, 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:29.209879+0000", 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:25:32.413 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:25:32.414 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "67", 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:25:32.415 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:25:32.416 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:25:32.417 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.4800000190734863, 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.4800000190734863, 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.559999942779541, 2024-09-21T08:25:32.418 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.559999942779541, 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:30.851833+0000", 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:25:32.419 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:25:32.420 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "69", 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:25:32.421 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:25:32.422 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:25:32.423 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.7100000381469727, 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.7100000381469727, 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.3399999141693115, 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.3399999141693115, 2024-09-21T08:25:32.424 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:32.798630+0000", 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:25:32.425 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:25:32.426 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "71", 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:25:32.427 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:25:32.428 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:25:32.429 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9800000190734863, 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9800000190734863, 2024-09-21T08:25:32.430 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:25:32.431 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-21T08:25:32.431 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-21T08:25:32.431 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:25:32.431 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:25:32.431 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:25:32.431 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.431 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:34.858938+0000", 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:25:32.432 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "73", 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:25:32.433 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:25:32.434 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:25:32.435 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.2400000095367432, 2024-09-21T08:25:32.436 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.2400000095367432, 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.4099999666213989, 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.4099999666213989, 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_0", 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:24:33.441067+0000", 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-21T08:25:32.437 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 16, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 16, 2024-09-21T08:25:32.438 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 16, 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 16, 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 16, 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "138", 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:25:32.439 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 27, 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 138, 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:25:32.440 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:25:32.441 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.4800000190734863, 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.4800000190734863, 2024-09-21T08:25:32.442 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.559999942779541, 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.559999942779541, 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977", 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:25:32.443 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 102, 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:25:32.444 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.445 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.446 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/3938795530", 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/3938795530", 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/3938795530", 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/3938795530", 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d", 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:25:32.447 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 102, 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.448 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.449 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.450 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/2743268361", 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/2743268361", 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/2743268361", 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/2743268361", 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d", 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:25:32.451 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 102, 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.453 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1500397557", 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1500397557", 2024-09-21T08:25:32.454 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1500397557", 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1500397557", 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "73f21606-38be-4578-a86b-bb417217c0e0", 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 25, 2024-09-21T08:25:32.455 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 84, 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.456 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:25:32.457 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/617155021", 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/617155021", 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/617155021", 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/617155021", 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae", 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 31, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 80, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:25:32.459 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6800", 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6801", 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.460 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6803", 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.461 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6802", 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6800/70360958", 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6801/70360958", 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6803/70360958", 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6802/70360958", 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd", 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 41, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 102, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.463 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.464 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6804", 2024-09-21T08:25:32.464 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:25:32.464 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.464 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.464 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:25:32.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:32.465 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:32.465 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:32.465 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6805", 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6807", 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.466 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6806", 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6804/1835779380", 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6805/1835779380", 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6807/1835779380", 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6806/1835779380", 2024-09-21T08:25:32.467 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b", 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:25:32.468 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 49, 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 94, 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6808", 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:25:32.469 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6809", 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6811", 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:25:32.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6810", 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6808/3820354208", 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6809/3820354208", 2024-09-21T08:25:32.471 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6811/3820354208", 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6810/3820354208", 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f", 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:25:32.472 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 59, 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 105, 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6812", 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6813", 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.474 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6815", 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6814", 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6812/1933524273", 2024-09-21T08:25:32.475 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6813/1933524273", 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6815/1933524273", 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6814/1933524273", 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:25:32.476 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:24:30.474500+0000", 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:24:44.425059+0000", 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:24:42.900661+0000", 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:25:32.478 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:25:25.742806+0000", 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:25:32.479 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:24:49.747151+0000", 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:24:58.685117+0000", 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:25:15.918748+0000", 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:25:32.481 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:24:37.692569+0000", 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [], 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1478868457": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1738156348": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:25:32.482 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2643910317": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/705906958": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3936900881": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/223603733": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1718301516": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2580000250": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3166961425": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/700535732": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1900990534": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1825289826": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3010106803": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3442439903": "2024-09-22T08:23:59.682278+0000" 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-21T08:25:32.483 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [ 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 13, 2024-09-21T08:25:32.484 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 17, 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 11 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [ 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 26, 2024-09-21T08:25:32.485 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [ 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 15, 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 2 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:25:32.486 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-21T08:25:32.487 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-21T08:25:32.487 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-21T08:25:32.487 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-21T08:25:32.487 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-21T08:25:32.487 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-21T08:25:32.487 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-21T08:25:32.487 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-21T08:25:32.487 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:32.487 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:25:32.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:25:32.488 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:32.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:32.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:32.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:32.588 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:32.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:32.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:32.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:32.712 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:32.733 INFO:tasks.rados.rados.0.smithi137.stdout:313: done (5 left) 2024-09-21T08:25:32.733 INFO:tasks.rados.rados.0.smithi137.stdout:314: done (4 left) 2024-09-21T08:25:32.733 INFO:tasks.rados.rados.0.smithi137.stdout:316: done (3 left) 2024-09-21T08:25:32.733 INFO:tasks.rados.rados.0.smithi137.stdout:317: done (2 left) 2024-09-21T08:25:32.733 INFO:tasks.rados.rados.0.smithi137.stdout:318: done (1 left) 2024-09-21T08:25:32.733 INFO:tasks.rados.rados.0.smithi137.stdout:321: done (0 left) 2024-09-21T08:25:32.733 INFO:tasks.rados.rados.0.smithi137.stdout:322: read oid 12 snap 26 2024-09-21T08:25:32.733 INFO:tasks.rados.rados.0.smithi137.stdout:322: expect deleted 2024-09-21T08:25:32.734 INFO:tasks.rados.rados.0.smithi137.stdout:323: read oid 35 snap -1 2024-09-21T08:25:32.734 INFO:tasks.rados.rados.0.smithi137.stdout:323: expect (ObjNum 139 snap 26 seq_num 139) 2024-09-21T08:25:32.734 INFO:tasks.rados.rados.0.smithi137.stdout:324: read oid 19 snap 26 2024-09-21T08:25:32.734 INFO:tasks.rados.rados.0.smithi137.stdout:324: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-21T08:25:32.734 INFO:tasks.rados.rados.0.smithi137.stdout:325: read oid 24 snap -1 2024-09-21T08:25:32.734 INFO:tasks.rados.rados.0.smithi137.stdout:325: expect (ObjNum 105 snap 21 seq_num 105) 2024-09-21T08:25:32.734 INFO:tasks.rados.rados.0.smithi137.stdout:326: read oid 40 snap -1 2024-09-21T08:25:32.734 INFO:tasks.rados.rados.0.smithi137.stdout:326: expect (ObjNum 133 snap 24 seq_num 133) 2024-09-21T08:25:32.734 INFO:tasks.rados.rados.0.smithi137.stdout:327: write oid 28 current snap is 27 2024-09-21T08:25:32.734 INFO:tasks.rados.rados.0.smithi137.stdout:327: seq_num 147 ranges {470629=472310,1418853=513377} 2024-09-21T08:25:32.739 INFO:tasks.rados.rados.0.smithi137.stdout:327: writing smithi13779221-28 from 470629 to 942939 tid 1 2024-09-21T08:25:32.747 INFO:tasks.rados.rados.0.smithi137.stdout:327: writing smithi13779221-28 from 1418853 to 1932230 tid 2 2024-09-21T08:25:32.747 INFO:tasks.rados.rados.0.smithi137.stdout:328: read oid 26 snap 26 2024-09-21T08:25:32.747 INFO:tasks.rados.rados.0.smithi137.stdout:328: expect deleted 2024-09-21T08:25:32.747 INFO:tasks.rados.rados.0.smithi137.stdout:329: copy_from oid 17 from oid 44 current snap is 27 2024-09-21T08:25:32.747 INFO:tasks.rados.rados.0.smithi137.stdout:322: done (7 left) 2024-09-21T08:25:32.748 INFO:tasks.rados.rados.0.smithi137.stdout:330: write oid 29 current snap is 27 2024-09-21T08:25:32.748 INFO:tasks.rados.rados.0.smithi137.stdout:324: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-21T08:25:32.813 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:32.820 INFO:tasks.rados.rados.0.smithi137.stdout:323: expect (ObjNum 139 snap 26 seq_num 139) 2024-09-21T08:25:32.834 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:32.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:32.835 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:32.849 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:32 smithi165 ceph-mon[32893]: pgmap v113: 148 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 135 active+clean; 139 MiB data, 482 MiB used, 447 GiB / 447 GiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T08:25:32.850 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:32 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/428090741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:25:32.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:32.949 INFO:tasks.rados.rados.0.smithi137.stdout:325: expect (ObjNum 105 snap 21 seq_num 105) 2024-09-21T08:25:32.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:32.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:32.958 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:32.966 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:25:32.966 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:25:33.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:32 smithi137 ceph-mon[29634]: pgmap v113: 148 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 135 active+clean; 139 MiB data, 482 MiB used, 447 GiB / 447 GiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T08:25:33.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:32 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/428090741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:25:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:32 smithi137 ceph-mon[41044]: pgmap v113: 148 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 135 active+clean; 139 MiB data, 482 MiB used, 447 GiB / 447 GiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T08:25:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:32 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/428090741' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:25:33.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:33.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:33.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:33.080 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:33.104 INFO:tasks.rados.rados.0.smithi137.stdout:326: expect (ObjNum 133 snap 24 seq_num 133) 2024-09-21T08:25:33.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:33.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:33.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:33.202 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:33.237 INFO:tasks.rados.rados.0.smithi137.stdout:327: finishing write tid 1 to smithi13779221-28 2024-09-21T08:25:33.237 INFO:tasks.rados.rados.0.smithi137.stdout:327: finishing write tid 2 to smithi13779221-28 2024-09-21T08:25:33.237 INFO:tasks.rados.rados.0.smithi137.stdout:327: finishing write tid 3 to smithi13779221-28 2024-09-21T08:25:33.237 INFO:tasks.rados.rados.0.smithi137.stdout:327: finishing write tid 4 to smithi13779221-28 2024-09-21T08:25:33.238 INFO:tasks.rados.rados.0.smithi137.stdout:327: oid 28 updating version 0 to 63 2024-09-21T08:25:33.238 INFO:tasks.rados.rados.0.smithi137.stdout:327: oid 28 version 63 is already newer than 61 2024-09-21T08:25:33.238 INFO:tasks.rados.rados.0.smithi137.stdout:327: oid 28 version 63 is already newer than 62 2024-09-21T08:25:33.238 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 63 (ObjNum 147 snap 27 seq_num 147) dirty exists 2024-09-21T08:25:33.238 INFO:tasks.rados.rados.0.smithi137.stdout:327: left oid 28 (ObjNum 147 snap 27 seq_num 147) 2024-09-21T08:25:33.238 INFO:tasks.rados.rados.0.smithi137.stdout:329: finishing copy_from racing read to smithi13779221-17 2024-09-21T08:25:33.238 INFO:tasks.rados.rados.0.smithi137.stdout:329: finishing copy_from to smithi13779221-17 2024-09-21T08:25:33.238 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 46 (ObjNum 98 snap 19 seq_num 98) dirty exists 2024-09-21T08:25:33.238 INFO:tasks.rados.rados.0.smithi137.stdout:330: seq_num 148 ranges {487436=547559,1760982=657035} 2024-09-21T08:25:33.243 INFO:tasks.rados.rados.0.smithi137.stdout:330: writing smithi13779221-29 from 487436 to 1034995 tid 1 2024-09-21T08:25:33.250 INFO:tasks.rados.rados.0.smithi137.stdout:330: writing smithi13779221-29 from 1760982 to 2418017 tid 2 2024-09-21T08:25:33.251 INFO:tasks.rados.rados.0.smithi137.stdout:323: done (7 left) 2024-09-21T08:25:33.251 INFO:tasks.rados.rados.0.smithi137.stdout:324: done (6 left) 2024-09-21T08:25:33.251 INFO:tasks.rados.rados.0.smithi137.stdout:325: done (5 left) 2024-09-21T08:25:33.251 INFO:tasks.rados.rados.0.smithi137.stdout:326: done (4 left) 2024-09-21T08:25:33.251 INFO:tasks.rados.rados.0.smithi137.stdout:327: done (3 left) 2024-09-21T08:25:33.251 INFO:tasks.rados.rados.0.smithi137.stdout:328: done (2 left) 2024-09-21T08:25:33.251 INFO:tasks.rados.rados.0.smithi137.stdout:329: done (1 left) 2024-09-21T08:25:33.251 INFO:tasks.rados.rados.0.smithi137.stdout:331: write (excl) oid 31 current snap is 27 2024-09-21T08:25:33.251 INFO:tasks.rados.rados.0.smithi137.stdout:330: finishing write tid 1 to smithi13779221-29 2024-09-21T08:25:33.251 INFO:tasks.rados.rados.0.smithi137.stdout:331: seq_num 149 ranges {504243=622808,1819464=517046} 2024-09-21T08:25:33.256 INFO:tasks.rados.rados.0.smithi137.stdout:331: writing smithi13779221-31 from 504243 to 1127051 tid 1 2024-09-21T08:25:33.264 INFO:tasks.rados.rados.0.smithi137.stdout:331: writing smithi13779221-31 from 1819464 to 2336510 tid 2 2024-09-21T08:25:33.264 INFO:tasks.rados.rados.0.smithi137.stdout:332: read oid 3 snap 26 2024-09-21T08:25:33.264 INFO:tasks.rados.rados.0.smithi137.stdout:332: expect (ObjNum 142 snap 26 seq_num 142) 2024-09-21T08:25:33.264 INFO:tasks.rados.rados.0.smithi137.stdout:333: rollback oid 49 current snap is 27 2024-09-21T08:25:33.264 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 49 to 26 2024-09-21T08:25:33.264 INFO:tasks.rados.rados.0.smithi137.stdout:330: finishing write tid 2 to smithi13779221-29 2024-09-21T08:25:33.264 INFO:tasks.rados.rados.0.smithi137.stdout:330: finishing write tid 3 to smithi13779221-29 2024-09-21T08:25:33.264 INFO:tasks.rados.rados.0.smithi137.stdout:330: finishing write tid 4 to smithi13779221-29 2024-09-21T08:25:33.264 INFO:tasks.rados.rados.0.smithi137.stdout:330: oid 29 updating version 0 to 143 2024-09-21T08:25:33.264 INFO:tasks.rados.rados.0.smithi137.stdout:330: oid 29 version 143 is already newer than 142 2024-09-21T08:25:33.265 INFO:tasks.rados.rados.0.smithi137.stdout:330: oid 29 updating version 143 to 144 2024-09-21T08:25:33.265 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 144 (ObjNum 148 snap 27 seq_num 148) dirty exists 2024-09-21T08:25:33.265 INFO:tasks.rados.rados.0.smithi137.stdout:330: left oid 29 (ObjNum 148 snap 27 seq_num 148) 2024-09-21T08:25:33.265 INFO:tasks.rados.rados.0.smithi137.stdout:331: finishing write tid 1 to smithi13779221-31 2024-09-21T08:25:33.265 INFO:tasks.rados.rados.0.smithi137.stdout:330: done (3 left) 2024-09-21T08:25:33.265 INFO:tasks.rados.rados.0.smithi137.stdout:334: copy_from oid 37 from oid 43 current snap is 27 2024-09-21T08:25:33.265 INFO:tasks.rados.rados.0.smithi137.stdout:335: write (excl) oid 23 current snap is 27 2024-09-21T08:25:33.265 INFO:tasks.rados.rados.0.smithi137.stdout:335: seq_num 150 ranges {521050=698057,1877946=643104} 2024-09-21T08:25:33.270 INFO:tasks.rados.rados.0.smithi137.stdout:335: writing smithi13779221-23 from 521050 to 1219107 tid 1 2024-09-21T08:25:33.278 INFO:tasks.rados.rados.0.smithi137.stdout:335: writing smithi13779221-23 from 1877946 to 2521050 tid 2 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:336: read oid 1 snap -1 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:336: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:337: rollback oid 47 current snap is 27 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 47 to 26 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:333: finishing rollback tid 1 to smithi13779221-49 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:334: finishing copy_from to smithi13779221-37 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:334: got expected ENOENT (src dne) 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:334: finishing copy_from racing read to smithi13779221-37 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:331: finishing write tid 2 to smithi13779221-31 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:331: finishing write tid 3 to smithi13779221-31 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:331: finishing write tid 4 to smithi13779221-31 2024-09-21T08:25:33.279 INFO:tasks.rados.rados.0.smithi137.stdout:331: oid 31 updating version 0 to 91 2024-09-21T08:25:33.280 INFO:tasks.rados.rados.0.smithi137.stdout:331: oid 31 version 91 is already newer than 90 2024-09-21T08:25:33.280 INFO:tasks.rados.rados.0.smithi137.stdout:331: oid 31 version 91 is already newer than 89 2024-09-21T08:25:33.280 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 91 (ObjNum 149 snap 27 seq_num 149) dirty exists 2024-09-21T08:25:33.280 INFO:tasks.rados.rados.0.smithi137.stdout:331: left oid 31 (ObjNum 149 snap 27 seq_num 149) 2024-09-21T08:25:33.280 INFO:tasks.rados.rados.0.smithi137.stdout:332: expect (ObjNum 142 snap 26 seq_num 142) 2024-09-21T08:25:33.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:33.325 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:33.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:33.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:33.386 INFO:tasks.thrashosds.thrasher:j is {'epoch': 138, 'fsid': 'ec8566d6-77f1-11ef-baf6-efdc52797490', 'created': '2024-09-21T08:18:21.794737+0000', 'modified': '2024-09-21T08:25:31.489911+0000', 'last_up_change': '2024-09-21T08:23:19.473450+0000', 'last_in_change': '2024-09-21T08:24:55.589261+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'jewel', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T08:21:15.037508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '122', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T08:23:28.246186+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T08:23:29.209879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 2.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T08:23:30.851833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.7100000381469727, 'score_stable': 3.7100000381469727, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.3399999141693115, 'raw_score_stable': 2.3399999141693115, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T08:23:32.798630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9800000190734863, 'score_stable': 1.9800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T08:23:34.858938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.240000009536743, 'score_stable': 2.240000009536743, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.409999966621399, 'raw_score_stable': 1.409999966621399, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-21T08:24:33.441067+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 16, 'pg_placement_num': 16, 'pg_placement_num_target': 16, 'pg_num_target': 16, 'pg_num_pending': 16, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '138', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 27, 'snap_epoch': 138, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.4800000190734863, 'score_stable': 2.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.559999942779541, 'raw_score_stable': 1.559999942779541, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '1b007d1d-6c6b-4a68-9e68-2c5fd7d26977', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 3938795530}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 3938795530}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 3938795530}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 3938795530}]}, 'public_addr': '172.21.15.137:6801/3938795530', 'cluster_addr': '172.21.15.137:6802/3938795530', 'heartbeat_back_addr': '172.21.15.137:6804/3938795530', 'heartbeat_front_addr': '172.21.15.137:6803/3938795530', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8651068e-694f-4bd4-9917-5c55670aca5d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 2743268361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 2743268361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 2743268361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 2743268361}]}, 'public_addr': '172.21.15.137:6805/2743268361', 'cluster_addr': '172.21.15.137:6806/2743268361', 'heartbeat_back_addr': '172.21.15.137:6808/2743268361', 'heartbeat_front_addr': '172.21.15.137:6807/2743268361', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2a22f0f7-c610-4995-98a7-d4785efb1e5d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1500397557}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1500397557}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1500397557}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1500397557}]}, 'public_addr': '172.21.15.137:6809/1500397557', 'cluster_addr': '172.21.15.137:6810/1500397557', 'heartbeat_back_addr': '172.21.15.137:6812/1500397557', 'heartbeat_front_addr': '172.21.15.137:6811/1500397557', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '73f21606-38be-4578-a86b-bb417217c0e0', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 84, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 617155021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 617155021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 617155021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 617155021}]}, 'public_addr': '172.21.15.137:6813/617155021', 'cluster_addr': '172.21.15.137:6814/617155021', 'heartbeat_back_addr': '172.21.15.137:6816/617155021', 'heartbeat_front_addr': '172.21.15.137:6815/617155021', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e5f6762b-65f5-4dc7-8e54-9087149ae6ae', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 70360958}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 70360958}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 70360958}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 70360958}]}, 'public_addr': '172.21.15.165:6800/70360958', 'cluster_addr': '172.21.15.165:6801/70360958', 'heartbeat_back_addr': '172.21.15.165:6803/70360958', 'heartbeat_front_addr': '172.21.15.165:6802/70360958', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f7f8c456-b77f-4435-a860-202e64dda8dd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 1835779380}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 1835779380}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 1835779380}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 1835779380}]}, 'public_addr': '172.21.15.165:6804/1835779380', 'cluster_addr': '172.21.15.165:6805/1835779380', 'heartbeat_back_addr': '172.21.15.165:6807/1835779380', 'heartbeat_front_addr': '172.21.15.165:6806/1835779380', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b8b3df16-73fa-41a5-a93f-117c599d488b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 3820354208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 3820354208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 3820354208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 3820354208}]}, 'public_addr': '172.21.15.165:6808/3820354208', 'cluster_addr': '172.21.15.165:6809/3820354208', 'heartbeat_back_addr': '172.21.15.165:6811/3820354208', 'heartbeat_front_addr': '172.21.15.165:6810/3820354208', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ddf234e4-1494-40bd-a31b-333b01717c3f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 105, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 1933524273}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 1933524273}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 1933524273}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 1933524273}]}, 'public_addr': '172.21.15.165:6812/1933524273', 'cluster_addr': '172.21.15.165:6813/1933524273', 'heartbeat_back_addr': '172.21.15.165:6815/1933524273', 'heartbeat_front_addr': '172.21.15.165:6814/1933524273', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:24:30.474500+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:24:44.425059+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:24:42.900661+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:25:25.742806+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:24:49.747151+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:24:58.685117+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:25:15.918748+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:24:37.692569+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:0/1478868457': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/1738156348': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/2643910317': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/705906958': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/3936900881': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/223603733': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/1718301516': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:6800/2580000250': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/3166961425': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/700535732': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:6800/1900990534': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:6800/1825289826': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/3010106803': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/3442439903': '2024-09-22T08:23:59.682278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 13, 'length': 1}, {'begin': 17, 'length': 11}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 26, 'length': 1}]}], 'new_purged_snaps': [{'pool': 7, 'snaps': [{'begin': 15, 'length': 2}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T08:25:33.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:25:33.396 INFO:tasks.rados.rados.0.smithi137.stdout:335: finishing write tid 1 to smithi13779221-23 2024-09-21T08:25:33.396 INFO:tasks.rados.rados.0.smithi137.stdout:331: done (6 left) 2024-09-21T08:25:33.396 INFO:tasks.rados.rados.0.smithi137.stdout:332: done (5 left) 2024-09-21T08:25:33.396 INFO:tasks.rados.rados.0.smithi137.stdout:333: done (4 left) 2024-09-21T08:25:33.396 INFO:tasks.rados.rados.0.smithi137.stdout:334: done (3 left) 2024-09-21T08:25:33.396 INFO:tasks.rados.rados.0.smithi137.stdout:338: snap_create 2024-09-21T08:25:33.396 INFO:tasks.rados.rados.0.smithi137.stdout:337: finishing rollback tid 0 to smithi13779221-47 2024-09-21T08:25:33.396 INFO:tasks.rados.rados.0.smithi137.stdout:337: finishing rollback tid 1 to smithi13779221-47 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:337: finishing rollback tid 2 to smithi13779221-47 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 168 (ObjNum 62 snap 4 seq_num 62) dirty exists 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:335: finishing write tid 2 to smithi13779221-23 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:335: finishing write tid 3 to smithi13779221-23 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:335: finishing write tid 4 to smithi13779221-23 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:335: oid 23 updating version 0 to 55 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:335: oid 23 version 55 is already newer than 53 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:335: oid 23 version 55 is already newer than 54 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 55 (ObjNum 150 snap 27 seq_num 150) dirty exists 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:335: left oid 23 (ObjNum 150 snap 27 seq_num 150) 2024-09-21T08:25:33.397 INFO:tasks.rados.rados.0.smithi137.stdout:336: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-21T08:25:33.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:33.448 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:33.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:33.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:33.504 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:33.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:33.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:33.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:33.574 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:33.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T08:25:33.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:33.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:33.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:33.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:33.731 INFO:tasks.rados.rados.0.smithi137.stdout:335: done (3 left) 2024-09-21T08:25:33.731 INFO:tasks.rados.rados.0.smithi137.stdout:336: done (2 left) 2024-09-21T08:25:33.731 INFO:tasks.rados.rados.0.smithi137.stdout:337: done (1 left) 2024-09-21T08:25:33.731 INFO:tasks.rados.rados.0.smithi137.stdout:338: done (0 left) 2024-09-21T08:25:33.732 INFO:tasks.rados.rados.0.smithi137.stdout:339: rollback oid 7 current snap is 28 2024-09-21T08:25:33.732 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 7 to 27 2024-09-21T08:25:33.732 INFO:tasks.rados.rados.0.smithi137.stdout:340: write oid 13 current snap is 28 2024-09-21T08:25:33.732 INFO:tasks.rados.rados.0.smithi137.stdout:340: seq_num 151 ranges {537857=773306,1936428=520715} 2024-09-21T08:25:33.739 INFO:tasks.rados.rados.0.smithi137.stdout:340: writing smithi13779221-13 from 537857 to 1311163 tid 1 2024-09-21T08:25:33.746 INFO:tasks.rados.rados.0.smithi137.stdout:340: writing smithi13779221-13 from 1936428 to 2457143 tid 2 2024-09-21T08:25:33.746 INFO:tasks.rados.rados.0.smithi137.stdout:341: copy_from oid 35 from oid 6 current snap is 28 2024-09-21T08:25:33.746 INFO:tasks.rados.rados.0.smithi137.stdout:339: finishing rollback tid 0 to smithi13779221-7 2024-09-21T08:25:33.746 INFO:tasks.rados.rados.0.smithi137.stdout:339: finishing rollback tid 1 to smithi13779221-7 2024-09-21T08:25:33.747 INFO:tasks.rados.rados.0.smithi137.stdout:339: finishing rollback tid 2 to smithi13779221-7 2024-09-21T08:25:33.747 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 148 (ObjNum 43 snap 0 seq_num 43) dirty exists 2024-09-21T08:25:33.747 INFO:tasks.rados.rados.0.smithi137.stdout:340: finishing write tid 1 to smithi13779221-13 2024-09-21T08:25:33.747 INFO:tasks.rados.rados.0.smithi137.stdout:339: done (2 left) 2024-09-21T08:25:33.747 INFO:tasks.rados.rados.0.smithi137.stdout:342: snap_create 2024-09-21T08:25:33.749 INFO:tasks.rados.rados.0.smithi137.stdout:340: finishing write tid 2 to smithi13779221-13 2024-09-21T08:25:33.749 INFO:tasks.rados.rados.0.smithi137.stdout:340: finishing write tid 3 to smithi13779221-13 2024-09-21T08:25:33.749 INFO:tasks.rados.rados.0.smithi137.stdout:340: finishing write tid 4 to smithi13779221-13 2024-09-21T08:25:33.749 INFO:tasks.rados.rados.0.smithi137.stdout:340: oid 13 updating version 0 to 151 2024-09-21T08:25:33.749 INFO:tasks.rados.rados.0.smithi137.stdout:340: oid 13 version 151 is already newer than 150 2024-09-21T08:25:33.749 INFO:tasks.rados.rados.0.smithi137.stdout:340: oid 13 updating version 151 to 152 2024-09-21T08:25:33.749 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 152 (ObjNum 151 snap 28 seq_num 151) dirty exists 2024-09-21T08:25:33.749 INFO:tasks.rados.rados.0.smithi137.stdout:340: left oid 13 (ObjNum 151 snap 28 seq_num 151) 2024-09-21T08:25:33.757 INFO:tasks.rados.rados.0.smithi137.stdout:341: finishing copy_from racing read to smithi13779221-35 2024-09-21T08:25:33.761 INFO:tasks.rados.rados.0.smithi137.stdout:341: finishing copy_from to smithi13779221-35 2024-09-21T08:25:33.761 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 13 (ObjNum 62 snap 4 seq_num 62) dirty exists 2024-09-21T08:25:33.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:33.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:33.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:33.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:33.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:33 smithi137 ceph-mon[29634]: osdmap e139: 8 total, 8 up, 5 in 2024-09-21T08:25:33.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:33 smithi137 ceph-mon[41044]: osdmap e139: 8 total, 8 up, 5 in 2024-09-21T08:25:33.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:33.942 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:33.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:33.943 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:33.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:25:34.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:33 smithi165 ceph-mon[32893]: osdmap e139: 8 total, 8 up, 5 in 2024-09-21T08:25:34.013 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:34.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:34.066 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:34.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:34.067 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:34.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:34.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:34.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:34.193 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:34.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:34.316 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:34.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:34.316 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:34.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:34.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:34.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:34.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:34.487 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:34.536 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:34.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:34.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:34.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:34.561 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:34.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:34.684 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:34.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:34.685 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:34.689 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:25:34.691 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:25:34.740 INFO:tasks.rados.rados.0.smithi137.stdout:340: done (2 left) 2024-09-21T08:25:34.740 INFO:tasks.rados.rados.0.smithi137.stdout:341: done (1 left) 2024-09-21T08:25:34.740 INFO:tasks.rados.rados.0.smithi137.stdout:342: done (0 left) 2024-09-21T08:25:34.740 INFO:tasks.rados.rados.0.smithi137.stdout:343: delete oid 45 current snap is 29 2024-09-21T08:25:34.740 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:25:34.741 INFO:tasks.rados.rados.0.smithi137.stdout:343: done (0 left) 2024-09-21T08:25:34.741 INFO:tasks.rados.rados.0.smithi137.stdout:344: write oid 19 current snap is 29 2024-09-21T08:25:34.742 INFO:tasks.rados.rados.0.smithi137.stdout:344: seq_num 153 ranges {571471=640157,1653392=524384} 2024-09-21T08:25:34.748 INFO:tasks.rados.rados.0.smithi137.stdout:344: writing smithi13779221-19 from 571471 to 1211628 tid 1 2024-09-21T08:25:34.752 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:34 smithi137 ceph-mon[41044]: osdmap e140: 8 total, 8 up, 5 in 2024-09-21T08:25:34.752 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:34 smithi137 ceph-mon[41044]: pgmap v116: 148 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 135 active+clean; 139 MiB data, 482 MiB used, 447 GiB / 447 GiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T08:25:34.752 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/139836824' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:25:34.752 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:34 smithi137 ceph-mon[41044]: from='client.44885 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:25:34.752 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:34 smithi137 ceph-mon[29634]: osdmap e140: 8 total, 8 up, 5 in 2024-09-21T08:25:34.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:34 smithi137 ceph-mon[29634]: pgmap v116: 148 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 135 active+clean; 139 MiB data, 482 MiB used, 447 GiB / 447 GiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T08:25:34.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/139836824' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:25:34.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:34 smithi137 ceph-mon[29634]: from='client.44885 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:25:34.754 INFO:tasks.rados.rados.0.smithi137.stdout:344: writing smithi13779221-19 from 1653392 to 2177776 tid 2 2024-09-21T08:25:34.754 INFO:tasks.rados.rados.0.smithi137.stdout:345: snap_create 2024-09-21T08:25:34.757 INFO:tasks.rados.rados.0.smithi137.stdout:344: finishing write tid 1 to smithi13779221-19 2024-09-21T08:25:34.758 INFO:tasks.rados.rados.0.smithi137.stdout:344: finishing write tid 2 to smithi13779221-19 2024-09-21T08:25:34.758 INFO:tasks.rados.rados.0.smithi137.stdout:344: finishing write tid 3 to smithi13779221-19 2024-09-21T08:25:34.758 INFO:tasks.rados.rados.0.smithi137.stdout:344: finishing write tid 4 to smithi13779221-19 2024-09-21T08:25:34.758 INFO:tasks.rados.rados.0.smithi137.stdout:344: oid 19 updating version 0 to 48 2024-09-21T08:25:34.758 INFO:tasks.rados.rados.0.smithi137.stdout:344: oid 19 updating version 48 to 50 2024-09-21T08:25:34.758 INFO:tasks.rados.rados.0.smithi137.stdout:344: oid 19 version 50 is already newer than 49 2024-09-21T08:25:34.758 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 50 (ObjNum 153 snap 29 seq_num 153) dirty exists 2024-09-21T08:25:34.758 INFO:tasks.rados.rados.0.smithi137.stdout:344: left oid 19 (ObjNum 153 snap 29 seq_num 153) 2024-09-21T08:25:34.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:34.807 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:34.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:34.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:34.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:34.920 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:34 smithi165 ceph-mon[32893]: osdmap e140: 8 total, 8 up, 5 in 2024-09-21T08:25:34.920 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:34 smithi165 ceph-mon[32893]: pgmap v116: 148 pgs: 7 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 135 active+clean; 139 MiB data, 482 MiB used, 447 GiB / 447 GiB avail; 8.1 MiB/s rd, 1.6 MiB/s wr, 17 op/s 2024-09-21T08:25:34.920 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/139836824' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:25:34.920 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:34 smithi165 ceph-mon[32893]: from='client.44885 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:25:34.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:34.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:35.032 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:35.053 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:35.053 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:25:35.053 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:25:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:35.054 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:35.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:35.176 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:35.176 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:35.177 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:35.277 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:35.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:35.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:35.300 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:35.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:35.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:35.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:35.423 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:35.524 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:35.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:35.547 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:35.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T08:25:35.615 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2024-09-21T08:25:34.053589+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":145966802,"num_objects":259,"num_object_clones":17,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":5,"num_read":1607,"num_read_kb":456517,"num_write":1192,"num_write_kb":214913,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":312,"num_bytes_recovered":69697317,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1542,"ondisk_log_size":1542,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":493992,"kb_used_data":333860,"kb_used_omap":7,"kb_used_meta":159992,"kb_avail":468231768,"statfs":{"total":479975178240,"available":479469330432,"internally_reserved":0,"allocated":341872640,"data_stored":375443363,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7970,"internal_metadata":163832030},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-47287154,"num_objects":-23,"num_object_clones":-19,"num_object_copies":-69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-23,"num_whiteouts":-2,"num_read":44,"num_read_kb":33213,"num_write":26,"num_write_kb":6615,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000793"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.713753+0000","last_change":"2024-09-21T08:23:30.549696+0000","last_active":"2024-09-21T08:25:29.713753+0000","last_peered":"2024-09-21T08:25:29.713753+0000","last_clean":"2024-09-21T08:25:29.713753+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:25:29.713753+0000","last_undegraded":"2024-09-21T08:25:29.713753+0000","last_fullsized":"2024-09-21T08:25:29.713753+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":123,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.709908+0000","last_change":"2024-09-21T08:24:56.570163+0000","last_active":"2024-09-21T08:25:29.709908+0000","last_peered":"2024-09-21T08:25:29.709908+0000","last_clean":"2024-09-21T08:25:29.709908+0000","last_became_active":"2024-09-21T08:24:56.524489+0000","last_became_peered":"2024-09-21T08:24:56.524489+0000","last_unstale":"2024-09-21T08:25:29.709908+0000","last_undegraded":"2024-09-21T08:25:29.709908+0000","last_fullsized":"2024-09-21T08:25:29.709908+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:24:50.572709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.572709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:06.339471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":142,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710545+0000","last_change":"2024-09-21T08:23:34.569082+0000","last_active":"2024-09-21T08:25:29.710545+0000","last_peered":"2024-09-21T08:25:29.710545+0000","last_clean":"2024-09-21T08:25:29.710545+0000","last_became_active":"2024-09-21T08:23:34.568762+0000","last_became_peered":"2024-09-21T08:23:34.568762+0000","last_unstale":"2024-09-21T08:25:29.710545+0000","last_undegraded":"2024-09-21T08:25:29.710545+0000","last_fullsized":"2024-09-21T08:25:29.710545+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436834+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":115,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.701887+0000","last_change":"2024-09-21T08:24:38.552223+0000","last_active":"2024-09-21T08:25:29.701887+0000","last_peered":"2024-09-21T08:25:29.701887+0000","last_clean":"2024-09-21T08:25:29.701887+0000","last_became_active":"2024-09-21T08:24:38.551819+0000","last_became_peered":"2024-09-21T08:24:38.551819+0000","last_unstale":"2024-09-21T08:25:29.701887+0000","last_undegraded":"2024-09-21T08:25:29.701887+0000","last_fullsized":"2024-09-21T08:25:29.701887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703278+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":99,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721276+0000","last_change":"2024-09-21T08:24:56.514713+0000","last_active":"2024-09-21T08:25:29.721276+0000","last_peered":"2024-09-21T08:25:29.721276+0000","last_clean":"2024-09-21T08:25:29.721276+0000","last_became_active":"2024-09-21T08:24:56.514170+0000","last_became_peered":"2024-09-21T08:24:56.514170+0000","last_unstale":"2024-09-21T08:25:29.721276+0000","last_undegraded":"2024-09-21T08:25:29.721276+0000","last_fullsized":"2024-09-21T08:25:29.721276+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":89,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.715723+0000","last_change":"2024-09-21T08:25:19.906990+0000","last_active":"2024-09-21T08:25:29.715723+0000","last_peered":"2024-09-21T08:25:29.715723+0000","last_clean":"2024-09-21T08:25:29.715723+0000","last_became_active":"2024-09-21T08:24:56.516951+0000","last_became_peered":"2024-09-21T08:24:56.516951+0000","last_unstale":"2024-09-21T08:25:29.715723+0000","last_undegraded":"2024-09-21T08:25:29.715723+0000","last_fullsized":"2024-09-21T08:25:29.715723+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:19.906906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:19.906906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.680233+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,7],"acting":[2,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.711758+0000","last_change":"2024-09-21T08:25:20.942089+0000","last_active":"2024-09-21T08:25:29.711758+0000","last_peered":"2024-09-21T08:25:29.711758+0000","last_clean":"2024-09-21T08:25:29.711758+0000","last_became_active":"2024-09-21T08:24:48.594855+0000","last_became_peered":"2024-09-21T08:24:48.594855+0000","last_unstale":"2024-09-21T08:25:29.711758+0000","last_undegraded":"2024-09-21T08:25:29.711758+0000","last_fullsized":"2024-09-21T08:25:29.711758+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:25:20.941990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:20.941990+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:29.954407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":80,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.709748+0000","last_change":"2024-09-21T08:24:56.519439+0000","last_active":"2024-09-21T08:25:29.709748+0000","last_peered":"2024-09-21T08:25:29.709748+0000","last_clean":"2024-09-21T08:25:29.709748+0000","last_became_active":"2024-09-21T08:24:56.517228+0000","last_became_peered":"2024-09-21T08:24:56.517228+0000","last_unstale":"2024-09-21T08:25:29.709748+0000","last_undegraded":"2024-09-21T08:25:29.709748+0000","last_fullsized":"2024-09-21T08:25:29.709748+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.640831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.640831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:16.137464+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":163,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702904+0000","last_change":"2024-09-21T08:24:56.510104+0000","last_active":"2024-09-21T08:25:29.702904+0000","last_peered":"2024-09-21T08:25:29.702904+0000","last_clean":"2024-09-21T08:25:29.702904+0000","last_became_active":"2024-09-21T08:24:56.509786+0000","last_became_peered":"2024-09-21T08:24:56.509786+0000","last_unstale":"2024-09-21T08:25:29.702904+0000","last_undegraded":"2024-09-21T08:25:29.702904+0000","last_fullsized":"2024-09-21T08:25:29.702904+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":154,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710772+0000","last_change":"2024-09-21T08:24:56.523774+0000","last_active":"2024-09-21T08:25:29.710772+0000","last_peered":"2024-09-21T08:25:29.710772+0000","last_clean":"2024-09-21T08:25:29.710772+0000","last_became_active":"2024-09-21T08:24:56.500171+0000","last_became_peered":"2024-09-21T08:24:56.500171+0000","last_unstale":"2024-09-21T08:25:29.710772+0000","last_undegraded":"2024-09-21T08:25:29.710772+0000","last_fullsized":"2024-09-21T08:25:29.710772+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":151,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702489+0000","last_change":"2024-09-21T08:25:22.581281+0000","last_active":"2024-09-21T08:25:29.702489+0000","last_peered":"2024-09-21T08:25:29.702489+0000","last_clean":"2024-09-21T08:25:29.702489+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:25:29.702489+0000","last_undegraded":"2024-09-21T08:25:29.702489+0000","last_fullsized":"2024-09-21T08:25:29.702489+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:33.300785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":138,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.711301+0000","last_change":"2024-09-21T08:23:36.582491+0000","last_active":"2024-09-21T08:25:29.711301+0000","last_peered":"2024-09-21T08:25:29.711301+0000","last_clean":"2024-09-21T08:25:29.711301+0000","last_became_active":"2024-09-21T08:23:36.581720+0000","last_became_peered":"2024-09-21T08:23:36.581720+0000","last_unstale":"2024-09-21T08:25:29.711301+0000","last_undegraded":"2024-09-21T08:25:29.711301+0000","last_fullsized":"2024-09-21T08:25:29.711301+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":171,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702496+0000","last_change":"2024-09-21T08:24:56.698348+0000","last_active":"2024-09-21T08:25:29.702496+0000","last_peered":"2024-09-21T08:25:29.702496+0000","last_clean":"2024-09-21T08:25:29.702496+0000","last_became_active":"2024-09-21T08:24:56.517077+0000","last_became_peered":"2024-09-21T08:24:56.517077+0000","last_unstale":"2024-09-21T08:25:29.702496+0000","last_undegraded":"2024-09-21T08:25:29.702496+0000","last_fullsized":"2024-09-21T08:25:29.702496+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.466677+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":129,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.719088+0000","last_change":"2024-09-21T08:24:48.602351+0000","last_active":"2024-09-21T08:25:29.719088+0000","last_peered":"2024-09-21T08:25:29.719088+0000","last_clean":"2024-09-21T08:25:29.719088+0000","last_became_active":"2024-09-21T08:24:48.587366+0000","last_became_peered":"2024-09-21T08:24:48.587366+0000","last_unstale":"2024-09-21T08:25:29.719088+0000","last_undegraded":"2024-09-21T08:25:29.719088+0000","last_fullsized":"2024-09-21T08:25:29.719088+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:24:32.569846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:32.569846+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:42.983663+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":147,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720473+0000","last_change":"2024-09-21T08:24:38.539036+0000","last_active":"2024-09-21T08:25:29.720473+0000","last_peered":"2024-09-21T08:25:29.720473+0000","last_clean":"2024-09-21T08:25:29.720473+0000","last_became_active":"2024-09-21T08:24:38.538724+0000","last_became_peered":"2024-09-21T08:24:38.538724+0000","last_unstale":"2024-09-21T08:25:29.720473+0000","last_undegraded":"2024-09-21T08:25:29.720473+0000","last_fullsized":"2024-09-21T08:25:29.720473+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.701760+0000","last_change":"2024-09-21T08:24:48.605875+0000","last_active":"2024-09-21T08:25:29.701760+0000","last_peered":"2024-09-21T08:25:29.701760+0000","last_clean":"2024-09-21T08:25:29.701760+0000","last_became_active":"2024-09-21T08:24:48.586120+0000","last_became_peered":"2024-09-21T08:24:48.586120+0000","last_unstale":"2024-09-21T08:25:29.701760+0000","last_undegraded":"2024-09-21T08:25:29.701760+0000","last_fullsized":"2024-09-21T08:25:29.701760+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703127+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":108,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702871+0000","last_change":"2024-09-21T08:24:56.500353+0000","last_active":"2024-09-21T08:25:29.702871+0000","last_peered":"2024-09-21T08:25:29.702871+0000","last_clean":"2024-09-21T08:25:29.702871+0000","last_became_active":"2024-09-21T08:24:56.499884+0000","last_became_peered":"2024-09-21T08:24:56.499884+0000","last_unstale":"2024-09-21T08:25:29.702871+0000","last_undegraded":"2024-09-21T08:25:29.702871+0000","last_fullsized":"2024-09-21T08:25:29.702871+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703078+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710914+0000","last_change":"2024-09-21T08:24:54.870568+0000","last_active":"2024-09-21T08:25:29.710914+0000","last_peered":"2024-09-21T08:25:29.710914+0000","last_clean":"2024-09-21T08:25:29.710914+0000","last_became_active":"2024-09-21T08:24:38.546590+0000","last_became_peered":"2024-09-21T08:24:38.546590+0000","last_unstale":"2024-09-21T08:25:29.710914+0000","last_undegraded":"2024-09-21T08:25:29.710914+0000","last_fullsized":"2024-09-21T08:25:29.710914+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.354297+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":201,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.715061+0000","last_change":"2024-09-21T08:25:24.945896+0000","last_active":"2024-09-21T08:25:29.715061+0000","last_peered":"2024-09-21T08:25:29.715061+0000","last_clean":"2024-09-21T08:25:29.715061+0000","last_became_active":"2024-09-21T08:24:48.601541+0000","last_became_peered":"2024-09-21T08:24:48.601541+0000","last_unstale":"2024-09-21T08:25:29.715061+0000","last_undegraded":"2024-09-21T08:25:29.715061+0000","last_fullsized":"2024-09-21T08:25:29.715061+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:25:24.945807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:24.945807+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:46.549490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":148,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.717222+0000","last_change":"2024-09-21T08:24:38.551744+0000","last_active":"2024-09-21T08:25:29.717222+0000","last_peered":"2024-09-21T08:25:29.717222+0000","last_clean":"2024-09-21T08:25:29.717222+0000","last_became_active":"2024-09-21T08:24:38.551452+0000","last_became_peered":"2024-09-21T08:24:38.551452+0000","last_unstale":"2024-09-21T08:25:29.717222+0000","last_undegraded":"2024-09-21T08:25:29.717222+0000","last_fullsized":"2024-09-21T08:25:29.717222+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,5],"acting":[1,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":80,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702949+0000","last_change":"2024-09-21T08:24:56.518401+0000","last_active":"2024-09-21T08:25:29.702949+0000","last_peered":"2024-09-21T08:25:29.702949+0000","last_clean":"2024-09-21T08:25:29.702949+0000","last_became_active":"2024-09-21T08:24:56.518102+0000","last_became_peered":"2024-09-21T08:24:56.518102+0000","last_unstale":"2024-09-21T08:25:29.702949+0000","last_undegraded":"2024-09-21T08:25:29.702949+0000","last_fullsized":"2024-09-21T08:25:29.702949+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":114,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716457+0000","last_change":"2024-09-21T08:24:38.554008+0000","last_active":"2024-09-21T08:25:29.716457+0000","last_peered":"2024-09-21T08:25:29.716457+0000","last_clean":"2024-09-21T08:25:29.716457+0000","last_became_active":"2024-09-21T08:24:38.551991+0000","last_became_peered":"2024-09-21T08:24:38.551991+0000","last_unstale":"2024-09-21T08:25:29.716457+0000","last_undegraded":"2024-09-21T08:25:29.716457+0000","last_fullsized":"2024-09-21T08:25:29.716457+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":174,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.717310+0000","last_change":"2024-09-21T08:25:21.424491+0000","last_active":"2024-09-21T08:25:29.717310+0000","last_peered":"2024-09-21T08:25:29.717310+0000","last_clean":"2024-09-21T08:25:29.717310+0000","last_became_active":"2024-09-21T08:24:56.519036+0000","last_became_peered":"2024-09-21T08:24:56.519036+0000","last_unstale":"2024-09-21T08:25:29.717310+0000","last_undegraded":"2024-09-21T08:25:29.717310+0000","last_fullsized":"2024-09-21T08:25:29.717310+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:21.424414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.424414+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:34.359099+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":104,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.714670+0000","last_change":"2024-09-21T08:24:55.797165+0000","last_active":"2024-09-21T08:25:29.714670+0000","last_peered":"2024-09-21T08:25:29.714670+0000","last_clean":"2024-09-21T08:25:29.714670+0000","last_became_active":"2024-09-21T08:24:48.598748+0000","last_became_peered":"2024-09-21T08:24:48.598748+0000","last_unstale":"2024-09-21T08:25:29.714670+0000","last_undegraded":"2024-09-21T08:25:29.714670+0000","last_fullsized":"2024-09-21T08:25:29.714670+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.797084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:06.836794+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":138,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720974+0000","last_change":"2024-09-21T08:23:36.584003+0000","last_active":"2024-09-21T08:25:29.720974+0000","last_peered":"2024-09-21T08:25:29.720974+0000","last_clean":"2024-09-21T08:25:29.720974+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:25:29.720974+0000","last_undegraded":"2024-09-21T08:25:29.720974+0000","last_fullsized":"2024-09-21T08:25:29.720974+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":181,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710728+0000","last_change":"2024-09-21T08:24:56.543188+0000","last_active":"2024-09-21T08:25:29.710728+0000","last_peered":"2024-09-21T08:25:29.710728+0000","last_clean":"2024-09-21T08:25:29.710728+0000","last_became_active":"2024-09-21T08:24:56.517078+0000","last_became_peered":"2024-09-21T08:24:56.517078+0000","last_unstale":"2024-09-21T08:25:29.710728+0000","last_undegraded":"2024-09-21T08:25:29.710728+0000","last_fullsized":"2024-09-21T08:25:29.710728+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436744+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":179,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.711325+0000","last_change":"2024-09-21T08:24:48.659647+0000","last_active":"2024-09-21T08:25:29.711325+0000","last_peered":"2024-09-21T08:25:29.711325+0000","last_clean":"2024-09-21T08:25:29.711325+0000","last_became_active":"2024-09-21T08:24:48.602672+0000","last_became_peered":"2024-09-21T08:24:48.602672+0000","last_unstale":"2024-09-21T08:25:29.711325+0000","last_undegraded":"2024-09-21T08:25:29.711325+0000","last_fullsized":"2024-09-21T08:25:29.711325+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"103'11","reported_seq":201,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721137+0000","last_change":"2024-09-21T08:24:56.555287+0000","last_active":"2024-09-21T08:25:29.721137+0000","last_peered":"2024-09-21T08:25:29.721137+0000","last_clean":"2024-09-21T08:25:29.721137+0000","last_became_active":"2024-09-21T08:24:56.507316+0000","last_became_peered":"2024-09-21T08:24:56.507316+0000","last_unstale":"2024-09-21T08:25:29.721137+0000","last_undegraded":"2024-09-21T08:25:29.721137+0000","last_fullsized":"2024-09-21T08:25:29.721137+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.594212+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:52.368992+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"135'45","reported_seq":328,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720707+0000","last_change":"2024-09-21T08:25:28.711593+0000","last_active":"2024-09-21T08:25:29.720707+0000","last_peered":"2024-09-21T08:25:29.720707+0000","last_clean":"2024-09-21T08:25:29.720707+0000","last_became_active":"2024-09-21T08:24:56.503681+0000","last_became_peered":"2024-09-21T08:24:56.503681+0000","last_unstale":"2024-09-21T08:25:29.720707+0000","last_undegraded":"2024-09-21T08:25:29.720707+0000","last_fullsized":"2024-09-21T08:25:29.720707+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:40.767977+0000","scrub_duration":0,"objects_trimmed":3,"snaptrim_duration":0.011528073,"stat_sum":{"num_bytes":6705993,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":48,"num_read_kb":47873,"num_write":28,"num_write_kb":14617,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":12634927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"b"}]},{"pgid":"4.f","version":"73'15","reported_seq":215,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.718569+0000","last_change":"2024-09-21T08:24:48.611807+0000","last_active":"2024-09-21T08:25:29.718569+0000","last_peered":"2024-09-21T08:25:29.718569+0000","last_clean":"2024-09-21T08:25:29.718569+0000","last_became_active":"2024-09-21T08:24:48.581857+0000","last_became_peered":"2024-09-21T08:24:48.581857+0000","last_unstale":"2024-09-21T08:25:29.718569+0000","last_undegraded":"2024-09-21T08:25:29.718569+0000","last_fullsized":"2024-09-21T08:25:29.718569+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":95,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710004+0000","last_change":"2024-09-21T08:24:48.580125+0000","last_active":"2024-09-21T08:25:29.710004+0000","last_peered":"2024-09-21T08:25:29.710004+0000","last_clean":"2024-09-21T08:25:29.710004+0000","last_became_active":"2024-09-21T08:24:48.579498+0000","last_became_peered":"2024-09-21T08:24:48.579498+0000","last_unstale":"2024-09-21T08:25:29.710004+0000","last_undegraded":"2024-09-21T08:25:29.710004+0000","last_fullsized":"2024-09-21T08:25:29.710004+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"85'9","reported_seq":163,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.703111+0000","last_change":"2024-09-21T08:25:21.631370+0000","last_active":"2024-09-21T08:25:29.703111+0000","last_peered":"2024-09-21T08:25:29.703111+0000","last_clean":"2024-09-21T08:25:29.703111+0000","last_became_active":"2024-09-21T08:24:38.548789+0000","last_became_peered":"2024-09-21T08:24:38.548789+0000","last_unstale":"2024-09-21T08:25:29.703111+0000","last_undegraded":"2024-09-21T08:25:29.703111+0000","last_fullsized":"2024-09-21T08:25:29.703111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:25:21.631254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.631254+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:44.705485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":139,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702614+0000","last_change":"2024-09-21T08:23:36.578919+0000","last_active":"2024-09-21T08:25:29.702614+0000","last_peered":"2024-09-21T08:25:29.702614+0000","last_clean":"2024-09-21T08:25:29.702614+0000","last_became_active":"2024-09-21T08:23:36.578648+0000","last_became_peered":"2024-09-21T08:23:36.578648+0000","last_unstale":"2024-09-21T08:25:29.702614+0000","last_undegraded":"2024-09-21T08:25:29.702614+0000","last_fullsized":"2024-09-21T08:25:29.702614+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"86'3","reported_seq":263,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.712302+0000","last_change":"2024-09-21T08:25:28.705330+0000","last_active":"2024-09-21T08:25:29.712302+0000","last_peered":"2024-09-21T08:25:29.712302+0000","last_clean":"2024-09-21T08:25:29.712302+0000","last_became_active":"2024-09-21T08:24:56.521044+0000","last_became_peered":"2024-09-21T08:24:56.521044+0000","last_unstale":"2024-09-21T08:25:29.712302+0000","last_undegraded":"2024-09-21T08:25:29.712302+0000","last_fullsized":"2024-09-21T08:25:29.712302+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:54.394520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00035529000000000002,"stat_sum":{"num_bytes":2403368,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":10649,"num_write":4,"num_write_kb":1254,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2565420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"b"}]},{"pgid":"4.0","version":"73'18","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720188+0000","last_change":"2024-09-21T08:24:48.650167+0000","last_active":"2024-09-21T08:25:29.720188+0000","last_peered":"2024-09-21T08:25:29.720188+0000","last_clean":"2024-09-21T08:25:29.720188+0000","last_became_active":"2024-09-21T08:24:48.585831+0000","last_became_peered":"2024-09-21T08:24:48.585831+0000","last_unstale":"2024-09-21T08:25:29.720188+0000","last_undegraded":"2024-09-21T08:25:29.720188+0000","last_fullsized":"2024-09-21T08:25:29.720188+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":102,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721367+0000","last_change":"2024-09-21T08:24:50.648706+0000","last_active":"2024-09-21T08:25:29.721367+0000","last_peered":"2024-09-21T08:25:29.721367+0000","last_clean":"2024-09-21T08:25:29.721367+0000","last_became_active":"2024-09-21T08:24:48.603740+0000","last_became_peered":"2024-09-21T08:24:48.603740+0000","last_unstale":"2024-09-21T08:25:29.721367+0000","last_undegraded":"2024-09-21T08:25:29.721367+0000","last_fullsized":"2024-09-21T08:25:29.721367+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.532278+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":95,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721073+0000","last_change":"2024-09-21T08:24:48.604125+0000","last_active":"2024-09-21T08:25:29.721073+0000","last_peered":"2024-09-21T08:25:29.721073+0000","last_clean":"2024-09-21T08:25:29.721073+0000","last_became_active":"2024-09-21T08:24:48.603825+0000","last_became_peered":"2024-09-21T08:24:48.603825+0000","last_unstale":"2024-09-21T08:25:29.721073+0000","last_undegraded":"2024-09-21T08:25:29.721073+0000","last_fullsized":"2024-09-21T08:25:29.721073+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":103,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.712916+0000","last_change":"2024-09-21T08:24:56.501652+0000","last_active":"2024-09-21T08:25:29.712916+0000","last_peered":"2024-09-21T08:25:29.712916+0000","last_clean":"2024-09-21T08:25:29.712916+0000","last_became_active":"2024-09-21T08:24:56.501237+0000","last_became_peered":"2024-09-21T08:24:56.501237+0000","last_unstale":"2024-09-21T08:25:29.712916+0000","last_undegraded":"2024-09-21T08:25:29.712916+0000","last_fullsized":"2024-09-21T08:25:29.712916+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"136'120","reported_seq":399,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:30.563274+0000","last_change":"2024-09-21T08:25:30.561865+0000","last_active":"2024-09-21T08:25:30.563274+0000","last_peered":"2024-09-21T08:25:30.563274+0000","last_clean":"2024-09-21T08:25:30.563274+0000","last_became_active":"2024-09-21T08:24:48.595901+0000","last_became_peered":"2024-09-21T08:24:48.595901+0000","last_unstale":"2024-09-21T08:25:30.563274+0000","last_undegraded":"2024-09-21T08:25:30.563274+0000","last_fullsized":"2024-09-21T08:25:30.563274+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":120,"log_dups_size":0,"ondisk_log_size":120,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.615833+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.86642876099999999,"stat_sum":{"num_bytes":23441545,"num_objects":10,"num_object_clones":4,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":2,"num_read":68,"num_read_kb":55484,"num_write":79,"num_write_kb":31418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":10580183,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"3"},{"start":"14","length":"1"}]},{"pgid":"4.1","version":"73'14","reported_seq":192,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.701811+0000","last_change":"2024-09-21T08:24:56.561414+0000","last_active":"2024-09-21T08:25:29.701811+0000","last_peered":"2024-09-21T08:25:29.701811+0000","last_clean":"2024-09-21T08:25:29.701811+0000","last_became_active":"2024-09-21T08:24:56.509169+0000","last_became_peered":"2024-09-21T08:24:56.509169+0000","last_unstale":"2024-09-21T08:25:29.701811+0000","last_undegraded":"2024-09-21T08:25:29.701811+0000","last_fullsized":"2024-09-21T08:25:29.701811+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703165+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":169,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.711650+0000","last_change":"2024-09-21T08:24:55.889139+0000","last_active":"2024-09-21T08:25:29.711650+0000","last_peered":"2024-09-21T08:25:29.711650+0000","last_clean":"2024-09-21T08:25:29.711650+0000","last_became_active":"2024-09-21T08:24:38.547546+0000","last_became_peered":"2024-09-21T08:24:38.547546+0000","last_unstale":"2024-09-21T08:25:29.711650+0000","last_undegraded":"2024-09-21T08:25:29.711650+0000","last_fullsized":"2024-09-21T08:25:29.711650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:20.158135+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":46,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"96'10","reported_seq":132,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710412+0000","last_change":"2024-09-21T08:24:48.622146+0000","last_active":"2024-09-21T08:25:29.710412+0000","last_peered":"2024-09-21T08:25:29.710412+0000","last_clean":"2024-09-21T08:25:29.710412+0000","last_became_active":"2024-09-21T08:24:48.585809+0000","last_became_peered":"2024-09-21T08:24:48.585809+0000","last_unstale":"2024-09-21T08:25:29.710412+0000","last_undegraded":"2024-09-21T08:25:29.710412+0000","last_fullsized":"2024-09-21T08:25:29.710412+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.437047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,2],"acting":[1,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":151,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720575+0000","last_change":"2024-09-21T08:25:21.561910+0000","last_active":"2024-09-21T08:25:29.720575+0000","last_peered":"2024-09-21T08:25:29.720575+0000","last_clean":"2024-09-21T08:25:29.720575+0000","last_became_active":"2024-09-21T08:24:56.513734+0000","last_became_peered":"2024-09-21T08:24:56.513734+0000","last_unstale":"2024-09-21T08:25:29.720575+0000","last_undegraded":"2024-09-21T08:25:29.720575+0000","last_fullsized":"2024-09-21T08:25:29.720575+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:21.561826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.561826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:22.343032+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"134'18","reported_seq":196,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:30.561761+0000","last_change":"2024-09-21T08:25:30.561806+0000","last_active":"2024-09-21T08:25:30.561761+0000","last_peered":"2024-09-21T08:25:30.561761+0000","last_clean":"2024-09-21T08:25:30.561761+0000","last_became_active":"2024-09-21T08:24:56.527516+0000","last_became_peered":"2024-09-21T08:24:56.527516+0000","last_unstale":"2024-09-21T08:25:30.561761+0000","last_undegraded":"2024-09-21T08:25:30.561761+0000","last_fullsized":"2024-09-21T08:25:30.561761+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.067549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.85161375800000005,"stat_sum":{"num_bytes":5260525,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":9554,"num_write":9,"num_write_kb":4636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1702450,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"4"},{"start":"14","length":"1"}]},{"pgid":"4.2","version":"73'10","reported_seq":179,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716733+0000","last_change":"2024-09-21T08:24:38.567048+0000","last_active":"2024-09-21T08:25:29.716733+0000","last_peered":"2024-09-21T08:25:29.716733+0000","last_clean":"2024-09-21T08:25:29.716733+0000","last_became_active":"2024-09-21T08:24:38.551531+0000","last_became_peered":"2024-09-21T08:24:38.551531+0000","last_unstale":"2024-09-21T08:25:29.716733+0000","last_undegraded":"2024-09-21T08:25:29.716733+0000","last_fullsized":"2024-09-21T08:25:29.716733+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":177,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.703003+0000","last_change":"2024-09-21T08:24:56.519880+0000","last_active":"2024-09-21T08:25:29.703003+0000","last_peered":"2024-09-21T08:25:29.703003+0000","last_clean":"2024-09-21T08:25:29.703003+0000","last_became_active":"2024-09-21T08:24:56.500882+0000","last_became_peered":"2024-09-21T08:24:56.500882+0000","last_unstale":"2024-09-21T08:25:29.703003+0000","last_undegraded":"2024-09-21T08:25:29.703003+0000","last_fullsized":"2024-09-21T08:25:29.703003+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"103'9","reported_seq":185,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710426+0000","last_change":"2024-09-21T08:24:56.541475+0000","last_active":"2024-09-21T08:25:29.710426+0000","last_peered":"2024-09-21T08:25:29.710426+0000","last_clean":"2024-09-21T08:25:29.710426+0000","last_became_active":"2024-09-21T08:24:56.518379+0000","last_became_peered":"2024-09-21T08:24:56.518379+0000","last_unstale":"2024-09-21T08:25:29.710426+0000","last_undegraded":"2024-09-21T08:25:29.710426+0000","last_fullsized":"2024-09-21T08:25:29.710426+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.299021+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":144,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710955+0000","last_change":"2024-09-21T08:24:48.597514+0000","last_active":"2024-09-21T08:25:29.710955+0000","last_peered":"2024-09-21T08:25:29.710955+0000","last_clean":"2024-09-21T08:25:29.710955+0000","last_became_active":"2024-09-21T08:24:48.597048+0000","last_became_peered":"2024-09-21T08:24:48.597048+0000","last_unstale":"2024-09-21T08:25:29.710955+0000","last_undegraded":"2024-09-21T08:25:29.710955+0000","last_fullsized":"2024-09-21T08:25:29.710955+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.299054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"135'41","reported_seq":331,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:29.701741+0000","last_change":"2024-09-21T08:25:28.708565+0000","last_active":"2024-09-21T08:25:29.701741+0000","last_peered":"2024-09-21T08:25:29.701741+0000","last_clean":"2024-09-21T08:25:29.701741+0000","last_became_active":"2024-09-21T08:24:48.586911+0000","last_became_peered":"2024-09-21T08:24:48.586911+0000","last_unstale":"2024-09-21T08:25:29.701741+0000","last_undegraded":"2024-09-21T08:25:29.701741+0000","last_fullsized":"2024-09-21T08:25:29.701741+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":41,"log_dups_size":0,"ondisk_log_size":41,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.956648+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.966044299,"stat_sum":{"num_bytes":5731121,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":23,"num_read_kb":6167,"num_write":34,"num_write_kb":10631,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":3259822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"8"},{"start":"18","length":"2"}]},{"pgid":"4.3","version":"73'19","reported_seq":182,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710485+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:25:29.710485+0000","last_peered":"2024-09-21T08:25:29.710485+0000","last_clean":"2024-09-21T08:25:29.710485+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:25:29.710485+0000","last_undegraded":"2024-09-21T08:25:29.710485+0000","last_fullsized":"2024-09-21T08:25:29.710485+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298498+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":158,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716548+0000","last_change":"2024-09-21T08:24:52.529765+0000","last_active":"2024-09-21T08:25:29.716548+0000","last_peered":"2024-09-21T08:25:29.716548+0000","last_clean":"2024-09-21T08:25:29.716548+0000","last_became_active":"2024-09-21T08:23:30.547290+0000","last_became_peered":"2024-09-21T08:23:30.547290+0000","last_unstale":"2024-09-21T08:25:29.716548+0000","last_undegraded":"2024-09-21T08:25:29.716548+0000","last_fullsized":"2024-09-21T08:25:29.716548+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.529672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.529672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.610202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":77,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.709863+0000","last_change":"2024-09-21T08:24:56.503913+0000","last_active":"2024-09-21T08:25:29.709863+0000","last_peered":"2024-09-21T08:25:29.709863+0000","last_clean":"2024-09-21T08:25:29.709863+0000","last_became_active":"2024-09-21T08:24:56.499628+0000","last_became_peered":"2024-09-21T08:24:56.499628+0000","last_unstale":"2024-09-21T08:25:29.709863+0000","last_undegraded":"2024-09-21T08:25:29.709863+0000","last_fullsized":"2024-09-21T08:25:29.709863+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298329+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":152,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710458+0000","last_change":"2024-09-21T08:25:22.425496+0000","last_active":"2024-09-21T08:25:29.710458+0000","last_peered":"2024-09-21T08:25:29.710458+0000","last_clean":"2024-09-21T08:25:29.710458+0000","last_became_active":"2024-09-21T08:24:56.517498+0000","last_became_peered":"2024-09-21T08:24:56.517498+0000","last_unstale":"2024-09-21T08:25:29.710458+0000","last_undegraded":"2024-09-21T08:25:29.710458+0000","last_fullsized":"2024-09-21T08:25:29.710458+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.425389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:50.516628+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"133'32","reported_seq":309,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:29.710942+0000","last_change":"2024-09-21T08:25:29.673067+0000","last_active":"2024-09-21T08:25:29.710942+0000","last_peered":"2024-09-21T08:25:29.710942+0000","last_clean":"2024-09-21T08:25:29.710942+0000","last_became_active":"2024-09-21T08:24:56.513790+0000","last_became_peered":"2024-09-21T08:24:56.513790+0000","last_unstale":"2024-09-21T08:25:29.710942+0000","last_undegraded":"2024-09-21T08:25:29.710942+0000","last_fullsized":"2024-09-21T08:25:29.710942+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.336094+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.96502903100000004,"stat_sum":{"num_bytes":3654966,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":16,"num_read_kb":11107,"num_write":20,"num_write_kb":4891,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4730540,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"8"}]},{"pgid":"4.4","version":"103'30","reported_seq":275,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716439+0000","last_change":"2024-09-21T08:24:56.561857+0000","last_active":"2024-09-21T08:25:29.716439+0000","last_peered":"2024-09-21T08:25:29.716439+0000","last_clean":"2024-09-21T08:25:29.716439+0000","last_became_active":"2024-09-21T08:24:56.519586+0000","last_became_peered":"2024-09-21T08:24:56.519586+0000","last_unstale":"2024-09-21T08:25:29.716439+0000","last_undegraded":"2024-09-21T08:25:29.716439+0000","last_fullsized":"2024-09-21T08:25:29.716439+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436907+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":48,"num_read_kb":33,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":119,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710694+0000","last_change":"2024-09-21T08:24:56.519301+0000","last_active":"2024-09-21T08:25:29.710694+0000","last_peered":"2024-09-21T08:25:29.710694+0000","last_clean":"2024-09-21T08:25:29.710694+0000","last_became_active":"2024-09-21T08:24:56.518876+0000","last_became_peered":"2024-09-21T08:24:56.518876+0000","last_unstale":"2024-09-21T08:25:29.710694+0000","last_undegraded":"2024-09-21T08:25:29.710694+0000","last_fullsized":"2024-09-21T08:25:29.710694+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":171,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702336+0000","last_change":"2024-09-21T08:24:56.556151+0000","last_active":"2024-09-21T08:25:29.702336+0000","last_peered":"2024-09-21T08:25:29.702336+0000","last_clean":"2024-09-21T08:25:29.702336+0000","last_became_active":"2024-09-21T08:24:56.508710+0000","last_became_peered":"2024-09-21T08:24:56.508710+0000","last_unstale":"2024-09-21T08:25:29.702336+0000","last_undegraded":"2024-09-21T08:25:29.702336+0000","last_fullsized":"2024-09-21T08:25:29.702336+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":154,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.713959+0000","last_change":"2024-09-21T08:24:56.516897+0000","last_active":"2024-09-21T08:25:29.713959+0000","last_peered":"2024-09-21T08:25:29.713959+0000","last_clean":"2024-09-21T08:25:29.713959+0000","last_became_active":"2024-09-21T08:24:56.516563+0000","last_became_peered":"2024-09-21T08:24:56.516563+0000","last_unstale":"2024-09-21T08:25:29.713959+0000","last_undegraded":"2024-09-21T08:25:29.713959+0000","last_fullsized":"2024-09-21T08:25:29.713959+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":141,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.722146+0000","last_change":"2024-09-21T08:25:23.602449+0000","last_active":"2024-09-21T08:25:29.722146+0000","last_peered":"2024-09-21T08:25:29.722146+0000","last_clean":"2024-09-21T08:25:29.722146+0000","last_became_active":"2024-09-21T08:24:56.505969+0000","last_became_peered":"2024-09-21T08:24:56.505969+0000","last_unstale":"2024-09-21T08:25:29.722146+0000","last_undegraded":"2024-09-21T08:25:29.722146+0000","last_fullsized":"2024-09-21T08:25:29.722146+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.602376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:29.483779+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,1],"acting":[7,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"132'28","reported_seq":305,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:29.713766+0000","last_change":"2024-09-21T08:25:29.672861+0000","last_active":"2024-09-21T08:25:29.713766+0000","last_peered":"2024-09-21T08:25:29.713766+0000","last_clean":"2024-09-21T08:25:29.713766+0000","last_became_active":"2024-09-21T08:24:56.517718+0000","last_became_peered":"2024-09-21T08:24:56.517718+0000","last_unstale":"2024-09-21T08:25:29.713766+0000","last_undegraded":"2024-09-21T08:25:29.713766+0000","last_fullsized":"2024-09-21T08:25:29.713766+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.202906+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.96477528999999995,"stat_sum":{"num_bytes":5217849,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":15,"num_read_kb":12586,"num_write":19,"num_write_kb":5122,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1312313,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"8"}]},{"pgid":"4.7","version":"73'13","reported_seq":173,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.718461+0000","last_change":"2024-09-21T08:23:32.560593+0000","last_active":"2024-09-21T08:25:29.718461+0000","last_peered":"2024-09-21T08:25:29.718461+0000","last_clean":"2024-09-21T08:25:29.718461+0000","last_became_active":"2024-09-21T08:23:32.560323+0000","last_became_peered":"2024-09-21T08:23:32.560323+0000","last_unstale":"2024-09-21T08:25:29.718461+0000","last_undegraded":"2024-09-21T08:25:29.718461+0000","last_fullsized":"2024-09-21T08:25:29.718461+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":30,"num_read_kb":19,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,0],"acting":[1,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.709699+0000","last_change":"2024-09-21T08:24:56.525402+0000","last_active":"2024-09-21T08:25:29.709699+0000","last_peered":"2024-09-21T08:25:29.709699+0000","last_clean":"2024-09-21T08:25:29.709699+0000","last_became_active":"2024-09-21T08:24:56.524958+0000","last_became_peered":"2024-09-21T08:24:56.524958+0000","last_unstale":"2024-09-21T08:25:29.709699+0000","last_undegraded":"2024-09-21T08:25:29.709699+0000","last_fullsized":"2024-09-21T08:25:29.709699+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":177,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.711885+0000","last_change":"2024-09-21T08:24:50.675752+0000","last_active":"2024-09-21T08:25:29.711885+0000","last_peered":"2024-09-21T08:25:29.711885+0000","last_clean":"2024-09-21T08:25:29.711885+0000","last_became_active":"2024-09-21T08:24:48.604993+0000","last_became_peered":"2024-09-21T08:24:48.604993+0000","last_unstale":"2024-09-21T08:25:29.711885+0000","last_undegraded":"2024-09-21T08:25:29.711885+0000","last_fullsized":"2024-09-21T08:25:29.711885+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:50.675673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.675673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.957118+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":142,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.715627+0000","last_change":"2024-09-21T08:23:34.567541+0000","last_active":"2024-09-21T08:25:29.715627+0000","last_peered":"2024-09-21T08:25:29.715627+0000","last_clean":"2024-09-21T08:25:29.715627+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:25:29.715627+0000","last_undegraded":"2024-09-21T08:25:29.715627+0000","last_fullsized":"2024-09-21T08:25:29.715627+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":149,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721357+0000","last_change":"2024-09-21T08:24:56.501148+0000","last_active":"2024-09-21T08:25:29.721357+0000","last_peered":"2024-09-21T08:25:29.721357+0000","last_clean":"2024-09-21T08:25:29.721357+0000","last_became_active":"2024-09-21T08:24:56.498476+0000","last_became_peered":"2024-09-21T08:24:56.498476+0000","last_unstale":"2024-09-21T08:25:29.721357+0000","last_undegraded":"2024-09-21T08:25:29.721357+0000","last_fullsized":"2024-09-21T08:25:29.721357+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"136'155","reported_seq":433,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.728972+0000","last_change":"2024-09-21T08:25:28.711633+0000","last_active":"2024-09-21T08:25:29.728972+0000","last_peered":"2024-09-21T08:25:29.728972+0000","last_clean":"2024-09-21T08:25:29.728972+0000","last_became_active":"2024-09-21T08:24:34.507710+0000","last_became_peered":"2024-09-21T08:24:34.507710+0000","last_unstale":"2024-09-21T08:25:29.728972+0000","last_undegraded":"2024-09-21T08:25:29.728972+0000","last_fullsized":"2024-09-21T08:25:29.728972+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:41.880605+0000","scrub_duration":0,"objects_trimmed":4,"snaptrim_duration":0.011143004,"stat_sum":{"num_bytes":8958032,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":81,"num_read_kb":65424,"num_write":83,"num_write_kb":34081,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"b"}]},{"pgid":"4.6","version":"73'12","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.711448+0000","last_change":"2024-09-21T08:23:32.561943+0000","last_active":"2024-09-21T08:25:29.711448+0000","last_peered":"2024-09-21T08:25:29.711448+0000","last_clean":"2024-09-21T08:25:29.711448+0000","last_became_active":"2024-09-21T08:23:32.561617+0000","last_became_peered":"2024-09-21T08:23:32.561617+0000","last_unstale":"2024-09-21T08:25:29.711448+0000","last_undegraded":"2024-09-21T08:25:29.711448+0000","last_fullsized":"2024-09-21T08:25:29.711448+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298396+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.714383+0000","last_change":"2024-09-21T08:24:48.601343+0000","last_active":"2024-09-21T08:25:29.714383+0000","last_peered":"2024-09-21T08:25:29.714383+0000","last_clean":"2024-09-21T08:25:29.714383+0000","last_became_active":"2024-09-21T08:24:48.601044+0000","last_became_peered":"2024-09-21T08:24:48.601044+0000","last_unstale":"2024-09-21T08:25:29.714383+0000","last_undegraded":"2024-09-21T08:25:29.714383+0000","last_fullsized":"2024-09-21T08:25:29.714383+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":284,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:30.653386+0000","last_change":"2024-09-21T08:24:47.740562+0000","last_active":"2024-09-21T08:25:30.653386+0000","last_peered":"2024-09-21T08:25:30.653386+0000","last_clean":"2024-09-21T08:25:30.653386+0000","last_became_active":"2024-09-21T08:23:29.532520+0000","last_became_peered":"2024-09-21T08:23:29.532520+0000","last_unstale":"2024-09-21T08:25:30.653386+0000","last_undegraded":"2024-09-21T08:25:30.653386+0000","last_fullsized":"2024-09-21T08:25:30.653386+0000","mapping_epoch":64,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":65,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:04.383800+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":115,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":143,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.703287+0000","last_change":"2024-09-21T08:23:34.570449+0000","last_active":"2024-09-21T08:25:29.703287+0000","last_peered":"2024-09-21T08:25:29.703287+0000","last_clean":"2024-09-21T08:25:29.703287+0000","last_became_active":"2024-09-21T08:23:34.569970+0000","last_became_peered":"2024-09-21T08:23:34.569970+0000","last_unstale":"2024-09-21T08:25:29.703287+0000","last_undegraded":"2024-09-21T08:25:29.703287+0000","last_fullsized":"2024-09-21T08:25:29.703287+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716003+0000","last_change":"2024-09-21T08:24:56.520951+0000","last_active":"2024-09-21T08:25:29.716003+0000","last_peered":"2024-09-21T08:25:29.716003+0000","last_clean":"2024-09-21T08:25:29.716003+0000","last_became_active":"2024-09-21T08:24:56.520320+0000","last_became_peered":"2024-09-21T08:24:56.520320+0000","last_unstale":"2024-09-21T08:25:29.716003+0000","last_undegraded":"2024-09-21T08:25:29.716003+0000","last_fullsized":"2024-09-21T08:25:29.716003+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.437061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"135'48","reported_seq":328,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:29.711274+0000","last_change":"2024-09-21T08:25:29.673105+0000","last_active":"2024-09-21T08:25:29.711274+0000","last_peered":"2024-09-21T08:25:29.711274+0000","last_clean":"2024-09-21T08:25:29.711274+0000","last_became_active":"2024-09-21T08:24:48.593246+0000","last_became_peered":"2024-09-21T08:24:48.593246+0000","last_unstale":"2024-09-21T08:25:29.711274+0000","last_undegraded":"2024-09-21T08:25:29.711274+0000","last_fullsized":"2024-09-21T08:25:29.711274+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.302667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.96375288999999997,"stat_sum":{"num_bytes":6218557,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":28,"num_read_kb":24701,"num_write":27,"num_write_kb":8445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4966638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"8"}]},{"pgid":"4.5","version":"103'18","reported_seq":146,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721965+0000","last_change":"2024-09-21T08:24:56.564202+0000","last_active":"2024-09-21T08:25:29.721965+0000","last_peered":"2024-09-21T08:25:29.721965+0000","last_clean":"2024-09-21T08:25:29.721965+0000","last_became_active":"2024-09-21T08:24:56.503984+0000","last_became_peered":"2024-09-21T08:24:56.503984+0000","last_unstale":"2024-09-21T08:25:29.721965+0000","last_undegraded":"2024-09-21T08:25:29.721965+0000","last_fullsized":"2024-09-21T08:25:29.721965+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-09-21T08:24:48.583137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.583137+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.538958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":101,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702241+0000","last_change":"2024-09-21T08:24:56.509057+0000","last_active":"2024-09-21T08:25:29.702241+0000","last_peered":"2024-09-21T08:25:29.702241+0000","last_clean":"2024-09-21T08:25:29.702241+0000","last_became_active":"2024-09-21T08:24:56.508272+0000","last_became_peered":"2024-09-21T08:24:56.508272+0000","last_unstale":"2024-09-21T08:25:29.702241+0000","last_undegraded":"2024-09-21T08:25:29.702241+0000","last_fullsized":"2024-09-21T08:25:29.702241+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":421,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720827+0000","last_change":"2024-09-21T08:24:59.498580+0000","last_active":"2024-09-21T08:25:29.720827+0000","last_peered":"2024-09-21T08:25:29.720827+0000","last_clean":"2024-09-21T08:25:29.720827+0000","last_became_active":"2024-09-21T08:24:59.498199+0000","last_became_peered":"2024-09-21T08:24:59.498199+0000","last_unstale":"2024-09-21T08:25:29.720827+0000","last_undegraded":"2024-09-21T08:25:29.720827+0000","last_fullsized":"2024-09-21T08:25:29.720827+0000","mapping_epoch":105,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":106,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:24:05.765804+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.323834+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":141,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721089+0000","last_change":"2024-09-21T08:23:34.568359+0000","last_active":"2024-09-21T08:25:29.721089+0000","last_peered":"2024-09-21T08:25:29.721089+0000","last_clean":"2024-09-21T08:25:29.721089+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:25:29.721089+0000","last_undegraded":"2024-09-21T08:25:29.721089+0000","last_fullsized":"2024-09-21T08:25:29.721089+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":159,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.703112+0000","last_change":"2024-09-21T08:24:48.726197+0000","last_active":"2024-09-21T08:25:29.703112+0000","last_peered":"2024-09-21T08:25:29.703112+0000","last_clean":"2024-09-21T08:25:29.703112+0000","last_became_active":"2024-09-21T08:24:48.575960+0000","last_became_peered":"2024-09-21T08:24:48.575960+0000","last_unstale":"2024-09-21T08:25:29.703112+0000","last_undegraded":"2024-09-21T08:25:29.703112+0000","last_fullsized":"2024-09-21T08:25:29.703112+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:02.092267+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"131'34","reported_seq":220,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:29.716800+0000","last_change":"2024-09-21T08:25:28.562881+0000","last_active":"2024-09-21T08:25:29.716800+0000","last_peered":"2024-09-21T08:25:29.716800+0000","last_clean":"2024-09-21T08:25:29.716800+0000","last_became_active":"2024-09-21T08:24:56.524759+0000","last_became_peered":"2024-09-21T08:24:56.524759+0000","last_unstale":"2024-09-21T08:25:29.716800+0000","last_undegraded":"2024-09-21T08:25:29.716800+0000","last_fullsized":"2024-09-21T08:25:29.716800+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:50.964651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.86652685799999996,"stat_sum":{"num_bytes":9862755,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":18,"num_read_kb":11024,"num_write":27,"num_write_kb":11474,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2039342,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"5"}]},{"pgid":"4.e","version":"73'11","reported_seq":128,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.717108+0000","last_change":"2024-09-21T08:24:56.538842+0000","last_active":"2024-09-21T08:25:29.717108+0000","last_peered":"2024-09-21T08:25:29.717108+0000","last_clean":"2024-09-21T08:25:29.717108+0000","last_became_active":"2024-09-21T08:24:56.510706+0000","last_became_peered":"2024-09-21T08:24:56.510706+0000","last_unstale":"2024-09-21T08:25:29.717108+0000","last_undegraded":"2024-09-21T08:25:29.717108+0000","last_fullsized":"2024-09-21T08:25:29.717108+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":122,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.711608+0000","last_change":"2024-09-21T08:24:48.756972+0000","last_active":"2024-09-21T08:25:29.711608+0000","last_peered":"2024-09-21T08:25:29.711608+0000","last_clean":"2024-09-21T08:25:29.711608+0000","last_became_active":"2024-09-21T08:24:38.548474+0000","last_became_peered":"2024-09-21T08:24:38.548474+0000","last_unstale":"2024-09-21T08:25:29.711608+0000","last_undegraded":"2024-09-21T08:25:29.711608+0000","last_fullsized":"2024-09-21T08:25:29.711608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.756905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.756905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.943320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.703105+0000","last_change":"2024-09-21T08:24:56.508737+0000","last_active":"2024-09-21T08:25:29.703105+0000","last_peered":"2024-09-21T08:25:29.703105+0000","last_clean":"2024-09-21T08:25:29.703105+0000","last_became_active":"2024-09-21T08:24:56.508405+0000","last_became_peered":"2024-09-21T08:24:56.508405+0000","last_unstale":"2024-09-21T08:25:29.703105+0000","last_undegraded":"2024-09-21T08:25:29.703105+0000","last_fullsized":"2024-09-21T08:25:29.703105+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703260+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":80,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702375+0000","last_change":"2024-09-21T08:24:56.517612+0000","last_active":"2024-09-21T08:25:29.702375+0000","last_peered":"2024-09-21T08:25:29.702375+0000","last_clean":"2024-09-21T08:25:29.702375+0000","last_became_active":"2024-09-21T08:24:56.516670+0000","last_became_peered":"2024-09-21T08:24:56.516670+0000","last_unstale":"2024-09-21T08:25:29.702375+0000","last_undegraded":"2024-09-21T08:25:29.702375+0000","last_fullsized":"2024-09-21T08:25:29.702375+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703231+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"135'59","reported_seq":327,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:29.715385+0000","last_change":"2024-09-21T08:25:28.705758+0000","last_active":"2024-09-21T08:25:29.715385+0000","last_peered":"2024-09-21T08:25:29.715385+0000","last_clean":"2024-09-21T08:25:29.715385+0000","last_became_active":"2024-09-21T08:24:34.508974+0000","last_became_peered":"2024-09-21T08:24:34.508974+0000","last_unstale":"2024-09-21T08:25:29.715385+0000","last_undegraded":"2024-09-21T08:25:29.715385+0000","last_fullsized":"2024-09-21T08:25:29.715385+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":59,"log_dups_size":0,"ondisk_log_size":59,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:47.608637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.97542452099999999,"stat_sum":{"num_bytes":3932805,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":18,"num_read_kb":7781,"num_write":24,"num_write_kb":9403,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"8"},{"start":"18","length":"2"}]},{"pgid":"4.d","version":"73'17","reported_seq":203,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.714231+0000","last_change":"2024-09-21T08:24:56.553183+0000","last_active":"2024-09-21T08:25:29.714231+0000","last_peered":"2024-09-21T08:25:29.714231+0000","last_clean":"2024-09-21T08:25:29.714231+0000","last_became_active":"2024-09-21T08:24:56.514487+0000","last_became_peered":"2024-09-21T08:24:56.514487+0000","last_unstale":"2024-09-21T08:25:29.714231+0000","last_undegraded":"2024-09-21T08:25:29.714231+0000","last_fullsized":"2024-09-21T08:25:29.714231+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,0],"acting":[2,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716554+0000","last_change":"2024-09-21T08:24:56.516700+0000","last_active":"2024-09-21T08:25:29.716554+0000","last_peered":"2024-09-21T08:25:29.716554+0000","last_clean":"2024-09-21T08:25:29.716554+0000","last_became_active":"2024-09-21T08:24:56.516326+0000","last_became_peered":"2024-09-21T08:24:56.516326+0000","last_unstale":"2024-09-21T08:25:29.716554+0000","last_undegraded":"2024-09-21T08:25:29.716554+0000","last_fullsized":"2024-09-21T08:25:29.716554+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.437015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":154,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716089+0000","last_change":"2024-09-21T08:24:56.526336+0000","last_active":"2024-09-21T08:25:29.716089+0000","last_peered":"2024-09-21T08:25:29.716089+0000","last_clean":"2024-09-21T08:25:29.716089+0000","last_became_active":"2024-09-21T08:24:56.524399+0000","last_became_peered":"2024-09-21T08:24:56.524399+0000","last_unstale":"2024-09-21T08:25:29.716089+0000","last_undegraded":"2024-09-21T08:25:29.716089+0000","last_fullsized":"2024-09-21T08:25:29.716089+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436712+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":164,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.713595+0000","last_change":"2024-09-21T08:24:56.520526+0000","last_active":"2024-09-21T08:25:29.713595+0000","last_peered":"2024-09-21T08:25:29.713595+0000","last_clean":"2024-09-21T08:25:29.713595+0000","last_became_active":"2024-09-21T08:24:56.517208+0000","last_became_peered":"2024-09-21T08:24:56.517208+0000","last_unstale":"2024-09-21T08:25:29.713595+0000","last_undegraded":"2024-09-21T08:25:29.713595+0000","last_fullsized":"2024-09-21T08:25:29.713595+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.766815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:01.409327+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,5],"acting":[2,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"100'7","reported_seq":273,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:29.711115+0000","last_change":"2024-09-21T08:25:29.672620+0000","last_active":"2024-09-21T08:25:29.711115+0000","last_peered":"2024-09-21T08:25:29.711115+0000","last_clean":"2024-09-21T08:25:29.711115+0000","last_became_active":"2024-09-21T08:24:48.586187+0000","last_became_peered":"2024-09-21T08:24:48.586187+0000","last_unstale":"2024-09-21T08:25:29.711115+0000","last_undegraded":"2024-09-21T08:25:29.711115+0000","last_fullsized":"2024-09-21T08:25:29.711115+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:49.675919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.96453612700000002,"stat_sum":{"num_bytes":2369754,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":3,"num_read_kb":1,"num_write":5,"num_write_kb":3244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":949927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"8"}]},{"pgid":"4.c","version":"73'10","reported_seq":126,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.715856+0000","last_change":"2024-09-21T08:24:56.567156+0000","last_active":"2024-09-21T08:25:29.715856+0000","last_peered":"2024-09-21T08:25:29.715856+0000","last_clean":"2024-09-21T08:25:29.715856+0000","last_became_active":"2024-09-21T08:24:56.521403+0000","last_became_peered":"2024-09-21T08:24:56.521403+0000","last_unstale":"2024-09-21T08:25:29.715856+0000","last_undegraded":"2024-09-21T08:25:29.715856+0000","last_fullsized":"2024-09-21T08:25:29.715856+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436775+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":139,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710593+0000","last_change":"2024-09-21T08:24:48.596461+0000","last_active":"2024-09-21T08:25:29.710593+0000","last_peered":"2024-09-21T08:25:29.710593+0000","last_clean":"2024-09-21T08:25:29.710593+0000","last_became_active":"2024-09-21T08:24:48.595725+0000","last_became_peered":"2024-09-21T08:24:48.595725+0000","last_unstale":"2024-09-21T08:25:29.710593+0000","last_undegraded":"2024-09-21T08:25:29.710593+0000","last_fullsized":"2024-09-21T08:25:29.710593+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":183,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.714125+0000","last_change":"2024-09-21T08:24:51.689976+0000","last_active":"2024-09-21T08:25:29.714125+0000","last_peered":"2024-09-21T08:25:29.714125+0000","last_clean":"2024-09-21T08:25:29.714125+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:25:29.714125+0000","last_undegraded":"2024-09-21T08:25:29.714125+0000","last_fullsized":"2024-09-21T08:25:29.714125+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:24:51.689901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.689901+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.850967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":115,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.717409+0000","last_change":"2024-09-21T08:24:38.550749+0000","last_active":"2024-09-21T08:25:29.717409+0000","last_peered":"2024-09-21T08:25:29.717409+0000","last_clean":"2024-09-21T08:25:29.717409+0000","last_became_active":"2024-09-21T08:24:38.550434+0000","last_became_peered":"2024-09-21T08:24:38.550434+0000","last_unstale":"2024-09-21T08:25:29.717409+0000","last_undegraded":"2024-09-21T08:25:29.717409+0000","last_fullsized":"2024-09-21T08:25:29.717409+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.437000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"135'84","reported_seq":385,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:29.726837+0000","last_change":"2024-09-21T08:25:28.704445+0000","last_active":"2024-09-21T08:25:29.726837+0000","last_peered":"2024-09-21T08:25:29.726837+0000","last_clean":"2024-09-21T08:25:29.726837+0000","last_became_active":"2024-09-21T08:24:48.590826+0000","last_became_peered":"2024-09-21T08:24:48.590826+0000","last_unstale":"2024-09-21T08:25:29.726837+0000","last_undegraded":"2024-09-21T08:25:29.726837+0000","last_fullsized":"2024-09-21T08:25:29.726837+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:42.425316+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.97678026100000004,"stat_sum":{"num_bytes":14050520,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":39,"num_read_kb":29695,"num_write":52,"num_write_kb":18936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8754687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"8"},{"start":"18","length":"2"}]},{"pgid":"4.b","version":"73'9","reported_seq":202,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710475+0000","last_change":"2024-09-21T08:25:22.369898+0000","last_active":"2024-09-21T08:25:29.710475+0000","last_peered":"2024-09-21T08:25:29.710475+0000","last_clean":"2024-09-21T08:25:29.710475+0000","last_became_active":"2024-09-21T08:24:48.590921+0000","last_became_peered":"2024-09-21T08:24:48.590921+0000","last_unstale":"2024-09-21T08:25:29.710475+0000","last_undegraded":"2024-09-21T08:25:29.710475+0000","last_fullsized":"2024-09-21T08:25:29.710475+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:25:22.369811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.369811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:39.214728+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":171,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702573+0000","last_change":"2024-09-21T08:24:58.665514+0000","last_active":"2024-09-21T08:25:29.702573+0000","last_peered":"2024-09-21T08:25:29.702573+0000","last_clean":"2024-09-21T08:25:29.702573+0000","last_became_active":"2024-09-21T08:24:56.517512+0000","last_became_peered":"2024-09-21T08:24:56.517512+0000","last_unstale":"2024-09-21T08:25:29.702573+0000","last_undegraded":"2024-09-21T08:25:29.702573+0000","last_fullsized":"2024-09-21T08:25:29.702573+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.665444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.665444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.770807+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.712922+0000","last_change":"2024-09-21T08:25:23.990040+0000","last_active":"2024-09-21T08:25:29.712922+0000","last_peered":"2024-09-21T08:25:29.712922+0000","last_clean":"2024-09-21T08:25:29.712922+0000","last_became_active":"2024-09-21T08:24:48.597800+0000","last_became_peered":"2024-09-21T08:24:48.597800+0000","last_unstale":"2024-09-21T08:25:29.712922+0000","last_undegraded":"2024-09-21T08:25:29.712922+0000","last_fullsized":"2024-09-21T08:25:29.712922+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.138500+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":138,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.714303+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:25:29.714303+0000","last_peered":"2024-09-21T08:25:29.714303+0000","last_clean":"2024-09-21T08:25:29.714303+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:25:29.714303+0000","last_undegraded":"2024-09-21T08:25:29.714303+0000","last_fullsized":"2024-09-21T08:25:29.714303+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"136'96","reported_seq":357,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:30.677888+0000","last_change":"2024-09-21T08:25:28.673666+0000","last_active":"2024-09-21T08:25:30.677888+0000","last_peered":"2024-09-21T08:25:30.677888+0000","last_clean":"2024-09-21T08:25:30.677888+0000","last_became_active":"2024-09-21T08:24:38.546433+0000","last_became_peered":"2024-09-21T08:24:38.546433+0000","last_unstale":"2024-09-21T08:25:30.677888+0000","last_undegraded":"2024-09-21T08:25:30.677888+0000","last_fullsized":"2024-09-21T08:25:30.677888+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:49.039397+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.77949019799999997,"stat_sum":{"num_bytes":19287145,"num_objects":8,"num_object_clones":3,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":1,"num_read":82,"num_read_kb":75533,"num_write":59,"num_write_kb":23387,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,7],"acting":[0,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"f","length":"3"},{"start":"14","length":"1"}]},{"pgid":"4.a","version":"73'19","reported_seq":148,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716835+0000","last_change":"2024-09-21T08:24:56.553824+0000","last_active":"2024-09-21T08:25:29.716835+0000","last_peered":"2024-09-21T08:25:29.716835+0000","last_clean":"2024-09-21T08:25:29.716835+0000","last_became_active":"2024-09-21T08:24:56.511681+0000","last_became_peered":"2024-09-21T08:24:56.511681+0000","last_unstale":"2024-09-21T08:25:29.716835+0000","last_undegraded":"2024-09-21T08:25:29.716835+0000","last_fullsized":"2024-09-21T08:25:29.716835+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":163,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721936+0000","last_change":"2024-09-21T08:24:56.507593+0000","last_active":"2024-09-21T08:25:29.721936+0000","last_peered":"2024-09-21T08:25:29.721936+0000","last_clean":"2024-09-21T08:25:29.721936+0000","last_became_active":"2024-09-21T08:24:56.505054+0000","last_became_peered":"2024-09-21T08:24:56.505054+0000","last_unstale":"2024-09-21T08:25:29.721936+0000","last_undegraded":"2024-09-21T08:25:29.721936+0000","last_fullsized":"2024-09-21T08:25:29.721936+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.740674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.740674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.640221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":142,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.712574+0000","last_change":"2024-09-21T08:23:34.564923+0000","last_active":"2024-09-21T08:25:29.712574+0000","last_peered":"2024-09-21T08:25:29.712574+0000","last_clean":"2024-09-21T08:25:29.712574+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:25:29.712574+0000","last_undegraded":"2024-09-21T08:25:29.712574+0000","last_fullsized":"2024-09-21T08:25:29.712574+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":157,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720741+0000","last_change":"2024-09-21T08:24:56.504692+0000","last_active":"2024-09-21T08:25:29.720741+0000","last_peered":"2024-09-21T08:25:29.720741+0000","last_clean":"2024-09-21T08:25:29.720741+0000","last_became_active":"2024-09-21T08:24:56.503564+0000","last_became_peered":"2024-09-21T08:24:56.503564+0000","last_unstale":"2024-09-21T08:25:29.720741+0000","last_undegraded":"2024-09-21T08:25:29.720741+0000","last_fullsized":"2024-09-21T08:25:29.720741+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:54.693321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"101'13","reported_seq":256,"reported_epoch":136,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:29.717310+0000","last_change":"2024-09-21T08:25:28.705657+0000","last_active":"2024-09-21T08:25:29.717310+0000","last_peered":"2024-09-21T08:25:29.717310+0000","last_clean":"2024-09-21T08:25:29.717310+0000","last_became_active":"2024-09-21T08:24:48.598068+0000","last_became_peered":"2024-09-21T08:24:48.598068+0000","last_unstale":"2024-09-21T08:25:29.717310+0000","last_undegraded":"2024-09-21T08:25:29.717310+0000","last_fullsized":"2024-09-21T08:25:29.717310+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:50.564491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.97548901399999999,"stat_sum":{"num_bytes":3159683,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":8893,"num_write":10,"num_write_kb":2495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":1161879,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"d","length":"1"},{"start":"f","length":"8"},{"start":"18","length":"2"}]},{"pgid":"4.9","version":"73'12","reported_seq":204,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710636+0000","last_change":"2024-09-21T08:24:56.577372+0000","last_active":"2024-09-21T08:25:29.710636+0000","last_peered":"2024-09-21T08:25:29.710636+0000","last_clean":"2024-09-21T08:25:29.710636+0000","last_became_active":"2024-09-21T08:24:56.526686+0000","last_became_peered":"2024-09-21T08:24:56.526686+0000","last_unstale":"2024-09-21T08:25:29.710636+0000","last_undegraded":"2024-09-21T08:25:29.710636+0000","last_fullsized":"2024-09-21T08:25:29.710636+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0,7],"acting":[1,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721543+0000","last_change":"2024-09-21T08:24:38.544579+0000","last_active":"2024-09-21T08:25:29.721543+0000","last_peered":"2024-09-21T08:25:29.721543+0000","last_clean":"2024-09-21T08:25:29.721543+0000","last_became_active":"2024-09-21T08:24:38.544215+0000","last_became_peered":"2024-09-21T08:24:38.544215+0000","last_unstale":"2024-09-21T08:25:29.721543+0000","last_undegraded":"2024-09-21T08:25:29.721543+0000","last_fullsized":"2024-09-21T08:25:29.721543+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,0],"acting":[7,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":142,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710009+0000","last_change":"2024-09-21T08:23:34.566472+0000","last_active":"2024-09-21T08:25:29.710009+0000","last_peered":"2024-09-21T08:25:29.710009+0000","last_clean":"2024-09-21T08:25:29.710009+0000","last_became_active":"2024-09-21T08:23:34.566193+0000","last_became_peered":"2024-09-21T08:23:34.566193+0000","last_unstale":"2024-09-21T08:25:29.710009+0000","last_undegraded":"2024-09-21T08:25:29.710009+0000","last_fullsized":"2024-09-21T08:25:29.710009+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,1],"acting":[2,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":107,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720373+0000","last_change":"2024-09-21T08:25:25.598422+0000","last_active":"2024-09-21T08:25:29.720373+0000","last_peered":"2024-09-21T08:25:29.720373+0000","last_clean":"2024-09-21T08:25:29.720373+0000","last_became_active":"2024-09-21T08:24:56.513965+0000","last_became_peered":"2024-09-21T08:24:56.513965+0000","last_unstale":"2024-09-21T08:25:29.720373+0000","last_undegraded":"2024-09-21T08:25:29.720373+0000","last_fullsized":"2024-09-21T08:25:29.720373+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:35.522812+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,5],"acting":[7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"136'137","reported_seq":433,"reported_epoch":136,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:30.564495+0000","last_change":"2024-09-21T08:25:28.562889+0000","last_active":"2024-09-21T08:25:30.564495+0000","last_peered":"2024-09-21T08:25:30.564495+0000","last_clean":"2024-09-21T08:25:30.564495+0000","last_became_active":"2024-09-21T08:24:56.524762+0000","last_became_peered":"2024-09-21T08:24:56.524762+0000","last_unstale":"2024-09-21T08:25:30.564495+0000","last_undegraded":"2024-09-21T08:25:30.564495+0000","last_fullsized":"2024-09-21T08:25:30.564495+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:52.286541+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.86412189800000005,"stat_sum":{"num_bytes":25115677,"num_objects":12,"num_object_clones":5,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":2,"num_read":103,"num_read_kb":79169,"num_write":97,"num_write_kb":28418,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":12669831,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"f","length":"3"}]},{"pgid":"4.8","version":"73'15","reported_seq":202,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.701741+0000","last_change":"2024-09-21T08:24:56.538033+0000","last_active":"2024-09-21T08:25:29.701741+0000","last_peered":"2024-09-21T08:25:29.701741+0000","last_clean":"2024-09-21T08:25:29.701741+0000","last_became_active":"2024-09-21T08:24:56.506378+0000","last_became_peered":"2024-09-21T08:24:56.506378+0000","last_unstale":"2024-09-21T08:25:29.701741+0000","last_undegraded":"2024-09-21T08:25:29.701741+0000","last_fullsized":"2024-09-21T08:25:29.701741+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.521582+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":181,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721384+0000","last_change":"2024-09-21T08:24:38.583223+0000","last_active":"2024-09-21T08:25:29.721384+0000","last_peered":"2024-09-21T08:25:29.721384+0000","last_clean":"2024-09-21T08:25:29.721384+0000","last_became_active":"2024-09-21T08:24:38.551180+0000","last_became_peered":"2024-09-21T08:24:38.551180+0000","last_unstale":"2024-09-21T08:25:29.721384+0000","last_undegraded":"2024-09-21T08:25:29.721384+0000","last_fullsized":"2024-09-21T08:25:29.721384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:32.960573+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"103'10","reported_seq":195,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720729+0000","last_change":"2024-09-21T08:24:56.562671+0000","last_active":"2024-09-21T08:25:29.720729+0000","last_peered":"2024-09-21T08:25:29.720729+0000","last_clean":"2024-09-21T08:25:29.720729+0000","last_became_active":"2024-09-21T08:24:56.511525+0000","last_became_peered":"2024-09-21T08:24:56.511525+0000","last_unstale":"2024-09-21T08:25:29.720729+0000","last_undegraded":"2024-09-21T08:25:29.720729+0000","last_fullsized":"2024-09-21T08:25:29.720729+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":145,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.701994+0000","last_change":"2024-09-21T08:24:56.500567+0000","last_active":"2024-09-21T08:25:29.701994+0000","last_peered":"2024-09-21T08:25:29.701994+0000","last_clean":"2024-09-21T08:25:29.701994+0000","last_became_active":"2024-09-21T08:24:56.500287+0000","last_became_peered":"2024-09-21T08:24:56.500287+0000","last_unstale":"2024-09-21T08:25:29.701994+0000","last_undegraded":"2024-09-21T08:25:29.701994+0000","last_fullsized":"2024-09-21T08:25:29.701994+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703311+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":77,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710239+0000","last_change":"2024-09-21T08:24:56.500240+0000","last_active":"2024-09-21T08:25:29.710239+0000","last_peered":"2024-09-21T08:25:29.710239+0000","last_clean":"2024-09-21T08:25:29.710239+0000","last_became_active":"2024-09-21T08:24:56.499114+0000","last_became_peered":"2024-09-21T08:24:56.499114+0000","last_unstale":"2024-09-21T08:25:29.710239+0000","last_undegraded":"2024-09-21T08:25:29.710239+0000","last_fullsized":"2024-09-21T08:25:29.710239+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":129,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721270+0000","last_change":"2024-09-21T08:24:48.621753+0000","last_active":"2024-09-21T08:25:29.721270+0000","last_peered":"2024-09-21T08:25:29.721270+0000","last_clean":"2024-09-21T08:25:29.721270+0000","last_became_active":"2024-09-21T08:24:48.590639+0000","last_became_peered":"2024-09-21T08:24:48.590639+0000","last_unstale":"2024-09-21T08:25:29.721270+0000","last_undegraded":"2024-09-21T08:25:29.721270+0000","last_fullsized":"2024-09-21T08:25:29.721270+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":155,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.703149+0000","last_change":"2024-09-21T08:24:56.508740+0000","last_active":"2024-09-21T08:25:29.703149+0000","last_peered":"2024-09-21T08:25:29.703149+0000","last_clean":"2024-09-21T08:25:29.703149+0000","last_became_active":"2024-09-21T08:24:56.508415+0000","last_became_peered":"2024-09-21T08:24:56.508415+0000","last_unstale":"2024-09-21T08:25:29.703149+0000","last_undegraded":"2024-09-21T08:25:29.703149+0000","last_fullsized":"2024-09-21T08:25:29.703149+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703093+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":149,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721259+0000","last_change":"2024-09-21T08:24:56.506870+0000","last_active":"2024-09-21T08:25:29.721259+0000","last_peered":"2024-09-21T08:25:29.721259+0000","last_clean":"2024-09-21T08:25:29.721259+0000","last_became_active":"2024-09-21T08:24:56.502937+0000","last_became_peered":"2024-09-21T08:24:56.502937+0000","last_unstale":"2024-09-21T08:25:29.721259+0000","last_undegraded":"2024-09-21T08:25:29.721259+0000","last_fullsized":"2024-09-21T08:25:29.721259+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":178,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.711539+0000","last_change":"2024-09-21T08:24:48.629077+0000","last_active":"2024-09-21T08:25:29.711539+0000","last_peered":"2024-09-21T08:25:29.711539+0000","last_clean":"2024-09-21T08:25:29.711539+0000","last_became_active":"2024-09-21T08:24:48.602176+0000","last_became_peered":"2024-09-21T08:24:48.602176+0000","last_unstale":"2024-09-21T08:25:29.711539+0000","last_undegraded":"2024-09-21T08:25:29.711539+0000","last_fullsized":"2024-09-21T08:25:29.711539+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":161,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720398+0000","last_change":"2024-09-21T08:24:56.505855+0000","last_active":"2024-09-21T08:25:29.720398+0000","last_peered":"2024-09-21T08:25:29.720398+0000","last_clean":"2024-09-21T08:25:29.720398+0000","last_became_active":"2024-09-21T08:24:56.505134+0000","last_became_peered":"2024-09-21T08:24:56.505134+0000","last_unstale":"2024-09-21T08:25:29.720398+0000","last_undegraded":"2024-09-21T08:25:29.720398+0000","last_fullsized":"2024-09-21T08:25:29.720398+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":94,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.718970+0000","last_change":"2024-09-21T08:24:48.579742+0000","last_active":"2024-09-21T08:25:29.718970+0000","last_peered":"2024-09-21T08:25:29.718970+0000","last_clean":"2024-09-21T08:25:29.718970+0000","last_became_active":"2024-09-21T08:24:48.579439+0000","last_became_peered":"2024-09-21T08:24:48.579439+0000","last_unstale":"2024-09-21T08:25:29.718970+0000","last_undegraded":"2024-09-21T08:25:29.718970+0000","last_fullsized":"2024-09-21T08:25:29.718970+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":152,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.714536+0000","last_change":"2024-09-21T08:24:53.746385+0000","last_active":"2024-09-21T08:25:29.714536+0000","last_peered":"2024-09-21T08:25:29.714536+0000","last_clean":"2024-09-21T08:25:29.714536+0000","last_became_active":"2024-09-21T08:24:38.539845+0000","last_became_peered":"2024-09-21T08:24:38.539845+0000","last_unstale":"2024-09-21T08:25:29.714536+0000","last_undegraded":"2024-09-21T08:25:29.714536+0000","last_fullsized":"2024-09-21T08:25:29.714536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.746295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.746295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.585736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":195,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702880+0000","last_change":"2024-09-21T08:24:56.529301+0000","last_active":"2024-09-21T08:25:29.702880+0000","last_peered":"2024-09-21T08:25:29.702880+0000","last_clean":"2024-09-21T08:25:29.702880+0000","last_became_active":"2024-09-21T08:24:56.499937+0000","last_became_peered":"2024-09-21T08:24:56.499937+0000","last_unstale":"2024-09-21T08:25:29.702880+0000","last_undegraded":"2024-09-21T08:25:29.702880+0000","last_fullsized":"2024-09-21T08:25:29.702880+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":156,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710914+0000","last_change":"2024-09-21T08:24:48.587676+0000","last_active":"2024-09-21T08:25:29.710914+0000","last_peered":"2024-09-21T08:25:29.710914+0000","last_clean":"2024-09-21T08:25:29.710914+0000","last_became_active":"2024-09-21T08:24:48.587393+0000","last_became_peered":"2024-09-21T08:24:48.587393+0000","last_unstale":"2024-09-21T08:25:29.710914+0000","last_undegraded":"2024-09-21T08:25:29.710914+0000","last_fullsized":"2024-09-21T08:25:29.710914+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298529+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,1],"acting":[0,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"101'10","reported_seq":140,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721131+0000","last_change":"2024-09-21T08:24:54.690664+0000","last_active":"2024-09-21T08:25:29.721131+0000","last_peered":"2024-09-21T08:25:29.721131+0000","last_clean":"2024-09-21T08:25:29.721131+0000","last_became_active":"2024-09-21T08:24:48.590539+0000","last_became_peered":"2024-09-21T08:24:48.590539+0000","last_unstale":"2024-09-21T08:25:29.721131+0000","last_undegraded":"2024-09-21T08:25:29.721131+0000","last_fullsized":"2024-09-21T08:25:29.721131+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:02.692436+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":115,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.713504+0000","last_change":"2024-09-21T08:24:38.543166+0000","last_active":"2024-09-21T08:25:29.713504+0000","last_peered":"2024-09-21T08:25:29.713504+0000","last_clean":"2024-09-21T08:25:29.713504+0000","last_became_active":"2024-09-21T08:24:38.542854+0000","last_became_peered":"2024-09-21T08:24:38.542854+0000","last_unstale":"2024-09-21T08:25:29.713504+0000","last_undegraded":"2024-09-21T08:25:29.713504+0000","last_fullsized":"2024-09-21T08:25:29.713504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"104'15","reported_seq":153,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716252+0000","last_change":"2024-09-21T08:24:57.624866+0000","last_active":"2024-09-21T08:25:29.716252+0000","last_peered":"2024-09-21T08:25:29.716252+0000","last_clean":"2024-09-21T08:25:29.716252+0000","last_became_active":"2024-09-21T08:24:48.598970+0000","last_became_peered":"2024-09-21T08:24:48.598970+0000","last_unstale":"2024-09-21T08:25:29.716252+0000","last_undegraded":"2024-09-21T08:25:29.716252+0000","last_fullsized":"2024-09-21T08:25:29.716252+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:23.937573+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":183,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721384+0000","last_change":"2024-09-21T08:24:56.709528+0000","last_active":"2024-09-21T08:25:29.721384+0000","last_peered":"2024-09-21T08:25:29.721384+0000","last_clean":"2024-09-21T08:25:29.721384+0000","last_became_active":"2024-09-21T08:24:56.504665+0000","last_became_peered":"2024-09-21T08:24:56.504665+0000","last_unstale":"2024-09-21T08:25:29.721384+0000","last_undegraded":"2024-09-21T08:25:29.721384+0000","last_fullsized":"2024-09-21T08:25:29.721384+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:56.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.709437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.598671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"73'8","reported_seq":184,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.702755+0000","last_change":"2024-09-21T08:25:18.620373+0000","last_active":"2024-09-21T08:25:29.702755+0000","last_peered":"2024-09-21T08:25:29.702755+0000","last_clean":"2024-09-21T08:25:29.702755+0000","last_became_active":"2024-09-21T08:23:34.566736+0000","last_became_peered":"2024-09-21T08:23:34.566736+0000","last_unstale":"2024-09-21T08:25:29.702755+0000","last_undegraded":"2024-09-21T08:25:29.702755+0000","last_fullsized":"2024-09-21T08:25:29.702755+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:35.593757+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":144,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710968+0000","last_change":"2024-09-21T08:24:48.592895+0000","last_active":"2024-09-21T08:25:29.710968+0000","last_peered":"2024-09-21T08:25:29.710968+0000","last_clean":"2024-09-21T08:25:29.710968+0000","last_became_active":"2024-09-21T08:24:48.592280+0000","last_became_peered":"2024-09-21T08:24:48.592280+0000","last_unstale":"2024-09-21T08:25:29.710968+0000","last_undegraded":"2024-09-21T08:25:29.710968+0000","last_fullsized":"2024-09-21T08:25:29.710968+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":128,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.718904+0000","last_change":"2024-09-21T08:24:56.539935+0000","last_active":"2024-09-21T08:25:29.718904+0000","last_peered":"2024-09-21T08:25:29.718904+0000","last_clean":"2024-09-21T08:25:29.718904+0000","last_became_active":"2024-09-21T08:24:56.517397+0000","last_became_peered":"2024-09-21T08:24:56.517397+0000","last_unstale":"2024-09-21T08:25:29.718904+0000","last_undegraded":"2024-09-21T08:25:29.718904+0000","last_fullsized":"2024-09-21T08:25:29.718904+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436820+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":119,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721278+0000","last_change":"2024-09-21T08:24:56.504037+0000","last_active":"2024-09-21T08:25:29.721278+0000","last_peered":"2024-09-21T08:25:29.721278+0000","last_clean":"2024-09-21T08:25:29.721278+0000","last_became_active":"2024-09-21T08:24:56.503232+0000","last_became_peered":"2024-09-21T08:24:56.503232+0000","last_unstale":"2024-09-21T08:25:29.721278+0000","last_undegraded":"2024-09-21T08:25:29.721278+0000","last_fullsized":"2024-09-21T08:25:29.721278+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":148,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716953+0000","last_change":"2024-09-21T08:24:48.591631+0000","last_active":"2024-09-21T08:25:29.716953+0000","last_peered":"2024-09-21T08:25:29.716953+0000","last_clean":"2024-09-21T08:25:29.716953+0000","last_became_active":"2024-09-21T08:24:48.591342+0000","last_became_peered":"2024-09-21T08:24:48.591342+0000","last_unstale":"2024-09-21T08:25:29.716953+0000","last_undegraded":"2024-09-21T08:25:29.716953+0000","last_fullsized":"2024-09-21T08:25:29.716953+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":93,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.714072+0000","last_change":"2024-09-21T08:24:48.600966+0000","last_active":"2024-09-21T08:25:29.714072+0000","last_peered":"2024-09-21T08:25:29.714072+0000","last_clean":"2024-09-21T08:25:29.714072+0000","last_became_active":"2024-09-21T08:24:48.600045+0000","last_became_peered":"2024-09-21T08:24:48.600045+0000","last_unstale":"2024-09-21T08:25:29.714072+0000","last_undegraded":"2024-09-21T08:25:29.714072+0000","last_fullsized":"2024-09-21T08:25:29.714072+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":178,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716640+0000","last_change":"2024-09-21T08:24:56.573418+0000","last_active":"2024-09-21T08:25:29.716640+0000","last_peered":"2024-09-21T08:25:29.716640+0000","last_clean":"2024-09-21T08:25:29.716640+0000","last_became_active":"2024-09-21T08:24:56.524898+0000","last_became_peered":"2024-09-21T08:24:56.524898+0000","last_unstale":"2024-09-21T08:25:29.716640+0000","last_undegraded":"2024-09-21T08:25:29.716640+0000","last_fullsized":"2024-09-21T08:25:29.716640+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436985+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":167,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.721844+0000","last_change":"2024-09-21T08:24:56.505458+0000","last_active":"2024-09-21T08:25:29.721844+0000","last_peered":"2024-09-21T08:25:29.721844+0000","last_clean":"2024-09-21T08:25:29.721844+0000","last_became_active":"2024-09-21T08:24:56.504584+0000","last_became_peered":"2024-09-21T08:24:56.504584+0000","last_unstale":"2024-09-21T08:25:29.721844+0000","last_undegraded":"2024-09-21T08:25:29.721844+0000","last_fullsized":"2024-09-21T08:25:29.721844+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":93,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710113+0000","last_change":"2024-09-21T08:24:48.591350+0000","last_active":"2024-09-21T08:25:29.710113+0000","last_peered":"2024-09-21T08:25:29.710113+0000","last_clean":"2024-09-21T08:25:29.710113+0000","last_became_active":"2024-09-21T08:24:48.590426+0000","last_became_peered":"2024-09-21T08:24:48.590426+0000","last_unstale":"2024-09-21T08:25:29.710113+0000","last_undegraded":"2024-09-21T08:25:29.710113+0000","last_fullsized":"2024-09-21T08:25:29.710113+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298561+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,5],"acting":[0,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":138,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.711446+0000","last_change":"2024-09-21T08:23:36.580460+0000","last_active":"2024-09-21T08:25:29.711446+0000","last_peered":"2024-09-21T08:25:29.711446+0000","last_clean":"2024-09-21T08:25:29.711446+0000","last_became_active":"2024-09-21T08:23:36.579949+0000","last_became_peered":"2024-09-21T08:23:36.579949+0000","last_unstale":"2024-09-21T08:25:29.711446+0000","last_undegraded":"2024-09-21T08:25:29.711446+0000","last_fullsized":"2024-09-21T08:25:29.711446+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":169,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720575+0000","last_change":"2024-09-21T08:24:56.531655+0000","last_active":"2024-09-21T08:25:29.720575+0000","last_peered":"2024-09-21T08:25:29.720575+0000","last_clean":"2024-09-21T08:25:29.720575+0000","last_became_active":"2024-09-21T08:24:56.503223+0000","last_became_peered":"2024-09-21T08:24:56.503223+0000","last_unstale":"2024-09-21T08:25:29.720575+0000","last_undegraded":"2024-09-21T08:25:29.720575+0000","last_fullsized":"2024-09-21T08:25:29.720575+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.478113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":151,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.703242+0000","last_change":"2024-09-21T08:23:30.544860+0000","last_active":"2024-09-21T08:25:29.703242+0000","last_peered":"2024-09-21T08:25:29.703242+0000","last_clean":"2024-09-21T08:25:29.703242+0000","last_became_active":"2024-09-21T08:23:30.544486+0000","last_became_peered":"2024-09-21T08:23:30.544486+0000","last_unstale":"2024-09-21T08:25:29.703242+0000","last_undegraded":"2024-09-21T08:25:29.703242+0000","last_fullsized":"2024-09-21T08:25:29.703242+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:31.703326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":161,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720062+0000","last_change":"2024-09-21T08:24:58.688115+0000","last_active":"2024-09-21T08:25:29.720062+0000","last_peered":"2024-09-21T08:25:29.720062+0000","last_clean":"2024-09-21T08:25:29.720062+0000","last_became_active":"2024-09-21T08:24:56.503123+0000","last_became_peered":"2024-09-21T08:24:56.503123+0000","last_unstale":"2024-09-21T08:25:29.720062+0000","last_undegraded":"2024-09-21T08:25:29.720062+0000","last_fullsized":"2024-09-21T08:25:29.720062+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.688046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.688046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.364343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":99,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710583+0000","last_change":"2024-09-21T08:24:56.508632+0000","last_active":"2024-09-21T08:25:29.710583+0000","last_peered":"2024-09-21T08:25:29.710583+0000","last_clean":"2024-09-21T08:25:29.710583+0000","last_became_active":"2024-09-21T08:24:56.508323+0000","last_became_peered":"2024-09-21T08:24:56.508323+0000","last_unstale":"2024-09-21T08:25:29.710583+0000","last_undegraded":"2024-09-21T08:25:29.710583+0000","last_fullsized":"2024-09-21T08:25:29.710583+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":137,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.710181+0000","last_change":"2024-09-21T08:24:56.546097+0000","last_active":"2024-09-21T08:25:29.710181+0000","last_peered":"2024-09-21T08:25:29.710181+0000","last_clean":"2024-09-21T08:25:29.710181+0000","last_became_active":"2024-09-21T08:24:56.517807+0000","last_became_peered":"2024-09-21T08:24:56.517807+0000","last_unstale":"2024-09-21T08:25:29.710181+0000","last_undegraded":"2024-09-21T08:25:29.710181+0000","last_fullsized":"2024-09-21T08:25:29.710181+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.437031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":162,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.713839+0000","last_change":"2024-09-21T08:24:56.517657+0000","last_active":"2024-09-21T08:25:29.713839+0000","last_peered":"2024-09-21T08:25:29.713839+0000","last_clean":"2024-09-21T08:25:29.713839+0000","last_became_active":"2024-09-21T08:24:56.517349+0000","last_became_peered":"2024-09-21T08:24:56.517349+0000","last_unstale":"2024-09-21T08:25:29.713839+0000","last_undegraded":"2024-09-21T08:25:29.713839+0000","last_fullsized":"2024-09-21T08:25:29.713839+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.298429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,1],"acting":[0,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":115,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720798+0000","last_change":"2024-09-21T08:24:56.507802+0000","last_active":"2024-09-21T08:25:29.720798+0000","last_peered":"2024-09-21T08:25:29.720798+0000","last_clean":"2024-09-21T08:25:29.720798+0000","last_became_active":"2024-09-21T08:24:56.506356+0000","last_became_peered":"2024-09-21T08:24:56.506356+0000","last_unstale":"2024-09-21T08:25:29.720798+0000","last_undegraded":"2024-09-21T08:25:29.720798+0000","last_fullsized":"2024-09-21T08:25:29.720798+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:59.651421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,1],"acting":[7,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":149,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.720371+0000","last_change":"2024-09-21T08:24:48.590701+0000","last_active":"2024-09-21T08:25:29.720371+0000","last_peered":"2024-09-21T08:25:29.720371+0000","last_clean":"2024-09-21T08:25:29.720371+0000","last_became_active":"2024-09-21T08:24:48.590283+0000","last_became_peered":"2024-09-21T08:24:48.590283+0000","last_unstale":"2024-09-21T08:25:29.720371+0000","last_undegraded":"2024-09-21T08:25:29.720371+0000","last_fullsized":"2024-09-21T08:25:29.720371+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":144,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.716961+0000","last_change":"2024-09-21T08:24:38.549206+0000","last_active":"2024-09-21T08:25:29.716961+0000","last_peered":"2024-09-21T08:25:29.716961+0000","last_clean":"2024-09-21T08:25:29.716961+0000","last_became_active":"2024-09-21T08:24:38.547039+0000","last_became_peered":"2024-09-21T08:24:38.547039+0000","last_unstale":"2024-09-21T08:25:29.716961+0000","last_undegraded":"2024-09-21T08:25:29.716961+0000","last_fullsized":"2024-09-21T08:25:29.716961+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:34.436726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":102,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.709436+0000","last_change":"2024-09-21T08:24:51.848588+0000","last_active":"2024-09-21T08:25:29.709436+0000","last_peered":"2024-09-21T08:25:29.709436+0000","last_clean":"2024-09-21T08:25:29.709436+0000","last_became_active":"2024-09-21T08:24:48.598312+0000","last_became_peered":"2024-09-21T08:24:48.598312+0000","last_unstale":"2024-09-21T08:25:29.709436+0000","last_undegraded":"2024-09-21T08:25:29.709436+0000","last_fullsized":"2024-09-21T08:25:29.709436+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:53.281030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,2],"acting":[0,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":128,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.703191+0000","last_change":"2024-09-21T08:24:56.568191+0000","last_active":"2024-09-21T08:25:29.703191+0000","last_peered":"2024-09-21T08:25:29.703191+0000","last_clean":"2024-09-21T08:25:29.703191+0000","last_became_active":"2024-09-21T08:24:56.516832+0000","last_became_peered":"2024-09-21T08:24:56.516832+0000","last_unstale":"2024-09-21T08:25:29.703191+0000","last_undegraded":"2024-09-21T08:25:29.703191+0000","last_fullsized":"2024-09-21T08:25:29.703191+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.301088+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,0],"acting":[5,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":150,"reported_epoch":136,"state":"active+clean","last_fresh":"2024-09-21T08:25:29.711155+0000","last_change":"2024-09-21T08:25:25.350265+0000","last_active":"2024-09-21T08:25:29.711155+0000","last_peered":"2024-09-21T08:25:29.711155+0000","last_clean":"2024-09-21T08:25:29.711155+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:25:29.711155+0000","last_undegraded":"2024-09-21T08:25:29.711155+0000","last_fullsized":"2024-09-21T08:25:29.711155+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.350197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.350197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:48.284282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":145370295,"num_objects":60,"num_object_clones":17,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":5,"num_read":577,"num_read_kb":455641,"num_write":577,"num_write_kb":212452,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":67327959,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":382750720,"data_stored":409753657,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":920,"ondisk_log_size":920,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":13,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":75,"ondisk_log_size":75,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":679,"num_read_kb":436,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":225,"num_bytes_recovered":4720,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":638976,"data_stored":16970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":402,"ondisk_log_size":402,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3134,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":7973,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":115,"num_read_kb":110,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":19,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":2361472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070493,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":113880,"kb_used_data":78528,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93631272,"statfs":{"total":95995035648,"available":95878422528,"internally_reserved":0,"allocated":80412672,"data_stored":86322953,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397539,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659174,"num_pgs":85,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":75496,"kb_used_data":44304,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93669656,"statfs":{"total":95995035648,"available":95917727744,"internally_reserved":0,"allocated":45367296,"data_stored":47055492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986220,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182448,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411382,"num_pgs":76,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102412,"kb_used_data":71216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93642740,"statfs":{"total":95995035648,"available":95890165760,"internally_reserved":0,"allocated":72925184,"data_stored":81130843,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574906,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112172,"kb_used_data":80976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93632980,"statfs":{"total":95995035648,"available":95880171520,"internally_reserved":0,"allocated":82919424,"data_stored":93471886,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738431,"num_pgs":90,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":90032,"kb_used_data":58836,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93655120,"statfs":{"total":95995035648,"available":95902842880,"internally_reserved":0,"allocated":60248064,"data_stored":67462189,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":56254464,"data_stored":62744832,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":75538432,"data_stored":81949970,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":68149248,"data_stored":73621389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32387072,"data_stored":32188068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":43945984,"data_stored":46351671,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":78381056,"data_stored":85027312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:25:35.621 INFO:tasks.thrashosds.thrasher:Setting 2.1 to [6, 4, 0, 5, 7, 3] 2024-09-21T08:25:35.621 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '2.1', '6', '4', '0', '5', '7', '3'] 2024-09-21T08:25:35.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 2.1 6 4 0 5 7 3 2024-09-21T08:25:35.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:35.668 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:35.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:35.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:35.748 INFO:tasks.rados.rados.0.smithi137.stdout:344: done (1 left) 2024-09-21T08:25:35.748 INFO:tasks.rados.rados.0.smithi137.stdout:345: done (0 left) 2024-09-21T08:25:35.748 INFO:tasks.rados.rados.0.smithi137.stdout:346: write (excl) oid 46 current snap is 30 2024-09-21T08:25:35.748 INFO:tasks.rados.rados.0.smithi137.stdout:346: seq_num 154 ranges {588278=715406,1711874=668042} 2024-09-21T08:25:35.755 INFO:tasks.rados.rados.0.smithi137.stdout:346: writing smithi13779221-46 from 588278 to 1303684 tid 1 2024-09-21T08:25:35.762 INFO:tasks.rados.rados.0.smithi137.stdout:346: writing smithi13779221-46 from 1711874 to 2379916 tid 2 2024-09-21T08:25:35.762 INFO:tasks.rados.rados.0.smithi137.stdout:347: copy_from oid 40 from oid 14 current snap is 30 2024-09-21T08:25:35.762 INFO:tasks.rados.rados.0.smithi137.stdout:346: finishing write tid 1 to smithi13779221-46 2024-09-21T08:25:35.762 INFO:tasks.rados.rados.0.smithi137.stdout:348: read oid 6 snap -1 2024-09-21T08:25:35.762 INFO:tasks.rados.rados.0.smithi137.stdout:348: expect (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:25:35.762 INFO:tasks.rados.rados.0.smithi137.stdout:349: rollback oid 39 current snap is 30 2024-09-21T08:25:35.762 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 39 to 27 2024-09-21T08:25:35.762 INFO:tasks.rados.rados.0.smithi137.stdout:350: delete oid 42 current snap is 30 2024-09-21T08:25:35.767 INFO:tasks.rados.rados.0.smithi137.stdout:350: done (4 left) 2024-09-21T08:25:35.767 INFO:tasks.rados.rados.0.smithi137.stdout:351: snap_create 2024-09-21T08:25:35.768 INFO:tasks.rados.rados.0.smithi137.stdout:349: finishing rollback tid 0 to smithi13779221-39 2024-09-21T08:25:35.768 INFO:tasks.rados.rados.0.smithi137.stdout:349: finishing rollback tid 1 to smithi13779221-39 2024-09-21T08:25:35.769 INFO:tasks.rados.rados.0.smithi137.stdout:348: expect (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:25:35.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:35.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:35.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:35.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:35.820 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:35.820 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:35.838 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:35.838 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:35.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:35.855 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:35.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[29634]: from='client.15000 v1:172.21.15.137:0/3531661646' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:25:35.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[29634]: from='client.44885 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:25:35.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[29634]: osdmap e141: 8 total, 8 up, 5 in 2024-09-21T08:25:35.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:35.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[29634]: 1.0 scrub starts 2024-09-21T08:25:35.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[29634]: 1.0 scrub ok 2024-09-21T08:25:35.856 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[41044]: from='client.15000 v1:172.21.15.137:0/3531661646' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:25:35.856 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[41044]: from='client.44885 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:25:35.856 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[41044]: osdmap e141: 8 total, 8 up, 5 in 2024-09-21T08:25:35.856 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:35.856 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[41044]: 1.0 scrub starts 2024-09-21T08:25:35.856 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:35 smithi137 ceph-mon[41044]: 1.0 scrub ok 2024-09-21T08:25:35.877 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:35.877 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:35.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:35.894 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:35.894 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:35.910 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:35.911 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:35.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:35.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:35.915 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:35.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:35 smithi165 ceph-mon[32893]: from='client.15000 v1:172.21.15.137:0/3531661646' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:25:35.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:35 smithi165 ceph-mon[32893]: from='client.44885 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:25:35.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:35 smithi165 ceph-mon[32893]: osdmap e141: 8 total, 8 up, 5 in 2024-09-21T08:25:35.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:35 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:35.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:35 smithi165 ceph-mon[32893]: 1.0 scrub starts 2024-09-21T08:25:35.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:35 smithi165 ceph-mon[32893]: 1.0 scrub ok 2024-09-21T08:25:35.928 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:35.928 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:35.932 INFO:tasks.rados.rados.0.smithi137.stdout:346: finishing write tid 2 to smithi13779221-46 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:346: finishing write tid 3 to smithi13779221-46 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:346: finishing write tid 4 to smithi13779221-46 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:346: oid 46 updating version 0 to 157 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:346: oid 46 updating version 157 to 158 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:346: oid 46 version 158 is already newer than 156 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 158 (ObjNum 154 snap 30 seq_num 154) dirty exists 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:346: left oid 46 (ObjNum 154 snap 30 seq_num 154) 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:349: finishing rollback tid 2 to smithi13779221-39 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 17 (ObjNum 69 snap 7 seq_num 69) dirty exists 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:347: finishing copy_from racing read to smithi13779221-40 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:347: finishing copy_from to smithi13779221-40 2024-09-21T08:25:35.933 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-21T08:25:35.945 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:35.945 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:36.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:36.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:36.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:36.039 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:36.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:36.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:36.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:36.162 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:36.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:36.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:36.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:36.285 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:36.366 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:25:36.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:36.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:36.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:36.410 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:36.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:36.534 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:36.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:36.535 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:36.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:36.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:36.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:36.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:36.752 INFO:tasks.rados.rados.0.smithi137.stdout:346: done (4 left) 2024-09-21T08:25:36.752 INFO:tasks.rados.rados.0.smithi137.stdout:347: done (3 left) 2024-09-21T08:25:36.752 INFO:tasks.rados.rados.0.smithi137.stdout:348: done (2 left) 2024-09-21T08:25:36.752 INFO:tasks.rados.rados.0.smithi137.stdout:349: done (1 left) 2024-09-21T08:25:36.752 INFO:tasks.rados.rados.0.smithi137.stdout:351: done (0 left) 2024-09-21T08:25:36.752 INFO:tasks.rados.rados.0.smithi137.stdout:352: read oid 33 snap -1 2024-09-21T08:25:36.752 INFO:tasks.rados.rados.0.smithi137.stdout:352: expect (ObjNum 112 snap 22 seq_num 112) 2024-09-21T08:25:36.752 INFO:tasks.rados.rados.0.smithi137.stdout:353: snap_remove snap 29 2024-09-21T08:25:36.755 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:36.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:36.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:36.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:36.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:36.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:36.886 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:25:36.886 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:25:36.886 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:25:36.886 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:25:36.886 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14568.0:8 2.0 2:353ec8f9:::gateway.conf:head [setxattr epoch (1) in=5b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e74)", 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:41.490634+0000", 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 115.39026737099999, 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006461774, 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14568", 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.165:0/3513922034", 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 8 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.887 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.490634+0000", 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.490634+0000", 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.490635+0000", 2024-09-21T08:25:36.888 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.119e-06 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.490645+0000", 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7629999999999995e-06 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.490655+0000", 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9960000000000005e-06 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.490660+0000", 2024-09-21T08:25:36.889 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1789999999999999e-06 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.490692+0000", 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2598000000000002e-05 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.490731+0000", 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8819999999999998e-05 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,1", 2024-09-21T08:25:36.890 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.490799+0000", 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.758e-05 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.491028+0000", 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022924600000000001 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.496929+0000", 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059005860000000002 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.891 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.496995+0000", 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6214e-05 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.497054+0000", 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9008999999999997e-05 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:41.497095+0000", 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1664e-05 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.892 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1086 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94837804044672 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159598+0000", 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 58.721302827999999, 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40422385100000002, 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1086 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.893 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159598+0000", 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159598+0000", 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159601+0000", 2024-09-21T08:25:36.894 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2270000000000002e-06 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159611+0000", 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9590000000000007e-06 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159634+0000", 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2931000000000001e-05 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.895 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159651+0000", 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6959000000000001e-05 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105437 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159772+0000", 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5319e-05 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.896 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559521+0000", 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39974934899999998 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559838+0000", 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031693299999999998 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.561288+0000", 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014493150000000001 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.897 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.561373+0000", 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4902999999999997e-05 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.563822+0000", 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024495189999999998 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.898 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:250 7.5 7:a9dd6c7d:::smithi13779221-47:head [write 557999~703664] snapc 4=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:45.497671+0000", 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 51.383230066000003, 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007295607, 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 250 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.899 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.497671+0000", 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.497671+0000", 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.497675+0000", 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8430000000000003e-06 2024-09-21T08:25:36.900 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.498471+0000", 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079575100000000003 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.498485+0000", 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4134000000000001e-05 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.498493+0000", 2024-09-21T08:25:36.901 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4060000000000005e-06 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.498585+0000", 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1728e-05 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.498591+0000", 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2740000000000004e-06 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.902 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.500741+0000", 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021502460000000002 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.500821+0000", 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9678999999999994e-05 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.500911+0000", 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0255999999999998e-05 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.903 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.502283+0000", 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013715520000000001 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.504854+0000", 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025709190000000001 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.504901+0000", 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7258000000000001e-05 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.904 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.504939+0000", 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7734000000000002e-05 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:45.504967+0000", 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7827e-05 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:277 7.5 7:a6c204bd:::smithi13779221-42:head [copy-from ver 6] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:25:36.905 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.588389+0000", 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.292512451, 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013364199, 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 277 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.906 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588389+0000", 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588389+0000", 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588391+0000", 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2759999999999999e-06 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.907 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588399+0000", 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5620000000000004e-06 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588413+0000", 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3451e-05 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588420+0000", 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6769999999999998e-06 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.908 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588486+0000", 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6549000000000003e-05 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588587+0000", 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100789 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.592068+0000", 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034810430000000001 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.909 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.594819+0000", 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027511670000000001 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.598876+0000", 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040573500000000004 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.601685+0000", 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028083639999999998 2024-09-21T08:25:36.910 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.601720+0000", 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.54e-05 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.601753+0000", 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2570999999999997e-05 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.911 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:342 7.5 7:aa5ad64c:::smithi13779221-49:head [sparse-read 0~1365420] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567039+0000", 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 48.313861625000001, 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070215970000000001, 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 342 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.912 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567039+0000", 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567039+0000", 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567041+0000", 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.116e-06 2024-09-21T08:25:36.913 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567046+0000", 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6829999999999996e-06 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567053+0000", 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4960000000000001e-06 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567059+0000", 2024-09-21T08:25:36.914 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.285e-06 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.572129+0000", 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050696650000000001 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.572217+0000", 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8523000000000006e-05 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.574061+0000", 2024-09-21T08:25:36.915 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001843829 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:343 7.5 7:aa5ad64c:::smithi13779221-49:head [read 0~2806736,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567068+0000", 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 48.313833224, 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010394037, 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.916 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 343 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567068+0000", 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567068+0000", 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.917 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567069+0000", 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7899999999999998e-07 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567073+0000", 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5569999999999997e-06 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567079+0000", 2024-09-21T08:25:36.918 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0900000000000001e-06 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567084+0000", 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2739999999999998e-06 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.574117+0000", 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070326829999999996 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.919 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.574229+0000", 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111501 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.577462+0000", 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032332530000000002 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:353 7.c e96/94 7:361ee62b:::smithi13779221-40:head v 96'14, mlcod=93'11)", 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.908212+0000", 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.972688666000003, 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.008046826, 2024-09-21T08:25:36.920 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.908212+0000", 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.908212+0000", 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.921 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.908213+0000", 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4700000000000002e-07 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.909706+0000", 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001493316 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.909725+0000", 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8861000000000001e-05 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.922 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.909735+0000", 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0287e-05 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.909808+0000", 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2986999999999998e-05 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.909833+0000", 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5256e-05 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.923 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916211+0000", 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063771970000000002 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916259+0000", 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8575000000000002e-05 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:427 7.5 7:a9dd6c7d:::smithi13779221-47:head [sparse-read 0~2179573,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:25:36.924 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:56.509397+0000", 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.371504319000003, 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006440126, 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 427 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.925 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509397+0000", 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509397+0000", 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509399+0000", 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2170000000000001e-06 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.926 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.927 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509405+0000", 2024-09-21T08:25:36.927 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4810000000000003e-06 2024-09-21T08:25:36.927 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.927 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.927 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509415+0000", 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8700000000000004e-06 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509424+0000", 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7020000000000007e-06 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.510808+0000", 2024-09-21T08:25:36.928 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001384031 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.510871+0000", 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2736000000000004e-05 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.515837+0000", 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049660889999999999 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.929 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:428 7.5 7:a9dd6c7d:::smithi13779221-47:head [sparse-read 0~2179573] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:56.509438+0000", 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.371463413999997, 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068355630000000002, 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 428 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.930 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509438+0000", 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509438+0000", 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509438+0000", 2024-09-21T08:25:36.931 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6599999999999995e-07 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509441+0000", 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0800000000000002e-06 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509447+0000", 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6840000000000001e-06 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.932 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509452+0000", 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9420000000000003e-06 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.515877+0000", 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064248810000000003 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.515984+0000", 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000106519 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.933 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.516273+0000", 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028969100000000002 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:429 7.5 7:a9dd6c7d:::smithi13779221-47:head [sparse-read 0~2179573] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:56.509459+0000", 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 40.371441648999998, 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071148419999999997, 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.934 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 429 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509459+0000", 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.935 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509459+0000", 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509460+0000", 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0699999999999997e-07 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509468+0000", 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.422e-06 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509474+0000", 2024-09-21T08:25:36.936 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6489999999999996e-06 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.509479+0000", 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0259999999999998e-06 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.516391+0000", 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069119960000000001 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.937 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.516461+0000", 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9723999999999999e-05 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:56.516574+0000", 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113418 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:557 7.5 7:aa5ad64c:::smithi13779221-49:head [write 796665~544891] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:09.536846+0000", 2024-09-21T08:25:36.938 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.344055104999999, 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074818769999999996, 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 557 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.536846+0000", 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.939 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.536846+0000", 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.536854+0000", 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5959999999999998e-06 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.537539+0000", 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00068562400000000002 2024-09-21T08:25:36.940 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.537559+0000", 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9732999999999999e-05 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.537586+0000", 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6937e-05 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.537685+0000", 2024-09-21T08:25:36.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9329999999999999e-05 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.537691+0000", 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8510000000000004e-06 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.540572+0000", 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002880985 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.540670+0000", 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8418000000000005e-05 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.540845+0000", 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017502699999999999 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.542521+0000", 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016757720000000001 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.543868+0000", 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013465969999999999 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544217+0000", 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034874500000000002 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544274+0000", 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7732000000000002e-05 2024-09-21T08:25:36.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544328+0000", 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3529999999999997e-05 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:552 7.d e116/102 7:b0b32695:::smithi13779221-19:head v 116'26, mlcod=107'23)", 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:09.540250+0000", 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.340651083000001, 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068591310000000001, 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.945 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.540250+0000", 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.540250+0000", 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.540250+0000", 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2500000000000001e-07 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544694+0000", 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044434139999999997 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544703+0000", 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4790000000000002e-06 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544708+0000", 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.203e-06 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544756+0000", 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7298e-05 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544778+0000", 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2365e-05 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.547071+0000", 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002292877 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.547109+0000", 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.807e-05 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.949 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:558 7.5 7:aa5ad64c:::smithi13779221-49:head [write 1759554~650689] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:09.544155+0000", 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 27.336745843999999, 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073056079999999999, 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 558 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.950 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544155+0000", 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544155+0000", 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544157+0000", 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.888e-06 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544836+0000", 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00067898500000000001 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544845+0000", 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8929999999999994e-06 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544850+0000", 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9919999999999998e-06 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544885+0000", 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4931000000000001e-05 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.544953+0000", 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8013000000000001e-05 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.545066+0000", 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113124 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.548892+0000", 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038257920000000002 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.551333+0000", 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024407769999999999 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.551387+0000", 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4101000000000003e-05 2024-09-21T08:25:36.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.551424+0000", 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7005000000000002e-05 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:09.551461+0000", 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7107000000000002e-05 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:571 7.1 e117/102 7:88bf4ad2:::smithi13779221-6:head v 117'8, mlcod=0'0)", 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:10.574774+0000", 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.306126715000001, 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080224489999999992, 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.574774+0000", 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.574774+0000", 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.574776+0000", 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0839999999999999e-06 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.580673+0000", 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058963469999999997 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.580697+0000", 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4680000000000001e-05 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.580719+0000", 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1926999999999999e-05 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.580771+0000", 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1795999999999997e-05 2024-09-21T08:25:36.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.580792+0000", 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1078e-05 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.582766+0000", 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019732970000000002 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.582797+0000", 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1239999999999999e-05 2024-09-21T08:25:36.959 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:592 7.5 7:aaa5c288:::smithi13779221-20:head [copy-from ver 40] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:10.738788+0000", 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.142113282, 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015504372000000001, 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.960 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 592 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738788+0000", 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738788+0000", 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738790+0000", 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7349999999999999e-06 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738795+0000", 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.271e-06 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738807+0000", 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1919000000000001e-05 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738829+0000", 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2436999999999999e-05 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738881+0000", 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2021000000000002e-05 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738943+0000", 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1517000000000001e-05 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.744334+0000", 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053909250000000004 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.747951+0000", 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036174990000000001 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.752148+0000", 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041967590000000004 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.754233+0000", 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020854340000000002 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.754266+0000", 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2697000000000002e-05 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.754292+0000", 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6157999999999999e-05 2024-09-21T08:25:36.965 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:593 7.5 7:aaa5c288:::smithi13779221-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:10.738842+0000", 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.142058845000001, 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083558569999999995, 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.966 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 593 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738842+0000", 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738842+0000", 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738843+0000", 2024-09-21T08:25:36.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3099999999999998e-07 2024-09-21T08:25:36.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738845+0000", 2024-09-21T08:25:36.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.03e-06 2024-09-21T08:25:36.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:36.969 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:36.969 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738849+0000", 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9790000000000004e-06 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738852+0000", 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.061e-06 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.739056+0000", 2024-09-21T08:25:36.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000203895 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.739087+0000", 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1112000000000003e-05 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.747035+0000", 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0079478599999999993 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.747112+0000", 2024-09-21T08:25:36.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7483000000000003e-05 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.747198+0000", 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5906e-05 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:622 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2890771,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e120)", 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:13.545665+0000", 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 23.335235634, 2024-09-21T08:25:36.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065128620000000003, 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 622 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.545665+0000", 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.973 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.545665+0000", 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.545666+0000", 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3000000000000001e-07 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.545670+0000", 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9009999999999996e-06 2024-09-21T08:25:36.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.545679+0000", 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8939999999999999e-06 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.545683+0000", 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8310000000000003e-06 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.545788+0000", 2024-09-21T08:25:36.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105276 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.545793+0000", 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7299999999999996e-06 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.551627+0000", 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0058340570000000001 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.551723+0000", 2024-09-21T08:25:36.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6453999999999995e-05 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:13.552178+0000", 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000455189 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:843 7.5 7:a6c204bd:::smithi13779221-42:head [rollback 1b] snapc 1b=[1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:29.723048+0000", 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.1578533899999996, 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005981676, 2024-09-21T08:25:36.977 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 843 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723048+0000", 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723048+0000", 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723049+0000", 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.119e-06 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723052+0000", 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2179999999999999e-06 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723059+0000", 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3309999999999998e-06 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723064+0000", 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1449999999999999e-06 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723975+0000", 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00091017599999999995 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.724070+0000", 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5285999999999995e-05 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.724223+0000", 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000153296 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.724924+0000", 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070125299999999999 2024-09-21T08:25:36.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.726847+0000", 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001922544 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.728912+0000", 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020649790000000002 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.728971+0000", 2024-09-21T08:25:36.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9077999999999999e-05 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.729029+0000", 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8251000000000002e-05 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:844 7.5 7:a6c204bd:::smithi13779221-42:head [append 0~0] snapc 1b=[1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:29.723079+0000", 2024-09-21T08:25:36.983 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.157821684, 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060466269999999997, 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 844 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723079+0000", 2024-09-21T08:25:36.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723079+0000", 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723080+0000", 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8100000000000003e-07 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723083+0000", 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6529999999999998e-06 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723088+0000", 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8200000000000002e-06 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.723093+0000", 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7809999999999996e-06 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.724564+0000", 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001470968 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.724650+0000", 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5978999999999998e-05 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.724754+0000", 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010402499999999999 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.725033+0000", 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000278776 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.726871+0000", 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018375889999999999 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.729067+0000", 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00219601 2024-09-21T08:25:36.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.729099+0000", 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2707999999999999e-05 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:29.729126+0000", 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6737000000000002e-05 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:849 7.5 7:aaa5c288:::smithi13779221-20:head [stat,delete] snapc 1b=[1a] ondisk+write+known_if_redirected+supports_pool_eio e137)", 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:30.723884+0000", 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 6.1570172960000003, 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073050959999999996, 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 849 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:25:36.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.723884+0000", 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.723884+0000", 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.723887+0000", 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.658e-06 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.723900+0000", 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2743e-05 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.723928+0000", 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7812000000000001e-05 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.723941+0000", 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3366999999999999e-05 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.724045+0000", 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103979 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.724053+0000", 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0169999999999996e-06 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.729819+0000", 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057659599999999997 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.729878+0000", 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8510000000000001e-05 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:30.731189+0000", 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00131105 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:25:36.994 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:25:36.995 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:36 smithi137 ceph-mon[29634]: osdmap e142: 8 total, 8 up, 5 in 2024-09-21T08:25:36.995 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:36 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1959101304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:36.995 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:36 smithi137 ceph-mon[29634]: pgmap v119: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 142 active+clean; 136 MiB data, 477 MiB used, 447 GiB / 447 GiB avail; 28 MiB/s rd, 6.2 MiB/s wr, 41 op/s 2024-09-21T08:25:36.995 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:36 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:36.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:36 smithi137 ceph-mon[41044]: osdmap e142: 8 total, 8 up, 5 in 2024-09-21T08:25:36.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:36 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1959101304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:36.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:36 smithi137 ceph-mon[41044]: pgmap v119: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 142 active+clean; 136 MiB data, 477 MiB used, 447 GiB / 447 GiB avail; 28 MiB/s rd, 6.2 MiB/s wr, 41 op/s 2024-09-21T08:25:36.996 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:36 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:36 smithi165 ceph-mon[32893]: osdmap e142: 8 total, 8 up, 5 in 2024-09-21T08:25:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:36 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1959101304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:36 smithi165 ceph-mon[32893]: pgmap v119: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 142 active+clean; 136 MiB data, 477 MiB used, 447 GiB / 447 GiB avail; 28 MiB/s rd, 6.2 MiB/s wr, 41 op/s 2024-09-21T08:25:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:36 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:37.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:37.092 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:37.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:37.093 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:37.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:37.215 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:37.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:37.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:37.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:37.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:37.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:37.339 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:37.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:37.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:37.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:37.463 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:37.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:37.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:37.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:37.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:37.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:37.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:37.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:37.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:37.751 INFO:teuthology.orchestra.run.smithi137.stderr:set 2.1 pg_upmap_items mapping to [6->4,0->5,7->3] 2024-09-21T08:25:37.754 INFO:tasks.rados.rados.0.smithi137.stdout:353: done (1 left) 2024-09-21T08:25:37.754 INFO:tasks.rados.rados.0.smithi137.stdout:354: copy_from oid 47 from oid 36 current snap is 31 2024-09-21T08:25:37.754 INFO:tasks.rados.rados.0.smithi137.stdout:352: expect (ObjNum 112 snap 22 seq_num 112) 2024-09-21T08:25:37.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:37.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T08:25:37.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:37.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:37.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:37.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:37.954 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:37.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:37.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:37 smithi165 ceph-mon[32893]: osdmap e143: 8 total, 8 up, 5 in 2024-09-21T08:25:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2347272435' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [6, 4, 0, 5, 7, 3]}]: dispatch 2024-09-21T08:25:38.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:37 smithi137 ceph-mon[29634]: osdmap e143: 8 total, 8 up, 5 in 2024-09-21T08:25:38.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2347272435' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [6, 4, 0, 5, 7, 3]}]: dispatch 2024-09-21T08:25:38.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:37 smithi137 ceph-mon[41044]: osdmap e143: 8 total, 8 up, 5 in 2024-09-21T08:25:38.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2347272435' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [6, 4, 0, 5, 7, 3]}]: dispatch 2024-09-21T08:25:38.031 INFO:tasks.rados.rados.0.smithi137.stdout:354: finishing copy_from racing read to smithi13779221-47 2024-09-21T08:25:38.031 INFO:tasks.rados.rados.0.smithi137.stdout:354: finishing copy_from to smithi13779221-47 2024-09-21T08:25:38.031 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 121 (ObjNum 121 snap 23 seq_num 121) dirty exists 2024-09-21T08:25:38.031 INFO:tasks.rados.rados.0.smithi137.stdout:352: done (1 left) 2024-09-21T08:25:38.032 INFO:tasks.rados.rados.0.smithi137.stdout:354: done (0 left) 2024-09-21T08:25:38.032 INFO:tasks.rados.rados.0.smithi137.stdout:355: read oid 3 snap -1 2024-09-21T08:25:38.032 INFO:tasks.rados.rados.0.smithi137.stdout:355: expect (ObjNum 142 snap 26 seq_num 142) 2024-09-21T08:25:38.032 INFO:tasks.rados.rados.0.smithi137.stdout:356: write (excl) oid 15 current snap is 31 2024-09-21T08:25:38.032 INFO:tasks.rados.rados.0.smithi137.stdout:356: seq_num 156 ranges {621892=465904,1712485=671711} 2024-09-21T08:25:38.037 INFO:tasks.rados.rados.0.smithi137.stdout:356: writing smithi13779221-15 from 621892 to 1087796 tid 1 2024-09-21T08:25:38.044 INFO:tasks.rados.rados.0.smithi137.stdout:356: writing smithi13779221-15 from 1712485 to 2384196 tid 2 2024-09-21T08:25:38.044 INFO:tasks.rados.rados.0.smithi137.stdout:357: write oid 4 current snap is 31 2024-09-21T08:25:38.044 INFO:tasks.rados.rados.0.smithi137.stdout:357: seq_num 157 ranges {638699=541153,1770967=531722} 2024-09-21T08:25:38.050 INFO:tasks.rados.rados.0.smithi137.stdout:357: writing smithi13779221-4 from 638699 to 1179852 tid 1 2024-09-21T08:25:38.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:38.057 INFO:tasks.rados.rados.0.smithi137.stdout:357: writing smithi13779221-4 from 1770967 to 2302689 tid 2 2024-09-21T08:25:38.057 INFO:tasks.rados.rados.0.smithi137.stdout:358: write oid 18 current snap is 31 2024-09-21T08:25:38.057 INFO:tasks.rados.rados.0.smithi137.stdout:358: seq_num 158 ranges {655506=616402,1829449=675380} 2024-09-21T08:25:38.064 INFO:tasks.rados.rados.0.smithi137.stdout:358: writing smithi13779221-18 from 655506 to 1271908 tid 1 2024-09-21T08:25:38.071 INFO:tasks.rados.rados.0.smithi137.stdout:358: writing smithi13779221-18 from 1829449 to 2504829 tid 2 2024-09-21T08:25:38.071 INFO:tasks.rados.rados.0.smithi137.stdout:359: snap_create 2024-09-21T08:25:38.071 INFO:tasks.rados.rados.0.smithi137.stdout:355: expect (ObjNum 142 snap 26 seq_num 142) 2024-09-21T08:25:38.078 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:38.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:38.079 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:38.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:356: finishing write tid 1 to smithi13779221-15 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:356: finishing write tid 2 to smithi13779221-15 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:356: finishing write tid 3 to smithi13779221-15 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:356: finishing write tid 4 to smithi13779221-15 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:356: oid 15 updating version 0 to 66 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:356: oid 15 version 66 is already newer than 64 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:356: oid 15 version 66 is already newer than 65 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 66 (ObjNum 156 snap 31 seq_num 156) dirty exists 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:356: left oid 15 (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:357: finishing write tid 1 to smithi13779221-4 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:357: finishing write tid 2 to smithi13779221-4 2024-09-21T08:25:38.184 INFO:tasks.rados.rados.0.smithi137.stdout:357: finishing write tid 3 to smithi13779221-4 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:357: finishing write tid 4 to smithi13779221-4 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:357: oid 4 updating version 0 to 96 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:357: oid 4 version 96 is already newer than 95 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:357: oid 4 updating version 96 to 97 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 97 (ObjNum 157 snap 31 seq_num 157) dirty exists 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:357: left oid 4 (ObjNum 157 snap 31 seq_num 157) 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:358: finishing write tid 1 to smithi13779221-18 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:358: finishing write tid 2 to smithi13779221-18 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:358: finishing write tid 3 to smithi13779221-18 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:358: finishing write tid 4 to smithi13779221-18 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:358: oid 18 updating version 0 to 42 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:358: oid 18 updating version 42 to 44 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:358: oid 18 version 44 is already newer than 43 2024-09-21T08:25:38.185 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 44 (ObjNum 158 snap 31 seq_num 158) dirty exists 2024-09-21T08:25:38.186 INFO:tasks.rados.rados.0.smithi137.stdout:358: left oid 18 (ObjNum 158 snap 31 seq_num 158) 2024-09-21T08:25:38.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:38.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:38.202 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:38.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:38.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:38.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:38.325 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:38.387 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:38.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:38.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:38.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:38.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:38.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:38.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:38.570 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:38.570 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:38.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:38.691 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:38.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:38.692 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:38.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:25:38.763 INFO:tasks.rados.rados.0.smithi137.stdout:355: done (4 left) 2024-09-21T08:25:38.763 INFO:tasks.rados.rados.0.smithi137.stdout:356: done (3 left) 2024-09-21T08:25:38.763 INFO:tasks.rados.rados.0.smithi137.stdout:357: done (2 left) 2024-09-21T08:25:38.763 INFO:tasks.rados.rados.0.smithi137.stdout:358: done (1 left) 2024-09-21T08:25:38.763 INFO:tasks.rados.rados.0.smithi137.stdout:359: done (0 left) 2024-09-21T08:25:38.763 INFO:tasks.rados.rados.0.smithi137.stdout:360: read oid 45 snap -1 2024-09-21T08:25:38.763 INFO:tasks.rados.rados.0.smithi137.stdout:360: expect deleted 2024-09-21T08:25:38.763 INFO:tasks.rados.rados.0.smithi137.stdout:361: delete oid 42 current snap is 32 2024-09-21T08:25:38.769 INFO:tasks.rados.rados.0.smithi137.stdout:360: done (1 left) 2024-09-21T08:25:38.769 INFO:tasks.rados.rados.0.smithi137.stdout:361: done (0 left) 2024-09-21T08:25:38.769 INFO:tasks.rados.rados.0.smithi137.stdout:362: rollback oid 3 current snap is 32 2024-09-21T08:25:38.769 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 3 to 27 2024-09-21T08:25:38.769 INFO:tasks.rados.rados.0.smithi137.stdout:363: write (excl) oid 38 current snap is 32 2024-09-21T08:25:38.769 INFO:tasks.rados.rados.0.smithi137.stdout:363: seq_num 160 ranges {689120=483253,1946413=679049} 2024-09-21T08:25:38.776 INFO:tasks.rados.rados.0.smithi137.stdout:363: writing smithi13779221-38 from 689120 to 1172373 tid 1 2024-09-21T08:25:38.784 INFO:tasks.rados.rados.0.smithi137.stdout:363: writing smithi13779221-38 from 1946413 to 2625462 tid 2 2024-09-21T08:25:38.784 INFO:tasks.rados.rados.0.smithi137.stdout:364: write oid 22 current snap is 32 2024-09-21T08:25:38.784 INFO:tasks.rados.rados.0.smithi137.stdout:362: finishing rollback tid 0 to smithi13779221-3 2024-09-21T08:25:38.784 INFO:tasks.rados.rados.0.smithi137.stdout:364: seq_num 161 ranges {705927=558502,2004895=422707} 2024-09-21T08:25:38.791 INFO:tasks.rados.rados.0.smithi137.stdout:364: writing smithi13779221-22 from 705927 to 1264429 tid 1 2024-09-21T08:25:38.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:38.797 INFO:tasks.rados.rados.0.smithi137.stdout:364: writing smithi13779221-22 from 2004895 to 2427602 tid 2 2024-09-21T08:25:38.797 INFO:tasks.rados.rados.0.smithi137.stdout:365: write (excl) oid 8 current snap is 32 2024-09-21T08:25:38.798 INFO:tasks.rados.rados.0.smithi137.stdout:362: finishing rollback tid 1 to smithi13779221-3 2024-09-21T08:25:38.798 INFO:tasks.rados.rados.0.smithi137.stdout:362: finishing rollback tid 2 to smithi13779221-3 2024-09-21T08:25:38.798 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 59 (ObjNum 142 snap 26 seq_num 142) dirty exists 2024-09-21T08:25:38.798 INFO:tasks.rados.rados.0.smithi137.stdout:363: finishing write tid 1 to smithi13779221-38 2024-09-21T08:25:38.798 INFO:tasks.rados.rados.0.smithi137.stdout:365: seq_num 162 ranges {722734=633751,2063377=659357} 2024-09-21T08:25:38.805 INFO:tasks.rados.rados.0.smithi137.stdout:365: writing smithi13779221-8 from 722734 to 1356485 tid 1 2024-09-21T08:25:38.813 INFO:tasks.rados.rados.0.smithi137.stdout:365: writing smithi13779221-8 from 2063377 to 2722734 tid 2 2024-09-21T08:25:38.813 INFO:tasks.rados.rados.0.smithi137.stdout:362: done (3 left) 2024-09-21T08:25:38.813 INFO:tasks.rados.rados.0.smithi137.stdout:366: snap_create 2024-09-21T08:25:38.813 INFO:tasks.rados.rados.0.smithi137.stdout:363: finishing write tid 2 to smithi13779221-38 2024-09-21T08:25:38.813 INFO:tasks.rados.rados.0.smithi137.stdout:363: finishing write tid 3 to smithi13779221-38 2024-09-21T08:25:38.813 INFO:tasks.rados.rados.0.smithi137.stdout:363: finishing write tid 4 to smithi13779221-38 2024-09-21T08:25:38.813 INFO:tasks.rados.rados.0.smithi137.stdout:363: oid 38 updating version 0 to 165 2024-09-21T08:25:38.813 INFO:tasks.rados.rados.0.smithi137.stdout:363: oid 38 version 165 is already newer than 164 2024-09-21T08:25:38.813 INFO:tasks.rados.rados.0.smithi137.stdout:363: oid 38 version 165 is already newer than 163 2024-09-21T08:25:38.813 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 165 (ObjNum 160 snap 32 seq_num 160) dirty exists 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:363: left oid 38 (ObjNum 160 snap 32 seq_num 160) 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:364: finishing write tid 1 to smithi13779221-22 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:364: finishing write tid 2 to smithi13779221-22 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:364: finishing write tid 3 to smithi13779221-22 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:364: finishing write tid 4 to smithi13779221-22 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:364: oid 22 updating version 0 to 100 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:364: oid 22 updating version 100 to 101 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:364: oid 22 version 101 is already newer than 99 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 101 (ObjNum 161 snap 32 seq_num 161) dirty exists 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:364: left oid 22 (ObjNum 161 snap 32 seq_num 161) 2024-09-21T08:25:38.814 INFO:tasks.rados.rados.0.smithi137.stdout:365: finishing write tid 1 to smithi13779221-8 2024-09-21T08:25:38.816 INFO:tasks.rados.rados.0.smithi137.stdout:365: finishing write tid 2 to smithi13779221-8 2024-09-21T08:25:38.816 INFO:tasks.rados.rados.0.smithi137.stdout:365: finishing write tid 3 to smithi13779221-8 2024-09-21T08:25:38.816 INFO:tasks.rados.rados.0.smithi137.stdout:365: finishing write tid 4 to smithi13779221-8 2024-09-21T08:25:38.817 INFO:tasks.rados.rados.0.smithi137.stdout:365: oid 8 updating version 0 to 105 2024-09-21T08:25:38.817 INFO:tasks.rados.rados.0.smithi137.stdout:365: oid 8 version 105 is already newer than 104 2024-09-21T08:25:38.817 INFO:tasks.rados.rados.0.smithi137.stdout:365: oid 8 version 105 is already newer than 103 2024-09-21T08:25:38.817 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 105 (ObjNum 162 snap 32 seq_num 162) dirty exists 2024-09-21T08:25:38.817 INFO:tasks.rados.rados.0.smithi137.stdout:365: left oid 8 (ObjNum 162 snap 32 seq_num 162) 2024-09-21T08:25:38.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:38.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:38.818 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:38.895 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:25:38.895 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:25:38.895 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:25:38.895 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:25:38.895 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:38 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2347272435' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [6, 4, 0, 5, 7, 3]}]': finished 2024-09-21T08:25:38.895 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:38 smithi137 ceph-mon[29634]: osdmap e144: 8 total, 8 up, 5 in 2024-09-21T08:25:38.895 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:38 smithi137 ceph-mon[29634]: pgmap v122: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 142 active+clean; 136 MiB data, 477 MiB used, 447 GiB / 447 GiB avail; 28 MiB/s rd, 6.2 MiB/s wr, 41 op/s 2024-09-21T08:25:38.895 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:38 smithi137 ceph-mon[29634]: 3.1f scrub starts 2024-09-21T08:25:38.896 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:38 smithi137 ceph-mon[29634]: 3.1f scrub ok 2024-09-21T08:25:38.896 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:38 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2347272435' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [6, 4, 0, 5, 7, 3]}]': finished 2024-09-21T08:25:38.896 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:38 smithi137 ceph-mon[41044]: osdmap e144: 8 total, 8 up, 5 in 2024-09-21T08:25:38.896 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:38 smithi137 ceph-mon[41044]: pgmap v122: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 142 active+clean; 136 MiB data, 477 MiB used, 447 GiB / 447 GiB avail; 28 MiB/s rd, 6.2 MiB/s wr, 41 op/s 2024-09-21T08:25:38.896 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:38 smithi137 ceph-mon[41044]: 3.1f scrub starts 2024-09-21T08:25:38.896 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:38 smithi137 ceph-mon[41044]: 3.1f scrub ok 2024-09-21T08:25:38.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:38.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:38.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:38.942 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:38 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2347272435' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "2.1", "id": [6, 4, 0, 5, 7, 3]}]': finished 2024-09-21T08:25:39.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:38 smithi165 ceph-mon[32893]: osdmap e144: 8 total, 8 up, 5 in 2024-09-21T08:25:39.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:38 smithi165 ceph-mon[32893]: pgmap v122: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 142 active+clean; 136 MiB data, 477 MiB used, 447 GiB / 447 GiB avail; 28 MiB/s rd, 6.2 MiB/s wr, 41 op/s 2024-09-21T08:25:39.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:38 smithi165 ceph-mon[32893]: 3.1f scrub starts 2024-09-21T08:25:39.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:38 smithi165 ceph-mon[32893]: 3.1f scrub ok 2024-09-21T08:25:39.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:39.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:39.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:39.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:39.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:39.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:39.188 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:39.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:39.289 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:39.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:39.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:39.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:39.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:39.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T08:25:39.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:39.460 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:39.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:39.531 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:39.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:39.582 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:39.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:39.583 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:39.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:25:39.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:39.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:39.707 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:39.707 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:39.770 INFO:tasks.rados.rados.0.smithi137.stdout:363: done (3 left) 2024-09-21T08:25:39.770 INFO:tasks.rados.rados.0.smithi137.stdout:364: done (2 left) 2024-09-21T08:25:39.770 INFO:tasks.rados.rados.0.smithi137.stdout:365: done (1 left) 2024-09-21T08:25:39.770 INFO:tasks.rados.rados.0.smithi137.stdout:366: done (0 left) 2024-09-21T08:25:39.770 INFO:tasks.rados.rados.0.smithi137.stdout:367: delete oid 30 current snap is 33 2024-09-21T08:25:39.776 INFO:tasks.rados.rados.0.smithi137.stdout:367: done (0 left) 2024-09-21T08:25:39.777 INFO:tasks.rados.rados.0.smithi137.stdout:368: snap_remove snap 30 2024-09-21T08:25:39.808 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:39.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:39.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:39.830 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:39.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:39.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:39.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:39.955 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:39 smithi165 ceph-mon[32893]: osdmap e145: 8 total, 8 up, 5 in 2024-09-21T08:25:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:39 smithi165 ceph-mon[32893]: 5.7 scrub starts 2024-09-21T08:25:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:39 smithi165 ceph-mon[32893]: 5.7 scrub ok 2024-09-21T08:25:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:39 smithi165 ceph-mon[32893]: 5.2 scrub starts 2024-09-21T08:25:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:39 smithi165 ceph-mon[32893]: 5.2 scrub ok 2024-09-21T08:25:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:39 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:39 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:39 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:25:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:39 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:25:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:39 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[29634]: osdmap e145: 8 total, 8 up, 5 in 2024-09-21T08:25:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[29634]: 5.7 scrub starts 2024-09-21T08:25:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[29634]: 5.7 scrub ok 2024-09-21T08:25:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[29634]: 5.2 scrub starts 2024-09-21T08:25:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[29634]: 5.2 scrub ok 2024-09-21T08:25:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:25:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:25:40.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[41044]: osdmap e145: 8 total, 8 up, 5 in 2024-09-21T08:25:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[41044]: 5.7 scrub starts 2024-09-21T08:25:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[41044]: 5.7 scrub ok 2024-09-21T08:25:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[41044]: 5.2 scrub starts 2024-09-21T08:25:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[41044]: 5.2 scrub ok 2024-09-21T08:25:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:40.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:40.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:25:40.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:25:40.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:39 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:40.051 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:40.054 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:40.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:40.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:40.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:40.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:40.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:40.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:40.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:40.202 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:40.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:40.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:40.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:40.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:40.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:40.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:40.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:40.451 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:40.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:40.574 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:40.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:40.575 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:40.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:40.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:40.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:40.699 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:40.714 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:25:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:25:40.771 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:25:40.781 INFO:tasks.rados.rados.0.smithi137.stdout:368: done (0 left) 2024-09-21T08:25:40.781 INFO:tasks.rados.rados.0.smithi137.stdout:369: copy_from oid 46 from oid 21 current snap is 33 2024-09-21T08:25:40.781 INFO:tasks.rados.rados.0.smithi137.stdout:370: read oid 29 snap -1 2024-09-21T08:25:40.781 INFO:tasks.rados.rados.0.smithi137.stdout:370: expect (ObjNum 148 snap 27 seq_num 148) 2024-09-21T08:25:40.781 INFO:tasks.rados.rados.0.smithi137.stdout:371: write (excl) oid 33 current snap is 33 2024-09-21T08:25:40.781 INFO:tasks.rados.rados.0.smithi137.stdout:371: seq_num 164 ranges {756348=784249,2063988=686387} 2024-09-21T08:25:40.790 INFO:tasks.rados.rados.0.smithi137.stdout:371: writing smithi13779221-33 from 756348 to 1540597 tid 1 2024-09-21T08:25:40.798 INFO:tasks.rados.rados.0.smithi137.stdout:371: writing smithi13779221-33 from 2063988 to 2750375 tid 2 2024-09-21T08:25:40.798 INFO:tasks.rados.rados.0.smithi137.stdout:372: read oid 27 snap -1 2024-09-21T08:25:40.798 INFO:tasks.rados.rados.0.smithi137.stdout:372: expect (ObjNum 124 snap 23 seq_num 124) 2024-09-21T08:25:40.798 INFO:tasks.rados.rados.0.smithi137.stdout:373: write oid 41 current snap is 33 2024-09-21T08:25:40.798 INFO:tasks.rados.rados.0.smithi137.stdout:373: seq_num 165 ranges {773155=459498,1722470=430045,2648676=124479} 2024-09-21T08:25:40.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:40.805 INFO:tasks.rados.rados.0.smithi137.stdout:373: writing smithi13779221-41 from 773155 to 1232653 tid 1 2024-09-21T08:25:40.810 INFO:tasks.rados.rados.0.smithi137.stdout:373: writing smithi13779221-41 from 1722470 to 2152515 tid 2 2024-09-21T08:25:40.814 INFO:tasks.rados.rados.0.smithi137.stdout:373: writing smithi13779221-41 from 2648676 to 2773155 tid 3 2024-09-21T08:25:40.814 INFO:tasks.rados.rados.0.smithi137.stdout:374: copy_from oid 18 from oid 14 current snap is 33 2024-09-21T08:25:40.814 INFO:tasks.rados.rados.0.smithi137.stdout:370: expect (ObjNum 148 snap 27 seq_num 148) 2024-09-21T08:25:40.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:40.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:40.822 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:40.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:40.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:40.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:40.947 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:369: finishing copy_from racing read to smithi13779221-46 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:369: finishing copy_from to smithi13779221-46 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 47 (ObjNum 130 snap 23 seq_num 130) dirty exists 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:371: finishing write tid 1 to smithi13779221-33 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:371: finishing write tid 2 to smithi13779221-33 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:371: finishing write tid 3 to smithi13779221-33 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:371: finishing write tid 4 to smithi13779221-33 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:371: oid 33 updating version 0 to 32 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:371: oid 33 updating version 32 to 34 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:371: oid 33 version 34 is already newer than 33 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 34 (ObjNum 164 snap 33 seq_num 164) dirty exists 2024-09-21T08:25:41.007 INFO:tasks.rados.rados.0.smithi137.stdout:371: left oid 33 (ObjNum 164 snap 33 seq_num 164) 2024-09-21T08:25:41.008 INFO:tasks.rados.rados.0.smithi137.stdout:372: expect (ObjNum 124 snap 23 seq_num 124) 2024-09-21T08:25:41.017 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: osdmap e146: 8 total, 8 up, 5 in 2024-09-21T08:25:41.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T08:25:41.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: Reconfiguring daemon alertmanager.a on smithi137 2024-09-21T08:25:41.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: 6.1e scrub starts 2024-09-21T08:25:41.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: 6.1e scrub ok 2024-09-21T08:25:41.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: 6.17 scrub starts 2024-09-21T08:25:41.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: 6.17 scrub ok 2024-09-21T08:25:41.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: pgmap v125: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 142 active+clean; 136 MiB data, 477 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:41.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3507128949' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: 4.a scrub starts 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[41044]: 4.a scrub ok 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: osdmap e146: 8 total, 8 up, 5 in 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: Reconfiguring daemon alertmanager.a on smithi137 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: 6.1e scrub starts 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: 6.1e scrub ok 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: 6.17 scrub starts 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: 6.17 scrub ok 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: pgmap v125: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 142 active+clean; 136 MiB data, 477 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:41.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3507128949' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:41.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: 4.a scrub starts 2024-09-21T08:25:41.020 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:40 smithi137 ceph-mon[29634]: 4.a scrub ok 2024-09-21T08:25:41.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:41.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:41.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:41.070 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:41.102 INFO:tasks.rados.rados.0.smithi137.stdout:373: finishing write tid 1 to smithi13779221-41 2024-09-21T08:25:41.102 INFO:tasks.rados.rados.0.smithi137.stdout:373: finishing write tid 2 to smithi13779221-41 2024-09-21T08:25:41.102 INFO:tasks.rados.rados.0.smithi137.stdout:373: finishing write tid 3 to smithi13779221-41 2024-09-21T08:25:41.102 INFO:tasks.rados.rados.0.smithi137.stdout:369: done (5 left) 2024-09-21T08:25:41.102 INFO:tasks.rados.rados.0.smithi137.stdout:370: done (4 left) 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:371: done (3 left) 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:372: done (2 left) 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:375: delete oid 10 current snap is 33 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:373: finishing write tid 4 to smithi13779221-41 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:373: finishing write tid 5 to smithi13779221-41 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:373: oid 41 updating version 0 to 63 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:373: oid 41 version 63 is already newer than 62 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:373: oid 41 version 63 is already newer than 61 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:373: oid 41 updating version 63 to 64 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 64 (ObjNum 165 snap 33 seq_num 165) dirty exists 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:373: left oid 41 (ObjNum 165 snap 33 seq_num 165) 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:374: finishing copy_from racing read to smithi13779221-18 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:374: finishing copy_from to smithi13779221-18 2024-09-21T08:25:41.103 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 6 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-21T08:25:41.104 INFO:tasks.rados.rados.0.smithi137.stdout:373: done (2 left) 2024-09-21T08:25:41.104 INFO:tasks.rados.rados.0.smithi137.stdout:374: done (1 left) 2024-09-21T08:25:41.104 INFO:tasks.rados.rados.0.smithi137.stdout:375: done (0 left) 2024-09-21T08:25:41.104 INFO:tasks.rados.rados.0.smithi137.stdout:376: read oid 15 snap -1 2024-09-21T08:25:41.104 INFO:tasks.rados.rados.0.smithi137.stdout:376: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:25:41.104 INFO:tasks.rados.rados.0.smithi137.stdout:377: delete oid 36 current snap is 33 2024-09-21T08:25:41.105 INFO:tasks.rados.rados.0.smithi137.stdout:377: done (1 left) 2024-09-21T08:25:41.105 INFO:tasks.rados.rados.0.smithi137.stdout:378: rollback oid 39 current snap is 33 2024-09-21T08:25:41.105 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 39 to 27 2024-09-21T08:25:41.105 INFO:tasks.rados.rados.0.smithi137.stdout:379: rollback oid 34 current snap is 33 2024-09-21T08:25:41.105 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 34 to 32 2024-09-21T08:25:41.105 INFO:tasks.rados.rados.0.smithi137.stdout:380: write oid 11 current snap is 33 2024-09-21T08:25:41.105 INFO:tasks.rados.rados.0.smithi137.stdout:380: seq_num 168 ranges {423576=401598,1497916=693725} 2024-09-21T08:25:41.110 INFO:tasks.rados.rados.0.smithi137.stdout:380: writing smithi13779221-11 from 423576 to 825174 tid 1 2024-09-21T08:25:41.118 INFO:tasks.rados.rados.0.smithi137.stdout:380: writing smithi13779221-11 from 1497916 to 2191641 tid 2 2024-09-21T08:25:41.118 INFO:tasks.rados.rados.0.smithi137.stdout:381: read oid 12 snap 32 2024-09-21T08:25:41.118 INFO:tasks.rados.rados.0.smithi137.stdout:381: expect deleted 2024-09-21T08:25:41.118 INFO:tasks.rados.rados.0.smithi137.stdout:382: copy_from oid 22 from oid 33 current snap is 33 2024-09-21T08:25:41.118 INFO:tasks.rados.rados.0.smithi137.stdout:378: finishing rollback tid 0 to smithi13779221-39 2024-09-21T08:25:41.118 INFO:tasks.rados.rados.0.smithi137.stdout:378: finishing rollback tid 1 to smithi13779221-39 2024-09-21T08:25:41.118 INFO:tasks.rados.rados.0.smithi137.stdout:378: finishing rollback tid 2 to smithi13779221-39 2024-09-21T08:25:41.118 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 23 (ObjNum 69 snap 7 seq_num 69) dirty exists 2024-09-21T08:25:41.118 INFO:tasks.rados.rados.0.smithi137.stdout:380: finishing write tid 1 to smithi13779221-11 2024-09-21T08:25:41.118 INFO:tasks.rados.rados.0.smithi137.stdout:376: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:25:41.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:41.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:41.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:41.194 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:41.245 INFO:tasks.rados.rados.0.smithi137.stdout:379: finishing rollback tid 0 to smithi13779221-34 2024-09-21T08:25:41.245 INFO:tasks.rados.rados.0.smithi137.stdout:379: finishing rollback tid 1 to smithi13779221-34 2024-09-21T08:25:41.245 INFO:tasks.rados.rados.0.smithi137.stdout:379: finishing rollback tid 2 to smithi13779221-34 2024-09-21T08:25:41.245 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 181 (ObjNum 143 snap 26 seq_num 143) dirty exists 2024-09-21T08:25:41.245 INFO:tasks.rados.rados.0.smithi137.stdout:376: done (5 left) 2024-09-21T08:25:41.245 INFO:tasks.rados.rados.0.smithi137.stdout:378: done (4 left) 2024-09-21T08:25:41.246 INFO:tasks.rados.rados.0.smithi137.stdout:379: done (3 left) 2024-09-21T08:25:41.246 INFO:tasks.rados.rados.0.smithi137.stdout:383: snap_remove snap 26 2024-09-21T08:25:41.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: osdmap e146: 8 total, 8 up, 5 in 2024-09-21T08:25:41.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: Reconfiguring alertmanager.a (dependencies changed)... 2024-09-21T08:25:41.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: Reconfiguring daemon alertmanager.a on smithi137 2024-09-21T08:25:41.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: 6.1e scrub starts 2024-09-21T08:25:41.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: 6.1e scrub ok 2024-09-21T08:25:41.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: 6.17 scrub starts 2024-09-21T08:25:41.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: 6.17 scrub ok 2024-09-21T08:25:41.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: pgmap v125: 148 pgs: 1 active+clean+scrubbing, 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 142 active+clean; 136 MiB data, 477 MiB used, 447 GiB / 447 GiB avail 2024-09-21T08:25:41.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3507128949' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:41.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: 4.a scrub starts 2024-09-21T08:25:41.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:40 smithi165 ceph-mon[32893]: 4.a scrub ok 2024-09-21T08:25:41.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:41.316 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:41.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:41.317 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:41.417 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:41.439 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:41.440 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:41.440 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:383: done (3 left) 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:384: snap_create 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:380: finishing write tid 2 to smithi13779221-11 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:380: finishing write tid 3 to smithi13779221-11 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:380: finishing write tid 4 to smithi13779221-11 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:380: oid 11 updating version 0 to 65 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:380: oid 11 updating version 65 to 67 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:380: oid 11 version 67 is already newer than 66 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 67 (ObjNum 168 snap 33 seq_num 168) dirty exists 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:380: left oid 11 (ObjNum 168 snap 33 seq_num 168) 2024-09-21T08:25:41.498 INFO:tasks.rados.rados.0.smithi137.stdout:382: finishing copy_from racing read to smithi13779221-22 2024-09-21T08:25:41.499 INFO:tasks.rados.rados.0.smithi137.stdout:382: finishing copy_from to smithi13779221-22 2024-09-21T08:25:41.499 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 34 (ObjNum 164 snap 33 seq_num 164) dirty exists 2024-09-21T08:25:41.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:41.563 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:41.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:41.564 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:41.665 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:41.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:41.687 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:41.688 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:41.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:41.811 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:41.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:41.812 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:41.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:41.934 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:41.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:41.935 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:42.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:25:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:25:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:25:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3507128949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-mon[32893]: osdmap e147: 8 total, 8 up, 5 in 2024-09-21T08:25:42.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-mon[32893]: 4.9 scrub starts 2024-09-21T08:25:42.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-mon[32893]: 4.9 scrub ok 2024-09-21T08:25:42.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-mon[32893]: osdmap e148: 8 total, 8 up, 5 in 2024-09-21T08:25:42.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:42.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-mon[32893]: 5.4 scrub starts 2024-09-21T08:25:42.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:41 smithi165 ceph-mon[32893]: 5.4 scrub ok 2024-09-21T08:25:42.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:42.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:25:42.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:25:42.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3507128949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:42.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[29634]: osdmap e147: 8 total, 8 up, 5 in 2024-09-21T08:25:42.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[29634]: 4.9 scrub starts 2024-09-21T08:25:42.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[29634]: 4.9 scrub ok 2024-09-21T08:25:42.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[29634]: osdmap e148: 8 total, 8 up, 5 in 2024-09-21T08:25:42.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:42.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[29634]: 5.4 scrub starts 2024-09-21T08:25:42.043 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[29634]: 5.4 scrub ok 2024-09-21T08:25:42.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:25:42.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:25:42.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3507128949' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:42.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[41044]: osdmap e147: 8 total, 8 up, 5 in 2024-09-21T08:25:42.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[41044]: 4.9 scrub starts 2024-09-21T08:25:42.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[41044]: 4.9 scrub ok 2024-09-21T08:25:42.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[41044]: osdmap e148: 8 total, 8 up, 5 in 2024-09-21T08:25:42.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:42.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[41044]: 5.4 scrub starts 2024-09-21T08:25:42.044 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:41 smithi137 ceph-mon[41044]: 5.4 scrub ok 2024-09-21T08:25:42.045 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:25:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:25:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:25:42.045 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:25:42.045 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:25:42.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:42.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:42.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:42.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:42.182 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:42.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:42.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:42.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:42.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:42.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:42.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:42.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:42.418 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:42.418 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:42.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:42.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:42.430 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:42.435 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:42.435 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:42.450 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:42.451 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:42.466 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:42.466 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:42.483 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:42.483 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:42.499 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:42.500 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:42.505 INFO:tasks.rados.rados.0.smithi137.stdout:380: done (3 left) 2024-09-21T08:25:42.505 INFO:tasks.rados.rados.0.smithi137.stdout:381: done (2 left) 2024-09-21T08:25:42.506 INFO:tasks.rados.rados.0.smithi137.stdout:382: done (1 left) 2024-09-21T08:25:42.506 INFO:tasks.rados.rados.0.smithi137.stdout:384: done (0 left) 2024-09-21T08:25:42.506 INFO:tasks.rados.rados.0.smithi137.stdout:385: write oid 49 current snap is 34 2024-09-21T08:25:42.506 INFO:tasks.rados.rados.0.smithi137.stdout:385: seq_num 169 ranges {440383=476847,1556398=437383,2474721=365662} 2024-09-21T08:25:42.510 INFO:tasks.rados.rados.0.smithi137.stdout:385: writing smithi13779221-49 from 440383 to 917230 tid 1 2024-09-21T08:25:42.516 INFO:tasks.rados.rados.0.smithi137.stdout:385: writing smithi13779221-49 from 1556398 to 1993781 tid 2 2024-09-21T08:25:42.520 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:42.520 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:42.521 INFO:tasks.rados.rados.0.smithi137.stdout:385: writing smithi13779221-49 from 2474721 to 2840383 tid 3 2024-09-21T08:25:42.522 INFO:tasks.rados.rados.0.smithi137.stdout:386: delete oid 18 current snap is 34 2024-09-21T08:25:42.522 INFO:tasks.rados.rados.0.smithi137.stdout:385: finishing write tid 1 to smithi13779221-49 2024-09-21T08:25:42.522 INFO:tasks.rados.rados.0.smithi137.stdout:385: finishing write tid 2 to smithi13779221-49 2024-09-21T08:25:42.524 INFO:tasks.rados.rados.0.smithi137.stdout:385: finishing write tid 3 to smithi13779221-49 2024-09-21T08:25:42.524 INFO:tasks.rados.rados.0.smithi137.stdout:385: finishing write tid 4 to smithi13779221-49 2024-09-21T08:25:42.524 INFO:tasks.rados.rados.0.smithi137.stdout:385: finishing write tid 5 to smithi13779221-49 2024-09-21T08:25:42.525 INFO:tasks.rados.rados.0.smithi137.stdout:385: oid 49 updating version 0 to 190 2024-09-21T08:25:42.525 INFO:tasks.rados.rados.0.smithi137.stdout:385: oid 49 updating version 190 to 191 2024-09-21T08:25:42.525 INFO:tasks.rados.rados.0.smithi137.stdout:385: oid 49 version 191 is already newer than 188 2024-09-21T08:25:42.525 INFO:tasks.rados.rados.0.smithi137.stdout:385: oid 49 version 191 is already newer than 189 2024-09-21T08:25:42.525 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 191 (ObjNum 169 snap 34 seq_num 169) dirty exists 2024-09-21T08:25:42.525 INFO:tasks.rados.rados.0.smithi137.stdout:385: left oid 49 (ObjNum 169 snap 34 seq_num 169) 2024-09-21T08:25:42.525 INFO:tasks.rados.rados.0.smithi137.stdout:385: done (1 left) 2024-09-21T08:25:42.526 INFO:tasks.rados.rados.0.smithi137.stdout:386: done (0 left) 2024-09-21T08:25:42.526 INFO:tasks.rados.rados.0.smithi137.stdout:387: snap_remove snap 32 2024-09-21T08:25:42.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:42.538 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:42.538 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:42.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:42.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:42.554 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:42.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:42.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:42.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:42.677 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:42.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:42.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:42.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:42.802 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:42.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[29634]: 5.6 scrub starts 2024-09-21T08:25:42.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[29634]: 5.6 scrub ok 2024-09-21T08:25:42.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[29634]: pgmap v128: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 139 MiB data, 489 MiB used, 447 GiB / 447 GiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T08:25:42.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4221336892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:42.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[29634]: osdmap e149: 8 total, 8 up, 5 in 2024-09-21T08:25:42.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[29634]: 5.9 scrub starts 2024-09-21T08:25:42.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[29634]: 5.9 scrub ok 2024-09-21T08:25:42.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[41044]: 5.6 scrub starts 2024-09-21T08:25:42.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[41044]: 5.6 scrub ok 2024-09-21T08:25:42.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[41044]: pgmap v128: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 139 MiB data, 489 MiB used, 447 GiB / 447 GiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T08:25:42.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4221336892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:42.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[41044]: osdmap e149: 8 total, 8 up, 5 in 2024-09-21T08:25:42.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[41044]: 5.9 scrub starts 2024-09-21T08:25:42.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:42 smithi137 ceph-mon[41044]: 5.9 scrub ok 2024-09-21T08:25:42.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:42.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:42.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:42.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:43.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:43.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:43.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:43.048 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:43.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:43.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:43.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:43.172 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:43.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:42 smithi165 ceph-mon[32893]: 5.6 scrub starts 2024-09-21T08:25:43.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:42 smithi165 ceph-mon[32893]: 5.6 scrub ok 2024-09-21T08:25:43.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:42 smithi165 ceph-mon[32893]: pgmap v128: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 139 MiB data, 489 MiB used, 447 GiB / 447 GiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T08:25:43.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:42 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4221336892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:43.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:42 smithi165 ceph-mon[32893]: osdmap e149: 8 total, 8 up, 5 in 2024-09-21T08:25:43.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:42 smithi165 ceph-mon[32893]: 5.9 scrub starts 2024-09-21T08:25:43.264 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:42 smithi165 ceph-mon[32893]: 5.9 scrub ok 2024-09-21T08:25:43.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:43.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:43.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:43.295 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:43.348 INFO:tasks.thrashosds.thrasher:in_osds: [0, 1, 2, 5, 7] out_osds: [4, 3, 6] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:25:43.348 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:25:43.348 INFO:tasks.thrashosds.thrasher:Removing osd 1, in_osds are: [0, 1, 2, 5, 7] 2024-09-21T08:25:43.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd out 1 2024-09-21T08:25:43.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:43.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:43.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:43.418 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:43.503 INFO:tasks.rados.rados.0.smithi137.stdout:387: done (0 left) 2024-09-21T08:25:43.503 INFO:tasks.rados.rados.0.smithi137.stdout:388: snap_remove snap 33 2024-09-21T08:25:43.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:43.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:43.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:43.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:43.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:43.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:43.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:43.665 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:43.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:43.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:43.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:43.787 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:43.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:43.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:25:43.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:43.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:43.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:44.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:44.034 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:44.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:44.035 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:44.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:44.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:44.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:44.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:44.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:44.284 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:44.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:44.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:44.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:44.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:44.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:44.410 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:44.507 INFO:tasks.rados.rados.0.smithi137.stdout:388: done (0 left) 2024-09-21T08:25:44.507 INFO:tasks.rados.rados.0.smithi137.stdout:389: write oid 50 current snap is 34 2024-09-21T08:25:44.507 INFO:tasks.rados.rados.0.smithi137.stdout:389: seq_num 171 ranges {473997=627345,1673362=441052,2529567=344430} 2024-09-21T08:25:44.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:44.513 INFO:tasks.rados.rados.0.smithi137.stdout:389: writing smithi13779221-50 from 473997 to 1101342 tid 1 2024-09-21T08:25:44.519 INFO:tasks.rados.rados.0.smithi137.stdout:389: writing smithi13779221-50 from 1673362 to 2114414 tid 2 2024-09-21T08:25:44.524 INFO:tasks.rados.rados.0.smithi137.stdout:389: writing smithi13779221-50 from 2529567 to 2873997 tid 3 2024-09-21T08:25:44.524 INFO:tasks.rados.rados.0.smithi137.stdout:390: rollback oid 32 current snap is 34 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 32 to 31 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:389: finishing write tid 1 to smithi13779221-50 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:389: finishing write tid 2 to smithi13779221-50 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:391: rollback oid 29 current snap is 34 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 29 to 31 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:392: read oid 37 snap 31 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:392: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:393: copy_from oid 7 from oid 15 current snap is 34 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:394: rollback oid 42 current snap is 34 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 42 to 27 2024-09-21T08:25:44.525 INFO:tasks.rados.rados.0.smithi137.stdout:395: delete oid 10 current snap is 34 2024-09-21T08:25:44.527 INFO:tasks.rados.rados.0.smithi137.stdout:394: finishing rollback tid 1 to smithi13779221-42 2024-09-21T08:25:44.527 INFO:tasks.rados.rados.0.smithi137.stdout:394: finishing rollback tid 2 to smithi13779221-42 2024-09-21T08:25:44.527 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 195 (ObjNum 114 snap 23 seq_num 114) dirty exists 2024-09-21T08:25:44.527 INFO:tasks.rados.rados.0.smithi137.stdout:389: finishing write tid 3 to smithi13779221-50 2024-09-21T08:25:44.527 INFO:tasks.rados.rados.0.smithi137.stdout:389: finishing write tid 4 to smithi13779221-50 2024-09-21T08:25:44.527 INFO:tasks.rados.rados.0.smithi137.stdout:389: finishing write tid 5 to smithi13779221-50 2024-09-21T08:25:44.527 INFO:tasks.rados.rados.0.smithi137.stdout:389: oid 50 updating version 0 to 136 2024-09-21T08:25:44.528 INFO:tasks.rados.rados.0.smithi137.stdout:389: oid 50 version 136 is already newer than 133 2024-09-21T08:25:44.528 INFO:tasks.rados.rados.0.smithi137.stdout:389: oid 50 version 136 is already newer than 134 2024-09-21T08:25:44.528 INFO:tasks.rados.rados.0.smithi137.stdout:389: oid 50 version 136 is already newer than 135 2024-09-21T08:25:44.528 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 136 (ObjNum 171 snap 34 seq_num 171) dirty exists 2024-09-21T08:25:44.528 INFO:tasks.rados.rados.0.smithi137.stdout:389: left oid 50 (ObjNum 171 snap 34 seq_num 171) 2024-09-21T08:25:44.530 INFO:tasks.rados.rados.0.smithi137.stdout:389: done (6 left) 2024-09-21T08:25:44.530 INFO:tasks.rados.rados.0.smithi137.stdout:394: done (5 left) 2024-09-21T08:25:44.530 INFO:tasks.rados.rados.0.smithi137.stdout:395: done (4 left) 2024-09-21T08:25:44.530 INFO:tasks.rados.rados.0.smithi137.stdout:396: read oid 2 snap 31 2024-09-21T08:25:44.530 INFO:tasks.rados.rados.0.smithi137.stdout:396: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-21T08:25:44.530 INFO:tasks.rados.rados.0.smithi137.stdout:397: read oid 21 snap -1 2024-09-21T08:25:44.530 INFO:tasks.rados.rados.0.smithi137.stdout:397: expect (ObjNum 130 snap 23 seq_num 130) 2024-09-21T08:25:44.530 INFO:tasks.rados.rados.0.smithi137.stdout:398: read oid 10 snap 28 2024-09-21T08:25:44.530 INFO:tasks.rados.rados.0.smithi137.stdout:398: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-21T08:25:44.530 INFO:tasks.rados.rados.0.smithi137.stdout:399: rollback oid 14 current snap is 34 2024-09-21T08:25:44.531 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 14 to 28 2024-09-21T08:25:44.531 INFO:tasks.rados.rados.0.smithi137.stdout:400: rollback oid 17 current snap is 34 2024-09-21T08:25:44.531 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 17 to 31 2024-09-21T08:25:44.531 INFO:tasks.rados.rados.0.smithi137.stdout:401: read oid 47 snap 31 2024-09-21T08:25:44.531 INFO:tasks.rados.rados.0.smithi137.stdout:401: expect (ObjNum 121 snap 23 seq_num 121) 2024-09-21T08:25:44.531 INFO:tasks.rados.rados.0.smithi137.stdout:402: read oid 50 snap -1 2024-09-21T08:25:44.531 INFO:tasks.rados.rados.0.smithi137.stdout:402: expect (ObjNum 171 snap 34 seq_num 171) 2024-09-21T08:25:44.531 INFO:tasks.rados.rados.0.smithi137.stdout:403: delete oid 25 current snap is 34 2024-09-21T08:25:44.531 INFO:tasks.rados.rados.0.smithi137.stdout:391: finishing rollback tid 0 to smithi13779221-29 2024-09-21T08:25:44.532 INFO:tasks.rados.rados.0.smithi137.stdout:391: finishing rollback tid 1 to smithi13779221-29 2024-09-21T08:25:44.532 INFO:tasks.rados.rados.0.smithi137.stdout:391: finishing rollback tid 2 to smithi13779221-29 2024-09-21T08:25:44.532 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 178 (ObjNum 148 snap 27 seq_num 148) dirty exists 2024-09-21T08:25:44.532 INFO:tasks.rados.rados.0.smithi137.stdout:390: finishing rollback tid 0 to smithi13779221-32 2024-09-21T08:25:44.535 INFO:tasks.rados.rados.0.smithi137.stdout:391: done (11 left) 2024-09-21T08:25:44.535 INFO:tasks.rados.rados.0.smithi137.stdout:403: done (10 left) 2024-09-21T08:25:44.535 INFO:tasks.rados.rados.0.smithi137.stdout:404: rollback oid 25 current snap is 34 2024-09-21T08:25:44.535 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 25 to 28 2024-09-21T08:25:44.535 INFO:tasks.rados.rados.0.smithi137.stdout:405: read oid 34 snap -1 2024-09-21T08:25:44.535 INFO:tasks.rados.rados.0.smithi137.stdout:405: expect (ObjNum 143 snap 26 seq_num 143) 2024-09-21T08:25:44.535 INFO:tasks.rados.rados.0.smithi137.stdout:406: delete oid 9 current snap is 34 2024-09-21T08:25:44.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:44.538 INFO:tasks.rados.rados.0.smithi137.stdout:392: expect (ObjNum 36 snap 0 seq_num 36) 2024-09-21T08:25:44.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:44.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:44.624 INFO:tasks.rados.rados.0.smithi137.stdout:397: expect (ObjNum 130 snap 23 seq_num 130) 2024-09-21T08:25:44.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:44.661 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:44.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:44.662 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:44.746 INFO:tasks.rados.rados.0.smithi137.stdout:404: finishing rollback tid 1 to smithi13779221-25 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:404: finishing rollback tid 2 to smithi13779221-25 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 7 (ObjNum 24 snap 0 seq_num 24) dirty exists 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:390: finishing rollback tid 1 to smithi13779221-32 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:390: finishing rollback tid 2 to smithi13779221-32 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 70 (ObjNum 125 snap 23 seq_num 125) dirty exists 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:400: finishing rollback tid 0 to smithi13779221-17 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:390: done (12 left) 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:392: done (11 left) 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:397: done (10 left) 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:404: done (9 left) 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:406: done (8 left) 2024-09-21T08:25:44.747 INFO:tasks.rados.rados.0.smithi137.stdout:407: write (excl) oid 16 current snap is 34 2024-09-21T08:25:44.748 INFO:tasks.rados.rados.0.smithi137.stdout:398: expect (ObjNum 9 snap 0 seq_num 9) 2024-09-21T08:25:44.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:44.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:44 smithi165 ceph-mon[32893]: 4.d scrub starts 2024-09-21T08:25:44.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:44 smithi165 ceph-mon[32893]: 4.d scrub ok 2024-09-21T08:25:44.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:44 smithi165 ceph-mon[32893]: osdmap e150: 8 total, 8 up, 5 in 2024-09-21T08:25:44.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:44 smithi137 ceph-mon[29634]: 4.d scrub starts 2024-09-21T08:25:44.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:44 smithi137 ceph-mon[29634]: 4.d scrub ok 2024-09-21T08:25:44.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:44 smithi137 ceph-mon[29634]: osdmap e150: 8 total, 8 up, 5 in 2024-09-21T08:25:44.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:44 smithi137 ceph-mon[41044]: 4.d scrub starts 2024-09-21T08:25:44.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:44 smithi137 ceph-mon[41044]: 4.d scrub ok 2024-09-21T08:25:44.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:44 smithi137 ceph-mon[41044]: osdmap e150: 8 total, 8 up, 5 in 2024-09-21T08:25:44.784 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:44.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:44.785 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:44.816 INFO:tasks.rados.rados.0.smithi137.stdout:401: expect (ObjNum 121 snap 23 seq_num 121) 2024-09-21T08:25:44.886 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:44.907 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:44.908 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:44.940 INFO:tasks.rados.rados.0.smithi137.stdout:400: finishing rollback tid 1 to smithi13779221-17 2024-09-21T08:25:44.940 INFO:tasks.rados.rados.0.smithi137.stdout:400: finishing rollback tid 2 to smithi13779221-17 2024-09-21T08:25:44.940 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 112 (ObjNum 98 snap 19 seq_num 98) dirty exists 2024-09-21T08:25:44.940 INFO:tasks.rados.rados.0.smithi137.stdout:399: finishing rollback tid 0 to smithi13779221-14 2024-09-21T08:25:44.940 INFO:tasks.rados.rados.0.smithi137.stdout:399: finishing rollback tid 1 to smithi13779221-14 2024-09-21T08:25:44.941 INFO:tasks.rados.rados.0.smithi137.stdout:399: finishing rollback tid 2 to smithi13779221-14 2024-09-21T08:25:44.941 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 22 (ObjNum 13 snap 0 seq_num 13) dirty exists 2024-09-21T08:25:44.941 INFO:tasks.rados.rados.0.smithi137.stdout:402: expect (ObjNum 171 snap 34 seq_num 171) 2024-09-21T08:25:45.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:45.029 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:45.030 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:45.032 INFO:tasks.rados.rados.0.smithi137.stdout:393: finishing copy_from racing read to smithi13779221-7 2024-09-21T08:25:45.032 INFO:tasks.rados.rados.0.smithi137.stdout:407: seq_num 175 ranges {541225=644694,1907290=448390} 2024-09-21T08:25:45.038 INFO:tasks.rados.rados.0.smithi137.stdout:407: writing smithi13779221-16 from 541225 to 1185919 tid 1 2024-09-21T08:25:45.045 INFO:tasks.rados.rados.0.smithi137.stdout:407: writing smithi13779221-16 from 1907290 to 2355680 tid 2 2024-09-21T08:25:45.045 INFO:tasks.rados.rados.0.smithi137.stdout:398: done (8 left) 2024-09-21T08:25:45.045 INFO:tasks.rados.rados.0.smithi137.stdout:399: done (7 left) 2024-09-21T08:25:45.045 INFO:tasks.rados.rados.0.smithi137.stdout:400: done (6 left) 2024-09-21T08:25:45.045 INFO:tasks.rados.rados.0.smithi137.stdout:401: done (5 left) 2024-09-21T08:25:45.045 INFO:tasks.rados.rados.0.smithi137.stdout:402: done (4 left) 2024-09-21T08:25:45.045 INFO:tasks.rados.rados.0.smithi137.stdout:408: delete oid 29 current snap is 34 2024-09-21T08:25:45.045 INFO:tasks.rados.rados.0.smithi137.stdout:405: expect (ObjNum 143 snap 26 seq_num 143) 2024-09-21T08:25:45.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:45.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:45.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:45.153 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:45.159 INFO:tasks.rados.rados.0.smithi137.stdout:393: finishing copy_from to smithi13779221-7 2024-09-21T08:25:45.159 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 66 (ObjNum 156 snap 31 seq_num 156) dirty exists 2024-09-21T08:25:45.159 INFO:tasks.rados.rados.0.smithi137.stdout:396: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-21T08:25:45.254 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:45.271 INFO:tasks.rados.rados.0.smithi137.stdout:393: done (4 left) 2024-09-21T08:25:45.271 INFO:tasks.rados.rados.0.smithi137.stdout:396: done (3 left) 2024-09-21T08:25:45.271 INFO:tasks.rados.rados.0.smithi137.stdout:405: done (2 left) 2024-09-21T08:25:45.271 INFO:tasks.rados.rados.0.smithi137.stdout:408: done (1 left) 2024-09-21T08:25:45.271 INFO:tasks.rados.rados.0.smithi137.stdout:409: rollback oid 6 current snap is 34 2024-09-21T08:25:45.271 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 6 to 31 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:407: finishing write tid 1 to smithi13779221-16 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:407: finishing write tid 2 to smithi13779221-16 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:407: finishing write tid 3 to smithi13779221-16 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:407: finishing write tid 4 to smithi13779221-16 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:407: oid 16 updating version 0 to 53 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:407: oid 16 updating version 53 to 54 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:407: oid 16 version 54 is already newer than 52 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 54 (ObjNum 175 snap 34 seq_num 175) dirty exists 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:407: left oid 16 (ObjNum 175 snap 34 seq_num 175) 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:407: done (1 left) 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:410: write oid 5 current snap is 34 2024-09-21T08:25:45.272 INFO:tasks.rados.rados.0.smithi137.stdout:410: seq_num 177 ranges {574839=795192,1907901=452059} 2024-09-21T08:25:45.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:45.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:45.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:45.278 INFO:tasks.rados.rados.0.smithi137.stdout:410: writing smithi13779221-5 from 574839 to 1370031 tid 1 2024-09-21T08:25:45.285 INFO:tasks.rados.rados.0.smithi137.stdout:410: writing smithi13779221-5 from 1907901 to 2359960 tid 2 2024-09-21T08:25:45.285 INFO:tasks.rados.rados.0.smithi137.stdout:411: read oid 46 snap -1 2024-09-21T08:25:45.285 INFO:tasks.rados.rados.0.smithi137.stdout:411: expect (ObjNum 130 snap 23 seq_num 130) 2024-09-21T08:25:45.285 INFO:tasks.rados.rados.0.smithi137.stdout:412: read oid 48 snap -1 2024-09-21T08:25:45.286 INFO:tasks.rados.rados.0.smithi137.stdout:412: expect (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:25:45.286 INFO:tasks.rados.rados.0.smithi137.stdout:413: read oid 16 snap -1 2024-09-21T08:25:45.286 INFO:tasks.rados.rados.0.smithi137.stdout:413: expect (ObjNum 175 snap 34 seq_num 175) 2024-09-21T08:25:45.286 INFO:tasks.rados.rados.0.smithi137.stdout:414: read oid 14 snap -1 2024-09-21T08:25:45.286 INFO:tasks.rados.rados.0.smithi137.stdout:414: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-21T08:25:45.286 INFO:tasks.rados.rados.0.smithi137.stdout:415: copy_from oid 8 from oid 30 current snap is 34 2024-09-21T08:25:45.287 INFO:tasks.rados.rados.0.smithi137.stdout:409: finishing rollback tid 0 to smithi13779221-6 2024-09-21T08:25:45.287 INFO:tasks.rados.rados.0.smithi137.stdout:409: finishing rollback tid 1 to smithi13779221-6 2024-09-21T08:25:45.287 INFO:tasks.rados.rados.0.smithi137.stdout:409: finishing rollback tid 2 to smithi13779221-6 2024-09-21T08:25:45.287 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 26 (ObjNum 62 snap 4 seq_num 62) dirty exists 2024-09-21T08:25:45.287 INFO:tasks.rados.rados.0.smithi137.stdout:410: finishing write tid 1 to smithi13779221-5 2024-09-21T08:25:45.287 INFO:tasks.rados.rados.0.smithi137.stdout:409: done (6 left) 2024-09-21T08:25:45.287 INFO:tasks.rados.rados.0.smithi137.stdout:416: copy_from oid 18 from oid 20 current snap is 34 2024-09-21T08:25:45.288 INFO:tasks.rados.rados.0.smithi137.stdout:417: snap_remove snap 27 2024-09-21T08:25:45.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T08:25:45.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:45.400 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:45.400 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:45.401 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:45.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:45.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:45.523 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:45.523 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:45.624 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:45.646 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:45.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:45.647 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:45.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:45.759 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:45 smithi137 ceph-mon[29634]: pgmap v131: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 139 MiB data, 489 MiB used, 447 GiB / 447 GiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T08:25:45.759 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:45 smithi137 ceph-mon[29634]: osdmap e151: 8 total, 8 up, 5 in 2024-09-21T08:25:45.759 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:45 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:45.759 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:45 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:45.759 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:45 smithi137 ceph-mon[41044]: pgmap v131: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 139 MiB data, 489 MiB used, 447 GiB / 447 GiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T08:25:45.760 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:45 smithi137 ceph-mon[41044]: osdmap e151: 8 total, 8 up, 5 in 2024-09-21T08:25:45.760 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:45 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:45.760 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:45 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:45.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:45 smithi165 ceph-mon[32893]: pgmap v131: 148 pgs: 1 active+clean+snaptrim, 147 active+clean; 139 MiB data, 489 MiB used, 447 GiB / 447 GiB avail; 7.7 MiB/s rd, 3.0 MiB/s wr, 19 op/s; 0 B/s, 0 objects/s recovering 2024-09-21T08:25:45.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:45 smithi165 ceph-mon[32893]: osdmap e151: 8 total, 8 up, 5 in 2024-09-21T08:25:45.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:45 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:45.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:45 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:25:45.769 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:45.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:45.770 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:45.853 INFO:tasks.rados.rados.0.smithi137.stdout:417: done (7 left) 2024-09-21T08:25:45.853 INFO:tasks.rados.rados.0.smithi137.stdout:418: write (excl) oid 4 current snap is 34 2024-09-21T08:25:45.853 INFO:tasks.rados.rados.0.smithi137.stdout:418: seq_num 178 ranges {591646=470441,1566383=595717,2888822=102824} 2024-09-21T08:25:45.859 INFO:tasks.rados.rados.0.smithi137.stdout:418: writing smithi13779221-4 from 591646 to 1062087 tid 1 2024-09-21T08:25:45.868 INFO:tasks.rados.rados.0.smithi137.stdout:418: writing smithi13779221-4 from 1566383 to 2162100 tid 2 2024-09-21T08:25:45.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:45.873 INFO:tasks.rados.rados.0.smithi137.stdout:418: writing smithi13779221-4 from 2888822 to 2991646 tid 3 2024-09-21T08:25:45.873 INFO:tasks.rados.rados.0.smithi137.stdout:419: snap_create 2024-09-21T08:25:45.873 INFO:tasks.rados.rados.0.smithi137.stdout:414: expect (ObjNum 13 snap 0 seq_num 13) 2024-09-21T08:25:45.886 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:45.887 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:45.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:45.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:45.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:45.955 INFO:tasks.rados.rados.0.smithi137.stdout:415: finishing copy_from to smithi13779221-8 2024-09-21T08:25:45.955 INFO:tasks.rados.rados.0.smithi137.stdout:415: got expected ENOENT (src dne) 2024-09-21T08:25:45.956 INFO:tasks.rados.rados.0.smithi137.stdout:415: finishing copy_from racing read to smithi13779221-8 2024-09-21T08:25:45.956 INFO:tasks.rados.rados.0.smithi137.stdout:411: expect (ObjNum 130 snap 23 seq_num 130) 2024-09-21T08:25:45.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:46.010 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:45 smithi137 systemd[1]: Stopping Ceph alertmanager.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:25:46.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:46.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:46.018 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:46.081 INFO:tasks.rados.rados.0.smithi137.stdout:410: finishing write tid 2 to smithi13779221-5 2024-09-21T08:25:46.081 INFO:tasks.rados.rados.0.smithi137.stdout:410: finishing write tid 3 to smithi13779221-5 2024-09-21T08:25:46.081 INFO:tasks.rados.rados.0.smithi137.stdout:410: finishing write tid 4 to smithi13779221-5 2024-09-21T08:25:46.081 INFO:tasks.rados.rados.0.smithi137.stdout:410: oid 5 updating version 0 to 144 2024-09-21T08:25:46.081 INFO:tasks.rados.rados.0.smithi137.stdout:410: oid 5 version 144 is already newer than 143 2024-09-21T08:25:46.081 INFO:tasks.rados.rados.0.smithi137.stdout:410: oid 5 updating version 144 to 145 2024-09-21T08:25:46.081 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 145 (ObjNum 177 snap 34 seq_num 177) dirty exists 2024-09-21T08:25:46.081 INFO:tasks.rados.rados.0.smithi137.stdout:410: left oid 5 (ObjNum 177 snap 34 seq_num 177) 2024-09-21T08:25:46.081 INFO:tasks.rados.rados.0.smithi137.stdout:412: expect (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:25:46.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:46.142 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:46.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:46.143 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:46.230 INFO:tasks.rados.rados.0.smithi137.stdout:413: expect (ObjNum 175 snap 34 seq_num 175) 2024-09-21T08:25:46.244 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:46.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:46.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:46.266 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:46.367 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:46.370 INFO:tasks.rados.rados.0.smithi137.stdout:416: finishing copy_from to smithi13779221-18 2024-09-21T08:25:46.370 INFO:tasks.rados.rados.0.smithi137.stdout:416: got expected ENOENT (src dne) 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:416: finishing copy_from racing read to smithi13779221-18 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:418: finishing write tid 1 to smithi13779221-4 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:418: finishing write tid 2 to smithi13779221-4 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:418: finishing write tid 3 to smithi13779221-4 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:418: finishing write tid 4 to smithi13779221-4 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:418: finishing write tid 5 to smithi13779221-4 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:418: oid 4 updating version 0 to 120 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:418: oid 4 version 120 is already newer than 119 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:418: oid 4 updating version 120 to 122 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:418: oid 4 version 122 is already newer than 121 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 122 (ObjNum 178 snap 34 seq_num 178) dirty exists 2024-09-21T08:25:46.371 INFO:tasks.rados.rados.0.smithi137.stdout:418: left oid 4 (ObjNum 178 snap 34 seq_num 178) 2024-09-21T08:25:46.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:46.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:46.390 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:46.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:46.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:46.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:46.517 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:46.553 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:46 smithi137 ceph-mon[29634]: 5.8 scrub starts 2024-09-21T08:25:46.553 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:46 smithi137 ceph-mon[29634]: 5.8 scrub ok 2024-09-21T08:25:46.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:46 smithi137 ceph-mon[29634]: 3.8 scrub starts 2024-09-21T08:25:46.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:46 smithi137 ceph-mon[29634]: 3.8 scrub ok 2024-09-21T08:25:46.554 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:46 smithi137 ceph-mon[29634]: osdmap e152: 8 total, 8 up, 5 in 2024-09-21T08:25:46.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:46 smithi137 ceph-mon[41044]: 5.8 scrub starts 2024-09-21T08:25:46.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:46 smithi137 ceph-mon[41044]: 5.8 scrub ok 2024-09-21T08:25:46.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:46 smithi137 ceph-mon[41044]: 3.8 scrub starts 2024-09-21T08:25:46.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:46 smithi137 ceph-mon[41044]: 3.8 scrub ok 2024-09-21T08:25:46.554 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:46 smithi137 ceph-mon[41044]: osdmap e152: 8 total, 8 up, 5 in 2024-09-21T08:25:46.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:46.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:46.641 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:46.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:46.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:46 smithi165 ceph-mon[32893]: 5.8 scrub starts 2024-09-21T08:25:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:46 smithi165 ceph-mon[32893]: 5.8 scrub ok 2024-09-21T08:25:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:46 smithi165 ceph-mon[32893]: 3.8 scrub starts 2024-09-21T08:25:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:46 smithi165 ceph-mon[32893]: 3.8 scrub ok 2024-09-21T08:25:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:46 smithi165 ceph-mon[32893]: osdmap e152: 8 total, 8 up, 5 in 2024-09-21T08:25:46.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:46.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:46.765 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:46.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:25:46.858 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:410: done (8 left) 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:411: done (7 left) 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:412: done (6 left) 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:413: done (5 left) 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:414: done (4 left) 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:415: done (3 left) 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:416: done (2 left) 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:418: done (1 left) 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:419: done (0 left) 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:420: write oid 13 current snap is 35 2024-09-21T08:25:46.864 INFO:tasks.rados.rados.0.smithi137.stdout:420: seq_num 179 ranges {608453=545690,1624865=455728,2632598=375855} 2024-09-21T08:25:46.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:46.871 INFO:tasks.rados.rados.0.smithi137.stdout:420: writing smithi13779221-13 from 608453 to 1154143 tid 1 2024-09-21T08:25:46.882 INFO:tasks.rados.rados.0.smithi137.stdout:420: writing smithi13779221-13 from 1624865 to 2080593 tid 2 2024-09-21T08:25:46.887 INFO:tasks.rados.rados.0.smithi137.stdout:420: writing smithi13779221-13 from 2632598 to 3008453 tid 3 2024-09-21T08:25:46.888 INFO:tasks.rados.rados.0.smithi137.stdout:420: finishing write tid 1 to smithi13779221-13 2024-09-21T08:25:46.888 INFO:tasks.rados.rados.0.smithi137.stdout:420: finishing write tid 2 to smithi13779221-13 2024-09-21T08:25:46.888 INFO:tasks.rados.rados.0.smithi137.stdout:421: snap_remove snap 34 2024-09-21T08:25:46.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:46.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:47.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:47.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:47.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:47.026 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:47.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:47.148 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:47.149 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:47.149 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:47.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:47.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:47.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:47.275 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:47.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:47.402 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:47.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:47.403 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:47.483 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29611]: 2024-09-21T08:25:47.121+0000 7f1058bb7640 -1 mon.a@0(leader).osd e153 definitely_dead 0 2024-09-21T08:25:47.483 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[76851]: ts=2024-09-21T08:25:47.217Z caller=main.go:583 level=info msg="Received SIGTERM, exiting gracefully..." 2024-09-21T08:25:47.483 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:47 smithi137 podman[91248]: 2024-09-21 08:25:47.238514408 +0000 UTC m=+0.521150855 container died c6bf9891e8f11ddd032990be72600cb03a35581fb46592c641c4174bfcd0427d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T08:25:47.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:47.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:47.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:47.528 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:47.628 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:47.654 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:47.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:47.655 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:47.756 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:47 smithi165 ceph-mon[32893]: pgmap v134: 148 pgs: 5 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 136 active+clean; 156 MiB data, 530 MiB used, 446 GiB / 447 GiB avail; 37 MiB/s rd, 4.4 MiB/s wr, 49 op/s 2024-09-21T08:25:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:47 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:25:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:47 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:25:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:47 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3782981036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:47 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:47 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3782981036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:25:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:47 smithi165 ceph-mon[32893]: osdmap e153: 8 total, 8 up, 5 in 2024-09-21T08:25:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:47 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2563304402' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-21T08:25:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:47 smithi165 ceph-mon[32893]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-21T08:25:47.768 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:47 smithi137 podman[91248]: 2024-09-21 08:25:47.647924603 +0000 UTC m=+0.930561043 container cleanup c6bf9891e8f11ddd032990be72600cb03a35581fb46592c641c4174bfcd0427d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T08:25:47.768 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:47 smithi137 bash[91248]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a 2024-09-21T08:25:47.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[29634]: pgmap v134: 148 pgs: 5 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 136 active+clean; 156 MiB data, 530 MiB used, 446 GiB / 447 GiB avail; 37 MiB/s rd, 4.4 MiB/s wr, 49 op/s 2024-09-21T08:25:47.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:25:47.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:25:47.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3782981036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:47.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:47.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3782981036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:25:47.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[29634]: osdmap e153: 8 total, 8 up, 5 in 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2563304402' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[29634]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[41044]: pgmap v134: 148 pgs: 5 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 136 active+clean; 156 MiB data, 530 MiB used, 446 GiB / 447 GiB avail; 37 MiB/s rd, 4.4 MiB/s wr, 49 op/s 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3782981036' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3782981036' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[41044]: osdmap e153: 8 total, 8 up, 5 in 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2563304402' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["1"]}]: dispatch 2024-09-21T08:25:47.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:47 smithi137 ceph-mon[41044]: Client client.admin marked osd.1 out, while it was still marked up 2024-09-21T08:25:47.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:47.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:47.782 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:47.860 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.1. 2024-09-21T08:25:47.878 INFO:tasks.rados.rados.0.smithi137.stdout:421: done (1 left) 2024-09-21T08:25:47.878 INFO:tasks.rados.rados.0.smithi137.stdout:422: write oid 25 current snap is 35 2024-09-21T08:25:47.878 INFO:tasks.rados.rados.0.smithi137.stdout:422: seq_num 180 ranges {625260=620939,1683347=599386,2943668=81592} 2024-09-21T08:25:47.882 INFO:tasks.rados.rados.0.smithi137.stdout:422: writing smithi13779221-25 from 625260 to 1246199 tid 1 2024-09-21T08:25:47.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:47.889 INFO:tasks.rados.rados.0.smithi137.stdout:422: writing smithi13779221-25 from 1683347 to 2282733 tid 2 2024-09-21T08:25:47.894 INFO:tasks.rados.rados.0.smithi137.stdout:422: writing smithi13779221-25 from 2943668 to 3025260 tid 3 2024-09-21T08:25:47.894 INFO:tasks.rados.rados.0.smithi137.stdout:420: finishing write tid 3 to smithi13779221-13 2024-09-21T08:25:47.895 INFO:tasks.rados.rados.0.smithi137.stdout:420: finishing write tid 4 to smithi13779221-13 2024-09-21T08:25:47.895 INFO:tasks.rados.rados.0.smithi137.stdout:420: finishing write tid 5 to smithi13779221-13 2024-09-21T08:25:47.895 INFO:tasks.rados.rados.0.smithi137.stdout:420: oid 13 updating version 0 to 187 2024-09-21T08:25:47.895 INFO:tasks.rados.rados.0.smithi137.stdout:420: oid 13 updating version 187 to 188 2024-09-21T08:25:47.895 INFO:tasks.rados.rados.0.smithi137.stdout:420: oid 13 updating version 188 to 189 2024-09-21T08:25:47.895 INFO:tasks.rados.rados.0.smithi137.stdout:420: oid 13 updating version 189 to 190 2024-09-21T08:25:47.895 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 190 (ObjNum 179 snap 35 seq_num 179) dirty exists 2024-09-21T08:25:47.895 INFO:tasks.rados.rados.0.smithi137.stdout:420: left oid 13 (ObjNum 179 snap 35 seq_num 179) 2024-09-21T08:25:47.895 INFO:tasks.rados.rados.0.smithi137.stdout:420: done (1 left) 2024-09-21T08:25:47.895 INFO:tasks.rados.rados.0.smithi137.stdout:423: snap_remove snap 31 2024-09-21T08:25:47.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:47.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:47.910 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:48.011 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:48.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:48.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:48.036 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:48.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:48.161 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:48.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:48.162 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:48.263 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:48.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:48.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:48.288 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:48.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:48.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:48.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:48.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:48.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:48.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:48.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:48.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:48.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:48.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:48.657 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:48.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:48 smithi165 ceph-mon[32893]: 3.11 scrub starts 2024-09-21T08:25:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:48 smithi165 ceph-mon[32893]: 3.11 scrub ok 2024-09-21T08:25:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:48 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2563304402' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-21T08:25:48.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:48 smithi165 ceph-mon[32893]: osdmap e154: 8 total, 8 up, 4 in 2024-09-21T08:25:48.767 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:48 smithi137 podman[91421]: 2024-09-21 08:25:48.529351571 +0000 UTC m=+1.283451030 container remove c6bf9891e8f11ddd032990be72600cb03a35581fb46592c641c4174bfcd0427d (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T08:25:48.767 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:48 smithi137 podman[91421]: 2024-09-21 08:25:48.62928798 +0000 UTC m=+1.383387439 volume remove 614db1e5393cfa1baeb63f0136c15d0de1cff9e988d11c521f5a03f2393312a2 2024-09-21T08:25:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:48 smithi137 ceph-mon[29634]: 3.11 scrub starts 2024-09-21T08:25:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:48 smithi137 ceph-mon[29634]: 3.11 scrub ok 2024-09-21T08:25:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:48 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2563304402' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-21T08:25:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:48 smithi137 ceph-mon[29634]: osdmap e154: 8 total, 8 up, 4 in 2024-09-21T08:25:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:48 smithi137 ceph-mon[41044]: 3.11 scrub starts 2024-09-21T08:25:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:48 smithi137 ceph-mon[41044]: 3.11 scrub ok 2024-09-21T08:25:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:48 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2563304402' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["1"]}]': finished 2024-09-21T08:25:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:48 smithi137 ceph-mon[41044]: osdmap e154: 8 total, 8 up, 4 in 2024-09-21T08:25:48.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:48.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:48.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:48.869 INFO:tasks.rados.rados.0.smithi137.stdout:423: done (1 left) 2024-09-21T08:25:48.869 INFO:tasks.rados.rados.0.smithi137.stdout:424: snap_remove snap 28 2024-09-21T08:25:48.881 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:48.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:48.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:48.925 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:49.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:49.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:49.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:49.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:49.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:49.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:49.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:49.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:49.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:49.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:49.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:49.294 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:49.327 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:49.327 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:49.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:49.418 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:49.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:49.419 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:49.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:49.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:49.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:49.542 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:49.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:49.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:49.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:49.666 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:49.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:49 smithi165 ceph-mon[32893]: pgmap v137: 148 pgs: 5 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 136 active+clean; 156 MiB data, 418 MiB used, 357 GiB / 358 GiB avail; 37 MiB/s rd, 4.4 MiB/s wr, 49 op/s 2024-09-21T08:25:49.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:49 smithi165 ceph-mon[32893]: osdmap e155: 8 total, 8 up, 4 in 2024-09-21T08:25:49.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:49.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:49 smithi137 ceph-mon[29634]: pgmap v137: 148 pgs: 5 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 136 active+clean; 156 MiB data, 418 MiB used, 357 GiB / 358 GiB avail; 37 MiB/s rd, 4.4 MiB/s wr, 49 op/s 2024-09-21T08:25:49.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:49 smithi137 ceph-mon[29634]: osdmap e155: 8 total, 8 up, 4 in 2024-09-21T08:25:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:49 smithi137 ceph-mon[41044]: pgmap v137: 148 pgs: 5 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 136 active+clean; 156 MiB data, 418 MiB used, 357 GiB / 358 GiB avail; 37 MiB/s rd, 4.4 MiB/s wr, 49 op/s 2024-09-21T08:25:49.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:49 smithi137 ceph-mon[41044]: osdmap e155: 8 total, 8 up, 4 in 2024-09-21T08:25:49.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:49.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:49.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:424: done (1 left) 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:425: snap_create 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:422: finishing write tid 1 to smithi13779221-25 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:422: finishing write tid 2 to smithi13779221-25 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:422: finishing write tid 3 to smithi13779221-25 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:422: finishing write tid 4 to smithi13779221-25 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:422: finishing write tid 5 to smithi13779221-25 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:422: oid 25 updating version 0 to 12 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:422: oid 25 updating version 12 to 13 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:422: oid 25 version 13 is already newer than 10 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:422: oid 25 version 13 is already newer than 11 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 13 (ObjNum 180 snap 35 seq_num 180) dirty exists 2024-09-21T08:25:49.876 INFO:tasks.rados.rados.0.smithi137.stdout:422: left oid 25 (ObjNum 180 snap 35 seq_num 180) 2024-09-21T08:25:49.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:49.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:49.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:49.913 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:50.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:50.036 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:50.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:50.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:50.090 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:50.090 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:50.106 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:50.106 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:50.122 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:50.122 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:50.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:50.138 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:50.139 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:50.154 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:50.154 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:50.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:50.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:50.159 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:50.164 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:25:50.170 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:50.170 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:50.187 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:50.187 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:50.204 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:50.204 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:50.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:50.282 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:50.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:50.283 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:50.380 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:25:50.380 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:25:50.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44375.0:2 3.12 3:49953fa1:::default.realm:head [read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e66)", 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:30.540899+0000", 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 139.83725607, 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0074166220000000003, 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44375", 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1756555978", 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.381 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.540899+0000", 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.540899+0000", 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.540904+0000", 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8520000000000003e-06 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.540910+0000", 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5539999999999997e-06 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.540921+0000", 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1344e-05 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.540930+0000", 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3139999999999993e-06 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.540980+0000", 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9926000000000003e-05 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.540988+0000", 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6290000000000001e-06 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.548230+0000", 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072417719999999996 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.548316+0000", 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6230999999999995e-05 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:257 7.9 7:970f66b6:::smithi13779221-24:head [copy-from ver 3] snapc 4=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.758223+0000", 2024-09-21T08:25:50.385 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 64.619932348000006, 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075312460000000001, 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 257 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.758223+0000", 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.758223+0000", 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.758227+0000", 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1590000000000003e-06 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.758240+0000", 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3305000000000001e-05 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.758259+0000", 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8671000000000001e-05 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.758268+0000", 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8699999999999998e-06 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.758341+0000", 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3362000000000007e-05 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.758446+0000", 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010471699999999999 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.761578+0000", 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031319680000000002 2024-09-21T08:25:50.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.764420+0000", 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028423559999999999 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.765449+0000", 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001029128 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.765662+0000", 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000212809 2024-09-21T08:25:50.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.765712+0000", 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9261000000000002e-05 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.765754+0000", 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2639999999999998e-05 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.391 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:277 7.5 e93/80 7:a6c204bd:::smithi13779221-42:head v 93'33, mlcod=92'30)", 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.592312+0000", 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 63.785842852999998, 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0091942240000000008, 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.592312+0000", 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.592312+0000", 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.592313+0000", 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.27e-07 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.598047+0000", 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057340159999999998 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.598061+0000", 2024-09-21T08:25:50.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3750999999999999e-05 2024-09-21T08:25:50.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:50.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.598069+0000", 2024-09-21T08:25:50.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8089999999999999e-06 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.598137+0000", 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7422000000000006e-05 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.598172+0000", 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4653000000000002e-05 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.601458+0000", 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00328634 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.601507+0000", 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8806e-05 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:336 7.9 7:9d4d36a0:::smithi13779221-27:head [copy-from ver 7] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 61.808429717999999, 2024-09-21T08:25:50.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050439811000000001, 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 336 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569727+0000", 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4899999999999999e-06 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569733+0000", 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9120000000000003e-06 2024-09-21T08:25:50.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569741+0000", 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4999999999999999e-06 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569749+0000", 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2099999999999996e-06 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.583356+0000", 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013607731 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584178+0000", 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082153700000000001 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.610330+0000", 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026152526999999998 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.617047+0000", 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067167470000000003 2024-09-21T08:25:50.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.619683+0000", 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026361039999999998 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620040+0000", 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035616499999999998 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620096+0000", 2024-09-21T08:25:50.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6922000000000002e-05 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620165+0000", 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8966000000000003e-05 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:337 7.9 7:9d4d36a0:::smithi13779221-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 61.800489886000001, 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.038882593999999999, 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.403 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 337 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577668+0000", 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2960000000000001e-06 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577674+0000", 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5479999999999998e-06 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577687+0000", 2024-09-21T08:25:50.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2829e-05 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577698+0000", 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0832999999999999e-05 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584334+0000", 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066355900000000002 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584379+0000", 2024-09-21T08:25:50.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5575999999999999e-05 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616393+0000", 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032013592 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616468+0000", 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5150999999999999e-05 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616548+0000", 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0179000000000006e-05 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:348 7.9 7:92b92e8a:::smithi13779221-35:head [copy-from ver 33] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:49.567369+0000", 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 60.810786544999999, 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014025052, 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.408 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 348 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567369+0000", 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567369+0000", 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567371+0000", 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4030000000000001e-06 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567378+0000", 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9280000000000004e-06 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567390+0000", 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2027e-05 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567399+0000", 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7930000000000003e-06 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567455+0000", 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5983000000000003e-05 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:25:50.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567462+0000", 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9430000000000003e-06 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.569717+0000", 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022547579999999999 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.569810+0000", 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3554999999999995e-05 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.575654+0000", 2024-09-21T08:25:50.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058440419999999998 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.580118+0000", 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044635400000000002 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.581293+0000", 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011749709999999999 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.581328+0000", 2024-09-21T08:25:50.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5018000000000003e-05 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.581366+0000", 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7889999999999998e-05 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.581394+0000", 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8201e-05 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:349 7.9 7:92b92e8a:::smithi13779221-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:49.567414+0000", 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 60.810741030000003, 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011978051, 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 349 2024-09-21T08:25:50.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567414+0000", 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567414+0000", 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567415+0000", 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6700000000000002e-07 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567418+0000", 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1219999999999999e-06 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567425+0000", 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4659999999999996e-06 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567431+0000", 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2489999999999998e-06 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567536+0000", 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104713 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567553+0000", 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7795000000000002e-05 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.570154+0000", 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026001900000000001 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.570197+0000", 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3000999999999997e-05 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.579201+0000", 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090041860000000008 2024-09-21T08:25:50.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.579285+0000", 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3881999999999994e-05 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.579392+0000", 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010758 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:483 7.6 7:6a4067f0:::smithi13779221-23:head [sparse-read 0~0,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106)", 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:59.491840+0000", 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.886315437999997, 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081383310000000004, 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 483 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491840+0000", 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491840+0000", 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491843+0000", 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.489e-06 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491863+0000", 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9585999999999998e-05 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491885+0000", 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2493e-05 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491898+0000", 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2947e-05 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491966+0000", 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7336999999999996e-05 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491971+0000", 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.711e-06 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.499843+0000", 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078712929999999997 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.499978+0000", 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000135475 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:484 7.6 7:6a4067f0:::smithi13779221-23:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106)", 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:59.491925+0000", 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.886230304000001, 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082087619999999997, 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.425 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 484 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491925+0000", 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491925+0000", 2024-09-21T08:25:50.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491926+0000", 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6299999999999993e-07 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491930+0000", 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6049999999999998e-06 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491938+0000", 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.225e-06 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491944+0000", 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2260000000000002e-06 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.492051+0000", 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106582 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.492056+0000", 2024-09-21T08:25:50.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4650000000000002e-06 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.500020+0000", 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079640609999999997 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.500134+0000", 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011363500000000001 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:485 7.6 7:6a4067f0:::smithi13779221-23:head [sparse-read 0~0] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e106)", 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:59.491954+0000", 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.886201679999999, 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0083074900000000007, 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "reached pg", 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 485 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.430 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491954+0000", 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491954+0000", 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491954+0000", 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1699999999999997e-07 2024-09-21T08:25:50.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491958+0000", 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.129e-06 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491964+0000", 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7139999999999996e-06 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.491970+0000", 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3730000000000001e-06 2024-09-21T08:25:50.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.492088+0000", 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011835799999999999 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.492107+0000", 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8635999999999999e-05 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.500161+0000", 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0080545119999999998 2024-09-21T08:25:50.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:59.500261+0000", 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9951000000000006e-05 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:552 7.d e116/102 7:b0b32695:::smithi13779221-19:head v 116'26, mlcod=107'23)", 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:09.539784+0000", 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 40.838370959999999, 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011090490999999999, 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.434 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.539784+0000", 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.539784+0000", 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.539785+0000", 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3000000000000002e-07 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.542541+0000", 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002755997 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.542553+0000", 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2150999999999999e-05 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.542561+0000", 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8860000000000004e-06 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.542644+0000", 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3232000000000005e-05 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.542663+0000", 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8635999999999999e-05 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.550806+0000", 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081433709999999999 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.550875+0000", 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8887999999999995e-05 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:569 7.e e117/80 7:78d05d33:::smithi13779221-41:head v 117'23, mlcod=100'19)", 2024-09-21T08:25:50.438 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.565684+0000", 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 39.812471332999998, 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013671371999999999, 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565684+0000", 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565684+0000", 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565685+0000", 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9899999999999999e-07 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567942+0000", 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022574980000000001 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567966+0000", 2024-09-21T08:25:50.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3577000000000001e-05 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567978+0000", 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2068e-05 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.568075+0000", 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7194000000000005e-05 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.568100+0000", 2024-09-21T08:25:50.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4593999999999999e-05 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579300+0000", 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011200217 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579355+0000", 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5325000000000003e-05 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:571 7.1 e117/102 7:88bf4ad2:::smithi13779221-6:head v 117'8, mlcod=0'0)", 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.574186+0000", 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 39.803969782999999, 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0099502229999999994, 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574186+0000", 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574186+0000", 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574187+0000", 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.032e-06 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576798+0000", 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026110069999999998 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576818+0000", 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.001e-05 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576833+0000", 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5396999999999998e-05 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.577194+0000", 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036094399999999998 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.577228+0000", 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3593000000000002e-05 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584053+0000", 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068257939999999996 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584136+0000", 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2446000000000003e-05 2024-09-21T08:25:50.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:592 7.5 e117/80 7:aaa5c288:::smithi13779221-20:head v 117'73, mlcod=116'70)", 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.744460+0000", 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 39.633695756999998, 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0096752599999999998, 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.744460+0000", 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.744460+0000", 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.744462+0000", 2024-09-21T08:25:50.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9999999999999999e-06 2024-09-21T08:25:50.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.749581+0000", 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051193150000000002 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.749596+0000", 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5434000000000002e-05 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.749603+0000", 2024-09-21T08:25:50.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9009999999999997e-06 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.749667+0000", 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3595000000000004e-05 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.749691+0000", 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4681999999999998e-05 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.754089+0000", 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043974959999999999 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.754135+0000", 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5837000000000002e-05 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:601 7.9 7:9d4d36a0:::smithi13779221-27:head [write 1651964~401685] snapc 15=[15,14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:11.503430+0000", 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 38.874724901999997, 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01254068, 2024-09-21T08:25:50.452 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 601 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503430+0000", 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503430+0000", 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503433+0000", 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3369999999999998e-06 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503559+0000", 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012637600000000001 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503581+0000", 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1925000000000001e-05 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503589+0000", 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3189999999999995e-06 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503647+0000", 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7720999999999998e-05 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503760+0000", 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112897 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503933+0000", 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017294200000000001 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.508305+0000", 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043725439999999999 2024-09-21T08:25:50.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.514072+0000", 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057668290000000002 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.515855+0000", 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017826249999999999 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.515910+0000", 2024-09-21T08:25:50.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4879000000000001e-05 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.515971+0000", 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1285999999999996e-05 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:711 7.6 7:611b1c4b:::smithi13779221-3:head [write 766419~549428] snapc 18=[17,16,d] ondisk+write+known_if_redirected+supports_pool_eio e129)", 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:22.674223+0000", 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 27.703932483999999, 2024-09-21T08:25:50.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011569104, 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 711 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.674223+0000", 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.674223+0000", 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.674226+0000", 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1389999999999999e-06 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.675000+0000", 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00077443100000000001 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.675016+0000", 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.607e-05 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.675023+0000", 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.002e-06 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.675074+0000", 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0832999999999999e-05 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.675079+0000", 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8760000000000004e-06 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.677668+0000", 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025890510000000002 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.678301+0000", 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00063255099999999997 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.678393+0000", 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2409000000000002e-05 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.679550+0000", 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001156439 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.682365+0000", 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028157600000000001 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.685676+0000", 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033101900000000002 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.685738+0000", 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2626999999999995e-05 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:22.685792+0000", 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3726e-05 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.464 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:833 7.6 7:611b1c4b:::smithi13779221-3:head [copy-from ver 0] snapc 1b=[1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.705983+0000", 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.672172062000001, 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013624030000000001, 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 833 2024-09-21T08:25:50.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705983+0000", 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705983+0000", 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705987+0000", 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8840000000000004e-06 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705997+0000", 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0370999999999999e-05 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706017+0000", 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9979e-05 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706028+0000", 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0128e-05 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706105+0000", 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7667e-05 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706114+0000", 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4540000000000007e-06 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.713988+0000", 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078742559999999996 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.714091+0000", 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010333200000000001 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.719607+0000", 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005515959 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.469 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:834 7.6 7:611b1c4b:::smithi13779221-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.67210373, 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014179074, 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 834 2024-09-21T08:25:50.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:25:50.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.23e-07 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706056+0000", 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.365e-06 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706064+0000", 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3399999999999998e-06 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706070+0000", 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0719999999999996e-06 2024-09-21T08:25:50.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706167+0000", 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6844999999999999e-05 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706174+0000", 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9630000000000001e-06 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.714277+0000", 2024-09-21T08:25:50.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081029469999999992 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.714328+0000", 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1270999999999998e-05 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.720039+0000", 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057105020000000001 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.720127+0000", 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7916000000000003e-05 2024-09-21T08:25:50.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.720231+0000", 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010403000000000001 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:974 7.5 e144/80 7:a9dd6c7d:::smithi13779221-47:head v 144'174, mlcod=142'170)", 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:37.771797+0000", 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 12.606358478000001, 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081165179999999996, 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.475 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:37.771797+0000", 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:37.771797+0000", 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:37.771800+0000", 2024-09-21T08:25:50.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6260000000000001e-06 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:37.775297+0000", 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034969559999999998 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:37.775325+0000", 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7433999999999999e-05 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:37.775338+0000", 2024-09-21T08:25:50.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3246e-05 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:37.775424+0000", 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6098000000000001e-05 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:37.775460+0000", 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5499e-05 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:37.779865+0000", 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044057619999999997 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:37.779913+0000", 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7896999999999999e-05 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1019 7.4 7:26d6e516:::smithi13779221-33:head [write 756348~784249,stat] snapc 22=[22,21,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:40.791263+0000", 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.5868924100000008, 2024-09-21T08:25:50.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0096660300000000008, 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1019 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.791263+0000", 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.791263+0000", 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.791267+0000", 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3560000000000003e-06 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.791945+0000", 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067770899999999997 2024-09-21T08:25:50.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.791982+0000", 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7271999999999998e-05 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.792077+0000", 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4776000000000002e-05 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.792196+0000", 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119079 2024-09-21T08:25:50.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.792340+0000", 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014390599999999999 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,5", 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.792566+0000", 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022559300000000001 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.794953+0000", 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002387054 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.796435+0000", 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001482489 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.800831+0000", 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043955399999999999 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.800887+0000", 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6372999999999997e-05 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:25:50.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:25:50.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.800929+0000", 2024-09-21T08:25:50.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1882999999999999e-05 2024-09-21T08:25:50.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.485 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.485 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:25:50.485 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:25:50.485 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:25:50.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:50.486 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:50.486 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:50.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:50.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:50.609 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:50.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:50.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:50.731 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:50.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:50.732 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:50.833 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:50.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:50.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:50.856 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:50.882 INFO:tasks.rados.rados.0.smithi137.stdout:422: done (1 left) 2024-09-21T08:25:50.882 INFO:tasks.rados.rados.0.smithi137.stdout:425: done (0 left) 2024-09-21T08:25:50.882 INFO:tasks.rados.rados.0.smithi137.stdout:426: write (excl) oid 36 current snap is 36 2024-09-21T08:25:50.883 INFO:tasks.rados.rados.0.smithi137.stdout:426: seq_num 181 ranges {642067=696188,2025476=459397,2971091=70976} 2024-09-21T08:25:50.889 INFO:tasks.rados.rados.0.smithi137.stdout:426: writing smithi13779221-36 from 642067 to 1338255 tid 1 2024-09-21T08:25:50.896 INFO:tasks.rados.rados.0.smithi137.stdout:426: writing smithi13779221-36 from 2025476 to 2484873 tid 2 2024-09-21T08:25:50.899 INFO:tasks.rados.rados.0.smithi137.stdout:426: writing smithi13779221-36 from 2971091 to 3042067 tid 3 2024-09-21T08:25:50.899 INFO:tasks.rados.rados.0.smithi137.stdout:427: write (excl) oid 47 current snap is 36 2024-09-21T08:25:50.899 INFO:tasks.rados.rados.0.smithi137.stdout:427: seq_num 182 ranges {658874=771437,2083958=603055} 2024-09-21T08:25:50.906 INFO:tasks.rados.rados.0.smithi137.stdout:427: writing smithi13779221-47 from 658874 to 1430311 tid 1 2024-09-21T08:25:50.914 INFO:tasks.rados.rados.0.smithi137.stdout:427: writing smithi13779221-47 from 2083958 to 2687013 tid 2 2024-09-21T08:25:50.914 INFO:tasks.rados.rados.0.smithi137.stdout:428: snap_create 2024-09-21T08:25:50.914 INFO:tasks.rados.rados.0.smithi137.stdout:427: finishing write tid 1 to smithi13779221-47 2024-09-21T08:25:50.917 INFO:tasks.rados.rados.0.smithi137.stdout:427: finishing write tid 2 to smithi13779221-47 2024-09-21T08:25:50.917 INFO:tasks.rados.rados.0.smithi137.stdout:427: finishing write tid 3 to smithi13779221-47 2024-09-21T08:25:50.917 INFO:tasks.rados.rados.0.smithi137.stdout:427: finishing write tid 4 to smithi13779221-47 2024-09-21T08:25:50.917 INFO:tasks.rados.rados.0.smithi137.stdout:427: oid 47 updating version 0 to 213 2024-09-21T08:25:50.917 INFO:tasks.rados.rados.0.smithi137.stdout:427: oid 47 version 213 is already newer than 211 2024-09-21T08:25:50.917 INFO:tasks.rados.rados.0.smithi137.stdout:427: oid 47 version 213 is already newer than 212 2024-09-21T08:25:50.917 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 213 (ObjNum 182 snap 36 seq_num 182) dirty exists 2024-09-21T08:25:50.917 INFO:tasks.rados.rados.0.smithi137.stdout:427: left oid 47 (ObjNum 182 snap 36 seq_num 182) 2024-09-21T08:25:50.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:50.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:50.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:50.978 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:51.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:50 smithi137 ceph-mon[29634]: osdmap e156: 8 total, 8 up, 4 in 2024-09-21T08:25:51.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:50 smithi137 ceph-mon[29634]: pgmap v140: 148 pgs: 5 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 136 active+clean; 156 MiB data, 418 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:25:51.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:50 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3821946521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:51.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:25:50 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:25:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:25:51.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:50 smithi137 ceph-mon[41044]: osdmap e156: 8 total, 8 up, 4 in 2024-09-21T08:25:51.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:50 smithi137 ceph-mon[41044]: pgmap v140: 148 pgs: 5 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 136 active+clean; 156 MiB data, 418 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:25:51.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:50 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3821946521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:51.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:51.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:51.101 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:51.101 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:51.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:51.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:51.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:51.224 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:50 smithi165 ceph-mon[32893]: osdmap e156: 8 total, 8 up, 4 in 2024-09-21T08:25:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:50 smithi165 ceph-mon[32893]: pgmap v140: 148 pgs: 5 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 136 active+clean; 156 MiB data, 418 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:25:51.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:50 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3821946521' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:51.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:51.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:51.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:51.347 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:51.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:51.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:51.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:51.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:51.502 INFO:tasks.rados.rados.0.smithi137.stdout:427: done (2 left) 2024-09-21T08:25:51.503 INFO:tasks.rados.rados.0.smithi137.stdout:428: done (1 left) 2024-09-21T08:25:51.503 INFO:tasks.rados.rados.0.smithi137.stdout:429: write oid 8 current snap is 37 2024-09-21T08:25:51.503 INFO:tasks.rados.rados.0.smithi137.stdout:429: seq_num 183 ranges {675681=563039,1858793=746713,3025937=49744} 2024-09-21T08:25:51.509 INFO:tasks.rados.rados.0.smithi137.stdout:429: writing smithi13779221-8 from 675681 to 1238720 tid 1 2024-09-21T08:25:51.518 INFO:tasks.rados.rados.0.smithi137.stdout:429: writing smithi13779221-8 from 1858793 to 2605506 tid 2 2024-09-21T08:25:51.522 INFO:tasks.rados.rados.0.smithi137.stdout:429: writing smithi13779221-8 from 3025937 to 3075681 tid 3 2024-09-21T08:25:51.522 INFO:tasks.rados.rados.0.smithi137.stdout:430: read oid 38 snap -1 2024-09-21T08:25:51.522 INFO:tasks.rados.rados.0.smithi137.stdout:430: expect (ObjNum 160 snap 32 seq_num 160) 2024-09-21T08:25:51.522 INFO:tasks.rados.rados.0.smithi137.stdout:431: copy_from oid 39 from oid 20 current snap is 37 2024-09-21T08:25:51.522 INFO:tasks.rados.rados.0.smithi137.stdout:426: finishing write tid 1 to smithi13779221-36 2024-09-21T08:25:51.522 INFO:tasks.rados.rados.0.smithi137.stdout:426: finishing write tid 2 to smithi13779221-36 2024-09-21T08:25:51.522 INFO:tasks.rados.rados.0.smithi137.stdout:429: finishing write tid 1 to smithi13779221-8 2024-09-21T08:25:51.522 INFO:tasks.rados.rados.0.smithi137.stdout:426: finishing write tid 3 to smithi13779221-36 2024-09-21T08:25:51.522 INFO:tasks.rados.rados.0.smithi137.stdout:426: finishing write tid 4 to smithi13779221-36 2024-09-21T08:25:51.522 INFO:tasks.rados.rados.0.smithi137.stdout:426: finishing write tid 5 to smithi13779221-36 2024-09-21T08:25:51.523 INFO:tasks.rados.rados.0.smithi137.stdout:426: oid 36 updating version 0 to 196 2024-09-21T08:25:51.523 INFO:tasks.rados.rados.0.smithi137.stdout:426: oid 36 updating version 196 to 198 2024-09-21T08:25:51.523 INFO:tasks.rados.rados.0.smithi137.stdout:426: oid 36 version 198 is already newer than 197 2024-09-21T08:25:51.523 INFO:tasks.rados.rados.0.smithi137.stdout:426: oid 36 version 198 is already newer than 195 2024-09-21T08:25:51.523 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 198 (ObjNum 181 snap 36 seq_num 181) dirty exists 2024-09-21T08:25:51.523 INFO:tasks.rados.rados.0.smithi137.stdout:426: left oid 36 (ObjNum 181 snap 36 seq_num 181) 2024-09-21T08:25:51.523 INFO:tasks.rados.rados.0.smithi137.stdout:426: done (3 left) 2024-09-21T08:25:51.523 INFO:tasks.rados.rados.0.smithi137.stdout:432: write oid 47 current snap is 37 2024-09-21T08:25:51.523 INFO:tasks.rados.rados.0.smithi137.stdout:432: seq_num 184 ranges {692488=638288,1917275=606724} 2024-09-21T08:25:51.528 INFO:tasks.rados.rados.0.smithi137.stdout:432: writing smithi13779221-47 from 692488 to 1330776 tid 1 2024-09-21T08:25:51.535 INFO:tasks.rados.rados.0.smithi137.stdout:432: writing smithi13779221-47 from 1917275 to 2523999 tid 2 2024-09-21T08:25:51.535 INFO:tasks.rados.rados.0.smithi137.stdout:433: write (excl) oid 41 current snap is 37 2024-09-21T08:25:51.535 INFO:tasks.rados.rados.0.smithi137.stdout:431: finishing copy_from to smithi13779221-39 2024-09-21T08:25:51.535 INFO:tasks.rados.rados.0.smithi137.stdout:431: got expected ENOENT (src dne) 2024-09-21T08:25:51.535 INFO:tasks.rados.rados.0.smithi137.stdout:433: seq_num 185 ranges {709295=713537,1859404=750382} 2024-09-21T08:25:51.543 INFO:tasks.rados.rados.0.smithi137.stdout:433: writing smithi13779221-41 from 709295 to 1422832 tid 1 2024-09-21T08:25:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:433: writing smithi13779221-41 from 1859404 to 2609786 tid 2 2024-09-21T08:25:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:434: copy_from oid 16 from oid 33 current snap is 37 2024-09-21T08:25:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:431: finishing copy_from racing read to smithi13779221-39 2024-09-21T08:25:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:429: finishing write tid 2 to smithi13779221-8 2024-09-21T08:25:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:429: finishing write tid 3 to smithi13779221-8 2024-09-21T08:25:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:429: finishing write tid 4 to smithi13779221-8 2024-09-21T08:25:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:429: finishing write tid 5 to smithi13779221-8 2024-09-21T08:25:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:429: oid 8 updating version 0 to 133 2024-09-21T08:25:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:429: oid 8 updating version 133 to 134 2024-09-21T08:25:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:429: oid 8 version 134 is already newer than 132 2024-09-21T08:25:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:429: oid 8 updating version 134 to 135 2024-09-21T08:25:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 135 (ObjNum 183 snap 37 seq_num 183) dirty exists 2024-09-21T08:25:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:429: left oid 8 (ObjNum 183 snap 37 seq_num 183) 2024-09-21T08:25:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:430: expect (ObjNum 160 snap 32 seq_num 160) 2024-09-21T08:25:51.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:51.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:51.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:51.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:51.641 INFO:tasks.rados.rados.0.smithi137.stdout:432: finishing write tid 1 to smithi13779221-47 2024-09-21T08:25:51.641 INFO:tasks.rados.rados.0.smithi137.stdout:432: finishing write tid 2 to smithi13779221-47 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:432: finishing write tid 3 to smithi13779221-47 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:432: finishing write tid 4 to smithi13779221-47 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:432: oid 47 updating version 0 to 215 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:432: oid 47 updating version 215 to 217 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:432: oid 47 version 217 is already newer than 216 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:432: left oid 47 (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:433: finishing write tid 1 to smithi13779221-41 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:433: finishing write tid 2 to smithi13779221-41 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:433: finishing write tid 3 to smithi13779221-41 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:433: finishing write tid 4 to smithi13779221-41 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:433: oid 41 updating version 0 to 71 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:433: oid 41 updating version 71 to 72 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:433: oid 41 version 72 is already newer than 70 2024-09-21T08:25:51.642 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 72 (ObjNum 185 snap 37 seq_num 185) dirty exists 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:433: left oid 41 (ObjNum 185 snap 37 seq_num 185) 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:434: finishing copy_from racing read to smithi13779221-16 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:434: finishing copy_from to smithi13779221-16 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 34 (ObjNum 164 snap 33 seq_num 164) dirty exists 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:429: done (5 left) 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:430: done (4 left) 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:431: done (3 left) 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:432: done (2 left) 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:433: done (1 left) 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:434: done (0 left) 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:435: write oid 27 current snap is 37 2024-09-21T08:25:51.643 INFO:tasks.rados.rados.0.smithi137.stdout:435: seq_num 186 ranges {726102=788786,1917886=610393,3108206=17896} 2024-09-21T08:25:51.649 INFO:tasks.rados.rados.0.smithi137.stdout:435: writing smithi13779221-27 from 726102 to 1514888 tid 1 2024-09-21T08:25:51.655 INFO:tasks.rados.rados.0.smithi137.stdout:435: writing smithi13779221-27 from 1917886 to 2528279 tid 2 2024-09-21T08:25:51.658 INFO:tasks.rados.rados.0.smithi137.stdout:435: writing smithi13779221-27 from 3108206 to 3126102 tid 3 2024-09-21T08:25:51.658 INFO:tasks.rados.rados.0.smithi137.stdout:436: write oid 17 current snap is 37 2024-09-21T08:25:51.659 INFO:tasks.rados.rados.0.smithi137.stdout:436: seq_num 187 ranges {742909=464035,1976368=754051} 2024-09-21T08:25:51.665 INFO:tasks.rados.rados.0.smithi137.stdout:436: writing smithi13779221-17 from 742909 to 1206944 tid 1 2024-09-21T08:25:51.673 INFO:tasks.rados.rados.0.smithi137.stdout:436: writing smithi13779221-17 from 1976368 to 2730419 tid 2 2024-09-21T08:25:51.673 INFO:tasks.rados.rados.0.smithi137.stdout:437: copy_from oid 3 from oid 50 current snap is 37 2024-09-21T08:25:51.673 INFO:tasks.rados.rados.0.smithi137.stdout:435: finishing write tid 1 to smithi13779221-27 2024-09-21T08:25:51.673 INFO:tasks.rados.rados.0.smithi137.stdout:438: snap_remove snap 36 2024-09-21T08:25:51.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:51.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:51.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:51.716 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:51.817 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:51.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:51.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:51.839 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:51.876 INFO:journalctl@ceph.osd.5.smithi165.stdout:Sep 21 08:25:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-5[44301]: 2024-09-21T08:25:51.515+0000 7f95a243a640 -1 snap_mapper.get_next_objects_to_trim::New Clone-Objects were added to Snap 1c after trimming was started 2024-09-21T08:25:51.877 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:25:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:25:51.940 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:51.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:51.963 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:51.963 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:52.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:51 smithi137 ceph-mon[41044]: osdmap e157: 8 total, 8 up, 4 in 2024-09-21T08:25:52.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:51 smithi137 ceph-mon[41044]: osdmap e158: 8 total, 8 up, 4 in 2024-09-21T08:25:52.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:51 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:51 smithi137 ceph-mon[29634]: osdmap e157: 8 total, 8 up, 4 in 2024-09-21T08:25:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:51 smithi137 ceph-mon[29634]: osdmap e158: 8 total, 8 up, 4 in 2024-09-21T08:25:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:51 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:52.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:52.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:52.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:52.086 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:52.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T08:25:52.121 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:52.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:52.208 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:52.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:52.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:52.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:51 smithi165 ceph-mon[32893]: osdmap e157: 8 total, 8 up, 4 in 2024-09-21T08:25:52.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:51 smithi165 ceph-mon[32893]: osdmap e158: 8 total, 8 up, 4 in 2024-09-21T08:25:52.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:51 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:25:52.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:52.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:52.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:52.332 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:52.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:52.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:52.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:52.454 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:438: done (3 left) 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:439: snap_create 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:435: finishing write tid 2 to smithi13779221-27 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:435: finishing write tid 3 to smithi13779221-27 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:435: finishing write tid 4 to smithi13779221-27 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:435: finishing write tid 5 to smithi13779221-27 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:435: oid 27 updating version 0 to 121 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:435: oid 27 version 121 is already newer than 119 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:435: oid 27 version 121 is already newer than 120 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:435: oid 27 version 121 is already newer than 118 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 121 (ObjNum 186 snap 37 seq_num 186) dirty exists 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:435: left oid 27 (ObjNum 186 snap 37 seq_num 186) 2024-09-21T08:25:52.520 INFO:tasks.rados.rados.0.smithi137.stdout:436: finishing write tid 1 to smithi13779221-17 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:436: finishing write tid 2 to smithi13779221-17 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:436: finishing write tid 3 to smithi13779221-17 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:436: finishing write tid 4 to smithi13779221-17 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:436: oid 17 updating version 0 to 125 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:436: oid 17 version 125 is already newer than 123 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:436: oid 17 version 125 is already newer than 124 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 125 (ObjNum 187 snap 37 seq_num 187) dirty exists 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:436: left oid 17 (ObjNum 187 snap 37 seq_num 187) 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:437: finishing copy_from racing read to smithi13779221-3 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:437: finishing copy_from to smithi13779221-3 2024-09-21T08:25:52.521 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 136 (ObjNum 171 snap 34 seq_num 171) dirty exists 2024-09-21T08:25:52.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:52.578 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:52.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:52.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:52.638 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:52.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:52.701 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:52.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:52.702 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:52.761 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:52 smithi137 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@alertmanager.a.service: Deactivated successfully. 2024-09-21T08:25:52.761 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:52 smithi137 systemd[1]: Stopped Ceph alertmanager.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:25:52.761 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:52 smithi137 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@alertmanager.a.service: Consumed 1.052s CPU time. 2024-09-21T08:25:52.761 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:52 smithi137 systemd[1]: Starting Ceph alertmanager.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:25:52.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:52.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:52.827 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:52.827 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:52.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:52.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:52.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:52.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:53.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:53.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:53.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:53.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:53.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:53.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:53.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:53.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:53.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:53.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:53.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:53.319 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:53.397 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:25:53.397 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:25:53.397 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:25:53.397 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:25:53.397 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:53 smithi137 podman[92290]: 2024-09-21 08:25:52.97208505 +0000 UTC m=+0.016931589 image pull c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19 quay.io/prometheus/alertmanager:v0.25.0 2024-09-21T08:25:53.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:53.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:53.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:53.443 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:53.506 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:25:53.511 INFO:tasks.rados.rados.0.smithi137.stdout:435: done (3 left) 2024-09-21T08:25:53.511 INFO:tasks.rados.rados.0.smithi137.stdout:436: done (2 left) 2024-09-21T08:25:53.511 INFO:tasks.rados.rados.0.smithi137.stdout:437: done (1 left) 2024-09-21T08:25:53.511 INFO:tasks.rados.rados.0.smithi137.stdout:439: done (0 left) 2024-09-21T08:25:53.512 INFO:tasks.rados.rados.0.smithi137.stdout:440: read oid 20 snap -1 2024-09-21T08:25:53.512 INFO:tasks.rados.rados.0.smithi137.stdout:440: expect deleted 2024-09-21T08:25:53.512 INFO:tasks.rados.rados.0.smithi137.stdout:441: rollback oid 43 current snap is 38 2024-09-21T08:25:53.512 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 43 to 37 2024-09-21T08:25:53.512 INFO:tasks.rados.rados.0.smithi137.stdout:442: delete oid 42 current snap is 38 2024-09-21T08:25:53.517 INFO:tasks.rados.rados.0.smithi137.stdout:441: finishing rollback tid 1 to smithi13779221-43 2024-09-21T08:25:53.517 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 0 (ObjNum 2513 snap 0 seq_num 379560512) dirty dne 2024-09-21T08:25:53.518 INFO:tasks.rados.rados.0.smithi137.stdout:440: done (2 left) 2024-09-21T08:25:53.518 INFO:tasks.rados.rados.0.smithi137.stdout:441: done (1 left) 2024-09-21T08:25:53.518 INFO:tasks.rados.rados.0.smithi137.stdout:442: done (0 left) 2024-09-21T08:25:53.518 INFO:tasks.rados.rados.0.smithi137.stdout:443: snap_remove snap 37 2024-09-21T08:25:53.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:53.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:53.566 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:53.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:53.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:53.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:53.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:53.690 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:53.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:25:53.751 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:53 smithi137 ceph-mon[29634]: pgmap v143: 148 pgs: 4 active+undersized+remapped, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 133 active+clean; 137 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 941 KiB/s wr, 3 op/s; 17 MiB/s, 41 objects/s recovering 2024-09-21T08:25:53.752 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:53 smithi137 ceph-mon[29634]: osdmap e159: 8 total, 8 up, 4 in 2024-09-21T08:25:53.752 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:53 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/716997592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:53.752 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:53 smithi137 ceph-mon[41044]: pgmap v143: 148 pgs: 4 active+undersized+remapped, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 133 active+clean; 137 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 941 KiB/s wr, 3 op/s; 17 MiB/s, 41 objects/s recovering 2024-09-21T08:25:53.752 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:53 smithi137 ceph-mon[41044]: osdmap e159: 8 total, 8 up, 4 in 2024-09-21T08:25:53.752 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:53 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/716997592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:53.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:53.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:53.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:53.813 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:53.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:53.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:53.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:53.936 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:53 smithi165 ceph-mon[32893]: pgmap v143: 148 pgs: 4 active+undersized+remapped, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 133 active+clean; 137 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 941 KiB/s wr, 3 op/s; 17 MiB/s, 41 objects/s recovering 2024-09-21T08:25:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:53 smithi165 ceph-mon[32893]: osdmap e159: 8 total, 8 up, 4 in 2024-09-21T08:25:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:53 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/716997592' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:25:54.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:54.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:54.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:54.059 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:54.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:54.180 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:54.181 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:54.181 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:54.277 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 3, 6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:25:54.277 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:25:54.277 INFO:tasks.thrashosds.thrasher:Shrinking pool unique_pool_0 2024-09-21T08:25:54.277 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool unique_pool_0 by 10 min 8 2024-09-21T08:25:54.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:25:54.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:54.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T08:25:54.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:54.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:54.351 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:54.452 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:54.478 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:54.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:54.479 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:54.518 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:54 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:54.518 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:54 smithi137 podman[92290]: 2024-09-21 08:25:54.427139434 +0000 UTC m=+1.471985945 volume create 056954df32193321b6765f8c634a86c063e3ef2d63c9f61321ba051812262ee3 2024-09-21T08:25:54.522 INFO:tasks.rados.rados.0.smithi137.stdout:443: done (0 left) 2024-09-21T08:25:54.522 INFO:tasks.rados.rados.0.smithi137.stdout:444: rollback oid 43 current snap is 38 2024-09-21T08:25:54.523 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 43 to 35 2024-09-21T08:25:54.523 INFO:tasks.rados.rados.0.smithi137.stdout:445: write (excl) oid 48 current snap is 38 2024-09-21T08:25:54.523 INFO:tasks.rados.rados.0.smithi137.stdout:445: seq_num 189 ranges {776523=614533,1976979=757720} 2024-09-21T08:25:54.530 INFO:tasks.rados.rados.0.smithi137.stdout:445: writing smithi13779221-48 from 776523 to 1391056 tid 1 2024-09-21T08:25:54.538 INFO:tasks.rados.rados.0.smithi137.stdout:445: writing smithi13779221-48 from 1976979 to 2734699 tid 2 2024-09-21T08:25:54.538 INFO:tasks.rados.rados.0.smithi137.stdout:446: delete oid 14 current snap is 38 2024-09-21T08:25:54.538 INFO:tasks.rados.rados.0.smithi137.stdout:444: finishing rollback tid 1 to smithi13779221-43 2024-09-21T08:25:54.538 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 0 (ObjNum 2513 snap 0 seq_num 379560512) dirty dne 2024-09-21T08:25:54.539 INFO:tasks.rados.rados.0.smithi137.stdout:445: finishing write tid 1 to smithi13779221-48 2024-09-21T08:25:54.540 INFO:tasks.rados.rados.0.smithi137.stdout:444: done (2 left) 2024-09-21T08:25:54.540 INFO:tasks.rados.rados.0.smithi137.stdout:446: done (1 left) 2024-09-21T08:25:54.540 INFO:tasks.rados.rados.0.smithi137.stdout:447: copy_from oid 18 from oid 31 current snap is 38 2024-09-21T08:25:54.540 INFO:tasks.rados.rados.0.smithi137.stdout:448: read oid 26 snap 35 2024-09-21T08:25:54.540 INFO:tasks.rados.rados.0.smithi137.stdout:448: expect deleted 2024-09-21T08:25:54.540 INFO:tasks.rados.rados.0.smithi137.stdout:449: snap_create 2024-09-21T08:25:54.541 INFO:tasks.rados.rados.0.smithi137.stdout:445: finishing write tid 2 to smithi13779221-48 2024-09-21T08:25:54.541 INFO:tasks.rados.rados.0.smithi137.stdout:445: finishing write tid 3 to smithi13779221-48 2024-09-21T08:25:54.541 INFO:tasks.rados.rados.0.smithi137.stdout:445: finishing write tid 4 to smithi13779221-48 2024-09-21T08:25:54.541 INFO:tasks.rados.rados.0.smithi137.stdout:445: oid 48 updating version 0 to 153 2024-09-21T08:25:54.541 INFO:tasks.rados.rados.0.smithi137.stdout:445: oid 48 version 153 is already newer than 151 2024-09-21T08:25:54.541 INFO:tasks.rados.rados.0.smithi137.stdout:445: oid 48 version 153 is already newer than 152 2024-09-21T08:25:54.542 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 153 (ObjNum 189 snap 38 seq_num 189) dirty exists 2024-09-21T08:25:54.542 INFO:tasks.rados.rados.0.smithi137.stdout:445: left oid 48 (ObjNum 189 snap 38 seq_num 189) 2024-09-21T08:25:54.550 INFO:tasks.rados.rados.0.smithi137.stdout:447: finishing copy_from racing read to smithi13779221-18 2024-09-21T08:25:54.556 INFO:tasks.rados.rados.0.smithi137.stdout:447: finishing copy_from to smithi13779221-18 2024-09-21T08:25:54.556 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 91 (ObjNum 149 snap 27 seq_num 149) dirty exists 2024-09-21T08:25:54.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:54.601 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:54.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:54.602 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:54.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:54.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:54.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:54.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:54 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/716997592' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:54 smithi137 ceph-mon[29634]: osdmap e160: 8 total, 8 up, 4 in 2024-09-21T08:25:54.768 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:54 smithi137 podman[92290]: 2024-09-21 08:25:54.544262959 +0000 UTC m=+1.589109468 container create 294634ea8fb9c44ba646a21e85f7ced5cf3733f362f7218cbc66d3e1642913fa (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T08:25:54.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:54 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:54.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:54 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/716997592' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:54.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:54 smithi137 ceph-mon[41044]: osdmap e160: 8 total, 8 up, 4 in 2024-09-21T08:25:54.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:54.849 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:54.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:54.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:54.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:54.973 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:54.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:54.974 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:54 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:25:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:54 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/716997592' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:25:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:54 smithi165 ceph-mon[32893]: osdmap e160: 8 total, 8 up, 4 in 2024-09-21T08:25:55.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:55.096 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:55.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:55.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:55.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:55.220 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:55.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:55.221 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:55.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:55.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:55.350 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:55.350 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:55.451 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:55.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:55.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:55.477 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:55.525 INFO:tasks.rados.rados.0.smithi137.stdout:445: done (3 left) 2024-09-21T08:25:55.525 INFO:tasks.rados.rados.0.smithi137.stdout:447: done (2 left) 2024-09-21T08:25:55.526 INFO:tasks.rados.rados.0.smithi137.stdout:448: done (1 left) 2024-09-21T08:25:55.526 INFO:tasks.rados.rados.0.smithi137.stdout:449: done (0 left) 2024-09-21T08:25:55.526 INFO:tasks.rados.rados.0.smithi137.stdout:450: snap_remove snap 38 2024-09-21T08:25:55.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:55.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:55.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:55.603 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:55.607 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:55.609 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:55.609 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:55.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:55.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:55.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:55.729 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:55.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-mon[29634]: pgmap v146: 148 pgs: 4 active+undersized+remapped, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 133 active+clean; 137 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 941 KiB/s wr, 3 op/s; 17 MiB/s, 41 objects/s recovering 2024-09-21T08:25:55.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-mon[29634]: osdmap e161: 8 total, 8 up, 4 in 2024-09-21T08:25:55.735 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-mon[41044]: pgmap v146: 148 pgs: 4 active+undersized+remapped, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 133 active+clean; 137 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 941 KiB/s wr, 3 op/s; 17 MiB/s, 41 objects/s recovering 2024-09-21T08:25:55.736 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-mon[41044]: osdmap e161: 8 total, 8 up, 4 in 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 podman[92290]: 2024-09-21 08:25:55.4177864 +0000 UTC m=+2.462632926 container init 294634ea8fb9c44ba646a21e85f7ced5cf3733f362f7218cbc66d3e1642913fa (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 podman[92290]: 2024-09-21 08:25:55.423231966 +0000 UTC m=+2.468078474 container start 294634ea8fb9c44ba646a21e85f7ced5cf3733f362f7218cbc66d3e1642913fa (image=quay.io/prometheus/alertmanager:v0.25.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a, maintainer=The Prometheus Authors ) 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[92623]: ts=2024-09-21T08:25:55.435Z caller=main.go:240 level=info msg="Starting Alertmanager" version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)" 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[92623]: ts=2024-09-21T08:25:55.435Z caller=main.go:241 level=info build_context="(go=go1.19.4, user=root@abe866dd5717, date=20221222-14:51:36)" 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[92623]: ts=2024-09-21T08:25:55.436Z caller=cluster.go:185 level=info component=cluster msg="setting advertise address explicitly" addr=172.21.15.137 port=9094 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[92623]: ts=2024-09-21T08:25:55.437Z caller=cluster.go:681 level=info component=cluster msg="Waiting for gossip to settle..." interval=2s 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[92623]: ts=2024-09-21T08:25:55.460Z caller=coordinator.go:113 level=info component=configuration msg="Loading configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[92623]: ts=2024-09-21T08:25:55.461Z caller=coordinator.go:126 level=info component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/alertmanager.yml 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[92623]: ts=2024-09-21T08:25:55.463Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9093 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[92623]: ts=2024-09-21T08:25:55.463Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9093 2024-09-21T08:25:55.736 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 bash[92290]: 294634ea8fb9c44ba646a21e85f7ced5cf3733f362f7218cbc66d3e1642913fa 2024-09-21T08:25:55.737 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:55 smithi137 systemd[1]: Started Ceph alertmanager.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:25:55.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:55.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:55.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:55.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:55.953 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:55.973 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:55.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:55.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:55 smithi165 ceph-mon[32893]: pgmap v146: 148 pgs: 4 active+undersized+remapped, 6 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 133 active+clean; 137 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 941 KiB/s wr, 3 op/s; 17 MiB/s, 41 objects/s recovering 2024-09-21T08:25:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:55 smithi165 ceph-mon[32893]: osdmap e161: 8 total, 8 up, 4 in 2024-09-21T08:25:56.076 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:56.102 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:56.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:56.103 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:56.203 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:56.225 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:56.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:56.226 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:56.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:56.347 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:56.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:56.348 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:56.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:56.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:56.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:56.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:56.532 INFO:tasks.rados.rados.0.smithi137.stdout:450: done (0 left) 2024-09-21T08:25:56.532 INFO:tasks.rados.rados.0.smithi137.stdout:451: read oid 35 snap -1 2024-09-21T08:25:56.532 INFO:tasks.rados.rados.0.smithi137.stdout:451: expect (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:25:56.532 INFO:tasks.rados.rados.0.smithi137.stdout:452: read oid 23 snap -1 2024-09-21T08:25:56.532 INFO:tasks.rados.rados.0.smithi137.stdout:452: expect (ObjNum 150 snap 27 seq_num 150) 2024-09-21T08:25:56.532 INFO:tasks.rados.rados.0.smithi137.stdout:453: snap_remove snap 35 2024-09-21T08:25:56.572 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:56.593 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:56.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:56.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:56.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:56.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:56.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:56.716 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:25:56.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:56 smithi137 ceph-mon[29634]: osdmap e162: 8 total, 8 up, 4 in 2024-09-21T08:25:56.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:56 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:56.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:56 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:56.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:56 smithi137 ceph-mon[29634]: Reconfiguring node-exporter.a (unknown last config time)... 2024-09-21T08:25:56.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:56 smithi137 ceph-mon[29634]: Reconfiguring daemon node-exporter.a on smithi137 2024-09-21T08:25:56.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:56 smithi137 ceph-mon[41044]: osdmap e162: 8 total, 8 up, 4 in 2024-09-21T08:25:56.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:56 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:56.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:56 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:56.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:56 smithi137 ceph-mon[41044]: Reconfiguring node-exporter.a (unknown last config time)... 2024-09-21T08:25:56.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:56 smithi137 ceph-mon[41044]: Reconfiguring daemon node-exporter.a on smithi137 2024-09-21T08:25:56.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:56.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:56.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:56.843 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:56.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:56.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:56.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:56.968 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:56 smithi165 ceph-mon[32893]: osdmap e162: 8 total, 8 up, 4 in 2024-09-21T08:25:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:56 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:56 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:25:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:56 smithi165 ceph-mon[32893]: Reconfiguring node-exporter.a (unknown last config time)... 2024-09-21T08:25:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:56 smithi165 ceph-mon[32893]: Reconfiguring daemon node-exporter.a on smithi137 2024-09-21T08:25:57.069 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:56 smithi137 systemd[1]: Stopping Ceph node-exporter.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:25:57.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:57.072 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:25:57.072 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:25:57.072 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:25:57.072 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:25:57.072 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:25:57.092 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:57.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:57.092 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:57.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:25:57.193 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:57.216 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:57.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:57.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:57.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:57.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:57.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:57.341 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:57.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:57.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:57.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:57.464 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:57.529 INFO:tasks.rados.rados.0.smithi137.stdout:453: done (2 left) 2024-09-21T08:25:57.529 INFO:tasks.rados.rados.0.smithi137.stdout:454: write oid 6 current snap is 39 2024-09-21T08:25:57.529 INFO:tasks.rados.rados.0.smithi137.stdout:454: seq_num 191 ranges {410137=481384,1410296=761389,2728968=481169} 2024-09-21T08:25:57.534 INFO:tasks.rados.rados.0.smithi137.stdout:454: writing smithi13779221-6 from 410137 to 891521 tid 1 2024-09-21T08:25:57.541 INFO:tasks.rados.rados.0.smithi137.stdout:454: writing smithi13779221-6 from 1410296 to 2171685 tid 2 2024-09-21T08:25:57.548 INFO:tasks.rados.rados.0.smithi137.stdout:454: writing smithi13779221-6 from 2728968 to 3210137 tid 3 2024-09-21T08:25:57.548 INFO:tasks.rados.rados.0.smithi137.stdout:455: write (excl) oid 17 current snap is 39 2024-09-21T08:25:57.548 INFO:tasks.rados.rados.0.smithi137.stdout:455: seq_num 192 ranges {426944=556633,1468778=621400,2872744=354200} 2024-09-21T08:25:57.554 INFO:tasks.rados.rados.0.smithi137.stdout:455: writing smithi13779221-17 from 426944 to 983577 tid 1 2024-09-21T08:25:57.560 INFO:tasks.rados.rados.0.smithi137.stdout:455: writing smithi13779221-17 from 1468778 to 2090178 tid 2 2024-09-21T08:25:57.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:57.566 INFO:tasks.rados.rados.0.smithi137.stdout:455: writing smithi13779221-17 from 2872744 to 3226944 tid 3 2024-09-21T08:25:57.566 INFO:tasks.rados.rados.0.smithi137.stdout:456: read oid 46 snap -1 2024-09-21T08:25:57.566 INFO:tasks.rados.rados.0.smithi137.stdout:456: expect (ObjNum 130 snap 23 seq_num 130) 2024-09-21T08:25:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:457: read oid 49 snap -1 2024-09-21T08:25:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:457: expect (ObjNum 169 snap 34 seq_num 169) 2024-09-21T08:25:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:458: write oid 19 current snap is 39 2024-09-21T08:25:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:458: seq_num 193 ranges {443751=631882,1810907=765058,3067461=176290} 2024-09-21T08:25:57.573 INFO:tasks.rados.rados.0.smithi137.stdout:458: writing smithi13779221-19 from 443751 to 1075633 tid 1 2024-09-21T08:25:57.581 INFO:tasks.rados.rados.0.smithi137.stdout:458: writing smithi13779221-19 from 1810907 to 2575965 tid 2 2024-09-21T08:25:57.585 INFO:tasks.rados.rados.0.smithi137.stdout:458: writing smithi13779221-19 from 3067461 to 3243751 tid 3 2024-09-21T08:25:57.585 INFO:tasks.rados.rados.0.smithi137.stdout:459: rollback oid 33 current snap is 39 2024-09-21T08:25:57.585 INFO:tasks.rados.rados.0.smithi137.stdout:459: done (7 left) 2024-09-21T08:25:57.585 INFO:tasks.rados.rados.0.smithi137.stdout:460: snap_create 2024-09-21T08:25:57.585 INFO:tasks.rados.rados.0.smithi137.stdout:452: expect (ObjNum 150 snap 27 seq_num 150) 2024-09-21T08:25:57.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:57.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:57.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:57.689 INFO:tasks.rados.rados.0.smithi137.stdout:451: expect (ObjNum 62 snap 4 seq_num 62) 2024-09-21T08:25:57.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:25:57.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:57.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:57.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:25:57.767 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:25:57 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[92623]: ts=2024-09-21T08:25:57.438Z caller=cluster.go:706 level=info component=cluster msg="gossip not settled" polls=0 before=0 now=1 elapsed=2.000595634s 2024-09-21T08:25:57.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:57 smithi137 ceph-mon[29634]: pgmap v149: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 464 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 17 op/s 2024-09-21T08:25:57.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:57 smithi137 ceph-mon[29634]: osdmap e163: 8 total, 8 up, 4 in 2024-09-21T08:25:57.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:57 smithi137 ceph-mon[29634]: osdmap e164: 8 total, 8 up, 4 in 2024-09-21T08:25:57.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:57 smithi137 ceph-mon[41044]: pgmap v149: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 464 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 17 op/s 2024-09-21T08:25:57.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:57 smithi137 ceph-mon[41044]: osdmap e163: 8 total, 8 up, 4 in 2024-09-21T08:25:57.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:57 smithi137 ceph-mon[41044]: osdmap e164: 8 total, 8 up, 4 in 2024-09-21T08:25:57.817 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:454: finishing write tid 1 to smithi13779221-6 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:454: finishing write tid 2 to smithi13779221-6 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:454: finishing write tid 3 to smithi13779221-6 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:454: finishing write tid 4 to smithi13779221-6 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:454: finishing write tid 5 to smithi13779221-6 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:454: oid 6 updating version 0 to 39 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:454: oid 6 version 39 is already newer than 37 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:454: oid 6 version 39 is already newer than 38 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:454: oid 6 updating version 39 to 40 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 40 (ObjNum 191 snap 39 seq_num 191) dirty exists 2024-09-21T08:25:57.831 INFO:tasks.rados.rados.0.smithi137.stdout:454: left oid 6 (ObjNum 191 snap 39 seq_num 191) 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:455: finishing write tid 1 to smithi13779221-17 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:455: finishing write tid 2 to smithi13779221-17 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:455: finishing write tid 3 to smithi13779221-17 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:455: finishing write tid 4 to smithi13779221-17 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:455: finishing write tid 5 to smithi13779221-17 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:455: oid 17 updating version 0 to 132 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:455: oid 17 updating version 132 to 134 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:455: oid 17 version 134 is already newer than 133 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:455: oid 17 updating version 134 to 135 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 135 (ObjNum 192 snap 39 seq_num 192) dirty exists 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:455: left oid 17 (ObjNum 192 snap 39 seq_num 192) 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:458: finishing write tid 1 to smithi13779221-19 2024-09-21T08:25:57.832 INFO:tasks.rados.rados.0.smithi137.stdout:456: expect (ObjNum 130 snap 23 seq_num 130) 2024-09-21T08:25:57.839 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:57.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:57.840 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:25:57.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T08:25:57.941 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:57.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:57.965 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:57.976 INFO:tasks.rados.rados.0.smithi137.stdout:457: expect (ObjNum 169 snap 34 seq_num 169) 2024-09-21T08:25:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:57 smithi165 ceph-mon[32893]: pgmap v149: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 464 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 17 op/s 2024-09-21T08:25:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:57 smithi165 ceph-mon[32893]: osdmap e163: 8 total, 8 up, 4 in 2024-09-21T08:25:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:57 smithi165 ceph-mon[32893]: osdmap e164: 8 total, 8 up, 4 in 2024-09-21T08:25:58.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:25:58.071 INFO:tasks.rados.rados.0.smithi137.stdout:458: finishing write tid 2 to smithi13779221-19 2024-09-21T08:25:58.071 INFO:tasks.rados.rados.0.smithi137.stdout:458: finishing write tid 3 to smithi13779221-19 2024-09-21T08:25:58.071 INFO:tasks.rados.rados.0.smithi137.stdout:458: finishing write tid 4 to smithi13779221-19 2024-09-21T08:25:58.071 INFO:tasks.rados.rados.0.smithi137.stdout:458: finishing write tid 5 to smithi13779221-19 2024-09-21T08:25:58.071 INFO:tasks.rados.rados.0.smithi137.stdout:458: oid 19 updating version 0 to 53 2024-09-21T08:25:58.072 INFO:tasks.rados.rados.0.smithi137.stdout:458: oid 19 version 53 is already newer than 52 2024-09-21T08:25:58.072 INFO:tasks.rados.rados.0.smithi137.stdout:458: oid 19 updating version 53 to 54 2024-09-21T08:25:58.072 INFO:tasks.rados.rados.0.smithi137.stdout:458: oid 19 version 54 is already newer than 51 2024-09-21T08:25:58.072 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 54 (ObjNum 193 snap 39 seq_num 193) dirty exists 2024-09-21T08:25:58.072 INFO:tasks.rados.rados.0.smithi137.stdout:458: left oid 19 (ObjNum 193 snap 39 seq_num 193) 2024-09-21T08:25:58.072 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:25:58.072 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:25:58.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:58.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:58.092 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:25:58.092 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:25:58.092 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:25:58.107 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:25:58.107 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:25:58.123 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:25:58.123 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:25:58.140 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:25:58.140 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:25:58.157 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:25:58.157 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:25:58.174 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:25:58.174 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:25:58.192 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:25:58.192 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:25:58.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:58.216 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:58.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:58.217 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:58.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:58.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:58.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:58.340 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:58.441 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:58.452 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:25:58.453 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:25:58.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:58.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:58.465 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:58.548 INFO:tasks.rados.rados.0.smithi137.stdout:451: done (7 left) 2024-09-21T08:25:58.548 INFO:tasks.rados.rados.0.smithi137.stdout:452: done (6 left) 2024-09-21T08:25:58.548 INFO:tasks.rados.rados.0.smithi137.stdout:454: done (5 left) 2024-09-21T08:25:58.549 INFO:tasks.rados.rados.0.smithi137.stdout:455: done (4 left) 2024-09-21T08:25:58.549 INFO:tasks.rados.rados.0.smithi137.stdout:456: done (3 left) 2024-09-21T08:25:58.549 INFO:tasks.rados.rados.0.smithi137.stdout:457: done (2 left) 2024-09-21T08:25:58.549 INFO:tasks.rados.rados.0.smithi137.stdout:458: done (1 left) 2024-09-21T08:25:58.549 INFO:tasks.rados.rados.0.smithi137.stdout:460: done (0 left) 2024-09-21T08:25:58.549 INFO:tasks.rados.rados.0.smithi137.stdout:461: read oid 23 snap 39 2024-09-21T08:25:58.549 INFO:tasks.rados.rados.0.smithi137.stdout:461: expect (ObjNum 150 snap 27 seq_num 150) 2024-09-21T08:25:58.549 INFO:tasks.rados.rados.0.smithi137.stdout:462: write oid 18 current snap is 40 2024-09-21T08:25:58.549 INFO:tasks.rados.rados.0.smithi137.stdout:462: seq_num 194 ranges {460558=707131,1869389=508716,3094884=165674} 2024-09-21T08:25:58.554 INFO:tasks.rados.rados.0.smithi137.stdout:462: writing smithi13779221-18 from 460558 to 1167689 tid 1 2024-09-21T08:25:58.561 INFO:tasks.rados.rados.0.smithi137.stdout:462: writing smithi13779221-18 from 1869389 to 2378105 tid 2 2024-09-21T08:25:58.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:58.567 INFO:tasks.rados.rados.0.smithi137.stdout:462: writing smithi13779221-18 from 3094884 to 3260558 tid 3 2024-09-21T08:25:58.567 INFO:tasks.rados.rados.0.smithi137.stdout:463: rollback oid 34 current snap is 40 2024-09-21T08:25:58.567 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 34 to 39 2024-09-21T08:25:58.567 INFO:tasks.rados.rados.0.smithi137.stdout:461: expect (ObjNum 150 snap 27 seq_num 150) 2024-09-21T08:25:58.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:58.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:58.592 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:462: finishing write tid 1 to smithi13779221-18 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:462: finishing write tid 2 to smithi13779221-18 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:462: finishing write tid 3 to smithi13779221-18 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:462: finishing write tid 4 to smithi13779221-18 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:462: finishing write tid 5 to smithi13779221-18 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:462: oid 18 updating version 0 to 94 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:462: oid 18 version 94 is already newer than 93 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:462: oid 18 version 94 is already newer than 92 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:462: oid 18 updating version 94 to 95 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 95 (ObjNum 194 snap 40 seq_num 194) dirty exists 2024-09-21T08:25:58.671 INFO:tasks.rados.rados.0.smithi137.stdout:462: left oid 18 (ObjNum 194 snap 40 seq_num 194) 2024-09-21T08:25:58.672 INFO:tasks.rados.rados.0.smithi137.stdout:463: finishing rollback tid 0 to smithi13779221-34 2024-09-21T08:25:58.672 INFO:tasks.rados.rados.0.smithi137.stdout:461: done (2 left) 2024-09-21T08:25:58.672 INFO:tasks.rados.rados.0.smithi137.stdout:462: done (1 left) 2024-09-21T08:25:58.672 INFO:tasks.rados.rados.0.smithi137.stdout:464: write oid 20 current snap is 40 2024-09-21T08:25:58.672 INFO:tasks.rados.rados.0.smithi137.stdout:463: finishing rollback tid 1 to smithi13779221-34 2024-09-21T08:25:58.672 INFO:tasks.rados.rados.0.smithi137.stdout:464: seq_num 195 ranges {477365=782380,1927871=768727,3122307=155058} 2024-09-21T08:25:58.677 INFO:tasks.rados.rados.0.smithi137.stdout:464: writing smithi13779221-20 from 477365 to 1259745 tid 1 2024-09-21T08:25:58.686 INFO:tasks.rados.rados.0.smithi137.stdout:464: writing smithi13779221-20 from 1927871 to 2696598 tid 2 2024-09-21T08:25:58.689 INFO:tasks.rados.rados.0.smithi137.stdout:464: writing smithi13779221-20 from 3122307 to 3277365 tid 3 2024-09-21T08:25:58.689 INFO:tasks.rados.rados.0.smithi137.stdout:465: read oid 2 snap 39 2024-09-21T08:25:58.689 INFO:tasks.rados.rados.0.smithi137.stdout:465: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-21T08:25:58.689 INFO:tasks.rados.rados.0.smithi137.stdout:466: write (excl) oid 28 current snap is 40 2024-09-21T08:25:58.690 INFO:tasks.rados.rados.0.smithi137.stdout:466: seq_num 196 ranges {494172=457629,1586353=512385,2749730=544442} 2024-09-21T08:25:58.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:58.696 INFO:tasks.rados.rados.0.smithi137.stdout:466: writing smithi13779221-28 from 494172 to 951801 tid 1 2024-09-21T08:25:58.703 INFO:tasks.rados.rados.0.smithi137.stdout:466: writing smithi13779221-28 from 1586353 to 2098738 tid 2 2024-09-21T08:25:58.711 INFO:tasks.rados.rados.0.smithi137.stdout:466: writing smithi13779221-28 from 2749730 to 3294172 tid 3 2024-09-21T08:25:58.711 INFO:tasks.rados.rados.0.smithi137.stdout:467: read oid 39 snap 39 2024-09-21T08:25:58.711 INFO:tasks.rados.rados.0.smithi137.stdout:467: expect (ObjNum 69 snap 7 seq_num 69) 2024-09-21T08:25:58.711 INFO:tasks.rados.rados.0.smithi137.stdout:468: write (excl) oid 35 current snap is 40 2024-09-21T08:25:58.711 INFO:tasks.rados.rados.0.smithi137.stdout:463: finishing rollback tid 2 to smithi13779221-34 2024-09-21T08:25:58.711 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 231 (ObjNum 143 snap 26 seq_num 143) dirty exists 2024-09-21T08:25:58.711 INFO:tasks.rados.rados.0.smithi137.stdout:464: finishing write tid 1 to smithi13779221-20 2024-09-21T08:25:58.711 INFO:tasks.rados.rados.0.smithi137.stdout:468: seq_num 197 ranges {510979=532878,1528482=772396,3060800=250179} 2024-09-21T08:25:58.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:58.717 INFO:tasks.rados.rados.0.smithi137.stdout:468: writing smithi13779221-35 from 510979 to 1043857 tid 1 2024-09-21T08:25:58.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:58.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:58.725 INFO:tasks.rados.rados.0.smithi137.stdout:468: writing smithi13779221-35 from 1528482 to 2300878 tid 2 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:468: writing smithi13779221-35 from 3060800 to 3310979 tid 3 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:463: done (5 left) 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:469: delete oid 24 current snap is 40 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:464: finishing write tid 2 to smithi13779221-20 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:464: finishing write tid 3 to smithi13779221-20 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:464: finishing write tid 4 to smithi13779221-20 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:464: finishing write tid 5 to smithi13779221-20 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:464: oid 20 updating version 0 to 235 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:464: oid 20 version 235 is already newer than 233 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:464: oid 20 version 235 is already newer than 234 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:464: oid 20 version 235 is already newer than 232 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 235 (ObjNum 195 snap 40 seq_num 195) dirty exists 2024-09-21T08:25:58.731 INFO:tasks.rados.rados.0.smithi137.stdout:464: left oid 20 (ObjNum 195 snap 40 seq_num 195) 2024-09-21T08:25:58.732 INFO:tasks.rados.rados.0.smithi137.stdout:465: expect (ObjNum 88 snap 11 seq_num 88) 2024-09-21T08:25:58.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:58 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:25:58.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:58 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:25:58.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:58 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1781893548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:58.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:58 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:25:58.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:58 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:25:58.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:58 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1781893548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:58.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:25:58.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:58.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:58.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:25:58.849 INFO:tasks.rados.rados.0.smithi137.stdout:466: finishing write tid 1 to smithi13779221-28 2024-09-21T08:25:58.850 INFO:tasks.rados.rados.0.smithi137.stdout:466: finishing write tid 2 to smithi13779221-28 2024-09-21T08:25:58.850 INFO:tasks.rados.rados.0.smithi137.stdout:464: done (5 left) 2024-09-21T08:25:58.850 INFO:tasks.rados.rados.0.smithi137.stdout:465: done (4 left) 2024-09-21T08:25:58.850 INFO:tasks.rados.rados.0.smithi137.stdout:469: done (3 left) 2024-09-21T08:25:58.850 INFO:tasks.rados.rados.0.smithi137.stdout:470: write oid 48 current snap is 40 2024-09-21T08:25:58.850 INFO:tasks.rados.rados.0.smithi137.stdout:466: finishing write tid 3 to smithi13779221-28 2024-09-21T08:25:58.850 INFO:tasks.rados.rados.0.smithi137.stdout:466: finishing write tid 4 to smithi13779221-28 2024-09-21T08:25:58.850 INFO:tasks.rados.rados.0.smithi137.stdout:470: seq_num 199 ranges {544593=799729,1761799=776065,2948352=396241} 2024-09-21T08:25:58.856 INFO:tasks.rados.rados.0.smithi137.stdout:470: writing smithi13779221-48 from 544593 to 1344322 tid 1 2024-09-21T08:25:58.863 INFO:tasks.rados.rados.0.smithi137.stdout:470: writing smithi13779221-48 from 1761799 to 2537864 tid 2 2024-09-21T08:25:58.867 INFO:tasks.rados.rados.0.smithi137.stdout:470: writing smithi13779221-48 from 2948352 to 3344593 tid 3 2024-09-21T08:25:58.867 INFO:tasks.rados.rados.0.smithi137.stdout:471: copy_from oid 25 from oid 14 current snap is 40 2024-09-21T08:25:58.868 INFO:tasks.rados.rados.0.smithi137.stdout:467: expect (ObjNum 69 snap 7 seq_num 69) 2024-09-21T08:25:58.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:58.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:58.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:58.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:58.969 INFO:tasks.rados.rados.0.smithi137.stdout:466: finishing write tid 5 to smithi13779221-28 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:466: oid 28 updating version 0 to 78 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:466: oid 28 version 78 is already newer than 77 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:466: oid 28 version 78 is already newer than 76 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:466: oid 28 version 78 is already newer than 75 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 78 (ObjNum 196 snap 40 seq_num 196) dirty exists 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:466: left oid 28 (ObjNum 196 snap 40 seq_num 196) 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:468: finishing write tid 1 to smithi13779221-35 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:468: finishing write tid 2 to smithi13779221-35 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:466: done (4 left) 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:467: done (3 left) 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:472: rollback oid 33 current snap is 40 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 33 to 39 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:468: finishing write tid 3 to smithi13779221-35 2024-09-21T08:25:58.970 INFO:tasks.rados.rados.0.smithi137.stdout:468: finishing write tid 4 to smithi13779221-35 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:468: finishing write tid 5 to smithi13779221-35 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:468: oid 35 updating version 0 to 137 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:468: oid 35 updating version 137 to 138 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:468: oid 35 updating version 138 to 139 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:468: oid 35 updating version 139 to 140 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 140 (ObjNum 197 snap 40 seq_num 197) dirty exists 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:468: left oid 35 (ObjNum 197 snap 40 seq_num 197) 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:470: finishing write tid 1 to smithi13779221-48 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:470: finishing write tid 2 to smithi13779221-48 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:471: finishing copy_from to smithi13779221-25 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:471: got expected ENOENT (src dne) 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:471: finishing copy_from racing read to smithi13779221-25 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:470: finishing write tid 3 to smithi13779221-48 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:470: finishing write tid 4 to smithi13779221-48 2024-09-21T08:25:58.971 INFO:tasks.rados.rados.0.smithi137.stdout:470: finishing write tid 5 to smithi13779221-48 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:470: oid 48 updating version 0 to 158 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:470: oid 48 updating version 158 to 159 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:470: oid 48 updating version 159 to 160 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:470: oid 48 version 160 is already newer than 157 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 160 (ObjNum 199 snap 40 seq_num 199) dirty exists 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:470: left oid 48 (ObjNum 199 snap 40 seq_num 199) 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:468: done (3 left) 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:470: done (2 left) 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:471: done (1 left) 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:473: read oid 49 snap -1 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:473: expect (ObjNum 169 snap 34 seq_num 169) 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:474: write (excl) oid 2 current snap is 40 2024-09-21T08:25:58.972 INFO:tasks.rados.rados.0.smithi137.stdout:474: seq_num 200 ranges {561400=474978,1820281=519723,2859422=457579} 2024-09-21T08:25:58.975 INFO:tasks.rados.rados.0.smithi137.stdout:474: writing smithi13779221-2 from 561400 to 1036378 tid 1 2024-09-21T08:25:58.983 INFO:tasks.rados.rados.0.smithi137.stdout:474: writing smithi13779221-2 from 1820281 to 2340004 tid 2 2024-09-21T08:25:58.988 INFO:tasks.rados.rados.0.smithi137.stdout:474: writing smithi13779221-2 from 2859422 to 3317001 tid 3 2024-09-21T08:25:58.989 INFO:tasks.rados.rados.0.smithi137.stdout:475: write oid 12 current snap is 40 2024-09-21T08:25:58.989 INFO:tasks.rados.rados.0.smithi137.stdout:472: finishing rollback tid 0 to smithi13779221-33 2024-09-21T08:25:58.989 INFO:tasks.rados.rados.0.smithi137.stdout:475: seq_num 201 ranges {578207=550227,1762410=779734,3286845=91362} 2024-09-21T08:25:58.995 INFO:tasks.rados.rados.0.smithi137.stdout:475: writing smithi13779221-12 from 578207 to 1128434 tid 1 2024-09-21T08:25:59.002 INFO:tasks.rados.rados.0.smithi137.stdout:475: writing smithi13779221-12 from 1762410 to 2542144 tid 2 2024-09-21T08:25:59.007 INFO:tasks.rados.rados.0.smithi137.stdout:475: writing smithi13779221-12 from 3286845 to 3378207 tid 3 2024-09-21T08:25:59.007 INFO:tasks.rados.rados.0.smithi137.stdout:476: rollback oid 48 current snap is 40 2024-09-21T08:25:59.008 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 48 to 39 2024-09-21T08:25:59.008 INFO:tasks.rados.rados.0.smithi137.stdout:472: finishing rollback tid 1 to smithi13779221-33 2024-09-21T08:25:59.008 INFO:tasks.rados.rados.0.smithi137.stdout:472: finishing rollback tid 2 to smithi13779221-33 2024-09-21T08:25:59.008 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 44 (ObjNum 164 snap 33 seq_num 164) dirty exists 2024-09-21T08:25:59.008 INFO:tasks.rados.rados.0.smithi137.stdout:473: expect (ObjNum 169 snap 34 seq_num 169) 2024-09-21T08:25:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:58 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:25:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:58 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:25:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:58 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1781893548' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:25:59.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:59.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:59.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:59.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:59.102 INFO:tasks.rados.rados.0.smithi137.stdout:474: finishing write tid 1 to smithi13779221-2 2024-09-21T08:25:59.102 INFO:tasks.rados.rados.0.smithi137.stdout:474: finishing write tid 2 to smithi13779221-2 2024-09-21T08:25:59.102 INFO:tasks.rados.rados.0.smithi137.stdout:472: done (4 left) 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:473: done (3 left) 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:477: write oid 38 current snap is 40 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:474: finishing write tid 3 to smithi13779221-2 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:474: finishing write tid 4 to smithi13779221-2 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:474: finishing write tid 5 to smithi13779221-2 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:474: oid 2 updating version 0 to 99 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:474: oid 2 version 99 is already newer than 98 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:474: oid 2 version 99 is already newer than 97 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:474: oid 2 version 99 is already newer than 96 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 99 (ObjNum 200 snap 40 seq_num 200) dirty exists 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:474: left oid 2 (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:475: finishing write tid 1 to smithi13779221-12 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:475: finishing write tid 2 to smithi13779221-12 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:475: finishing write tid 3 to smithi13779221-12 2024-09-21T08:25:59.103 INFO:tasks.rados.rados.0.smithi137.stdout:475: finishing write tid 4 to smithi13779221-12 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:475: finishing write tid 5 to smithi13779221-12 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:475: oid 12 updating version 0 to 161 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:475: oid 12 updating version 161 to 164 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:475: oid 12 version 164 is already newer than 163 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:475: oid 12 version 164 is already newer than 162 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 164 (ObjNum 201 snap 40 seq_num 201) dirty exists 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:475: left oid 12 (ObjNum 201 snap 40 seq_num 201) 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:476: finishing rollback tid 0 to smithi13779221-48 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:476: finishing rollback tid 1 to smithi13779221-48 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:476: finishing rollback tid 2 to smithi13779221-48 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 167 (ObjNum 189 snap 38 seq_num 189) dirty exists 2024-09-21T08:25:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:477: seq_num 202 ranges {595014=625476,1820892=523392,2797915=596476} 2024-09-21T08:25:59.109 INFO:tasks.rados.rados.0.smithi137.stdout:477: writing smithi13779221-38 from 595014 to 1220490 tid 1 2024-09-21T08:25:59.115 INFO:tasks.rados.rados.0.smithi137.stdout:477: writing smithi13779221-38 from 1820892 to 2344284 tid 2 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:477: writing smithi13779221-38 from 2797915 to 3394391 tid 3 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:474: done (3 left) 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:475: done (2 left) 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:476: done (1 left) 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:478: read oid 30 snap -1 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:478: expect deleted 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:479: read oid 7 snap 39 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:479: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:480: write (excl) oid 34 current snap is 40 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:477: finishing write tid 1 to smithi13779221-38 2024-09-21T08:25:59.123 INFO:tasks.rados.rados.0.smithi137.stdout:480: seq_num 203 ranges {611821=700725,1879374=783403,3341691=70130} 2024-09-21T08:25:59.130 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":152,"stamp":"2024-09-21T08:25:58.058215+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":142305731,"num_objects":259,"num_object_clones":18,"num_object_copies":777,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":259,"num_whiteouts":3,"num_read":1927,"num_read_kb":743044,"num_write":1388,"num_write_kb":292766,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":142984584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1995,"ondisk_log_size":1995,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":474640,"kb_used_data":337384,"kb_used_omap":6,"kb_used_meta":137145,"kb_avail":374505968,"statfs":{"total":383980142592,"available":383494111232,"internally_reserved":0,"allocated":345481216,"data_stored":381127466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6375,"internal_metadata":140437273},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-1391684,"num_objects":-9,"num_object_clones":-8,"num_object_copies":-27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-9,"num_whiteouts":-1,"num_read":31,"num_read_kb":11661,"num_write":40,"num_write_kb":18381,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001075"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":211,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694286+0000","last_change":"2024-09-21T08:25:38.378517+0000","last_active":"2024-09-21T08:25:55.694286+0000","last_peered":"2024-09-21T08:25:55.694286+0000","last_clean":"2024-09-21T08:25:55.694286+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:25:55.694286+0000","last_undegraded":"2024-09-21T08:25:55.694286+0000","last_fullsized":"2024-09-21T08:25:55.694286+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:52.783162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":110,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693344+0000","last_change":"2024-09-21T08:25:48.982638+0000","last_active":"2024-09-21T08:25:55.693344+0000","last_peered":"2024-09-21T08:25:55.693344+0000","last_clean":"2024-09-21T08:25:55.693344+0000","last_became_active":"2024-09-21T08:25:48.910812+0000","last_became_peered":"2024-09-21T08:25:48.910812+0000","last_unstale":"2024-09-21T08:25:55.693344+0000","last_undegraded":"2024-09-21T08:25:55.693344+0000","last_fullsized":"2024-09-21T08:25:55.693344+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:24:50.572709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.572709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:10.553204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":26,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693592+0000","last_change":"2024-09-21T08:25:48.912078+0000","last_active":"2024-09-21T08:25:55.693592+0000","last_peered":"2024-09-21T08:25:55.693592+0000","last_clean":"2024-09-21T08:25:55.693592+0000","last_became_active":"2024-09-21T08:25:48.911674+0000","last_became_peered":"2024-09-21T08:25:48.911674+0000","last_unstale":"2024-09-21T08:25:55.693592+0000","last_undegraded":"2024-09-21T08:25:55.693592+0000","last_fullsized":"2024-09-21T08:25:55.693592+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":173,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693776+0000","last_change":"2024-09-21T08:25:48.914509+0000","last_active":"2024-09-21T08:25:55.693776+0000","last_peered":"2024-09-21T08:25:55.693776+0000","last_clean":"2024-09-21T08:25:55.693776+0000","last_became_active":"2024-09-21T08:25:48.911853+0000","last_became_peered":"2024-09-21T08:25:48.911853+0000","last_unstale":"2024-09-21T08:25:55.693776+0000","last_undegraded":"2024-09-21T08:25:55.693776+0000","last_fullsized":"2024-09-21T08:25:55.693776+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":157,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617739+0000","last_change":"2024-09-21T08:25:48.900959+0000","last_active":"2024-09-21T08:25:55.617739+0000","last_peered":"2024-09-21T08:25:55.617739+0000","last_clean":"2024-09-21T08:25:55.617739+0000","last_became_active":"2024-09-21T08:25:48.900647+0000","last_became_peered":"2024-09-21T08:25:48.900647+0000","last_unstale":"2024-09-21T08:25:55.617739+0000","last_undegraded":"2024-09-21T08:25:55.617739+0000","last_fullsized":"2024-09-21T08:25:55.617739+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759445+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":147,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.696170+0000","last_change":"2024-09-21T08:25:48.884959+0000","last_active":"2024-09-21T08:25:55.696170+0000","last_peered":"2024-09-21T08:25:55.696170+0000","last_clean":"2024-09-21T08:25:55.696170+0000","last_became_active":"2024-09-21T08:25:48.884673+0000","last_became_peered":"2024-09-21T08:25:48.884673+0000","last_unstale":"2024-09-21T08:25:55.696170+0000","last_undegraded":"2024-09-21T08:25:55.696170+0000","last_fullsized":"2024-09-21T08:25:55.696170+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:19.906906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:19.906906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:22.705067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":214,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695631+0000","last_change":"2024-09-21T08:25:20.942089+0000","last_active":"2024-09-21T08:25:55.695631+0000","last_peered":"2024-09-21T08:25:55.695631+0000","last_clean":"2024-09-21T08:25:55.695631+0000","last_became_active":"2024-09-21T08:24:48.594855+0000","last_became_peered":"2024-09-21T08:24:48.594855+0000","last_unstale":"2024-09-21T08:25:55.695631+0000","last_undegraded":"2024-09-21T08:25:55.695631+0000","last_fullsized":"2024-09-21T08:25:55.695631+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:25:20.941990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:20.941990+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:29.954407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":31,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.692806+0000","last_change":"2024-09-21T08:25:48.912528+0000","last_active":"2024-09-21T08:25:55.692806+0000","last_peered":"2024-09-21T08:25:55.692806+0000","last_clean":"2024-09-21T08:25:55.692806+0000","last_became_active":"2024-09-21T08:25:48.911583+0000","last_became_peered":"2024-09-21T08:25:48.911583+0000","last_unstale":"2024-09-21T08:25:55.692806+0000","last_undegraded":"2024-09-21T08:25:55.692806+0000","last_fullsized":"2024-09-21T08:25:55.692806+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.640831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.640831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":221,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694274+0000","last_change":"2024-09-21T08:25:48.907786+0000","last_active":"2024-09-21T08:25:55.694274+0000","last_peered":"2024-09-21T08:25:55.694274+0000","last_clean":"2024-09-21T08:25:55.694274+0000","last_became_active":"2024-09-21T08:25:48.906922+0000","last_became_peered":"2024-09-21T08:25:48.906922+0000","last_unstale":"2024-09-21T08:25:55.694274+0000","last_undegraded":"2024-09-21T08:25:55.694274+0000","last_fullsized":"2024-09-21T08:25:55.694274+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":206,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693764+0000","last_change":"2024-09-21T08:24:56.523774+0000","last_active":"2024-09-21T08:25:55.693764+0000","last_peered":"2024-09-21T08:25:55.693764+0000","last_clean":"2024-09-21T08:25:55.693764+0000","last_became_active":"2024-09-21T08:24:56.500171+0000","last_became_peered":"2024-09-21T08:24:56.500171+0000","last_unstale":"2024-09-21T08:25:55.693764+0000","last_undegraded":"2024-09-21T08:25:55.693764+0000","last_fullsized":"2024-09-21T08:25:55.693764+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":203,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694695+0000","last_change":"2024-09-21T08:25:22.581281+0000","last_active":"2024-09-21T08:25:55.694695+0000","last_peered":"2024-09-21T08:25:55.694695+0000","last_clean":"2024-09-21T08:25:55.694695+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:25:55.694695+0000","last_undegraded":"2024-09-21T08:25:55.694695+0000","last_fullsized":"2024-09-21T08:25:55.694695+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:33.300785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":196,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695082+0000","last_change":"2024-09-21T08:25:48.885985+0000","last_active":"2024-09-21T08:25:55.695082+0000","last_peered":"2024-09-21T08:25:55.695082+0000","last_clean":"2024-09-21T08:25:55.695082+0000","last_became_active":"2024-09-21T08:25:48.885136+0000","last_became_peered":"2024-09-21T08:25:48.885136+0000","last_unstale":"2024-09-21T08:25:55.695082+0000","last_undegraded":"2024-09-21T08:25:55.695082+0000","last_fullsized":"2024-09-21T08:25:55.695082+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352649+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":229,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694168+0000","last_change":"2024-09-21T08:25:48.912437+0000","last_active":"2024-09-21T08:25:55.694168+0000","last_peered":"2024-09-21T08:25:55.694168+0000","last_clean":"2024-09-21T08:25:55.694168+0000","last_became_active":"2024-09-21T08:25:48.912043+0000","last_became_peered":"2024-09-21T08:25:48.912043+0000","last_unstale":"2024-09-21T08:25:55.694168+0000","last_undegraded":"2024-09-21T08:25:55.694168+0000","last_fullsized":"2024-09-21T08:25:55.694168+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:16.938344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":63,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695320+0000","last_change":"2024-09-21T08:25:48.937429+0000","last_active":"2024-09-21T08:25:55.695320+0000","last_peered":"2024-09-21T08:25:55.695320+0000","last_clean":"2024-09-21T08:25:55.695320+0000","last_became_active":"2024-09-21T08:25:48.907461+0000","last_became_peered":"2024-09-21T08:25:48.907461+0000","last_unstale":"2024-09-21T08:25:55.695320+0000","last_undegraded":"2024-09-21T08:25:55.695320+0000","last_fullsized":"2024-09-21T08:25:55.695320+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:24:32.569846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:32.569846+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.932644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":205,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617740+0000","last_change":"2024-09-21T08:25:48.886137+0000","last_active":"2024-09-21T08:25:55.617740+0000","last_peered":"2024-09-21T08:25:55.617740+0000","last_clean":"2024-09-21T08:25:55.617740+0000","last_became_active":"2024-09-21T08:25:48.885803+0000","last_became_peered":"2024-09-21T08:25:48.885803+0000","last_unstale":"2024-09-21T08:25:55.617740+0000","last_undegraded":"2024-09-21T08:25:55.617740+0000","last_fullsized":"2024-09-21T08:25:55.617740+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":212,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693595+0000","last_change":"2024-09-21T08:25:48.966469+0000","last_active":"2024-09-21T08:25:55.693595+0000","last_peered":"2024-09-21T08:25:55.693595+0000","last_clean":"2024-09-21T08:25:55.693595+0000","last_became_active":"2024-09-21T08:25:48.911981+0000","last_became_peered":"2024-09-21T08:25:48.911981+0000","last_unstale":"2024-09-21T08:25:55.693595+0000","last_undegraded":"2024-09-21T08:25:55.693595+0000","last_fullsized":"2024-09-21T08:25:55.693595+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":168,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694795+0000","last_change":"2024-09-21T08:25:39.855515+0000","last_active":"2024-09-21T08:25:55.694795+0000","last_peered":"2024-09-21T08:25:55.694795+0000","last_clean":"2024-09-21T08:25:55.694795+0000","last_became_active":"2024-09-21T08:24:56.499884+0000","last_became_peered":"2024-09-21T08:24:56.499884+0000","last_unstale":"2024-09-21T08:25:55.694795+0000","last_undegraded":"2024-09-21T08:25:55.694795+0000","last_fullsized":"2024-09-21T08:25:55.694795+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.855448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.855448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:04.367456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":222,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693250+0000","last_change":"2024-09-21T08:25:48.891123+0000","last_active":"2024-09-21T08:25:55.693250+0000","last_peered":"2024-09-21T08:25:55.693250+0000","last_clean":"2024-09-21T08:25:55.693250+0000","last_became_active":"2024-09-21T08:25:48.890600+0000","last_became_peered":"2024-09-21T08:25:48.890600+0000","last_unstale":"2024-09-21T08:25:55.693250+0000","last_undegraded":"2024-09-21T08:25:55.693250+0000","last_fullsized":"2024-09-21T08:25:55.693250+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:17.893561+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":275,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695975+0000","last_change":"2024-09-21T08:25:48.949361+0000","last_active":"2024-09-21T08:25:55.695975+0000","last_peered":"2024-09-21T08:25:55.695975+0000","last_clean":"2024-09-21T08:25:55.695975+0000","last_became_active":"2024-09-21T08:25:48.909222+0000","last_became_peered":"2024-09-21T08:25:48.909222+0000","last_unstale":"2024-09-21T08:25:55.695975+0000","last_undegraded":"2024-09-21T08:25:55.695975+0000","last_fullsized":"2024-09-21T08:25:55.695975+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:25:24.945807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:24.945807+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:43.015505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":63,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695105+0000","last_change":"2024-09-21T08:25:48.905281+0000","last_active":"2024-09-21T08:25:55.695105+0000","last_peered":"2024-09-21T08:25:55.695105+0000","last_clean":"2024-09-21T08:25:55.695105+0000","last_became_active":"2024-09-21T08:25:48.904956+0000","last_became_peered":"2024-09-21T08:25:48.904956+0000","last_unstale":"2024-09-21T08:25:55.695105+0000","last_undegraded":"2024-09-21T08:25:55.695105+0000","last_fullsized":"2024-09-21T08:25:55.695105+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":138,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694967+0000","last_change":"2024-09-21T08:25:48.914865+0000","last_active":"2024-09-21T08:25:55.694967+0000","last_peered":"2024-09-21T08:25:55.694967+0000","last_clean":"2024-09-21T08:25:55.694967+0000","last_became_active":"2024-09-21T08:25:48.912721+0000","last_became_peered":"2024-09-21T08:25:48.912721+0000","last_unstale":"2024-09-21T08:25:55.694967+0000","last_undegraded":"2024-09-21T08:25:55.694967+0000","last_fullsized":"2024-09-21T08:25:55.694967+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":28,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695989+0000","last_change":"2024-09-21T08:25:48.900151+0000","last_active":"2024-09-21T08:25:55.695989+0000","last_peered":"2024-09-21T08:25:55.695989+0000","last_clean":"2024-09-21T08:25:55.695989+0000","last_became_active":"2024-09-21T08:25:48.899796+0000","last_became_peered":"2024-09-21T08:25:48.899796+0000","last_unstale":"2024-09-21T08:25:55.695989+0000","last_undegraded":"2024-09-21T08:25:55.695989+0000","last_fullsized":"2024-09-21T08:25:55.695989+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.932560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":82,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694810+0000","last_change":"2024-09-21T08:25:48.964877+0000","last_active":"2024-09-21T08:25:55.694810+0000","last_peered":"2024-09-21T08:25:55.694810+0000","last_clean":"2024-09-21T08:25:55.694810+0000","last_became_active":"2024-09-21T08:25:48.913458+0000","last_became_peered":"2024-09-21T08:25:48.913458+0000","last_unstale":"2024-09-21T08:25:55.694810+0000","last_undegraded":"2024-09-21T08:25:55.694810+0000","last_fullsized":"2024-09-21T08:25:55.694810+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:21.424414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.424414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:21.922543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":162,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.696213+0000","last_change":"2024-09-21T08:25:48.892022+0000","last_active":"2024-09-21T08:25:55.696213+0000","last_peered":"2024-09-21T08:25:55.696213+0000","last_clean":"2024-09-21T08:25:55.696213+0000","last_became_active":"2024-09-21T08:25:48.891706+0000","last_became_peered":"2024-09-21T08:25:48.891706+0000","last_unstale":"2024-09-21T08:25:55.696213+0000","last_undegraded":"2024-09-21T08:25:55.696213+0000","last_fullsized":"2024-09-21T08:25:55.696213+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.797084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:22.450359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":190,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618720+0000","last_change":"2024-09-21T08:23:36.584003+0000","last_active":"2024-09-21T08:25:55.618720+0000","last_peered":"2024-09-21T08:25:55.618720+0000","last_clean":"2024-09-21T08:25:55.618720+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:25:55.618720+0000","last_undegraded":"2024-09-21T08:25:55.618720+0000","last_fullsized":"2024-09-21T08:25:55.618720+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":106,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618165+0000","last_change":"2024-09-21T08:25:48.971328+0000","last_active":"2024-09-21T08:25:55.618165+0000","last_peered":"2024-09-21T08:25:55.618165+0000","last_clean":"2024-09-21T08:25:55.618165+0000","last_became_active":"2024-09-21T08:25:48.896963+0000","last_became_peered":"2024-09-21T08:25:48.896963+0000","last_unstale":"2024-09-21T08:25:55.618165+0000","last_undegraded":"2024-09-21T08:25:55.618165+0000","last_fullsized":"2024-09-21T08:25:55.618165+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":250,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694281+0000","last_change":"2024-09-21T08:25:48.989640+0000","last_active":"2024-09-21T08:25:55.694281+0000","last_peered":"2024-09-21T08:25:55.694281+0000","last_clean":"2024-09-21T08:25:55.694281+0000","last_became_active":"2024-09-21T08:25:48.908487+0000","last_became_peered":"2024-09-21T08:25:48.908487+0000","last_unstale":"2024-09-21T08:25:55.694281+0000","last_undegraded":"2024-09-21T08:25:55.694281+0000","last_fullsized":"2024-09-21T08:25:55.694281+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"156'12","reported_seq":268,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619498+0000","last_change":"2024-09-21T08:25:48.961434+0000","last_active":"2024-09-21T08:25:55.619498+0000","last_peered":"2024-09-21T08:25:55.619498+0000","last_clean":"2024-09-21T08:25:55.619498+0000","last_became_active":"2024-09-21T08:25:48.896649+0000","last_became_peered":"2024-09-21T08:25:48.896649+0000","last_unstale":"2024-09-21T08:25:55.619498+0000","last_undegraded":"2024-09-21T08:25:55.619498+0000","last_fullsized":"2024-09-21T08:25:55.619498+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.594212+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:59.176997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"156'73","reported_seq":493,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618002+0000","last_change":"2024-09-21T08:25:54.534947+0000","last_active":"2024-09-21T08:25:55.618002+0000","last_peered":"2024-09-21T08:25:55.618002+0000","last_clean":"2024-09-21T08:25:55.618002+0000","last_became_active":"2024-09-21T08:24:56.503681+0000","last_became_peered":"2024-09-21T08:24:56.503681+0000","last_unstale":"2024-09-21T08:25:55.618002+0000","last_undegraded":"2024-09-21T08:25:55.618002+0000","last_fullsized":"2024-09-21T08:25:55.618002+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010890260000000001,"stat_sum":{"num_bytes":7311012,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":78,"num_read_kb":79912,"num_write":37,"num_write_kb":18859,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":12634927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1c","length":"9"},{"start":"26","length":"2"}]},{"pgid":"4.f","version":"73'15","reported_seq":108,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619331+0000","last_change":"2024-09-21T08:25:48.935201+0000","last_active":"2024-09-21T08:25:55.619331+0000","last_peered":"2024-09-21T08:25:55.619331+0000","last_clean":"2024-09-21T08:25:55.619331+0000","last_became_active":"2024-09-21T08:25:48.894852+0000","last_became_peered":"2024-09-21T08:25:48.894852+0000","last_unstale":"2024-09-21T08:25:55.619331+0000","last_undegraded":"2024-09-21T08:25:55.619331+0000","last_fullsized":"2024-09-21T08:25:55.619331+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":29,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617227+0000","last_change":"2024-09-21T08:25:48.897595+0000","last_active":"2024-09-21T08:25:55.617227+0000","last_peered":"2024-09-21T08:25:55.617227+0000","last_clean":"2024-09-21T08:25:55.617227+0000","last_became_active":"2024-09-21T08:25:48.897249+0000","last_became_peered":"2024-09-21T08:25:48.897249+0000","last_unstale":"2024-09-21T08:25:55.617227+0000","last_undegraded":"2024-09-21T08:25:55.617227+0000","last_fullsized":"2024-09-21T08:25:55.617227+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:45.434055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:45.434055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:59.844465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"85'9","reported_seq":220,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694787+0000","last_change":"2024-09-21T08:25:21.631370+0000","last_active":"2024-09-21T08:25:55.694787+0000","last_peered":"2024-09-21T08:25:55.694787+0000","last_clean":"2024-09-21T08:25:55.694787+0000","last_became_active":"2024-09-21T08:24:38.548789+0000","last_became_peered":"2024-09-21T08:24:38.548789+0000","last_unstale":"2024-09-21T08:25:55.694787+0000","last_undegraded":"2024-09-21T08:25:55.694787+0000","last_fullsized":"2024-09-21T08:25:55.694787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:25:21.631254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.631254+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:44.705485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":198,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694654+0000","last_change":"2024-09-21T08:25:48.914809+0000","last_active":"2024-09-21T08:25:55.694654+0000","last_peered":"2024-09-21T08:25:55.694654+0000","last_clean":"2024-09-21T08:25:55.694654+0000","last_became_active":"2024-09-21T08:25:48.914453+0000","last_became_peered":"2024-09-21T08:25:48.914453+0000","last_unstale":"2024-09-21T08:25:55.694654+0000","last_undegraded":"2024-09-21T08:25:55.694654+0000","last_fullsized":"2024-09-21T08:25:55.694654+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"158'15","reported_seq":395,"reported_epoch":162,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:55.695210+0000","last_change":"2024-09-21T08:25:53.708033+0000","last_active":"2024-09-21T08:25:55.695210+0000","last_peered":"2024-09-21T08:25:55.695210+0000","last_clean":"2024-09-21T08:25:55.695210+0000","last_became_active":"2024-09-21T08:25:48.901490+0000","last_became_peered":"2024-09-21T08:25:48.901490+0000","last_unstale":"2024-09-21T08:25:55.695210+0000","last_undegraded":"2024-09-21T08:25:55.695210+0000","last_fullsized":"2024-09-21T08:25:55.695210+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:56.232437+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.19386793699999999,"stat_sum":{"num_bytes":3025260,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":11,"num_read_kb":10650,"num_write":11,"num_write_kb":2527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5130840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1c","length":"5"}]},{"pgid":"4.0","version":"73'18","reported_seq":238,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617219+0000","last_change":"2024-09-21T08:25:48.942497+0000","last_active":"2024-09-21T08:25:55.617219+0000","last_peered":"2024-09-21T08:25:55.617219+0000","last_clean":"2024-09-21T08:25:55.617219+0000","last_became_active":"2024-09-21T08:25:48.895885+0000","last_became_peered":"2024-09-21T08:25:48.895885+0000","last_unstale":"2024-09-21T08:25:55.617219+0000","last_undegraded":"2024-09-21T08:25:55.617219+0000","last_fullsized":"2024-09-21T08:25:55.617219+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":154,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619628+0000","last_change":"2024-09-21T08:24:50.648706+0000","last_active":"2024-09-21T08:25:55.619628+0000","last_peered":"2024-09-21T08:25:55.619628+0000","last_clean":"2024-09-21T08:25:55.619628+0000","last_became_active":"2024-09-21T08:24:48.603740+0000","last_became_peered":"2024-09-21T08:24:48.603740+0000","last_unstale":"2024-09-21T08:25:55.619628+0000","last_undegraded":"2024-09-21T08:25:55.619628+0000","last_fullsized":"2024-09-21T08:25:55.619628+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.532278+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":147,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618021+0000","last_change":"2024-09-21T08:24:48.604125+0000","last_active":"2024-09-21T08:25:55.618021+0000","last_peered":"2024-09-21T08:25:55.618021+0000","last_clean":"2024-09-21T08:25:55.618021+0000","last_became_active":"2024-09-21T08:24:48.603825+0000","last_became_peered":"2024-09-21T08:24:48.603825+0000","last_unstale":"2024-09-21T08:25:55.618021+0000","last_undegraded":"2024-09-21T08:25:55.618021+0000","last_fullsized":"2024-09-21T08:25:55.618021+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":155,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.696337+0000","last_change":"2024-09-21T08:24:56.501652+0000","last_active":"2024-09-21T08:25:55.696337+0000","last_peered":"2024-09-21T08:25:55.696337+0000","last_clean":"2024-09-21T08:25:55.696337+0000","last_became_active":"2024-09-21T08:24:56.501237+0000","last_became_peered":"2024-09-21T08:24:56.501237+0000","last_unstale":"2024-09-21T08:25:55.696337+0000","last_undegraded":"2024-09-21T08:25:55.696337+0000","last_fullsized":"2024-09-21T08:25:55.696337+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.883417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"161'153","reported_seq":608,"reported_epoch":162,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:55.696658+0000","last_change":"2024-09-21T08:25:53.708076+0000","last_active":"2024-09-21T08:25:55.696658+0000","last_peered":"2024-09-21T08:25:55.696658+0000","last_clean":"2024-09-21T08:25:55.696658+0000","last_became_active":"2024-09-21T08:25:51.507328+0000","last_became_peered":"2024-09-21T08:25:51.507328+0000","last_unstale":"2024-09-21T08:25:55.696658+0000","last_undegraded":"2024-09-21T08:25:55.696658+0000","last_fullsized":"2024-09-21T08:25:55.696658+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":153,"log_dups_size":0,"ondisk_log_size":153,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.060037163999999997,"stat_sum":{"num_bytes":12548875,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":89,"num_read_kb":76474,"num_write":94,"num_write_kb":35359,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":22739695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1c","length":"1"},{"start":"1f","length":"2"},{"start":"22","length":"1"}]},{"pgid":"4.1","version":"73'14","reported_seq":244,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693732+0000","last_change":"2024-09-21T08:24:56.561414+0000","last_active":"2024-09-21T08:25:55.693732+0000","last_peered":"2024-09-21T08:25:55.693732+0000","last_clean":"2024-09-21T08:25:55.693732+0000","last_became_active":"2024-09-21T08:24:56.509169+0000","last_became_peered":"2024-09-21T08:24:56.509169+0000","last_unstale":"2024-09-21T08:25:55.693732+0000","last_undegraded":"2024-09-21T08:25:55.693732+0000","last_fullsized":"2024-09-21T08:25:55.693732+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":232,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693980+0000","last_change":"2024-09-21T08:25:48.919957+0000","last_active":"2024-09-21T08:25:55.693980+0000","last_peered":"2024-09-21T08:25:55.693980+0000","last_clean":"2024-09-21T08:25:55.693980+0000","last_became_active":"2024-09-21T08:25:48.893688+0000","last_became_peered":"2024-09-21T08:25:48.893688+0000","last_unstale":"2024-09-21T08:25:55.693980+0000","last_undegraded":"2024-09-21T08:25:55.693980+0000","last_fullsized":"2024-09-21T08:25:55.693980+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.923056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"156'11","reported_seq":92,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695330+0000","last_change":"2024-09-21T08:25:48.973195+0000","last_active":"2024-09-21T08:25:55.695330+0000","last_peered":"2024-09-21T08:25:55.695330+0000","last_clean":"2024-09-21T08:25:55.695330+0000","last_became_active":"2024-09-21T08:25:48.906808+0000","last_became_peered":"2024-09-21T08:25:48.906808+0000","last_unstale":"2024-09-21T08:25:55.695330+0000","last_undegraded":"2024-09-21T08:25:55.695330+0000","last_fullsized":"2024-09-21T08:25:55.695330+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":210,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618039+0000","last_change":"2024-09-21T08:25:48.900309+0000","last_active":"2024-09-21T08:25:55.618039+0000","last_peered":"2024-09-21T08:25:55.618039+0000","last_clean":"2024-09-21T08:25:55.618039+0000","last_became_active":"2024-09-21T08:25:48.899971+0000","last_became_peered":"2024-09-21T08:25:48.899971+0000","last_unstale":"2024-09-21T08:25:55.618039+0000","last_undegraded":"2024-09-21T08:25:55.618039+0000","last_fullsized":"2024-09-21T08:25:55.618039+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:21.561826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.561826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:44.050410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"161'36","reported_seq":367,"reported_epoch":162,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:55.695100+0000","last_change":"2024-09-21T08:25:53.902500+0000","last_active":"2024-09-21T08:25:55.695100+0000","last_peered":"2024-09-21T08:25:55.695100+0000","last_clean":"2024-09-21T08:25:55.695100+0000","last_became_active":"2024-09-21T08:25:48.908454+0000","last_became_peered":"2024-09-21T08:25:48.908454+0000","last_unstale":"2024-09-21T08:25:55.695100+0000","last_undegraded":"2024-09-21T08:25:55.695100+0000","last_fullsized":"2024-09-21T08:25:55.695100+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.40033116099999999,"stat_sum":{"num_bytes":5260525,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":32,"num_read_kb":33966,"num_write":14,"num_write_kb":4636,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5646358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"2"},{"start":"1f","length":"2"},{"start":"22","length":"2"}]},{"pgid":"4.2","version":"73'10","reported_seq":72,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693554+0000","last_change":"2024-09-21T08:25:48.929418+0000","last_active":"2024-09-21T08:25:55.693554+0000","last_peered":"2024-09-21T08:25:55.693554+0000","last_clean":"2024-09-21T08:25:55.693554+0000","last_became_active":"2024-09-21T08:25:48.894952+0000","last_became_peered":"2024-09-21T08:25:48.894952+0000","last_unstale":"2024-09-21T08:25:55.693554+0000","last_undegraded":"2024-09-21T08:25:55.693554+0000","last_fullsized":"2024-09-21T08:25:55.693554+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":230,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694005+0000","last_change":"2024-09-21T08:24:56.519880+0000","last_active":"2024-09-21T08:25:55.694005+0000","last_peered":"2024-09-21T08:25:55.694005+0000","last_clean":"2024-09-21T08:25:55.694005+0000","last_became_active":"2024-09-21T08:24:56.500882+0000","last_became_peered":"2024-09-21T08:24:56.500882+0000","last_unstale":"2024-09-21T08:25:55.694005+0000","last_undegraded":"2024-09-21T08:25:55.694005+0000","last_fullsized":"2024-09-21T08:25:55.694005+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"156'10","reported_seq":253,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693487+0000","last_change":"2024-09-21T08:25:48.928203+0000","last_active":"2024-09-21T08:25:55.693487+0000","last_peered":"2024-09-21T08:25:55.693487+0000","last_clean":"2024-09-21T08:25:55.693487+0000","last_became_active":"2024-09-21T08:25:48.902651+0000","last_became_peered":"2024-09-21T08:25:48.902651+0000","last_unstale":"2024-09-21T08:25:55.693487+0000","last_undegraded":"2024-09-21T08:25:55.693487+0000","last_fullsized":"2024-09-21T08:25:55.693487+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":197,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694861+0000","last_change":"2024-09-21T08:24:48.597514+0000","last_active":"2024-09-21T08:25:55.694861+0000","last_peered":"2024-09-21T08:25:55.694861+0000","last_clean":"2024-09-21T08:25:55.694861+0000","last_became_active":"2024-09-21T08:24:48.597048+0000","last_became_peered":"2024-09-21T08:24:48.597048+0000","last_unstale":"2024-09-21T08:25:55.694861+0000","last_undegraded":"2024-09-21T08:25:55.694861+0000","last_fullsized":"2024-09-21T08:25:55.694861+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"161'69","reported_seq":522,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693352+0000","last_change":"2024-09-21T08:25:55.004696+0000","last_active":"2024-09-21T08:25:55.693352+0000","last_peered":"2024-09-21T08:25:55.693352+0000","last_clean":"2024-09-21T08:25:55.693352+0000","last_became_active":"2024-09-21T08:25:48.911864+0000","last_became_peered":"2024-09-21T08:25:48.911864+0000","last_unstale":"2024-09-21T08:25:55.693352+0000","last_undegraded":"2024-09-21T08:25:55.693352+0000","last_fullsized":"2024-09-21T08:25:55.693352+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":69,"log_dups_size":0,"ondisk_log_size":69,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:00.026474+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.47962495900000002,"stat_sum":{"num_bytes":11680832,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":48,"num_read_kb":33337,"num_write":46,"num_write_kb":20266,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":8143226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1c","length":"9"},{"start":"26","length":"2"}]},{"pgid":"4.3","version":"73'19","reported_seq":235,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693362+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:25:55.693362+0000","last_peered":"2024-09-21T08:25:55.693362+0000","last_clean":"2024-09-21T08:25:55.693362+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:25:55.693362+0000","last_undegraded":"2024-09-21T08:25:55.693362+0000","last_fullsized":"2024-09-21T08:25:55.693362+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":27,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695930+0000","last_change":"2024-09-21T08:25:48.906191+0000","last_active":"2024-09-21T08:25:55.695930+0000","last_peered":"2024-09-21T08:25:55.695930+0000","last_clean":"2024-09-21T08:25:55.695930+0000","last_became_active":"2024-09-21T08:25:48.905818+0000","last_became_peered":"2024-09-21T08:25:48.905818+0000","last_unstale":"2024-09-21T08:25:55.695930+0000","last_undegraded":"2024-09-21T08:25:55.695930+0000","last_fullsized":"2024-09-21T08:25:55.695930+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.529672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.529672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:10.620945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":137,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693486+0000","last_change":"2024-09-21T08:25:39.339196+0000","last_active":"2024-09-21T08:25:55.693486+0000","last_peered":"2024-09-21T08:25:55.693486+0000","last_clean":"2024-09-21T08:25:55.693486+0000","last_became_active":"2024-09-21T08:24:56.499628+0000","last_became_peered":"2024-09-21T08:24:56.499628+0000","last_unstale":"2024-09-21T08:25:55.693486+0000","last_undegraded":"2024-09-21T08:25:55.693486+0000","last_fullsized":"2024-09-21T08:25:55.693486+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.339126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.339126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:57.963164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":95,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695541+0000","last_change":"2024-09-21T08:25:48.885420+0000","last_active":"2024-09-21T08:25:55.695541+0000","last_peered":"2024-09-21T08:25:55.695541+0000","last_clean":"2024-09-21T08:25:55.695541+0000","last_became_active":"2024-09-21T08:25:48.885120+0000","last_became_peered":"2024-09-21T08:25:48.885120+0000","last_unstale":"2024-09-21T08:25:55.695541+0000","last_undegraded":"2024-09-21T08:25:55.695541+0000","last_fullsized":"2024-09-21T08:25:55.695541+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.425389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:44.826050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"159'57","reported_seq":476,"reported_epoch":162,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:55.694789+0000","last_change":"2024-09-21T08:25:54.966034+0000","last_active":"2024-09-21T08:25:55.694789+0000","last_peered":"2024-09-21T08:25:55.694789+0000","last_clean":"2024-09-21T08:25:55.694789+0000","last_became_active":"2024-09-21T08:25:48.892621+0000","last_became_peered":"2024-09-21T08:25:48.892621+0000","last_unstale":"2024-09-21T08:25:55.694789+0000","last_undegraded":"2024-09-21T08:25:55.694789+0000","last_fullsized":"2024-09-21T08:25:55.694789+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":57,"log_dups_size":0,"ondisk_log_size":57,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.440521522,"stat_sum":{"num_bytes":4924451,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":18,"num_read_kb":11108,"num_write":27,"num_write_kb":9639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":10964761,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"6"}]},{"pgid":"4.4","version":"156'31","reported_seq":119,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.696091+0000","last_change":"2024-09-21T08:25:48.954047+0000","last_active":"2024-09-21T08:25:55.696091+0000","last_peered":"2024-09-21T08:25:55.696091+0000","last_clean":"2024-09-21T08:25:55.696091+0000","last_became_active":"2024-09-21T08:25:48.910276+0000","last_became_peered":"2024-09-21T08:25:48.910276+0000","last_unstale":"2024-09-21T08:25:55.696091+0000","last_undegraded":"2024-09-21T08:25:55.696091+0000","last_fullsized":"2024-09-21T08:25:55.696091+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.932582+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":177,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694156+0000","last_change":"2024-09-21T08:25:48.905376+0000","last_active":"2024-09-21T08:25:55.694156+0000","last_peered":"2024-09-21T08:25:55.694156+0000","last_clean":"2024-09-21T08:25:55.694156+0000","last_became_active":"2024-09-21T08:25:48.905056+0000","last_became_peered":"2024-09-21T08:25:48.905056+0000","last_unstale":"2024-09-21T08:25:55.694156+0000","last_undegraded":"2024-09-21T08:25:55.694156+0000","last_fullsized":"2024-09-21T08:25:55.694156+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352721+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":234,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693317+0000","last_change":"2024-09-21T08:25:48.911456+0000","last_active":"2024-09-21T08:25:55.693317+0000","last_peered":"2024-09-21T08:25:55.693317+0000","last_clean":"2024-09-21T08:25:55.693317+0000","last_became_active":"2024-09-21T08:25:48.894174+0000","last_became_peered":"2024-09-21T08:25:48.894174+0000","last_unstale":"2024-09-21T08:25:55.693317+0000","last_undegraded":"2024-09-21T08:25:55.693317+0000","last_fullsized":"2024-09-21T08:25:55.693317+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:23:28.509258+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":213,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695218+0000","last_change":"2024-09-21T08:25:48.893035+0000","last_active":"2024-09-21T08:25:55.695218+0000","last_peered":"2024-09-21T08:25:55.695218+0000","last_clean":"2024-09-21T08:25:55.695218+0000","last_became_active":"2024-09-21T08:25:48.892728+0000","last_became_peered":"2024-09-21T08:25:48.892728+0000","last_unstale":"2024-09-21T08:25:55.695218+0000","last_undegraded":"2024-09-21T08:25:55.695218+0000","last_fullsized":"2024-09-21T08:25:55.695218+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":199,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619181+0000","last_change":"2024-09-21T08:25:48.897361+0000","last_active":"2024-09-21T08:25:55.619181+0000","last_peered":"2024-09-21T08:25:55.619181+0000","last_clean":"2024-09-21T08:25:55.619181+0000","last_became_active":"2024-09-21T08:25:48.896898+0000","last_became_peered":"2024-09-21T08:25:48.896898+0000","last_unstale":"2024-09-21T08:25:55.619181+0000","last_undegraded":"2024-09-21T08:25:55.619181+0000","last_fullsized":"2024-09-21T08:25:55.619181+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.602376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:43.388968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"156'40","reported_seq":463,"reported_epoch":162,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:55.694660+0000","last_change":"2024-09-21T08:25:53.902471+0000","last_active":"2024-09-21T08:25:55.694660+0000","last_peered":"2024-09-21T08:25:55.694660+0000","last_clean":"2024-09-21T08:25:55.694660+0000","last_became_active":"2024-09-21T08:25:48.906238+0000","last_became_peered":"2024-09-21T08:25:48.906238+0000","last_unstale":"2024-09-21T08:25:55.694660+0000","last_undegraded":"2024-09-21T08:25:55.694660+0000","last_fullsized":"2024-09-21T08:25:55.694660+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":40,"log_dups_size":0,"ondisk_log_size":40,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.38365893099999998,"stat_sum":{"num_bytes":2756348,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":18843,"num_write":23,"num_write_kb":6559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4916498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"5"},{"start":"22","length":"1"}]},{"pgid":"4.7","version":"73'13","reported_seq":84,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694550+0000","last_change":"2024-09-21T08:25:48.937399+0000","last_active":"2024-09-21T08:25:55.694550+0000","last_peered":"2024-09-21T08:25:55.694550+0000","last_clean":"2024-09-21T08:25:55.694550+0000","last_became_active":"2024-09-21T08:25:48.908351+0000","last_became_peered":"2024-09-21T08:25:48.908351+0000","last_unstale":"2024-09-21T08:25:55.694550+0000","last_undegraded":"2024-09-21T08:25:55.694550+0000","last_fullsized":"2024-09-21T08:25:55.694550+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":107,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.692774+0000","last_change":"2024-09-21T08:25:48.912816+0000","last_active":"2024-09-21T08:25:55.692774+0000","last_peered":"2024-09-21T08:25:55.692774+0000","last_clean":"2024-09-21T08:25:55.692774+0000","last_became_active":"2024-09-21T08:25:48.911245+0000","last_became_peered":"2024-09-21T08:25:48.911245+0000","last_unstale":"2024-09-21T08:25:55.692774+0000","last_undegraded":"2024-09-21T08:25:55.692774+0000","last_fullsized":"2024-09-21T08:25:55.692774+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.932611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":250,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695720+0000","last_change":"2024-09-21T08:25:48.926199+0000","last_active":"2024-09-21T08:25:55.695720+0000","last_peered":"2024-09-21T08:25:55.695720+0000","last_clean":"2024-09-21T08:25:55.695720+0000","last_became_active":"2024-09-21T08:25:48.905148+0000","last_became_peered":"2024-09-21T08:25:48.905148+0000","last_unstale":"2024-09-21T08:25:55.695720+0000","last_undegraded":"2024-09-21T08:25:55.695720+0000","last_fullsized":"2024-09-21T08:25:55.695720+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:50.675673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.675673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.883457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":203,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.696143+0000","last_change":"2024-09-21T08:25:42.021037+0000","last_active":"2024-09-21T08:25:55.696143+0000","last_peered":"2024-09-21T08:25:55.696143+0000","last_clean":"2024-09-21T08:25:55.696143+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:25:55.696143+0000","last_undegraded":"2024-09-21T08:25:55.696143+0000","last_fullsized":"2024-09-21T08:25:55.696143+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:42.020928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.020928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:55.206204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":202,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618918+0000","last_change":"2024-09-21T08:24:56.501148+0000","last_active":"2024-09-21T08:25:55.618918+0000","last_peered":"2024-09-21T08:25:55.618918+0000","last_clean":"2024-09-21T08:25:55.618918+0000","last_became_active":"2024-09-21T08:24:56.498476+0000","last_became_peered":"2024-09-21T08:24:56.498476+0000","last_unstale":"2024-09-21T08:25:55.618918+0000","last_undegraded":"2024-09-21T08:25:55.618918+0000","last_fullsized":"2024-09-21T08:25:55.618918+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"161'223","reported_seq":634,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619297+0000","last_change":"2024-09-21T08:25:54.536500+0000","last_active":"2024-09-21T08:25:55.619297+0000","last_peered":"2024-09-21T08:25:55.619297+0000","last_clean":"2024-09-21T08:25:55.619297+0000","last_became_active":"2024-09-21T08:24:34.507710+0000","last_became_peered":"2024-09-21T08:24:34.507710+0000","last_unstale":"2024-09-21T08:25:55.619297+0000","last_undegraded":"2024-09-21T08:25:55.619297+0000","last_fullsized":"2024-09-21T08:25:55.619297+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":223,"log_dups_size":0,"ondisk_log_size":223,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0013325570000000001,"stat_sum":{"num_bytes":13140079,"num_objects":6,"num_object_clones":2,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":106,"num_read_kb":82772,"num_write":109,"num_write_kb":44644,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1c","length":"9"},{"start":"26","length":"2"}]},{"pgid":"4.6","version":"73'12","reported_seq":237,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694186+0000","last_change":"2024-09-21T08:25:48.924592+0000","last_active":"2024-09-21T08:25:55.694186+0000","last_peered":"2024-09-21T08:25:55.694186+0000","last_clean":"2024-09-21T08:25:55.694186+0000","last_became_active":"2024-09-21T08:25:48.888592+0000","last_became_peered":"2024-09-21T08:25:48.888592+0000","last_unstale":"2024-09-21T08:25:55.694186+0000","last_undegraded":"2024-09-21T08:25:55.694186+0000","last_fullsized":"2024-09-21T08:25:55.694186+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352558+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":215,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694994+0000","last_change":"2024-09-21T08:24:48.601343+0000","last_active":"2024-09-21T08:25:55.694994+0000","last_peered":"2024-09-21T08:25:55.694994+0000","last_clean":"2024-09-21T08:25:55.694994+0000","last_became_active":"2024-09-21T08:24:48.601044+0000","last_became_peered":"2024-09-21T08:24:48.601044+0000","last_unstale":"2024-09-21T08:25:55.694994+0000","last_undegraded":"2024-09-21T08:25:55.694994+0000","last_fullsized":"2024-09-21T08:25:55.694994+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":370,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617375+0000","last_change":"2024-09-21T08:25:48.914747+0000","last_active":"2024-09-21T08:25:55.617375+0000","last_peered":"2024-09-21T08:25:55.617375+0000","last_clean":"2024-09-21T08:25:55.617375+0000","last_became_active":"2024-09-21T08:25:48.894703+0000","last_became_peered":"2024-09-21T08:25:48.894703+0000","last_unstale":"2024-09-21T08:25:55.617375+0000","last_undegraded":"2024-09-21T08:25:55.617375+0000","last_fullsized":"2024-09-21T08:25:55.617375+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":138,"num_read_kb":133,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":210,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694589+0000","last_change":"2024-09-21T08:25:48.909001+0000","last_active":"2024-09-21T08:25:55.694589+0000","last_peered":"2024-09-21T08:25:55.694589+0000","last_clean":"2024-09-21T08:25:55.694589+0000","last_became_active":"2024-09-21T08:25:48.908492+0000","last_became_peered":"2024-09-21T08:25:48.908492+0000","last_unstale":"2024-09-21T08:25:55.694589+0000","last_undegraded":"2024-09-21T08:25:55.694589+0000","last_fullsized":"2024-09-21T08:25:55.694589+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.812789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.812789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:57.580731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":101,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694226+0000","last_change":"2024-09-21T08:25:48.914203+0000","last_active":"2024-09-21T08:25:55.694226+0000","last_peered":"2024-09-21T08:25:55.694226+0000","last_clean":"2024-09-21T08:25:55.694226+0000","last_became_active":"2024-09-21T08:25:48.912863+0000","last_became_peered":"2024-09-21T08:25:48.912863+0000","last_unstale":"2024-09-21T08:25:55.694226+0000","last_undegraded":"2024-09-21T08:25:55.694226+0000","last_fullsized":"2024-09-21T08:25:55.694226+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"158'71","reported_seq":477,"reported_epoch":162,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:55.694680+0000","last_change":"2024-09-21T08:25:54.966741+0000","last_active":"2024-09-21T08:25:55.694680+0000","last_peered":"2024-09-21T08:25:55.694680+0000","last_clean":"2024-09-21T08:25:55.694680+0000","last_became_active":"2024-09-21T08:24:48.593246+0000","last_became_peered":"2024-09-21T08:24:48.593246+0000","last_unstale":"2024-09-21T08:25:55.694680+0000","last_undegraded":"2024-09-21T08:25:55.694680+0000","last_fullsized":"2024-09-21T08:25:55.694680+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44131728399999998,"stat_sum":{"num_bytes":7781641,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":42135,"num_write":38,"num_write_kb":13384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4966638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"6"}]},{"pgid":"4.5","version":"103'18","reported_seq":203,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618707+0000","last_change":"2024-09-21T08:24:56.564202+0000","last_active":"2024-09-21T08:25:55.618707+0000","last_peered":"2024-09-21T08:25:55.618707+0000","last_clean":"2024-09-21T08:25:55.618707+0000","last_became_active":"2024-09-21T08:24:56.503984+0000","last_became_peered":"2024-09-21T08:24:56.503984+0000","last_unstale":"2024-09-21T08:25:55.618707+0000","last_undegraded":"2024-09-21T08:25:55.618707+0000","last_fullsized":"2024-09-21T08:25:55.618707+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-09-21T08:24:48.583137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.583137+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.538958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":159,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693913+0000","last_change":"2024-09-21T08:25:48.914796+0000","last_active":"2024-09-21T08:25:55.693913+0000","last_peered":"2024-09-21T08:25:55.693913+0000","last_clean":"2024-09-21T08:25:55.693913+0000","last_became_active":"2024-09-21T08:25:48.912615+0000","last_became_peered":"2024-09-21T08:25:48.912615+0000","last_unstale":"2024-09-21T08:25:55.693913+0000","last_undegraded":"2024-09-21T08:25:55.693913+0000","last_fullsized":"2024-09-21T08:25:55.693913+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":496,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617558+0000","last_change":"2024-09-21T08:25:51.506619+0000","last_active":"2024-09-21T08:25:55.617558+0000","last_peered":"2024-09-21T08:25:55.617558+0000","last_clean":"2024-09-21T08:25:55.617558+0000","last_became_active":"2024-09-21T08:25:51.506298+0000","last_became_peered":"2024-09-21T08:25:51.506298+0000","last_unstale":"2024-09-21T08:25:55.617558+0000","last_undegraded":"2024-09-21T08:25:55.617558+0000","last_fullsized":"2024-09-21T08:25:55.617558+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:59.195279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":202,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618838+0000","last_change":"2024-09-21T08:25:41.700634+0000","last_active":"2024-09-21T08:25:55.618838+0000","last_peered":"2024-09-21T08:25:55.618838+0000","last_clean":"2024-09-21T08:25:55.618838+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:25:55.618838+0000","last_undegraded":"2024-09-21T08:25:55.618838+0000","last_fullsized":"2024-09-21T08:25:55.618838+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:56.978983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":218,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694762+0000","last_change":"2024-09-21T08:25:48.907243+0000","last_active":"2024-09-21T08:25:55.694762+0000","last_peered":"2024-09-21T08:25:55.694762+0000","last_clean":"2024-09-21T08:25:55.694762+0000","last_became_active":"2024-09-21T08:25:48.906778+0000","last_became_peered":"2024-09-21T08:25:48.906778+0000","last_unstale":"2024-09-21T08:25:55.694762+0000","last_undegraded":"2024-09-21T08:25:55.694762+0000","last_fullsized":"2024-09-21T08:25:55.694762+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.519349+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"157'46","reported_seq":374,"reported_epoch":162,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:55.695074+0000","last_change":"2024-09-21T08:25:54.966537+0000","last_active":"2024-09-21T08:25:55.695074+0000","last_peered":"2024-09-21T08:25:55.695074+0000","last_clean":"2024-09-21T08:25:55.695074+0000","last_became_active":"2024-09-21T08:25:48.906387+0000","last_became_peered":"2024-09-21T08:25:48.906387+0000","last_unstale":"2024-09-21T08:25:55.695074+0000","last_undegraded":"2024-09-21T08:25:55.695074+0000","last_fullsized":"2024-09-21T08:25:55.695074+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.440851349,"stat_sum":{"num_bytes":4756381,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":38,"num_read_kb":34342,"num_write":31,"num_write_kb":12613,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":6854523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"5"},{"start":"22","length":"2"}]},{"pgid":"4.e","version":"73'11","reported_seq":117,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694975+0000","last_change":"2024-09-21T08:25:48.949515+0000","last_active":"2024-09-21T08:25:55.694975+0000","last_peered":"2024-09-21T08:25:55.694975+0000","last_clean":"2024-09-21T08:25:55.694975+0000","last_became_active":"2024-09-21T08:25:48.911473+0000","last_became_peered":"2024-09-21T08:25:48.911473+0000","last_unstale":"2024-09-21T08:25:55.694975+0000","last_undegraded":"2024-09-21T08:25:55.694975+0000","last_fullsized":"2024-09-21T08:25:55.694975+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":180,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695621+0000","last_change":"2024-09-21T08:25:48.907133+0000","last_active":"2024-09-21T08:25:55.695621+0000","last_peered":"2024-09-21T08:25:55.695621+0000","last_clean":"2024-09-21T08:25:55.695621+0000","last_became_active":"2024-09-21T08:25:48.906832+0000","last_became_peered":"2024-09-21T08:25:48.906832+0000","last_unstale":"2024-09-21T08:25:55.695621+0000","last_undegraded":"2024-09-21T08:25:55.695621+0000","last_fullsized":"2024-09-21T08:25:55.695621+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.756905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.756905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.932691+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":214,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694888+0000","last_change":"2024-09-21T08:25:48.917071+0000","last_active":"2024-09-21T08:25:55.694888+0000","last_peered":"2024-09-21T08:25:55.694888+0000","last_clean":"2024-09-21T08:25:55.694888+0000","last_became_active":"2024-09-21T08:25:48.915146+0000","last_became_peered":"2024-09-21T08:25:48.915146+0000","last_unstale":"2024-09-21T08:25:55.694888+0000","last_undegraded":"2024-09-21T08:25:55.694888+0000","last_fullsized":"2024-09-21T08:25:55.694888+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":138,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693785+0000","last_change":"2024-09-21T08:25:48.914706+0000","last_active":"2024-09-21T08:25:55.693785+0000","last_peered":"2024-09-21T08:25:55.693785+0000","last_clean":"2024-09-21T08:25:55.693785+0000","last_became_active":"2024-09-21T08:25:48.912765+0000","last_became_peered":"2024-09-21T08:25:48.912765+0000","last_unstale":"2024-09-21T08:25:55.693785+0000","last_undegraded":"2024-09-21T08:25:55.693785+0000","last_fullsized":"2024-09-21T08:25:55.693785+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"161'75","reported_seq":453,"reported_epoch":162,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:55.695969+0000","last_change":"2024-09-21T08:25:53.898353+0000","last_active":"2024-09-21T08:25:55.695969+0000","last_peered":"2024-09-21T08:25:55.695969+0000","last_clean":"2024-09-21T08:25:55.695969+0000","last_became_active":"2024-09-21T08:24:34.508974+0000","last_became_peered":"2024-09-21T08:24:34.508974+0000","last_unstale":"2024-09-21T08:25:55.695969+0000","last_undegraded":"2024-09-21T08:25:55.695969+0000","last_fullsized":"2024-09-21T08:25:55.695969+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":75,"log_dups_size":0,"ondisk_log_size":75,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.38181957300000002,"stat_sum":{"num_bytes":4573214,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":21,"num_read_kb":7783,"num_write":33,"num_write_kb":11824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1c","length":"5"}]},{"pgid":"4.d","version":"73'17","reported_seq":287,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694544+0000","last_change":"2024-09-21T08:25:48.934395+0000","last_active":"2024-09-21T08:25:55.694544+0000","last_peered":"2024-09-21T08:25:55.694544+0000","last_clean":"2024-09-21T08:25:55.694544+0000","last_became_active":"2024-09-21T08:25:48.904254+0000","last_became_peered":"2024-09-21T08:25:48.904254+0000","last_unstale":"2024-09-21T08:25:55.694544+0000","last_undegraded":"2024-09-21T08:25:55.694544+0000","last_fullsized":"2024-09-21T08:25:55.694544+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-09-21T08:25:42.978787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.978787+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:48.389318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":101,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693814+0000","last_change":"2024-09-21T08:25:48.907580+0000","last_active":"2024-09-21T08:25:55.693814+0000","last_peered":"2024-09-21T08:25:55.693814+0000","last_clean":"2024-09-21T08:25:55.693814+0000","last_became_active":"2024-09-21T08:25:48.907227+0000","last_became_peered":"2024-09-21T08:25:48.907227+0000","last_unstale":"2024-09-21T08:25:55.693814+0000","last_undegraded":"2024-09-21T08:25:55.693814+0000","last_fullsized":"2024-09-21T08:25:55.693814+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":105,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694770+0000","last_change":"2024-09-21T08:25:48.889493+0000","last_active":"2024-09-21T08:25:55.694770+0000","last_peered":"2024-09-21T08:25:55.694770+0000","last_clean":"2024-09-21T08:25:55.694770+0000","last_became_active":"2024-09-21T08:25:48.889206+0000","last_became_peered":"2024-09-21T08:25:48.889206+0000","last_unstale":"2024-09-21T08:25:55.694770+0000","last_undegraded":"2024-09-21T08:25:55.694770+0000","last_fullsized":"2024-09-21T08:25:55.694770+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":223,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693875+0000","last_change":"2024-09-21T08:25:48.909997+0000","last_active":"2024-09-21T08:25:55.693875+0000","last_peered":"2024-09-21T08:25:55.693875+0000","last_clean":"2024-09-21T08:25:55.693875+0000","last_became_active":"2024-09-21T08:25:48.908631+0000","last_became_peered":"2024-09-21T08:25:48.908631+0000","last_unstale":"2024-09-21T08:25:55.693875+0000","last_undegraded":"2024-09-21T08:25:55.693875+0000","last_fullsized":"2024-09-21T08:25:55.693875+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.766815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.605769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"100'7","reported_seq":414,"reported_epoch":162,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:55.695162+0000","last_change":"2024-09-21T08:25:54.966318+0000","last_active":"2024-09-21T08:25:55.695162+0000","last_peered":"2024-09-21T08:25:55.695162+0000","last_clean":"2024-09-21T08:25:55.695162+0000","last_became_active":"2024-09-21T08:24:48.586187+0000","last_became_peered":"2024-09-21T08:24:48.586187+0000","last_unstale":"2024-09-21T08:25:55.695162+0000","last_undegraded":"2024-09-21T08:25:55.695162+0000","last_fullsized":"2024-09-21T08:25:55.695162+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.44087270000000001,"stat_sum":{"num_bytes":2369754,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":11580,"num_write":5,"num_write_kb":3244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":949927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"9"},{"start":"26","length":"1"}]},{"pgid":"4.c","version":"73'10","reported_seq":74,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617796+0000","last_change":"2024-09-21T08:25:48.987805+0000","last_active":"2024-09-21T08:25:55.617796+0000","last_peered":"2024-09-21T08:25:55.617796+0000","last_clean":"2024-09-21T08:25:55.617796+0000","last_became_active":"2024-09-21T08:25:48.899148+0000","last_became_peered":"2024-09-21T08:25:48.899148+0000","last_unstale":"2024-09-21T08:25:55.617796+0000","last_undegraded":"2024-09-21T08:25:55.617796+0000","last_fullsized":"2024-09-21T08:25:55.617796+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":197,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693267+0000","last_change":"2024-09-21T08:25:48.891359+0000","last_active":"2024-09-21T08:25:55.693267+0000","last_peered":"2024-09-21T08:25:55.693267+0000","last_clean":"2024-09-21T08:25:55.693267+0000","last_became_active":"2024-09-21T08:25:48.890817+0000","last_became_peered":"2024-09-21T08:25:48.890817+0000","last_unstale":"2024-09-21T08:25:55.693267+0000","last_undegraded":"2024-09-21T08:25:55.693267+0000","last_fullsized":"2024-09-21T08:25:55.693267+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352621+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":241,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694371+0000","last_change":"2024-09-21T08:24:51.689976+0000","last_active":"2024-09-21T08:25:55.694371+0000","last_peered":"2024-09-21T08:25:55.694371+0000","last_clean":"2024-09-21T08:25:55.694371+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:25:55.694371+0000","last_undegraded":"2024-09-21T08:25:55.694371+0000","last_fullsized":"2024-09-21T08:25:55.694371+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:24:51.689901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.689901+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.850967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":29,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695982+0000","last_change":"2024-09-21T08:25:48.903834+0000","last_active":"2024-09-21T08:25:55.695982+0000","last_peered":"2024-09-21T08:25:55.695982+0000","last_clean":"2024-09-21T08:25:55.695982+0000","last_became_active":"2024-09-21T08:25:48.903536+0000","last_became_peered":"2024-09-21T08:25:48.903536+0000","last_unstale":"2024-09-21T08:25:55.695982+0000","last_undegraded":"2024-09-21T08:25:55.695982+0000","last_fullsized":"2024-09-21T08:25:55.695982+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.932597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"158'135","reported_seq":542,"reported_epoch":162,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:25:55.693220+0000","last_change":"2024-09-21T08:25:51.896879+0000","last_active":"2024-09-21T08:25:55.693220+0000","last_peered":"2024-09-21T08:25:55.693220+0000","last_clean":"2024-09-21T08:25:55.693220+0000","last_became_active":"2024-09-21T08:24:48.590826+0000","last_became_peered":"2024-09-21T08:24:48.590826+0000","last_unstale":"2024-09-21T08:25:55.693220+0000","last_undegraded":"2024-09-21T08:25:55.693220+0000","last_fullsized":"2024-09-21T08:25:55.693220+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":135,"log_dups_size":0,"ondisk_log_size":135,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.391775596,"stat_sum":{"num_bytes":19279882,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":58,"num_read_kb":34650,"num_write":84,"num_write_kb":29903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8754687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1c","length":"2"},{"start":"1f","length":"2"}]},{"pgid":"4.b","version":"73'9","reported_seq":271,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693445+0000","last_change":"2024-09-21T08:25:48.969652+0000","last_active":"2024-09-21T08:25:55.693445+0000","last_peered":"2024-09-21T08:25:55.693445+0000","last_clean":"2024-09-21T08:25:55.693445+0000","last_became_active":"2024-09-21T08:25:48.906741+0000","last_became_peered":"2024-09-21T08:25:48.906741+0000","last_unstale":"2024-09-21T08:25:55.693445+0000","last_undegraded":"2024-09-21T08:25:55.693445+0000","last_fullsized":"2024-09-21T08:25:55.693445+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:25:22.369811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.369811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:41.468044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":230,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694291+0000","last_change":"2024-09-21T08:25:48.916673+0000","last_active":"2024-09-21T08:25:55.694291+0000","last_peered":"2024-09-21T08:25:55.694291+0000","last_clean":"2024-09-21T08:25:55.694291+0000","last_became_active":"2024-09-21T08:25:48.914811+0000","last_became_peered":"2024-09-21T08:25:48.914811+0000","last_unstale":"2024-09-21T08:25:55.694291+0000","last_undegraded":"2024-09-21T08:25:55.694291+0000","last_fullsized":"2024-09-21T08:25:55.694291+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.665444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.665444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.303831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":221,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.696248+0000","last_change":"2024-09-21T08:25:48.900965+0000","last_active":"2024-09-21T08:25:55.696248+0000","last_peered":"2024-09-21T08:25:55.696248+0000","last_clean":"2024-09-21T08:25:55.696248+0000","last_became_active":"2024-09-21T08:25:48.900241+0000","last_became_peered":"2024-09-21T08:25:48.900241+0000","last_unstale":"2024-09-21T08:25:55.696248+0000","last_undegraded":"2024-09-21T08:25:55.696248+0000","last_fullsized":"2024-09-21T08:25:55.696248+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:48.797848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":191,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693912+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:25:55.693912+0000","last_peered":"2024-09-21T08:25:55.693912+0000","last_clean":"2024-09-21T08:25:55.693912+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:25:55.693912+0000","last_undegraded":"2024-09-21T08:25:55.693912+0000","last_fullsized":"2024-09-21T08:25:55.693912+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"161'127","reported_seq":578,"reported_epoch":162,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:55.694447+0000","last_change":"2024-09-21T08:25:54.966757+0000","last_active":"2024-09-21T08:25:55.694447+0000","last_peered":"2024-09-21T08:25:55.694447+0000","last_clean":"2024-09-21T08:25:55.694447+0000","last_became_active":"2024-09-21T08:25:51.507296+0000","last_became_peered":"2024-09-21T08:25:51.507296+0000","last_unstale":"2024-09-21T08:25:55.694447+0000","last_undegraded":"2024-09-21T08:25:55.694447+0000","last_fullsized":"2024-09-21T08:25:55.694447+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":127,"log_dups_size":0,"ondisk_log_size":127,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.38413264400000002,"stat_sum":{"num_bytes":19400097,"num_objects":7,"num_object_clones":2,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":110,"num_read_kb":108300,"num_write":72,"num_write_kb":32497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":13898927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1c","length":"2"},{"start":"1f","length":"2"},{"start":"22","length":"1"}]},{"pgid":"4.a","version":"73'19","reported_seq":96,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617600+0000","last_change":"2024-09-21T08:25:48.964103+0000","last_active":"2024-09-21T08:25:55.617600+0000","last_peered":"2024-09-21T08:25:55.617600+0000","last_clean":"2024-09-21T08:25:55.617600+0000","last_became_active":"2024-09-21T08:25:48.895647+0000","last_became_peered":"2024-09-21T08:25:48.895647+0000","last_unstale":"2024-09-21T08:25:55.617600+0000","last_undegraded":"2024-09-21T08:25:55.617600+0000","last_fullsized":"2024-09-21T08:25:55.617600+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:43.390605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":216,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618774+0000","last_change":"2024-09-21T08:24:56.507593+0000","last_active":"2024-09-21T08:25:55.618774+0000","last_peered":"2024-09-21T08:25:55.618774+0000","last_clean":"2024-09-21T08:25:55.618774+0000","last_became_active":"2024-09-21T08:24:56.505054+0000","last_became_peered":"2024-09-21T08:24:56.505054+0000","last_unstale":"2024-09-21T08:25:55.618774+0000","last_undegraded":"2024-09-21T08:25:55.618774+0000","last_fullsized":"2024-09-21T08:25:55.618774+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.740674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.740674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:58.640221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":195,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695470+0000","last_change":"2024-09-21T08:23:34.564923+0000","last_active":"2024-09-21T08:25:55.695470+0000","last_peered":"2024-09-21T08:25:55.695470+0000","last_clean":"2024-09-21T08:25:55.695470+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:25:55.695470+0000","last_undegraded":"2024-09-21T08:25:55.695470+0000","last_fullsized":"2024-09-21T08:25:55.695470+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.883399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":210,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617500+0000","last_change":"2024-09-21T08:24:56.504692+0000","last_active":"2024-09-21T08:25:55.617500+0000","last_peered":"2024-09-21T08:25:55.617500+0000","last_clean":"2024-09-21T08:25:55.617500+0000","last_became_active":"2024-09-21T08:24:56.503564+0000","last_became_peered":"2024-09-21T08:24:56.503564+0000","last_unstale":"2024-09-21T08:25:55.617500+0000","last_undegraded":"2024-09-21T08:25:55.617500+0000","last_fullsized":"2024-09-21T08:25:55.617500+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:59.727934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"160'31","reported_seq":401,"reported_epoch":162,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:55.696663+0000","last_change":"2024-09-21T08:25:53.898383+0000","last_active":"2024-09-21T08:25:55.696663+0000","last_peered":"2024-09-21T08:25:55.696663+0000","last_clean":"2024-09-21T08:25:55.696663+0000","last_became_active":"2024-09-21T08:25:48.910246+0000","last_became_peered":"2024-09-21T08:25:48.910246+0000","last_unstale":"2024-09-21T08:25:55.696663+0000","last_undegraded":"2024-09-21T08:25:55.696663+0000","last_fullsized":"2024-09-21T08:25:55.696663+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.20208298599999999,"stat_sum":{"num_bytes":3159683,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":8893,"num_write":14,"num_write_kb":2495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2551608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1c","length":"2"},{"start":"1f","length":"2"},{"start":"22","length":"1"}]},{"pgid":"4.9","version":"73'12","reported_seq":121,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695212+0000","last_change":"2024-09-21T08:25:48.944363+0000","last_active":"2024-09-21T08:25:55.695212+0000","last_peered":"2024-09-21T08:25:55.695212+0000","last_clean":"2024-09-21T08:25:55.695212+0000","last_became_active":"2024-09-21T08:25:48.894334+0000","last_became_peered":"2024-09-21T08:25:48.894334+0000","last_unstale":"2024-09-21T08:25:55.695212+0000","last_undegraded":"2024-09-21T08:25:55.695212+0000","last_fullsized":"2024-09-21T08:25:55.695212+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:41.463820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.463820+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.215785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":214,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618867+0000","last_change":"2024-09-21T08:25:48.896592+0000","last_active":"2024-09-21T08:25:55.618867+0000","last_peered":"2024-09-21T08:25:55.618867+0000","last_clean":"2024-09-21T08:25:55.618867+0000","last_became_active":"2024-09-21T08:25:48.896222+0000","last_became_peered":"2024-09-21T08:25:48.896222+0000","last_unstale":"2024-09-21T08:25:55.618867+0000","last_undegraded":"2024-09-21T08:25:55.618867+0000","last_fullsized":"2024-09-21T08:25:55.618867+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759430+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":209,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693632+0000","last_change":"2024-09-21T08:25:48.913293+0000","last_active":"2024-09-21T08:25:55.693632+0000","last_peered":"2024-09-21T08:25:55.693632+0000","last_clean":"2024-09-21T08:25:55.693632+0000","last_became_active":"2024-09-21T08:25:48.912993+0000","last_became_peered":"2024-09-21T08:25:48.912993+0000","last_unstale":"2024-09-21T08:25:55.693632+0000","last_undegraded":"2024-09-21T08:25:55.693632+0000","last_fullsized":"2024-09-21T08:25:55.693632+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:44.914394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:44.914394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:07.844366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":165,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618246+0000","last_change":"2024-09-21T08:25:48.901068+0000","last_active":"2024-09-21T08:25:55.618246+0000","last_peered":"2024-09-21T08:25:55.618246+0000","last_clean":"2024-09-21T08:25:55.618246+0000","last_became_active":"2024-09-21T08:25:48.900751+0000","last_became_peered":"2024-09-21T08:25:48.900751+0000","last_unstale":"2024-09-21T08:25:55.618246+0000","last_undegraded":"2024-09-21T08:25:55.618246+0000","last_fullsized":"2024-09-21T08:25:55.618246+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:38.201536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"161'208","reported_seq":724,"reported_epoch":162,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:25:55.694339+0000","last_change":"2024-09-21T08:25:55.523290+0000","last_active":"2024-09-21T08:25:55.694339+0000","last_peered":"2024-09-21T08:25:55.694339+0000","last_clean":"2024-09-21T08:25:55.694339+0000","last_became_active":"2024-09-21T08:25:51.507525+0000","last_became_peered":"2024-09-21T08:25:51.507525+0000","last_unstale":"2024-09-21T08:25:55.694339+0000","last_undegraded":"2024-09-21T08:25:55.694339+0000","last_fullsized":"2024-09-21T08:25:55.694339+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":208,"log_dups_size":0,"ondisk_log_size":208,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":9,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:56.569499+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99840456700000002,"stat_sum":{"num_bytes":19741190,"num_objects":8,"num_object_clones":2,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":171,"num_read_kb":147404,"num_write":135,"num_write_kb":41856,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31871318,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1c","length":"2"}]},{"pgid":"4.8","version":"73'15","reported_seq":277,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693117+0000","last_change":"2024-09-21T08:25:48.957137+0000","last_active":"2024-09-21T08:25:55.693117+0000","last_peered":"2024-09-21T08:25:55.693117+0000","last_clean":"2024-09-21T08:25:55.693117+0000","last_became_active":"2024-09-21T08:25:48.911675+0000","last_became_peered":"2024-09-21T08:25:48.911675+0000","last_unstale":"2024-09-21T08:25:55.693117+0000","last_undegraded":"2024-09-21T08:25:55.693117+0000","last_fullsized":"2024-09-21T08:25:55.693117+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":246,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619629+0000","last_change":"2024-09-21T08:25:48.975855+0000","last_active":"2024-09-21T08:25:55.619629+0000","last_peered":"2024-09-21T08:25:55.619629+0000","last_clean":"2024-09-21T08:25:55.619629+0000","last_became_active":"2024-09-21T08:25:48.897974+0000","last_became_peered":"2024-09-21T08:25:48.897974+0000","last_unstale":"2024-09-21T08:25:55.619629+0000","last_undegraded":"2024-09-21T08:25:55.619629+0000","last_fullsized":"2024-09-21T08:25:55.619629+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.484877+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"156'11","reported_seq":271,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617565+0000","last_change":"2024-09-21T08:25:48.977564+0000","last_active":"2024-09-21T08:25:55.617565+0000","last_peered":"2024-09-21T08:25:55.617565+0000","last_clean":"2024-09-21T08:25:55.617565+0000","last_became_active":"2024-09-21T08:25:48.897436+0000","last_became_peered":"2024-09-21T08:25:48.897436+0000","last_unstale":"2024-09-21T08:25:55.617565+0000","last_undegraded":"2024-09-21T08:25:55.617565+0000","last_fullsized":"2024-09-21T08:25:55.617565+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'10","last_scrub_stamp":"2024-09-21T08:25:42.708458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.708458+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:56.347110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":198,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693722+0000","last_change":"2024-09-21T08:24:56.500567+0000","last_active":"2024-09-21T08:25:55.693722+0000","last_peered":"2024-09-21T08:25:55.693722+0000","last_clean":"2024-09-21T08:25:55.693722+0000","last_became_active":"2024-09-21T08:24:56.500287+0000","last_became_peered":"2024-09-21T08:24:56.500287+0000","last_unstale":"2024-09-21T08:25:55.693722+0000","last_undegraded":"2024-09-21T08:25:55.693722+0000","last_fullsized":"2024-09-21T08:25:55.693722+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":129,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694764+0000","last_change":"2024-09-21T08:24:56.500240+0000","last_active":"2024-09-21T08:25:55.694764+0000","last_peered":"2024-09-21T08:25:55.694764+0000","last_clean":"2024-09-21T08:25:55.694764+0000","last_became_active":"2024-09-21T08:24:56.499114+0000","last_became_peered":"2024-09-21T08:24:56.499114+0000","last_unstale":"2024-09-21T08:25:55.694764+0000","last_undegraded":"2024-09-21T08:25:55.694764+0000","last_fullsized":"2024-09-21T08:25:55.694764+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":195,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619565+0000","last_change":"2024-09-21T08:25:48.980718+0000","last_active":"2024-09-21T08:25:55.619565+0000","last_peered":"2024-09-21T08:25:55.619565+0000","last_clean":"2024-09-21T08:25:55.619565+0000","last_became_active":"2024-09-21T08:25:48.898422+0000","last_became_peered":"2024-09-21T08:25:48.898422+0000","last_unstale":"2024-09-21T08:25:55.619565+0000","last_undegraded":"2024-09-21T08:25:55.619565+0000","last_fullsized":"2024-09-21T08:25:55.619565+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":213,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694540+0000","last_change":"2024-09-21T08:25:48.908032+0000","last_active":"2024-09-21T08:25:55.694540+0000","last_peered":"2024-09-21T08:25:55.694540+0000","last_clean":"2024-09-21T08:25:55.694540+0000","last_became_active":"2024-09-21T08:25:48.907510+0000","last_became_peered":"2024-09-21T08:25:48.907510+0000","last_unstale":"2024-09-21T08:25:55.694540+0000","last_undegraded":"2024-09-21T08:25:55.694540+0000","last_fullsized":"2024-09-21T08:25:55.694540+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":201,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619073+0000","last_change":"2024-09-21T08:24:56.506870+0000","last_active":"2024-09-21T08:25:55.619073+0000","last_peered":"2024-09-21T08:25:55.619073+0000","last_clean":"2024-09-21T08:25:55.619073+0000","last_became_active":"2024-09-21T08:24:56.502937+0000","last_became_peered":"2024-09-21T08:24:56.502937+0000","last_unstale":"2024-09-21T08:25:55.619073+0000","last_undegraded":"2024-09-21T08:25:55.619073+0000","last_fullsized":"2024-09-21T08:25:55.619073+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759524+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":230,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694089+0000","last_change":"2024-09-21T08:24:48.629077+0000","last_active":"2024-09-21T08:25:55.694089+0000","last_peered":"2024-09-21T08:25:55.694089+0000","last_clean":"2024-09-21T08:25:55.694089+0000","last_became_active":"2024-09-21T08:24:48.602176+0000","last_became_peered":"2024-09-21T08:24:48.602176+0000","last_unstale":"2024-09-21T08:25:55.694089+0000","last_undegraded":"2024-09-21T08:25:55.694089+0000","last_fullsized":"2024-09-21T08:25:55.694089+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":222,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617321+0000","last_change":"2024-09-21T08:25:46.719639+0000","last_active":"2024-09-21T08:25:55.617321+0000","last_peered":"2024-09-21T08:25:55.617321+0000","last_clean":"2024-09-21T08:25:55.617321+0000","last_became_active":"2024-09-21T08:24:56.505134+0000","last_became_peered":"2024-09-21T08:24:56.505134+0000","last_unstale":"2024-09-21T08:25:55.617321+0000","last_undegraded":"2024-09-21T08:25:55.617321+0000","last_fullsized":"2024-09-21T08:25:55.617321+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:46.719545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:46.719545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:51.390395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":28,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619532+0000","last_change":"2024-09-21T08:25:48.896617+0000","last_active":"2024-09-21T08:25:55.619532+0000","last_peered":"2024-09-21T08:25:55.619532+0000","last_clean":"2024-09-21T08:25:55.619532+0000","last_became_active":"2024-09-21T08:25:48.896076+0000","last_became_peered":"2024-09-21T08:25:48.896076+0000","last_unstale":"2024-09-21T08:25:55.619532+0000","last_undegraded":"2024-09-21T08:25:55.619532+0000","last_fullsized":"2024-09-21T08:25:55.619532+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759208+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":204,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.696267+0000","last_change":"2024-09-21T08:24:53.746385+0000","last_active":"2024-09-21T08:25:55.696267+0000","last_peered":"2024-09-21T08:25:55.696267+0000","last_clean":"2024-09-21T08:25:55.696267+0000","last_became_active":"2024-09-21T08:24:38.539845+0000","last_became_peered":"2024-09-21T08:24:38.539845+0000","last_unstale":"2024-09-21T08:25:55.696267+0000","last_undegraded":"2024-09-21T08:25:55.696267+0000","last_fullsized":"2024-09-21T08:25:55.696267+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.746295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.746295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.585736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":248,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694188+0000","last_change":"2024-09-21T08:24:56.529301+0000","last_active":"2024-09-21T08:25:55.694188+0000","last_peered":"2024-09-21T08:25:55.694188+0000","last_clean":"2024-09-21T08:25:55.694188+0000","last_became_active":"2024-09-21T08:24:56.499937+0000","last_became_peered":"2024-09-21T08:24:56.499937+0000","last_unstale":"2024-09-21T08:25:55.694188+0000","last_undegraded":"2024-09-21T08:25:55.694188+0000","last_fullsized":"2024-09-21T08:25:55.694188+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":215,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693876+0000","last_change":"2024-09-21T08:25:48.889025+0000","last_active":"2024-09-21T08:25:55.693876+0000","last_peered":"2024-09-21T08:25:55.693876+0000","last_clean":"2024-09-21T08:25:55.693876+0000","last_became_active":"2024-09-21T08:25:48.888733+0000","last_became_peered":"2024-09-21T08:25:48.888733+0000","last_unstale":"2024-09-21T08:25:55.693876+0000","last_undegraded":"2024-09-21T08:25:55.693876+0000","last_fullsized":"2024-09-21T08:25:55.693876+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"156'11","reported_seq":207,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619002+0000","last_change":"2024-09-21T08:25:48.958643+0000","last_active":"2024-09-21T08:25:55.619002+0000","last_peered":"2024-09-21T08:25:55.619002+0000","last_clean":"2024-09-21T08:25:55.619002+0000","last_became_active":"2024-09-21T08:25:48.896000+0000","last_became_peered":"2024-09-21T08:25:48.896000+0000","last_unstale":"2024-09-21T08:25:55.619002+0000","last_undegraded":"2024-09-21T08:25:55.619002+0000","last_fullsized":"2024-09-21T08:25:55.619002+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:16.416651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":175,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693738+0000","last_change":"2024-09-21T08:25:39.989977+0000","last_active":"2024-09-21T08:25:55.693738+0000","last_peered":"2024-09-21T08:25:55.693738+0000","last_clean":"2024-09-21T08:25:55.693738+0000","last_became_active":"2024-09-21T08:24:38.542854+0000","last_became_peered":"2024-09-21T08:24:38.542854+0000","last_unstale":"2024-09-21T08:25:55.693738+0000","last_undegraded":"2024-09-21T08:25:55.693738+0000","last_fullsized":"2024-09-21T08:25:55.693738+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.989838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.989838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:49.118031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"104'15","reported_seq":117,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619002+0000","last_change":"2024-09-21T08:25:48.935811+0000","last_active":"2024-09-21T08:25:55.619002+0000","last_peered":"2024-09-21T08:25:55.619002+0000","last_clean":"2024-09-21T08:25:55.619002+0000","last_became_active":"2024-09-21T08:25:48.894839+0000","last_became_peered":"2024-09-21T08:25:48.894839+0000","last_unstale":"2024-09-21T08:25:55.619002+0000","last_undegraded":"2024-09-21T08:25:55.619002+0000","last_fullsized":"2024-09-21T08:25:55.619002+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:03.068335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":236,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617641+0000","last_change":"2024-09-21T08:24:56.709528+0000","last_active":"2024-09-21T08:25:55.617641+0000","last_peered":"2024-09-21T08:25:55.617641+0000","last_clean":"2024-09-21T08:25:55.617641+0000","last_became_active":"2024-09-21T08:24:56.504665+0000","last_became_peered":"2024-09-21T08:24:56.504665+0000","last_unstale":"2024-09-21T08:25:55.617641+0000","last_undegraded":"2024-09-21T08:25:55.617641+0000","last_fullsized":"2024-09-21T08:25:55.617641+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:56.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.709437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.598671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"156'9","reported_seq":251,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694436+0000","last_change":"2024-09-21T08:25:48.931191+0000","last_active":"2024-09-21T08:25:55.694436+0000","last_peered":"2024-09-21T08:25:55.694436+0000","last_clean":"2024-09-21T08:25:55.694436+0000","last_became_active":"2024-09-21T08:25:48.908349+0000","last_became_peered":"2024-09-21T08:25:48.908349+0000","last_unstale":"2024-09-21T08:25:55.694436+0000","last_undegraded":"2024-09-21T08:25:55.694436+0000","last_fullsized":"2024-09-21T08:25:55.694436+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:31.979617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":196,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694913+0000","last_change":"2024-09-21T08:24:48.592895+0000","last_active":"2024-09-21T08:25:55.694913+0000","last_peered":"2024-09-21T08:25:55.694913+0000","last_clean":"2024-09-21T08:25:55.694913+0000","last_became_active":"2024-09-21T08:24:48.592280+0000","last_became_peered":"2024-09-21T08:24:48.592280+0000","last_unstale":"2024-09-21T08:25:55.694913+0000","last_undegraded":"2024-09-21T08:25:55.694913+0000","last_fullsized":"2024-09-21T08:25:55.694913+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":78,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694076+0000","last_change":"2024-09-21T08:25:48.979008+0000","last_active":"2024-09-21T08:25:55.694076+0000","last_peered":"2024-09-21T08:25:55.694076+0000","last_clean":"2024-09-21T08:25:55.694076+0000","last_became_active":"2024-09-21T08:25:48.914638+0000","last_became_peered":"2024-09-21T08:25:48.914638+0000","last_unstale":"2024-09-21T08:25:55.694076+0000","last_undegraded":"2024-09-21T08:25:55.694076+0000","last_fullsized":"2024-09-21T08:25:55.694076+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":171,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.619175+0000","last_change":"2024-09-21T08:24:56.504037+0000","last_active":"2024-09-21T08:25:55.619175+0000","last_peered":"2024-09-21T08:25:55.619175+0000","last_clean":"2024-09-21T08:25:55.619175+0000","last_became_active":"2024-09-21T08:24:56.503232+0000","last_became_peered":"2024-09-21T08:24:56.503232+0000","last_unstale":"2024-09-21T08:25:55.619175+0000","last_undegraded":"2024-09-21T08:25:55.619175+0000","last_fullsized":"2024-09-21T08:25:55.619175+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759492+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":83,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693658+0000","last_change":"2024-09-21T08:25:48.917229+0000","last_active":"2024-09-21T08:25:55.693658+0000","last_peered":"2024-09-21T08:25:55.693658+0000","last_clean":"2024-09-21T08:25:55.693658+0000","last_became_active":"2024-09-21T08:25:48.916802+0000","last_became_peered":"2024-09-21T08:25:48.916802+0000","last_unstale":"2024-09-21T08:25:55.693658+0000","last_undegraded":"2024-09-21T08:25:55.693658+0000","last_fullsized":"2024-09-21T08:25:55.693658+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":145,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694072+0000","last_change":"2024-09-21T08:24:48.600966+0000","last_active":"2024-09-21T08:25:55.694072+0000","last_peered":"2024-09-21T08:25:55.694072+0000","last_clean":"2024-09-21T08:25:55.694072+0000","last_became_active":"2024-09-21T08:24:48.600045+0000","last_became_peered":"2024-09-21T08:24:48.600045+0000","last_unstale":"2024-09-21T08:25:55.694072+0000","last_undegraded":"2024-09-21T08:25:55.694072+0000","last_fullsized":"2024-09-21T08:25:55.694072+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":71,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.696135+0000","last_change":"2024-09-21T08:25:48.953827+0000","last_active":"2024-09-21T08:25:55.696135+0000","last_peered":"2024-09-21T08:25:55.696135+0000","last_clean":"2024-09-21T08:25:55.696135+0000","last_became_active":"2024-09-21T08:25:48.910041+0000","last_became_peered":"2024-09-21T08:25:48.910041+0000","last_unstale":"2024-09-21T08:25:55.696135+0000","last_undegraded":"2024-09-21T08:25:55.696135+0000","last_fullsized":"2024-09-21T08:25:55.696135+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.932629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":220,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.618535+0000","last_change":"2024-09-21T08:24:56.505458+0000","last_active":"2024-09-21T08:25:55.618535+0000","last_peered":"2024-09-21T08:25:55.618535+0000","last_clean":"2024-09-21T08:25:55.618535+0000","last_became_active":"2024-09-21T08:24:56.504584+0000","last_became_peered":"2024-09-21T08:24:56.504584+0000","last_unstale":"2024-09-21T08:25:55.618535+0000","last_undegraded":"2024-09-21T08:25:55.618535+0000","last_fullsized":"2024-09-21T08:25:55.618535+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":151,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693651+0000","last_change":"2024-09-21T08:25:48.904228+0000","last_active":"2024-09-21T08:25:55.693651+0000","last_peered":"2024-09-21T08:25:55.693651+0000","last_clean":"2024-09-21T08:25:55.693651+0000","last_became_active":"2024-09-21T08:25:48.903934+0000","last_became_peered":"2024-09-21T08:25:48.903934+0000","last_unstale":"2024-09-21T08:25:55.693651+0000","last_undegraded":"2024-09-21T08:25:55.693651+0000","last_fullsized":"2024-09-21T08:25:55.693651+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":196,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.695233+0000","last_change":"2024-09-21T08:25:48.902778+0000","last_active":"2024-09-21T08:25:55.695233+0000","last_peered":"2024-09-21T08:25:55.695233+0000","last_clean":"2024-09-21T08:25:55.695233+0000","last_became_active":"2024-09-21T08:25:48.902435+0000","last_became_peered":"2024-09-21T08:25:48.902435+0000","last_unstale":"2024-09-21T08:25:55.695233+0000","last_undegraded":"2024-09-21T08:25:55.695233+0000","last_fullsized":"2024-09-21T08:25:55.695233+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":221,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617398+0000","last_change":"2024-09-21T08:24:56.531655+0000","last_active":"2024-09-21T08:25:55.617398+0000","last_peered":"2024-09-21T08:25:55.617398+0000","last_clean":"2024-09-21T08:25:55.617398+0000","last_became_active":"2024-09-21T08:24:56.503223+0000","last_became_peered":"2024-09-21T08:24:56.503223+0000","last_unstale":"2024-09-21T08:25:55.617398+0000","last_undegraded":"2024-09-21T08:25:55.617398+0000","last_fullsized":"2024-09-21T08:25:55.617398+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.478113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":210,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694504+0000","last_change":"2024-09-21T08:25:48.914674+0000","last_active":"2024-09-21T08:25:55.694504+0000","last_peered":"2024-09-21T08:25:55.694504+0000","last_clean":"2024-09-21T08:25:55.694504+0000","last_became_active":"2024-09-21T08:25:48.912523+0000","last_became_peered":"2024-09-21T08:25:48.912523+0000","last_unstale":"2024-09-21T08:25:55.694504+0000","last_undegraded":"2024-09-21T08:25:55.694504+0000","last_fullsized":"2024-09-21T08:25:55.694504+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":214,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617134+0000","last_change":"2024-09-21T08:24:58.688115+0000","last_active":"2024-09-21T08:25:55.617134+0000","last_peered":"2024-09-21T08:25:55.617134+0000","last_clean":"2024-09-21T08:25:55.617134+0000","last_became_active":"2024-09-21T08:24:56.503123+0000","last_became_peered":"2024-09-21T08:24:56.503123+0000","last_unstale":"2024-09-21T08:25:55.617134+0000","last_undegraded":"2024-09-21T08:25:55.617134+0000","last_fullsized":"2024-09-21T08:25:55.617134+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.688046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.688046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.364343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":151,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693353+0000","last_change":"2024-09-21T08:24:56.508632+0000","last_active":"2024-09-21T08:25:55.693353+0000","last_peered":"2024-09-21T08:25:55.693353+0000","last_clean":"2024-09-21T08:25:55.693353+0000","last_became_active":"2024-09-21T08:24:56.508323+0000","last_became_peered":"2024-09-21T08:24:56.508323+0000","last_unstale":"2024-09-21T08:25:55.693353+0000","last_undegraded":"2024-09-21T08:25:55.693353+0000","last_fullsized":"2024-09-21T08:25:55.693353+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":59,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.693301+0000","last_change":"2024-09-21T08:25:48.973819+0000","last_active":"2024-09-21T08:25:55.693301+0000","last_peered":"2024-09-21T08:25:55.693301+0000","last_clean":"2024-09-21T08:25:55.693301+0000","last_became_active":"2024-09-21T08:25:48.913344+0000","last_became_peered":"2024-09-21T08:25:48.913344+0000","last_unstale":"2024-09-21T08:25:55.693301+0000","last_undegraded":"2024-09-21T08:25:55.693301+0000","last_fullsized":"2024-09-21T08:25:55.693301+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":221,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694233+0000","last_change":"2024-09-21T08:25:48.907831+0000","last_active":"2024-09-21T08:25:55.694233+0000","last_peered":"2024-09-21T08:25:55.694233+0000","last_clean":"2024-09-21T08:25:55.694233+0000","last_became_active":"2024-09-21T08:25:48.907530+0000","last_became_peered":"2024-09-21T08:25:48.907530+0000","last_unstale":"2024-09-21T08:25:55.694233+0000","last_undegraded":"2024-09-21T08:25:55.694233+0000","last_fullsized":"2024-09-21T08:25:55.694233+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.352635+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":173,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617572+0000","last_change":"2024-09-21T08:25:48.899072+0000","last_active":"2024-09-21T08:25:55.617572+0000","last_peered":"2024-09-21T08:25:55.617572+0000","last_clean":"2024-09-21T08:25:55.617572+0000","last_became_active":"2024-09-21T08:25:48.898419+0000","last_became_peered":"2024-09-21T08:25:48.898419+0000","last_unstale":"2024-09-21T08:25:55.617572+0000","last_undegraded":"2024-09-21T08:25:55.617572+0000","last_fullsized":"2024-09-21T08:25:55.617572+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":207,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.617954+0000","last_change":"2024-09-21T08:25:48.896046+0000","last_active":"2024-09-21T08:25:55.617954+0000","last_peered":"2024-09-21T08:25:55.617954+0000","last_clean":"2024-09-21T08:25:55.617954+0000","last_became_active":"2024-09-21T08:25:48.895489+0000","last_became_peered":"2024-09-21T08:25:48.895489+0000","last_unstale":"2024-09-21T08:25:55.617954+0000","last_undegraded":"2024-09-21T08:25:55.617954+0000","last_fullsized":"2024-09-21T08:25:55.617954+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:55.759381+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":59,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694227+0000","last_change":"2024-09-21T08:25:48.913489+0000","last_active":"2024-09-21T08:25:55.694227+0000","last_peered":"2024-09-21T08:25:55.694227+0000","last_clean":"2024-09-21T08:25:55.694227+0000","last_became_active":"2024-09-21T08:25:48.912542+0000","last_became_peered":"2024-09-21T08:25:48.912542+0000","last_unstale":"2024-09-21T08:25:55.694227+0000","last_undegraded":"2024-09-21T08:25:55.694227+0000","last_fullsized":"2024-09-21T08:25:55.694227+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":160,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694494+0000","last_change":"2024-09-21T08:25:48.885761+0000","last_active":"2024-09-21T08:25:55.694494+0000","last_peered":"2024-09-21T08:25:55.694494+0000","last_clean":"2024-09-21T08:25:55.694494+0000","last_became_active":"2024-09-21T08:25:48.884347+0000","last_became_peered":"2024-09-21T08:25:48.884347+0000","last_unstale":"2024-09-21T08:25:55.694494+0000","last_undegraded":"2024-09-21T08:25:55.694494+0000","last_fullsized":"2024-09-21T08:25:55.694494+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":198,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694713+0000","last_change":"2024-09-21T08:25:48.942921+0000","last_active":"2024-09-21T08:25:55.694713+0000","last_peered":"2024-09-21T08:25:55.694713+0000","last_clean":"2024-09-21T08:25:55.694713+0000","last_became_active":"2024-09-21T08:25:48.908941+0000","last_became_peered":"2024-09-21T08:25:48.908941+0000","last_unstale":"2024-09-21T08:25:55.694713+0000","last_undegraded":"2024-09-21T08:25:55.694713+0000","last_fullsized":"2024-09-21T08:25:55.694713+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:25:57.274472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":203,"reported_epoch":162,"state":"active+clean","last_fresh":"2024-09-21T08:25:55.694432+0000","last_change":"2024-09-21T08:25:25.350265+0000","last_active":"2024-09-21T08:25:55.694432+0000","last_peered":"2024-09-21T08:25:55.694432+0000","last_clean":"2024-09-21T08:25:55.694432+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:25:55.694432+0000","last_undegraded":"2024-09-21T08:25:55.694432+0000","last_fullsized":"2024-09-21T08:25:55.694432+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.350197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.350197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:48.284282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":141709224,"num_objects":60,"num_object_clones":18,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":3,"num_read":878,"num_read_kb":742149,"num_write":773,"num_write_kb":290305,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":140023933,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":430571520,"data_stored":468931993,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1366,"ondisk_log_size":1366,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":675,"num_read_kb":432,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":5132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":19638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":8157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":138,"num_read_kb":133,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070498,"num_pgs":113,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116340,"kb_used_data":80984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93628812,"statfs":{"total":95995035648,"available":95875903488,"internally_reserved":0,"allocated":82927616,"data_stored":94116075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397544,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659179,"num_pgs":109,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":94736,"kb_used_data":63540,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93650416,"statfs":{"total":95995035648,"available":95898025984,"internally_reserved":0,"allocated":65064960,"data_stored":69194451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986225,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182453,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411387,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":131804,"kb_used_data":96456,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93613348,"statfs":{"total":95995035648,"available":95860068352,"internally_reserved":0,"allocated":98770944,"data_stored":112279680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574911,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738436,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":131760,"kb_used_data":96404,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93613392,"statfs":{"total":95995035648,"available":95860113408,"internally_reserved":0,"allocated":98717696,"data_stored":105537260,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":3262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":91627520,"data_stored":98989372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45531136,"data_stored":53470237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":92774400,"data_stored":103136158,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32387072,"data_stored":32188068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":59604992,"data_stored":64287686,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":80551936,"data_stored":88990057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:25:59.135 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj 6, currently 16 2024-09-21T08:25:59.135 INFO:tasks.rados.rados.0.smithi137.stdout:480: writing smithi13779221-34 from 611821 to 1312546 tid 1 2024-09-21T08:25:59.138 INFO:tasks.rados.rados.0.smithi137.stdout:480: writing smithi13779221-34 from 1879374 to 2662777 tid 2 2024-09-21T08:25:59.142 INFO:tasks.rados.rados.0.smithi137.stdout:480: writing smithi13779221-34 from 3341691 to 3411821 tid 3 2024-09-21T08:25:59.142 INFO:tasks.rados.rados.0.smithi137.stdout:481: snap_create 2024-09-21T08:25:59.142 INFO:tasks.rados.rados.0.smithi137.stdout:477: finishing write tid 2 to smithi13779221-38 2024-09-21T08:25:59.143 INFO:tasks.rados.rados.0.smithi137.stdout:477: finishing write tid 3 to smithi13779221-38 2024-09-21T08:25:59.143 INFO:tasks.rados.rados.0.smithi137.stdout:477: finishing write tid 4 to smithi13779221-38 2024-09-21T08:25:59.143 INFO:tasks.rados.rados.0.smithi137.stdout:477: finishing write tid 5 to smithi13779221-38 2024-09-21T08:25:59.143 INFO:tasks.rados.rados.0.smithi137.stdout:477: oid 38 updating version 0 to 214 2024-09-21T08:25:59.143 INFO:tasks.rados.rados.0.smithi137.stdout:477: oid 38 updating version 214 to 215 2024-09-21T08:25:59.143 INFO:tasks.rados.rados.0.smithi137.stdout:477: oid 38 version 215 is already newer than 212 2024-09-21T08:25:59.143 INFO:tasks.rados.rados.0.smithi137.stdout:477: oid 38 version 215 is already newer than 213 2024-09-21T08:25:59.143 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 215 (ObjNum 202 snap 40 seq_num 202) dirty exists 2024-09-21T08:25:59.143 INFO:tasks.rados.rados.0.smithi137.stdout:477: left oid 38 (ObjNum 202 snap 40 seq_num 202) 2024-09-21T08:25:59.143 INFO:tasks.rados.rados.0.smithi137.stdout:479: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:25:59.179 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:25:59.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:59.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:59.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:59.214 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:480: finishing write tid 1 to smithi13779221-34 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:480: finishing write tid 2 to smithi13779221-34 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:480: finishing write tid 3 to smithi13779221-34 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:480: finishing write tid 4 to smithi13779221-34 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:480: finishing write tid 5 to smithi13779221-34 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:480: oid 34 updating version 0 to 237 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:480: oid 34 version 237 is already newer than 236 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:480: oid 34 updating version 237 to 239 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:480: oid 34 version 239 is already newer than 238 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 239 (ObjNum 203 snap 40 seq_num 203) dirty exists 2024-09-21T08:25:59.278 INFO:tasks.rados.rados.0.smithi137.stdout:480: left oid 34 (ObjNum 203 snap 40 seq_num 203) 2024-09-21T08:25:59.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:59.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:59.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:59.337 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:59.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:25:59.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:59.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:59.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:25:59.551 INFO:tasks.rados.rados.0.smithi137.stdout:477: done (4 left) 2024-09-21T08:25:59.551 INFO:tasks.rados.rados.0.smithi137.stdout:478: done (3 left) 2024-09-21T08:25:59.551 INFO:tasks.rados.rados.0.smithi137.stdout:479: done (2 left) 2024-09-21T08:25:59.551 INFO:tasks.rados.rados.0.smithi137.stdout:480: done (1 left) 2024-09-21T08:25:59.551 INFO:tasks.rados.rados.0.smithi137.stdout:481: done (0 left) 2024-09-21T08:25:59.551 INFO:tasks.rados.rados.0.smithi137.stdout:482: delete oid 24 current snap is 41 2024-09-21T08:25:59.557 INFO:tasks.rados.rados.0.smithi137.stdout:482: done (0 left) 2024-09-21T08:25:59.557 INFO:tasks.rados.rados.0.smithi137.stdout:483: write oid 5 current snap is 41 2024-09-21T08:25:59.557 INFO:tasks.rados.rados.0.smithi137.stdout:483: seq_num 205 ranges {645435=451223,1879985=787072,3280184=165251} 2024-09-21T08:25:59.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:25:59.563 INFO:tasks.rados.rados.0.smithi137.stdout:483: writing smithi13779221-5 from 645435 to 1096658 tid 1 2024-09-21T08:25:59.572 INFO:tasks.rados.rados.0.smithi137.stdout:483: writing smithi13779221-5 from 1879985 to 2667057 tid 2 2024-09-21T08:25:59.576 INFO:tasks.rados.rados.0.smithi137.stdout:483: writing smithi13779221-5 from 3280184 to 3445435 tid 3 2024-09-21T08:25:59.576 INFO:tasks.rados.rados.0.smithi137.stdout:484: read oid 17 snap -1 2024-09-21T08:25:59.577 INFO:tasks.rados.rados.0.smithi137.stdout:484: expect (ObjNum 192 snap 39 seq_num 192) 2024-09-21T08:25:59.577 INFO:tasks.rados.rados.0.smithi137.stdout:485: read oid 6 snap -1 2024-09-21T08:25:59.577 INFO:tasks.rados.rados.0.smithi137.stdout:485: expect (ObjNum 191 snap 39 seq_num 191) 2024-09-21T08:25:59.577 INFO:tasks.rados.rados.0.smithi137.stdout:486: read oid 48 snap -1 2024-09-21T08:25:59.577 INFO:tasks.rados.rados.0.smithi137.stdout:486: expect (ObjNum 189 snap 38 seq_num 189) 2024-09-21T08:25:59.577 INFO:tasks.rados.rados.0.smithi137.stdout:487: write oid 10 current snap is 41 2024-09-21T08:25:59.577 INFO:tasks.rados.rados.0.smithi137.stdout:483: finishing write tid 1 to smithi13779221-5 2024-09-21T08:25:59.577 INFO:tasks.rados.rados.0.smithi137.stdout:487: seq_num 206 ranges {662242=642825,2054820=530730,3307607=154635} 2024-09-21T08:25:59.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:59.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:59.584 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:25:59.584 INFO:tasks.rados.rados.0.smithi137.stdout:487: writing smithi13779221-10 from 662242 to 1305067 tid 1 2024-09-21T08:25:59.591 INFO:tasks.rados.rados.0.smithi137.stdout:487: writing smithi13779221-10 from 2054820 to 2585550 tid 2 2024-09-21T08:25:59.597 INFO:tasks.rados.rados.0.smithi137.stdout:487: writing smithi13779221-10 from 3307607 to 3462242 tid 3 2024-09-21T08:25:59.597 INFO:tasks.rados.rados.0.smithi137.stdout:488: read oid 43 snap 39 2024-09-21T08:25:59.597 INFO:tasks.rados.rados.0.smithi137.stdout:488: expect deleted 2024-09-21T08:25:59.597 INFO:tasks.rados.rados.0.smithi137.stdout:489: read oid 1 snap -1 2024-09-21T08:25:59.597 INFO:tasks.rados.rados.0.smithi137.stdout:489: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-21T08:25:59.597 INFO:tasks.rados.rados.0.smithi137.stdout:490: rollback oid 3 current snap is 41 2024-09-21T08:25:59.597 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 3 to 39 2024-09-21T08:25:59.597 INFO:tasks.rados.rados.0.smithi137.stdout:483: finishing write tid 2 to smithi13779221-5 2024-09-21T08:25:59.597 INFO:tasks.rados.rados.0.smithi137.stdout:484: expect (ObjNum 192 snap 39 seq_num 192) 2024-09-21T08:25:59.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:25:59.707 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:25:59.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:59.708 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:25:59.808 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:25:59.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:59 smithi137 ceph-mon[29634]: pgmap v152: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 464 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 17 op/s 2024-09-21T08:25:59.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:59 smithi137 ceph-mon[29634]: from='client.15060 v1:172.21.15.137:0/2920048726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:25:59.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:25:59 smithi137 ceph-mon[29634]: osdmap e165: 8 total, 8 up, 4 in 2024-09-21T08:25:59.882 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:59 smithi137 ceph-mon[41044]: pgmap v152: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 464 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 17 op/s 2024-09-21T08:25:59.882 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:59 smithi137 ceph-mon[41044]: from='client.15060 v1:172.21.15.137:0/2920048726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:25:59.882 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:25:59 smithi137 ceph-mon[41044]: osdmap e165: 8 total, 8 up, 4 in 2024-09-21T08:25:59.883 INFO:tasks.rados.rados.0.smithi137.stdout:484: done (7 left) 2024-09-21T08:25:59.883 INFO:tasks.rados.rados.0.smithi137.stdout:491: snap_remove snap 40 2024-09-21T08:25:59.904 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:25:59.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:25:59.905 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:00.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:59 smithi165 ceph-mon[32893]: pgmap v152: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 464 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 4.5 MiB/s wr, 17 op/s 2024-09-21T08:26:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:59 smithi165 ceph-mon[32893]: from='client.15060 v1:172.21.15.137:0/2920048726' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:26:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:25:59 smithi165 ceph-mon[32893]: osdmap e165: 8 total, 8 up, 4 in 2024-09-21T08:26:00.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:00.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:00.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:00.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:00.152 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:00.152 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:25:59 smithi137 podman[93501]: 2024-09-21 08:25:59.820797026 +0000 UTC m=+0.645088936 container died 3a02837850704e13e22adccfb498f3c7f1f618c743f0abfc726946d3fc9c6b38 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:00.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:00.152 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:00.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:00.274 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:00.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:00.275 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:00.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:00.398 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:00.398 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:00.399 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:00.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:00.517 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:00 smithi137 podman[93501]: 2024-09-21 08:26:00.327543702 +0000 UTC m=+1.151835651 container cleanup 3a02837850704e13e22adccfb498f3c7f1f618c743f0abfc726946d3fc9c6b38 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:00.518 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:00 smithi137 bash[93501]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a 2024-09-21T08:26:00.521 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:00.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:00.522 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:00.556 INFO:tasks.rados.rados.0.smithi137.stdout:491: done (7 left) 2024-09-21T08:26:00.556 INFO:tasks.rados.rados.0.smithi137.stdout:492: write (excl) oid 31 current snap is 41 2024-09-21T08:26:00.556 INFO:tasks.rados.rados.0.smithi137.stdout:492: seq_num 207 ranges {679049=718074,2113302=674388,3335030=144019} 2024-09-21T08:26:00.564 INFO:tasks.rados.rados.0.smithi137.stdout:492: writing smithi13779221-31 from 679049 to 1397123 tid 1 2024-09-21T08:26:00.565 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:00.572 INFO:tasks.rados.rados.0.smithi137.stdout:492: writing smithi13779221-31 from 2113302 to 2787690 tid 2 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:492: writing smithi13779221-31 from 3335030 to 3479049 tid 3 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:493: delete oid 24 current snap is 41 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:483: finishing write tid 3 to smithi13779221-5 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:483: finishing write tid 4 to smithi13779221-5 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:483: finishing write tid 5 to smithi13779221-5 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:483: oid 5 updating version 0 to 171 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:483: oid 5 version 171 is already newer than 170 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:483: oid 5 updating version 171 to 172 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:483: oid 5 version 172 is already newer than 169 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 172 (ObjNum 205 snap 41 seq_num 205) dirty exists 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:483: left oid 5 (ObjNum 205 snap 41 seq_num 205) 2024-09-21T08:26:00.577 INFO:tasks.rados.rados.0.smithi137.stdout:486: expect (ObjNum 189 snap 38 seq_num 189) 2024-09-21T08:26:00.623 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:00.645 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:00.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:00.645 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:00.746 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:00.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:00.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:00.768 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:00.780 INFO:tasks.rados.rados.0.smithi137.stdout:487: finishing write tid 1 to smithi13779221-10 2024-09-21T08:26:00.780 INFO:tasks.rados.rados.0.smithi137.stdout:485: expect (ObjNum 191 snap 39 seq_num 191) 2024-09-21T08:26:00.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:00 smithi137 ceph-mon[29634]: osdmap e166: 8 total, 8 up, 4 in 2024-09-21T08:26:00.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:00.811 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:00.811 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:00 smithi137 ceph-mon[41044]: osdmap e166: 8 total, 8 up, 4 in 2024-09-21T08:26:00.811 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:00.811 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:00.812 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:00 smithi137 podman[93583]: 2024-09-21 08:26:00.545654356 +0000 UTC m=+0.728765505 container remove 3a02837850704e13e22adccfb498f3c7f1f618c743f0abfc726946d3fc9c6b38 (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:00.812 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:00 smithi137 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service: Main process exited, code=exited, status=143/n/a 2024-09-21T08:26:00.812 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:26:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:26:00.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:00.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:00.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:00.892 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:00.954 INFO:tasks.rados.rados.0.smithi137.stdout:487: finishing write tid 2 to smithi13779221-10 2024-09-21T08:26:00.954 INFO:tasks.rados.rados.0.smithi137.stdout:487: finishing write tid 3 to smithi13779221-10 2024-09-21T08:26:00.954 INFO:tasks.rados.rados.0.smithi137.stdout:487: finishing write tid 4 to smithi13779221-10 2024-09-21T08:26:00.954 INFO:tasks.rados.rados.0.smithi137.stdout:487: finishing write tid 5 to smithi13779221-10 2024-09-21T08:26:00.954 INFO:tasks.rados.rados.0.smithi137.stdout:487: oid 10 updating version 0 to 175 2024-09-21T08:26:00.954 INFO:tasks.rados.rados.0.smithi137.stdout:487: oid 10 version 175 is already newer than 174 2024-09-21T08:26:00.955 INFO:tasks.rados.rados.0.smithi137.stdout:487: oid 10 version 175 is already newer than 173 2024-09-21T08:26:00.955 INFO:tasks.rados.rados.0.smithi137.stdout:487: oid 10 updating version 175 to 176 2024-09-21T08:26:00.955 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 176 (ObjNum 206 snap 41 seq_num 206) dirty exists 2024-09-21T08:26:00.955 INFO:tasks.rados.rados.0.smithi137.stdout:487: left oid 10 (ObjNum 206 snap 41 seq_num 206) 2024-09-21T08:26:00.955 INFO:tasks.rados.rados.0.smithi137.stdout:489: expect (ObjNum 22 snap 0 seq_num 22) 2024-09-21T08:26:00.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:00 smithi165 ceph-mon[32893]: osdmap e166: 8 total, 8 up, 4 in 2024-09-21T08:26:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:01.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:01.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:01.016 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:01.032 INFO:tasks.rados.rados.0.smithi137.stdout:490: finishing rollback tid 0 to smithi13779221-3 2024-09-21T08:26:01.032 INFO:tasks.rados.rados.0.smithi137.stdout:490: finishing rollback tid 1 to smithi13779221-3 2024-09-21T08:26:01.032 INFO:tasks.rados.rados.0.smithi137.stdout:490: finishing rollback tid 2 to smithi13779221-3 2024-09-21T08:26:01.032 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 139 (ObjNum 171 snap 34 seq_num 171) dirty exists 2024-09-21T08:26:01.032 INFO:tasks.rados.rados.0.smithi137.stdout:492: finishing write tid 1 to smithi13779221-31 2024-09-21T08:26:01.032 INFO:tasks.rados.rados.0.smithi137.stdout:492: finishing write tid 2 to smithi13779221-31 2024-09-21T08:26:01.032 INFO:tasks.rados.rados.0.smithi137.stdout:492: finishing write tid 3 to smithi13779221-31 2024-09-21T08:26:01.032 INFO:tasks.rados.rados.0.smithi137.stdout:492: finishing write tid 4 to smithi13779221-31 2024-09-21T08:26:01.032 INFO:tasks.rados.rados.0.smithi137.stdout:492: finishing write tid 5 to smithi13779221-31 2024-09-21T08:26:01.032 INFO:tasks.rados.rados.0.smithi137.stdout:492: oid 31 updating version 0 to 146 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:492: oid 31 version 146 is already newer than 143 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:492: oid 31 version 146 is already newer than 144 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:492: oid 31 version 146 is already newer than 145 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 146 (ObjNum 207 snap 41 seq_num 207) dirty exists 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:492: left oid 31 (ObjNum 207 snap 41 seq_num 207) 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:483: done (8 left) 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:485: done (7 left) 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:486: done (6 left) 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:487: done (5 left) 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:488: done (4 left) 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:489: done (3 left) 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:490: done (2 left) 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:492: done (1 left) 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:493: done (0 left) 2024-09-21T08:26:01.033 INFO:tasks.rados.rados.0.smithi137.stdout:494: snap_remove snap 39 2024-09-21T08:26:01.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:01.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:01.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:01.139 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:01.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:01.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:01.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:01.261 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:01.362 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:01.383 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:01.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:01.384 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:01.439 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:01.439 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:01.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:01.499 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:26:01.505 INFO:tasks.rados.rados.0.smithi137.stdout:494: done (0 left) 2024-09-21T08:26:01.505 INFO:tasks.rados.rados.0.smithi137.stdout:495: delete oid 11 current snap is 41 2024-09-21T08:26:01.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:01.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:01.507 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:01.508 INFO:tasks.rados.rados.0.smithi137.stdout:495: done (0 left) 2024-09-21T08:26:01.508 INFO:tasks.rados.rados.0.smithi137.stdout:496: snap_create 2024-09-21T08:26:01.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:01.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:01.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:01.631 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:01.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:26:01.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[29634]: pgmap v155: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 464 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:01.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:26:01.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:26:01.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[29634]: osdmap e167: 8 total, 8 up, 4 in 2024-09-21T08:26:01.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4258556222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:01.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:01.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4258556222' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:26:01.715 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[29634]: osdmap e168: 8 total, 8 up, 4 in 2024-09-21T08:26:01.715 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:01 smithi137 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service: Failed with result 'exit-code'. 2024-09-21T08:26:01.716 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:01 smithi137 systemd[1]: Stopped Ceph node-exporter.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:26:01.716 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:01 smithi137 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@node-exporter.a.service: Consumed 1.942s CPU time. 2024-09-21T08:26:01.716 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:01 smithi137 systemd[1]: Starting Ceph node-exporter.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:26:01.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[41044]: pgmap v155: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 464 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:01.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:26:01.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:26:01.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[41044]: osdmap e167: 8 total, 8 up, 4 in 2024-09-21T08:26:01.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4258556222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:01.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:01.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4258556222' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:26:01.716 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:01 smithi137 ceph-mon[41044]: osdmap e168: 8 total, 8 up, 4 in 2024-09-21T08:26:01.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:01.752 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:01.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:01.753 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:01.854 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:01.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:01.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:01.877 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:01.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:01.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:02.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:02.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:02.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:26:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:26:02.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:01 smithi165 ceph-mon[32893]: pgmap v155: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 464 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:02.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:01 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:26:02.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:01 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:26:02.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:01 smithi165 ceph-mon[32893]: osdmap e167: 8 total, 8 up, 4 in 2024-09-21T08:26:02.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:01 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4258556222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:02.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:01 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:02.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:01 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4258556222' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:26:02.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:01 smithi165 ceph-mon[32893]: osdmap e168: 8 total, 8 up, 4 in 2024-09-21T08:26:02.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:02.123 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:02.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:02.124 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:02.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:02.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:02.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:02.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:02.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:02.370 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:02.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:02.371 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:02.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:02.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:02.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:02.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:02.506 INFO:tasks.rados.rados.0.smithi137.stdout:496: done (0 left) 2024-09-21T08:26:02.506 INFO:tasks.rados.rados.0.smithi137.stdout:497: snap_create 2024-09-21T08:26:02.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:02.616 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:02.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:02.617 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:02.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:02.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:02.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:02.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:02.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:02.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:02.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:02.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:02.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T08:26:02.911 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:02.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:02.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:02.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:02.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:03.090 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:03.111 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:03.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:03.112 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:03.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:03.234 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:03.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:03.235 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:03.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:03.356 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:03.356 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:03.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:03.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:03.357 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:03.375 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:03.375 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:03.395 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:03.395 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:03.397 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:03.414 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:03.414 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:03.434 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:03.434 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:03.451 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:03.451 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:03.459 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:03.468 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:03.468 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:03.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:03.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:03.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:03.484 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:03.485 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:03.509 INFO:tasks.rados.rados.0.smithi137.stdout:497: done (0 left) 2024-09-21T08:26:03.509 INFO:tasks.rados.rados.0.smithi137.stdout:498: write (excl) oid 16 current snap is 43 2024-09-21T08:26:03.509 INFO:tasks.rados.rados.0.smithi137.stdout:498: seq_num 210 ranges {729470=543821,1772395=538068,2900946=403005} 2024-09-21T08:26:03.517 INFO:tasks.rados.rados.0.smithi137.stdout:498: writing smithi13779221-16 from 729470 to 1273291 tid 1 2024-09-21T08:26:03.523 INFO:tasks.rados.rados.0.smithi137.stdout:498: writing smithi13779221-16 from 1772395 to 2310463 tid 2 2024-09-21T08:26:03.528 INFO:tasks.rados.rados.0.smithi137.stdout:498: writing smithi13779221-16 from 2900946 to 3303951 tid 3 2024-09-21T08:26:03.529 INFO:tasks.rados.rados.0.smithi137.stdout:499: write oid 40 current snap is 43 2024-09-21T08:26:03.529 INFO:tasks.rados.rados.0.smithi137.stdout:498: finishing write tid 1 to smithi13779221-16 2024-09-21T08:26:03.529 INFO:tasks.rados.rados.0.smithi137.stdout:499: seq_num 211 ranges {746277=619070,1830877=681726,2928369=617908} 2024-09-21T08:26:03.536 INFO:tasks.rados.rados.0.smithi137.stdout:499: writing smithi13779221-40 from 746277 to 1365347 tid 1 2024-09-21T08:26:03.542 INFO:tasks.rados.rados.0.smithi137.stdout:499: writing smithi13779221-40 from 1830877 to 2512603 tid 2 2024-09-21T08:26:03.548 INFO:tasks.rados.rados.0.smithi137.stdout:499: writing smithi13779221-40 from 2928369 to 3546277 tid 3 2024-09-21T08:26:03.548 INFO:tasks.rados.rados.0.smithi137.stdout:500: snap_remove snap 42 2024-09-21T08:26:03.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:03.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:03.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:03.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:03.652 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:03.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[41044]: pgmap v158: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 451 MiB used, 357 GiB / 358 GiB avail; 40 MiB/s rd, 5.6 MiB/s wr, 56 op/s 2024-09-21T08:26:03.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[41044]: osdmap e169: 8 total, 8 up, 4 in 2024-09-21T08:26:03.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:26:03.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:26:03.653 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1073325488' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:03.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:03.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[29634]: pgmap v158: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 451 MiB used, 357 GiB / 358 GiB avail; 40 MiB/s rd, 5.6 MiB/s wr, 56 op/s 2024-09-21T08:26:03.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[29634]: osdmap e169: 8 total, 8 up, 4 in 2024-09-21T08:26:03.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:26:03.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:26:03.654 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1073325488' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:03.654 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 podman[94196]: 2024-09-21 08:26:03.415145124 +0000 UTC m=+0.016619142 image pull 0da6a335fe1356545476b749c68f022c897de3a2139e8f0054f6937349ee2b83 quay.io/prometheus/node-exporter:v1.5.0 2024-09-21T08:26:03.654 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 podman[94196]: 2024-09-21 08:26:03.523698881 +0000 UTC m=+0.125172905 container create a7a2e764bc7c09ba45b758926ab70ca080c0ba9958375bc838e79c9201a30fba (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:03.708 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:03.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:03.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:03.730 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:03 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:03 smithi165 ceph-mon[32893]: pgmap v158: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 451 MiB used, 357 GiB / 358 GiB avail; 40 MiB/s rd, 5.6 MiB/s wr, 56 op/s 2024-09-21T08:26:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:03 smithi165 ceph-mon[32893]: osdmap e169: 8 total, 8 up, 4 in 2024-09-21T08:26:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:03 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:26:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:03 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:26:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:03 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1073325488' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:03.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:03.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:03.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:03.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:03.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:03.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:03.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:03.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:04.018 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 podman[94196]: 2024-09-21 08:26:03.979169069 +0000 UTC m=+0.580643092 container init a7a2e764bc7c09ba45b758926ab70ca080c0ba9958375bc838e79c9201a30fba (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:04.018 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 podman[94196]: 2024-09-21 08:26:03.984746819 +0000 UTC m=+0.586220835 container start a7a2e764bc7c09ba45b758926ab70ca080c0ba9958375bc838e79c9201a30fba (image=quay.io/prometheus/node-exporter:v1.5.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:04.018 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.988Z caller=node_exporter.go:180 level=info msg="Starting node_exporter" version="(version=1.5.0, branch=HEAD, revision=1b48970ffcf5630534fb00bb0687d73c66d1c959)" 2024-09-21T08:26:04.018 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.988Z caller=node_exporter.go:181 level=info msg="Build context" build_context="(go=go1.19.3, user=root@6e7732a7b81b, date=20221129-18:59:09)" 2024-09-21T08:26:04.018 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.989Z caller=diskstats_common.go:111 level=info collector=diskstats msg="Parsed flag --collector.diskstats.device-exclude" flag=^(ram|loop|fd|(h|s|v|xv)d[a-z]|nvme\d+n\d+p)\d+$ 2024-09-21T08:26:04.018 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.989Z caller=diskstats_linux.go:264 level=error collector=diskstats msg="Failed to open directory, disabling udev device properties" path=/run/udev/data 2024-09-21T08:26:04.018 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.989Z caller=filesystem_common.go:111 level=info collector=filesystem msg="Parsed flag --collector.filesystem.mount-points-exclude" flag=^/(dev|proc|run/credentials/.+|sys|var/lib/docker/.+|var/lib/containers/storage/.+)($|/) 2024-09-21T08:26:04.019 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.989Z caller=filesystem_common.go:113 level=info collector=filesystem msg="Parsed flag --collector.filesystem.fs-types-exclude" flag=^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$ 2024-09-21T08:26:04.019 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.989Z caller=node_exporter.go:110 level=info msg="Enabled collectors" 2024-09-21T08:26:04.019 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.989Z caller=node_exporter.go:117 level=info collector=arp 2024-09-21T08:26:04.019 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.989Z caller=node_exporter.go:117 level=info collector=bcache 2024-09-21T08:26:04.019 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.989Z caller=node_exporter.go:117 level=info collector=bonding 2024-09-21T08:26:04.019 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=btrfs 2024-09-21T08:26:04.019 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=conntrack 2024-09-21T08:26:04.019 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=cpu 2024-09-21T08:26:04.019 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=cpufreq 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=diskstats 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=dmi 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=edac 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=entropy 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=fibrechannel 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=filefd 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=filesystem 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=hwmon 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=infiniband 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=ipvs 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=loadavg 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=mdadm 2024-09-21T08:26:04.020 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=meminfo 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=netclass 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=netdev 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=netstat 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=nfs 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=nfsd 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=nvme 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=os 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=powersupplyclass 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=pressure 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=rapl 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=schedstat 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=selinux 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=sockstat 2024-09-21T08:26:04.021 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=softnet 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=stat 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=tapestats 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=textfile 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=thermal_zone 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=time 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=udp_queues 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=uname 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=vmstat 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=xfs 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=node_exporter.go:117 level=info collector=zfs 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=tls_config.go:232 level=info msg="Listening on" address=[::]:9100 2024-09-21T08:26:04.022 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-node-exporter-a[94243]: ts=2024-09-21T08:26:03.990Z caller=tls_config.go:235 level=info msg="TLS is disabled." http2=false address=[::]:9100 2024-09-21T08:26:04.079 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:04.100 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:04.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:04.101 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:04.136 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 3, 6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:26:04.137 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:26:04.137 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-21T08:26:04.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:26:04.201 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:26:04.201 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:26:04.201 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:26:04.201 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:26:04.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:04.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:04.224 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:04.225 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:04.325 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:04.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:04.348 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:04.348 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:04.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:04.461 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:26:04.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:04.471 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:04.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:04.516 INFO:tasks.rados.rados.0.smithi137.stdout:500: done (2 left) 2024-09-21T08:26:04.516 INFO:tasks.rados.rados.0.smithi137.stdout:501: snap_remove snap 41 2024-09-21T08:26:04.572 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:04.593 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:04.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:04.594 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:04.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:04.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:04.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:04.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:04.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:04 smithi165 ceph-mon[32893]: 2.2 scrub starts 2024-09-21T08:26:04.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:04 smithi165 ceph-mon[32893]: 2.2 scrub ok 2024-09-21T08:26:04.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:04 smithi165 ceph-mon[32893]: osdmap e170: 8 total, 8 up, 4 in 2024-09-21T08:26:04.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:04 smithi137 ceph-mon[29634]: 2.2 scrub starts 2024-09-21T08:26:04.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:04 smithi137 ceph-mon[29634]: 2.2 scrub ok 2024-09-21T08:26:04.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:04 smithi137 ceph-mon[29634]: osdmap e170: 8 total, 8 up, 4 in 2024-09-21T08:26:04.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:04 smithi137 ceph-mon[41044]: 2.2 scrub starts 2024-09-21T08:26:04.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:04 smithi137 ceph-mon[41044]: 2.2 scrub ok 2024-09-21T08:26:04.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:04 smithi137 ceph-mon[41044]: osdmap e170: 8 total, 8 up, 4 in 2024-09-21T08:26:04.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:04 smithi137 bash[94196]: a7a2e764bc7c09ba45b758926ab70ca080c0ba9958375bc838e79c9201a30fba 2024-09-21T08:26:04.768 INFO:journalctl@ceph.node-exporter.a.smithi137.stdout:Sep 21 08:26:04 smithi137 systemd[1]: Started Ceph node-exporter.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:26:04.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:04.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:04.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:04.843 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:04.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T08:26:04.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:04.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:04.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:04.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:05.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:05.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:05.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:05.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:05.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:05.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:05.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:05.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:05.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:05.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:05.337 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:05.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:05.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:05.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:05.460 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:05.511 INFO:journalctl@ceph.alertmanager.a.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-alertmanager-a[92623]: ts=2024-09-21T08:26:05.440Z caller=cluster.go:698 level=info component=cluster msg="gossip settled; proceeding" elapsed=10.002988599s 2024-09-21T08:26:05.517 INFO:tasks.rados.rados.0.smithi137.stdout:501: done (2 left) 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:502: copy_from oid 35 from oid 15 current snap is 43 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:498: finishing write tid 2 to smithi13779221-16 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:498: finishing write tid 3 to smithi13779221-16 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:498: finishing write tid 4 to smithi13779221-16 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:498: finishing write tid 5 to smithi13779221-16 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:498: oid 16 updating version 0 to 100 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:498: oid 16 updating version 100 to 103 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:498: oid 16 version 103 is already newer than 101 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:498: oid 16 version 103 is already newer than 102 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 103 (ObjNum 210 snap 43 seq_num 210) dirty exists 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:498: left oid 16 (ObjNum 210 snap 43 seq_num 210) 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:499: finishing write tid 1 to smithi13779221-40 2024-09-21T08:26:05.518 INFO:tasks.rados.rados.0.smithi137.stdout:499: finishing write tid 2 to smithi13779221-40 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:499: finishing write tid 3 to smithi13779221-40 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:499: finishing write tid 4 to smithi13779221-40 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:499: finishing write tid 5 to smithi13779221-40 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:499: oid 40 updating version 0 to 82 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:499: oid 40 updating version 82 to 83 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:499: oid 40 updating version 83 to 84 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:499: oid 40 updating version 84 to 85 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 85 (ObjNum 211 snap 43 seq_num 211) dirty exists 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:499: left oid 40 (ObjNum 211 snap 43 seq_num 211) 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:498: done (2 left) 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:499: done (1 left) 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:503: write oid 31 current snap is 43 2024-09-21T08:26:05.519 INFO:tasks.rados.rados.0.smithi137.stdout:503: seq_num 212 ranges {763084=694319,1889359=541737,2955792=541902} 2024-09-21T08:26:05.526 INFO:tasks.rados.rados.0.smithi137.stdout:503: writing smithi13779221-31 from 763084 to 1457403 tid 1 2024-09-21T08:26:05.532 INFO:tasks.rados.rados.0.smithi137.stdout:503: writing smithi13779221-31 from 1889359 to 2431096 tid 2 2024-09-21T08:26:05.538 INFO:tasks.rados.rados.0.smithi137.stdout:503: writing smithi13779221-31 from 2955792 to 3497694 tid 3 2024-09-21T08:26:05.538 INFO:tasks.rados.rados.0.smithi137.stdout:504: write (excl) oid 27 current snap is 43 2024-09-21T08:26:05.538 INFO:tasks.rados.rados.0.smithi137.stdout:503: finishing write tid 1 to smithi13779221-31 2024-09-21T08:26:05.538 INFO:tasks.rados.rados.0.smithi137.stdout:504: seq_num 213 ranges {779891=485921,1947841=685395,3383215=196676} 2024-09-21T08:26:05.545 INFO:tasks.rados.rados.0.smithi137.stdout:504: writing smithi13779221-27 from 779891 to 1265812 tid 1 2024-09-21T08:26:05.552 INFO:tasks.rados.rados.0.smithi137.stdout:504: writing smithi13779221-27 from 1947841 to 2633236 tid 2 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:504: writing smithi13779221-27 from 3383215 to 3579891 tid 3 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:505: delete oid 30 current snap is 43 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:502: finishing copy_from racing read to smithi13779221-35 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:502: finishing copy_from to smithi13779221-35 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 66 (ObjNum 156 snap 31 seq_num 156) dirty exists 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:503: finishing write tid 2 to smithi13779221-31 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:503: finishing write tid 3 to smithi13779221-31 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:503: finishing write tid 4 to smithi13779221-31 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:503: finishing write tid 5 to smithi13779221-31 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:503: oid 31 updating version 0 to 154 2024-09-21T08:26:05.558 INFO:tasks.rados.rados.0.smithi137.stdout:503: oid 31 version 154 is already newer than 151 2024-09-21T08:26:05.559 INFO:tasks.rados.rados.0.smithi137.stdout:503: oid 31 version 154 is already newer than 152 2024-09-21T08:26:05.559 INFO:tasks.rados.rados.0.smithi137.stdout:503: oid 31 version 154 is already newer than 153 2024-09-21T08:26:05.559 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 154 (ObjNum 212 snap 43 seq_num 212) dirty exists 2024-09-21T08:26:05.559 INFO:tasks.rados.rados.0.smithi137.stdout:503: left oid 31 (ObjNum 212 snap 43 seq_num 212) 2024-09-21T08:26:05.559 INFO:tasks.rados.rados.0.smithi137.stdout:504: finishing write tid 1 to smithi13779221-27 2024-09-21T08:26:05.559 INFO:tasks.rados.rados.0.smithi137.stdout:504: finishing write tid 2 to smithi13779221-27 2024-09-21T08:26:05.560 INFO:tasks.rados.rados.0.smithi137.stdout:502: done (3 left) 2024-09-21T08:26:05.560 INFO:tasks.rados.rados.0.smithi137.stdout:503: done (2 left) 2024-09-21T08:26:05.560 INFO:tasks.rados.rados.0.smithi137.stdout:505: done (1 left) 2024-09-21T08:26:05.560 INFO:tasks.rados.rados.0.smithi137.stdout:506: delete oid 10 current snap is 43 2024-09-21T08:26:05.560 INFO:tasks.rados.rados.0.smithi137.stdout:504: finishing write tid 3 to smithi13779221-27 2024-09-21T08:26:05.560 INFO:tasks.rados.rados.0.smithi137.stdout:504: finishing write tid 4 to smithi13779221-27 2024-09-21T08:26:05.561 INFO:tasks.rados.rados.0.smithi137.stdout:504: finishing write tid 5 to smithi13779221-27 2024-09-21T08:26:05.561 INFO:tasks.rados.rados.0.smithi137.stdout:504: oid 27 updating version 0 to 155 2024-09-21T08:26:05.561 INFO:tasks.rados.rados.0.smithi137.stdout:504: oid 27 updating version 155 to 157 2024-09-21T08:26:05.561 INFO:tasks.rados.rados.0.smithi137.stdout:504: oid 27 version 157 is already newer than 156 2024-09-21T08:26:05.561 INFO:tasks.rados.rados.0.smithi137.stdout:504: oid 27 version 157 is already newer than 154 2024-09-21T08:26:05.561 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 157 (ObjNum 213 snap 43 seq_num 213) dirty exists 2024-09-21T08:26:05.561 INFO:tasks.rados.rados.0.smithi137.stdout:504: left oid 27 (ObjNum 213 snap 43 seq_num 213) 2024-09-21T08:26:05.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:05.562 INFO:tasks.rados.rados.0.smithi137.stdout:504: done (1 left) 2024-09-21T08:26:05.562 INFO:tasks.rados.rados.0.smithi137.stdout:506: done (0 left) 2024-09-21T08:26:05.562 INFO:tasks.rados.rados.0.smithi137.stdout:507: copy_from oid 28 from oid 37 current snap is 43 2024-09-21T08:26:05.562 INFO:tasks.rados.rados.0.smithi137.stdout:508: snap_create 2024-09-21T08:26:05.568 INFO:tasks.rados.rados.0.smithi137.stdout:507: finishing copy_from racing read to smithi13779221-28 2024-09-21T08:26:05.576 INFO:tasks.rados.rados.0.smithi137.stdout:507: finishing copy_from to smithi13779221-28 2024-09-21T08:26:05.576 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 18 (ObjNum 36 snap 0 seq_num 36) dirty exists 2024-09-21T08:26:05.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:05.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:05.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:05.608 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:05.616 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:05.619 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:05.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:05.705 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:05 smithi165 ceph-mon[32893]: 3.16 scrub starts 2024-09-21T08:26:05.705 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:05 smithi165 ceph-mon[32893]: 3.16 scrub ok 2024-09-21T08:26:05.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:05 smithi165 ceph-mon[32893]: pgmap v161: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 451 MiB used, 357 GiB / 358 GiB avail; 40 MiB/s rd, 5.6 MiB/s wr, 56 op/s 2024-09-21T08:26:05.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:05 smithi165 ceph-mon[32893]: osdmap e171: 8 total, 8 up, 4 in 2024-09-21T08:26:05.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:05 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:05.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:05 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:05.706 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:05 smithi165 ceph-mon[32893]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T08:26:05.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:05.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:05.708 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[29634]: 3.16 scrub starts 2024-09-21T08:26:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[29634]: 3.16 scrub ok 2024-09-21T08:26:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[29634]: pgmap v161: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 451 MiB used, 357 GiB / 358 GiB avail; 40 MiB/s rd, 5.6 MiB/s wr, 56 op/s 2024-09-21T08:26:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[29634]: osdmap e171: 8 total, 8 up, 4 in 2024-09-21T08:26:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[29634]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T08:26:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[41044]: 3.16 scrub starts 2024-09-21T08:26:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[41044]: 3.16 scrub ok 2024-09-21T08:26:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[41044]: pgmap v161: 148 pgs: 8 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 135 active+clean; 136 MiB data, 451 MiB used, 357 GiB / 358 GiB avail; 40 MiB/s rd, 5.6 MiB/s wr, 56 op/s 2024-09-21T08:26:05.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[41044]: osdmap e171: 8 total, 8 up, 4 in 2024-09-21T08:26:05.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:05.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:05.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:05 smithi137 ceph-mon[41044]: Reconfiguring prometheus.a (dependencies changed)... 2024-09-21T08:26:05.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:05.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:05.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:05.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:05.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:05.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:05.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:05.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:06.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:06.077 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:06.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:06.077 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:06.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:06.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:06.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:06.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:06.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:06.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:06.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:06.324 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:06.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:06.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:06.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:06.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:06.523 INFO:tasks.rados.rados.0.smithi137.stdout:507: done (1 left) 2024-09-21T08:26:06.523 INFO:tasks.rados.rados.0.smithi137.stdout:508: done (0 left) 2024-09-21T08:26:06.523 INFO:tasks.rados.rados.0.smithi137.stdout:509: write (excl) oid 26 current snap is 44 2024-09-21T08:26:06.523 INFO:tasks.rados.rados.0.smithi137.stdout:509: seq_num 216 ranges {430312=711668,1723287=549075,2781837=703343} 2024-09-21T08:26:06.529 INFO:tasks.rados.rados.0.smithi137.stdout:509: writing smithi13779221-26 from 430312 to 1141980 tid 1 2024-09-21T08:26:06.536 INFO:tasks.rados.rados.0.smithi137.stdout:509: writing smithi13779221-26 from 1723287 to 2272362 tid 2 2024-09-21T08:26:06.543 INFO:tasks.rados.rados.0.smithi137.stdout:509: writing smithi13779221-26 from 2781837 to 3485180 tid 3 2024-09-21T08:26:06.543 INFO:tasks.rados.rados.0.smithi137.stdout:510: read oid 36 snap -1 2024-09-21T08:26:06.543 INFO:tasks.rados.rados.0.smithi137.stdout:510: expect (ObjNum 181 snap 36 seq_num 181) 2024-09-21T08:26:06.543 INFO:tasks.rados.rados.0.smithi137.stdout:511: read oid 8 snap -1 2024-09-21T08:26:06.544 INFO:tasks.rados.rados.0.smithi137.stdout:511: expect (ObjNum 183 snap 37 seq_num 183) 2024-09-21T08:26:06.544 INFO:tasks.rados.rados.0.smithi137.stdout:512: snap_remove snap 43 2024-09-21T08:26:06.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:06.569 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:06.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:06.570 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:06.578 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:26:06.578 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:26:06.578 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:26:06.578 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:26:06.578 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:26:06.645 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:06 smithi165 ceph-mon[32893]: Reconfiguring daemon prometheus.a on smithi165 2024-09-21T08:26:06.646 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:06 smithi165 ceph-mon[32893]: osdmap e172: 8 total, 8 up, 4 in 2024-09-21T08:26:06.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:06.693 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:06.693 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:06.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:06 smithi137 ceph-mon[29634]: Reconfiguring daemon prometheus.a on smithi165 2024-09-21T08:26:06.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:06 smithi137 ceph-mon[29634]: osdmap e172: 8 total, 8 up, 4 in 2024-09-21T08:26:06.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:06 smithi137 ceph-mon[41044]: Reconfiguring daemon prometheus.a on smithi165 2024-09-21T08:26:06.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:06 smithi137 ceph-mon[41044]: osdmap e172: 8 total, 8 up, 4 in 2024-09-21T08:26:06.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:06.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:06.816 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:06.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:06.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:06.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:06.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:07.038 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:07.059 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:07.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:07.060 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:07.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:07.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:07.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:07.183 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:07.284 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:07.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:07.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:07.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:07.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:07.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:07.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:07.428 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:07.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T08:26:07.529 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:07.532 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:26:07.532 INFO:tasks.rados.rados.0.smithi137.stdout:512: done (3 left) 2024-09-21T08:26:07.532 INFO:tasks.rados.rados.0.smithi137.stdout:513: read oid 9 snap -1 2024-09-21T08:26:07.532 INFO:tasks.rados.rados.0.smithi137.stdout:513: expect deleted 2024-09-21T08:26:07.534 INFO:tasks.rados.rados.0.smithi137.stdout:514: delete oid 32 current snap is 44 2024-09-21T08:26:07.534 INFO:tasks.rados.rados.0.smithi137.stdout:509: finishing write tid 1 to smithi13779221-26 2024-09-21T08:26:07.534 INFO:tasks.rados.rados.0.smithi137.stdout:509: finishing write tid 2 to smithi13779221-26 2024-09-21T08:26:07.534 INFO:tasks.rados.rados.0.smithi137.stdout:509: finishing write tid 3 to smithi13779221-26 2024-09-21T08:26:07.534 INFO:tasks.rados.rados.0.smithi137.stdout:509: finishing write tid 4 to smithi13779221-26 2024-09-21T08:26:07.535 INFO:tasks.rados.rados.0.smithi137.stdout:509: finishing write tid 5 to smithi13779221-26 2024-09-21T08:26:07.535 INFO:tasks.rados.rados.0.smithi137.stdout:509: oid 26 updating version 0 to 81 2024-09-21T08:26:07.535 INFO:tasks.rados.rados.0.smithi137.stdout:509: oid 26 version 81 is already newer than 78 2024-09-21T08:26:07.535 INFO:tasks.rados.rados.0.smithi137.stdout:509: oid 26 version 81 is already newer than 79 2024-09-21T08:26:07.535 INFO:tasks.rados.rados.0.smithi137.stdout:509: oid 26 version 81 is already newer than 80 2024-09-21T08:26:07.535 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 81 (ObjNum 216 snap 44 seq_num 216) dirty exists 2024-09-21T08:26:07.535 INFO:tasks.rados.rados.0.smithi137.stdout:509: left oid 26 (ObjNum 216 snap 44 seq_num 216) 2024-09-21T08:26:07.535 INFO:tasks.rados.rados.0.smithi137.stdout:511: expect (ObjNum 183 snap 37 seq_num 183) 2024-09-21T08:26:07.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:07.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:07.558 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:07.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:26:07.617 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:26:07.618 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:26:07.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[41044]: 4.2 scrub starts 2024-09-21T08:26:07.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[41044]: 4.2 scrub ok 2024-09-21T08:26:07.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[41044]: 5.17 scrub starts 2024-09-21T08:26:07.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[41044]: 5.17 scrub ok 2024-09-21T08:26:07.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[41044]: pgmap v164: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 115 MiB data, 408 MiB used, 357 GiB / 358 GiB avail; 2.2 KiB/s rd, 2.8 MiB/s wr, 9 op/s 2024-09-21T08:26:07.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[41044]: osdmap e173: 8 total, 8 up, 4 in 2024-09-21T08:26:07.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[41044]: from='client.44969 v1:172.21.15.137:0/472570113' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:26:07.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/653284359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:07.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[41044]: from='client.44975 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:07.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[29634]: 4.2 scrub starts 2024-09-21T08:26:07.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[29634]: 4.2 scrub ok 2024-09-21T08:26:07.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[29634]: 5.17 scrub starts 2024-09-21T08:26:07.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[29634]: 5.17 scrub ok 2024-09-21T08:26:07.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[29634]: pgmap v164: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 115 MiB data, 408 MiB used, 357 GiB / 358 GiB avail; 2.2 KiB/s rd, 2.8 MiB/s wr, 9 op/s 2024-09-21T08:26:07.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[29634]: osdmap e173: 8 total, 8 up, 4 in 2024-09-21T08:26:07.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[29634]: from='client.44969 v1:172.21.15.137:0/472570113' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:26:07.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/653284359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:07.621 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:07 smithi137 ceph-mon[29634]: from='client.44975 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:07.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:07.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:07.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:07.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:07.712 INFO:tasks.rados.rados.0.smithi137.stdout:510: expect (ObjNum 181 snap 36 seq_num 181) 2024-09-21T08:26:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:07 smithi165 ceph-mon[32893]: 4.2 scrub starts 2024-09-21T08:26:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:07 smithi165 ceph-mon[32893]: 4.2 scrub ok 2024-09-21T08:26:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:07 smithi165 ceph-mon[32893]: 5.17 scrub starts 2024-09-21T08:26:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:07 smithi165 ceph-mon[32893]: 5.17 scrub ok 2024-09-21T08:26:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:07 smithi165 ceph-mon[32893]: pgmap v164: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 115 MiB data, 408 MiB used, 357 GiB / 358 GiB avail; 2.2 KiB/s rd, 2.8 MiB/s wr, 9 op/s 2024-09-21T08:26:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:07 smithi165 ceph-mon[32893]: osdmap e173: 8 total, 8 up, 4 in 2024-09-21T08:26:07.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:07 smithi165 ceph-mon[32893]: from='client.44969 v1:172.21.15.137:0/472570113' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:26:07.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:07 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/653284359' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:07.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:07 smithi165 ceph-mon[32893]: from='client.44975 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:07.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:07.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:07.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:07.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:07.824 INFO:tasks.rados.rados.0.smithi137.stdout:509: done (4 left) 2024-09-21T08:26:07.824 INFO:tasks.rados.rados.0.smithi137.stdout:510: done (3 left) 2024-09-21T08:26:07.824 INFO:tasks.rados.rados.0.smithi137.stdout:511: done (2 left) 2024-09-21T08:26:07.824 INFO:tasks.rados.rados.0.smithi137.stdout:514: done (1 left) 2024-09-21T08:26:07.824 INFO:tasks.rados.rados.0.smithi137.stdout:515: rollback oid 46 current snap is 44 2024-09-21T08:26:07.824 INFO:tasks.rados.rados.0.smithi137.stdout:515: done (1 left) 2024-09-21T08:26:07.824 INFO:tasks.rados.rados.0.smithi137.stdout:516: read oid 20 snap -1 2024-09-21T08:26:07.824 INFO:tasks.rados.rados.0.smithi137.stdout:516: expect (ObjNum 195 snap 40 seq_num 195) 2024-09-21T08:26:07.824 INFO:tasks.rados.rados.0.smithi137.stdout:517: read oid 41 snap -1 2024-09-21T08:26:07.825 INFO:tasks.rados.rados.0.smithi137.stdout:517: expect (ObjNum 185 snap 37 seq_num 185) 2024-09-21T08:26:07.825 INFO:tasks.rados.rados.0.smithi137.stdout:518: read oid 31 snap -1 2024-09-21T08:26:07.825 INFO:tasks.rados.rados.0.smithi137.stdout:518: expect (ObjNum 212 snap 43 seq_num 212) 2024-09-21T08:26:07.825 INFO:tasks.rados.rados.0.smithi137.stdout:519: read oid 26 snap -1 2024-09-21T08:26:07.825 INFO:tasks.rados.rados.0.smithi137.stdout:519: expect (ObjNum 216 snap 44 seq_num 216) 2024-09-21T08:26:07.825 INFO:tasks.rados.rados.0.smithi137.stdout:520: snap_create 2024-09-21T08:26:07.835 INFO:tasks.rados.rados.0.smithi137.stdout:517: expect (ObjNum 185 snap 37 seq_num 185) 2024-09-21T08:26:07.905 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:07.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:07.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:07.930 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:08.026 INFO:tasks.rados.rados.0.smithi137.stdout:516: expect (ObjNum 195 snap 40 seq_num 195) 2024-09-21T08:26:08.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:08.053 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:08.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:08.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:08.128 INFO:tasks.rados.rados.0.smithi137.stdout:518: expect (ObjNum 212 snap 43 seq_num 212) 2024-09-21T08:26:08.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:08.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:08.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:08.176 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:08.217 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:07 smithi165 systemd[1]: Stopping Ceph prometheus.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:26:08.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:08.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:08.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:08.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:08.360 INFO:tasks.rados.rados.0.smithi137.stdout:519: expect (ObjNum 216 snap 44 seq_num 216) 2024-09-21T08:26:08.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:08.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:08.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:08.420 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:08.499 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.216Z caller=main.go:840 level=warn msg="Received SIGTERM, exiting gracefully..." 2024-09-21T08:26:08.499 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.216Z caller=main.go:864 level=info msg="Stopping scrape discovery manager..." 2024-09-21T08:26:08.499 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.216Z caller=main.go:878 level=info msg="Stopping notify discovery manager..." 2024-09-21T08:26:08.499 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.216Z caller=manager.go:988 level=info component="rule manager" msg="Stopping rule manager..." 2024-09-21T08:26:08.499 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.216Z caller=main.go:860 level=info msg="Scrape discovery manager stopped" 2024-09-21T08:26:08.500 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.216Z caller=manager.go:998 level=info component="rule manager" msg="Rule manager stopped" 2024-09-21T08:26:08.500 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.216Z caller=main.go:915 level=info msg="Stopping scrape manager..." 2024-09-21T08:26:08.500 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.216Z caller=main.go:874 level=info msg="Notify discovery manager stopped" 2024-09-21T08:26:08.500 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.216Z caller=main.go:907 level=info msg="Scrape manager stopped" 2024-09-21T08:26:08.500 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.217Z caller=notifier.go:606 level=info component=notifier msg="Stopping notification manager..." 2024-09-21T08:26:08.500 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.217Z caller=main.go:1135 level=info msg="Notifier manager stopped" 2024-09-21T08:26:08.500 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[62799]: ts=2024-09-21T08:26:08.217Z caller=main.go:1147 level=info msg="See you next time!" 2024-09-21T08:26:08.500 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 podman[74642]: 2024-09-21 08:26:08.227008577 +0000 UTC m=+0.121287195 container died ae6d54d2591339eeab1590759b85b64aa6cbaa47f70b4754ac4fc11a30658232 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:08.520 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:08.534 INFO:tasks.rados.rados.0.smithi137.stdout:513: done (5 left) 2024-09-21T08:26:08.534 INFO:tasks.rados.rados.0.smithi137.stdout:516: done (4 left) 2024-09-21T08:26:08.534 INFO:tasks.rados.rados.0.smithi137.stdout:517: done (3 left) 2024-09-21T08:26:08.534 INFO:tasks.rados.rados.0.smithi137.stdout:518: done (2 left) 2024-09-21T08:26:08.534 INFO:tasks.rados.rados.0.smithi137.stdout:519: done (1 left) 2024-09-21T08:26:08.534 INFO:tasks.rados.rados.0.smithi137.stdout:520: done (0 left) 2024-09-21T08:26:08.535 INFO:tasks.rados.rados.0.smithi137.stdout:521: read oid 7 snap 44 2024-09-21T08:26:08.535 INFO:tasks.rados.rados.0.smithi137.stdout:521: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:26:08.535 INFO:tasks.rados.rados.0.smithi137.stdout:522: rollback oid 27 current snap is 45 2024-09-21T08:26:08.535 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 27 to 44 2024-09-21T08:26:08.535 INFO:tasks.rados.rados.0.smithi137.stdout:523: write (excl) oid 46 current snap is 45 2024-09-21T08:26:08.535 INFO:tasks.rados.rados.0.smithi137.stdout:523: seq_num 218 ranges {463926=462166,1723898=436391,2603977=725887} 2024-09-21T08:26:08.540 INFO:tasks.rados.rados.0.smithi137.stdout:523: writing smithi13779221-46 from 463926 to 926092 tid 1 2024-09-21T08:26:08.544 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:08.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:08.545 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:08.547 INFO:tasks.rados.rados.0.smithi137.stdout:523: writing smithi13779221-46 from 1723898 to 2160289 tid 2 2024-09-21T08:26:08.554 INFO:tasks.rados.rados.0.smithi137.stdout:523: writing smithi13779221-46 from 2603977 to 3329864 tid 3 2024-09-21T08:26:08.554 INFO:tasks.rados.rados.0.smithi137.stdout:524: rollback oid 36 current snap is 45 2024-09-21T08:26:08.554 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 36 to 44 2024-09-21T08:26:08.554 INFO:tasks.rados.rados.0.smithi137.stdout:522: finishing rollback tid 0 to smithi13779221-27 2024-09-21T08:26:08.555 INFO:tasks.rados.rados.0.smithi137.stdout:522: finishing rollback tid 1 to smithi13779221-27 2024-09-21T08:26:08.555 INFO:tasks.rados.rados.0.smithi137.stdout:522: finishing rollback tid 2 to smithi13779221-27 2024-09-21T08:26:08.555 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 161 (ObjNum 213 snap 43 seq_num 213) dirty exists 2024-09-21T08:26:08.555 INFO:tasks.rados.rados.0.smithi137.stdout:521: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:26:08.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:08.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:08.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:08.670 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:523: finishing write tid 1 to smithi13779221-46 2024-09-21T08:26:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:523: finishing write tid 2 to smithi13779221-46 2024-09-21T08:26:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:521: done (3 left) 2024-09-21T08:26:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:522: done (2 left) 2024-09-21T08:26:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:525: delete oid 43 current snap is 45 2024-09-21T08:26:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:523: finishing write tid 3 to smithi13779221-46 2024-09-21T08:26:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:523: finishing write tid 4 to smithi13779221-46 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:523: finishing write tid 5 to smithi13779221-46 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:523: oid 46 updating version 0 to 223 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:523: oid 46 version 223 is already newer than 222 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:523: oid 46 updating version 223 to 225 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:523: oid 46 version 225 is already newer than 224 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 225 (ObjNum 218 snap 45 seq_num 218) dirty exists 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:523: left oid 46 (ObjNum 218 snap 45 seq_num 218) 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:524: finishing rollback tid 0 to smithi13779221-36 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:524: finishing rollback tid 1 to smithi13779221-36 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:524: finishing rollback tid 2 to smithi13779221-36 2024-09-21T08:26:08.686 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 229 (ObjNum 181 snap 36 seq_num 181) dirty exists 2024-09-21T08:26:08.687 INFO:tasks.rados.rados.0.smithi137.stdout:523: done (2 left) 2024-09-21T08:26:08.687 INFO:tasks.rados.rados.0.smithi137.stdout:524: done (1 left) 2024-09-21T08:26:08.687 INFO:tasks.rados.rados.0.smithi137.stdout:525: done (0 left) 2024-09-21T08:26:08.687 INFO:tasks.rados.rados.0.smithi137.stdout:526: rollback oid 27 current snap is 45 2024-09-21T08:26:08.687 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 27 to 44 2024-09-21T08:26:08.687 INFO:tasks.rados.rados.0.smithi137.stdout:527: rollback oid 10 current snap is 45 2024-09-21T08:26:08.687 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 10 to 44 2024-09-21T08:26:08.688 INFO:tasks.rados.rados.0.smithi137.stdout:528: read oid 41 snap 44 2024-09-21T08:26:08.688 INFO:tasks.rados.rados.0.smithi137.stdout:528: expect (ObjNum 185 snap 37 seq_num 185) 2024-09-21T08:26:08.688 INFO:tasks.rados.rados.0.smithi137.stdout:529: snap_create 2024-09-21T08:26:08.689 INFO:tasks.rados.rados.0.smithi137.stdout:527: finishing rollback tid 1 to smithi13779221-10 2024-09-21T08:26:08.689 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 0 (ObjNum 529 snap 0 seq_num 3760921552) dirty dne 2024-09-21T08:26:08.690 INFO:tasks.rados.rados.0.smithi137.stdout:528: expect (ObjNum 185 snap 37 seq_num 185) 2024-09-21T08:26:08.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 podman[74642]: 2024-09-21 08:26:08.615160646 +0000 UTC m=+0.509439263 container cleanup ae6d54d2591339eeab1590759b85b64aa6cbaa47f70b4754ac4fc11a30658232 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:08.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 bash[74642]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a 2024-09-21T08:26:08.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-mon[32893]: 4.6 scrub starts 2024-09-21T08:26:08.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-mon[32893]: 4.6 scrub ok 2024-09-21T08:26:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:26:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:26:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-mon[32893]: from='client.44975 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:08 smithi165 ceph-mon[32893]: osdmap e174: 8 total, 8 up, 4 in 2024-09-21T08:26:08.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:08.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:08.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:08.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:08.879 INFO:tasks.rados.rados.0.smithi137.stdout:526: finishing rollback tid 0 to smithi13779221-27 2024-09-21T08:26:08.879 INFO:tasks.rados.rados.0.smithi137.stdout:526: finishing rollback tid 1 to smithi13779221-27 2024-09-21T08:26:08.879 INFO:tasks.rados.rados.0.smithi137.stdout:526: finishing rollback tid 2 to smithi13779221-27 2024-09-21T08:26:08.879 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 164 (ObjNum 213 snap 43 seq_num 213) dirty exists 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[29634]: 4.6 scrub starts 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[29634]: 4.6 scrub ok 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[29634]: from='client.44975 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[29634]: osdmap e174: 8 total, 8 up, 4 in 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[41044]: 4.6 scrub starts 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[41044]: 4.6 scrub ok 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:26:08.889 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[41044]: from='client.44975 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:08.890 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:08 smithi137 ceph-mon[41044]: osdmap e174: 8 total, 8 up, 4 in 2024-09-21T08:26:08.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:08.914 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:08.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:08.915 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:08.915 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":165,"stamp":"2024-09-21T08:26:06.519537+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":120367522,"num_objects":240,"num_object_clones":2,"num_object_copies":720,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":240,"num_whiteouts":0,"num_read":2082,"num_read_kb":907978,"num_write":1496,"num_write_kb":327149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":479,"num_bytes_recovered":142984584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2177,"ondisk_log_size":2177,"up":444,"acting":444,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":444,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":418068,"kb_used_data":276672,"kb_used_omap":6,"kb_used_meta":141305,"kb_avail":374562540,"statfs":{"total":383980142592,"available":383552040960,"internally_reserved":0,"allocated":283312128,"data_stored":288276022,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6375,"internal_metadata":144697113},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-21916506,"num_objects":-15,"num_object_clones":-12,"num_object_copies":-45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-15,"num_whiteouts":-1,"num_read":14,"num_read_kb":9,"num_write":24,"num_write_kb":11509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.017929"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":231,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521728+0000","last_change":"2024-09-21T08:25:38.378517+0000","last_active":"2024-09-21T08:26:05.521728+0000","last_peered":"2024-09-21T08:26:05.521728+0000","last_clean":"2024-09-21T08:26:05.521728+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:26:05.521728+0000","last_undegraded":"2024-09-21T08:26:05.521728+0000","last_fullsized":"2024-09-21T08:26:05.521728+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:52.783162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":130,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516200+0000","last_change":"2024-09-21T08:25:48.982638+0000","last_active":"2024-09-21T08:26:05.516200+0000","last_peered":"2024-09-21T08:26:05.516200+0000","last_clean":"2024-09-21T08:26:05.516200+0000","last_became_active":"2024-09-21T08:25:48.910812+0000","last_became_peered":"2024-09-21T08:25:48.910812+0000","last_unstale":"2024-09-21T08:26:05.516200+0000","last_undegraded":"2024-09-21T08:26:05.516200+0000","last_fullsized":"2024-09-21T08:26:05.516200+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:24:50.572709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.572709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:10.553204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":46,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529272+0000","last_change":"2024-09-21T08:25:48.912078+0000","last_active":"2024-09-21T08:26:05.529272+0000","last_peered":"2024-09-21T08:26:05.529272+0000","last_clean":"2024-09-21T08:26:05.529272+0000","last_became_active":"2024-09-21T08:25:48.911674+0000","last_became_peered":"2024-09-21T08:25:48.911674+0000","last_unstale":"2024-09-21T08:26:05.529272+0000","last_undegraded":"2024-09-21T08:26:05.529272+0000","last_fullsized":"2024-09-21T08:26:05.529272+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":193,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529457+0000","last_change":"2024-09-21T08:25:48.914509+0000","last_active":"2024-09-21T08:26:05.529457+0000","last_peered":"2024-09-21T08:26:05.529457+0000","last_clean":"2024-09-21T08:26:05.529457+0000","last_became_active":"2024-09-21T08:25:48.911853+0000","last_became_peered":"2024-09-21T08:25:48.911853+0000","last_unstale":"2024-09-21T08:26:05.529457+0000","last_undegraded":"2024-09-21T08:26:05.529457+0000","last_fullsized":"2024-09-21T08:26:05.529457+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":177,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516842+0000","last_change":"2024-09-21T08:25:48.900959+0000","last_active":"2024-09-21T08:26:05.516842+0000","last_peered":"2024-09-21T08:26:05.516842+0000","last_clean":"2024-09-21T08:26:05.516842+0000","last_became_active":"2024-09-21T08:25:48.900647+0000","last_became_peered":"2024-09-21T08:25:48.900647+0000","last_unstale":"2024-09-21T08:26:05.516842+0000","last_undegraded":"2024-09-21T08:26:05.516842+0000","last_fullsized":"2024-09-21T08:26:05.516842+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":167,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517891+0000","last_change":"2024-09-21T08:25:48.884959+0000","last_active":"2024-09-21T08:26:05.517891+0000","last_peered":"2024-09-21T08:26:05.517891+0000","last_clean":"2024-09-21T08:26:05.517891+0000","last_became_active":"2024-09-21T08:25:48.884673+0000","last_became_peered":"2024-09-21T08:25:48.884673+0000","last_unstale":"2024-09-21T08:26:05.517891+0000","last_undegraded":"2024-09-21T08:26:05.517891+0000","last_fullsized":"2024-09-21T08:26:05.517891+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:19.906906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:19.906906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:22.705067+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516620+0000","last_change":"2024-09-21T08:25:20.942089+0000","last_active":"2024-09-21T08:26:05.516620+0000","last_peered":"2024-09-21T08:26:05.516620+0000","last_clean":"2024-09-21T08:26:05.516620+0000","last_became_active":"2024-09-21T08:24:48.594855+0000","last_became_peered":"2024-09-21T08:24:48.594855+0000","last_unstale":"2024-09-21T08:26:05.516620+0000","last_undegraded":"2024-09-21T08:26:05.516620+0000","last_fullsized":"2024-09-21T08:26:05.516620+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:25:20.941990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:20.941990+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:29.954407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":51,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.515925+0000","last_change":"2024-09-21T08:25:48.912528+0000","last_active":"2024-09-21T08:26:05.515925+0000","last_peered":"2024-09-21T08:26:05.515925+0000","last_clean":"2024-09-21T08:26:05.515925+0000","last_became_active":"2024-09-21T08:25:48.911583+0000","last_became_peered":"2024-09-21T08:25:48.911583+0000","last_unstale":"2024-09-21T08:26:05.515925+0000","last_undegraded":"2024-09-21T08:26:05.515925+0000","last_fullsized":"2024-09-21T08:26:05.515925+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.640831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.640831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":242,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530229+0000","last_change":"2024-09-21T08:25:48.907786+0000","last_active":"2024-09-21T08:26:05.530229+0000","last_peered":"2024-09-21T08:26:05.530229+0000","last_clean":"2024-09-21T08:26:05.530229+0000","last_became_active":"2024-09-21T08:25:48.906922+0000","last_became_peered":"2024-09-21T08:25:48.906922+0000","last_unstale":"2024-09-21T08:26:05.530229+0000","last_undegraded":"2024-09-21T08:26:05.530229+0000","last_fullsized":"2024-09-21T08:26:05.530229+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":226,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.519995+0000","last_change":"2024-09-21T08:24:56.523774+0000","last_active":"2024-09-21T08:26:05.519995+0000","last_peered":"2024-09-21T08:26:05.519995+0000","last_clean":"2024-09-21T08:26:05.519995+0000","last_became_active":"2024-09-21T08:24:56.500171+0000","last_became_peered":"2024-09-21T08:24:56.500171+0000","last_unstale":"2024-09-21T08:26:05.519995+0000","last_undegraded":"2024-09-21T08:26:05.519995+0000","last_fullsized":"2024-09-21T08:26:05.519995+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":224,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529784+0000","last_change":"2024-09-21T08:25:22.581281+0000","last_active":"2024-09-21T08:26:05.529784+0000","last_peered":"2024-09-21T08:26:05.529784+0000","last_clean":"2024-09-21T08:26:05.529784+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:26:05.529784+0000","last_undegraded":"2024-09-21T08:26:05.529784+0000","last_fullsized":"2024-09-21T08:26:05.529784+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:33.300785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":217,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520228+0000","last_change":"2024-09-21T08:25:48.885985+0000","last_active":"2024-09-21T08:26:05.520228+0000","last_peered":"2024-09-21T08:26:05.520228+0000","last_clean":"2024-09-21T08:26:05.520228+0000","last_became_active":"2024-09-21T08:25:48.885136+0000","last_became_peered":"2024-09-21T08:25:48.885136+0000","last_unstale":"2024-09-21T08:26:05.520228+0000","last_undegraded":"2024-09-21T08:26:05.520228+0000","last_fullsized":"2024-09-21T08:26:05.520228+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":250,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530004+0000","last_change":"2024-09-21T08:25:48.912437+0000","last_active":"2024-09-21T08:26:05.530004+0000","last_peered":"2024-09-21T08:26:05.530004+0000","last_clean":"2024-09-21T08:26:05.530004+0000","last_became_active":"2024-09-21T08:25:48.912043+0000","last_became_peered":"2024-09-21T08:25:48.912043+0000","last_unstale":"2024-09-21T08:26:05.530004+0000","last_undegraded":"2024-09-21T08:26:05.530004+0000","last_fullsized":"2024-09-21T08:26:05.530004+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:16.938344+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":83,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516591+0000","last_change":"2024-09-21T08:25:48.937429+0000","last_active":"2024-09-21T08:26:05.516591+0000","last_peered":"2024-09-21T08:26:05.516591+0000","last_clean":"2024-09-21T08:26:05.516591+0000","last_became_active":"2024-09-21T08:25:48.907461+0000","last_became_peered":"2024-09-21T08:25:48.907461+0000","last_unstale":"2024-09-21T08:26:05.516591+0000","last_undegraded":"2024-09-21T08:26:05.516591+0000","last_fullsized":"2024-09-21T08:26:05.516591+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:24:32.569846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:32.569846+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":226,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517402+0000","last_change":"2024-09-21T08:25:48.886137+0000","last_active":"2024-09-21T08:26:05.517402+0000","last_peered":"2024-09-21T08:26:05.517402+0000","last_clean":"2024-09-21T08:26:05.517402+0000","last_became_active":"2024-09-21T08:25:48.885803+0000","last_became_peered":"2024-09-21T08:25:48.885803+0000","last_unstale":"2024-09-21T08:26:05.517402+0000","last_undegraded":"2024-09-21T08:26:05.517402+0000","last_fullsized":"2024-09-21T08:26:05.517402+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":233,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529408+0000","last_change":"2024-09-21T08:25:48.966469+0000","last_active":"2024-09-21T08:26:05.529408+0000","last_peered":"2024-09-21T08:26:05.529408+0000","last_clean":"2024-09-21T08:26:05.529408+0000","last_became_active":"2024-09-21T08:25:48.911981+0000","last_became_peered":"2024-09-21T08:25:48.911981+0000","last_unstale":"2024-09-21T08:26:05.529408+0000","last_undegraded":"2024-09-21T08:26:05.529408+0000","last_fullsized":"2024-09-21T08:26:05.529408+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":188,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530516+0000","last_change":"2024-09-21T08:25:39.855515+0000","last_active":"2024-09-21T08:26:05.530516+0000","last_peered":"2024-09-21T08:26:05.530516+0000","last_clean":"2024-09-21T08:26:05.530516+0000","last_became_active":"2024-09-21T08:24:56.499884+0000","last_became_peered":"2024-09-21T08:24:56.499884+0000","last_unstale":"2024-09-21T08:26:05.530516+0000","last_undegraded":"2024-09-21T08:26:05.530516+0000","last_fullsized":"2024-09-21T08:26:05.530516+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.855448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.855448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:04.367456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":243,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.519756+0000","last_change":"2024-09-21T08:25:48.891123+0000","last_active":"2024-09-21T08:26:05.519756+0000","last_peered":"2024-09-21T08:26:05.519756+0000","last_clean":"2024-09-21T08:26:05.519756+0000","last_became_active":"2024-09-21T08:25:48.890600+0000","last_became_peered":"2024-09-21T08:25:48.890600+0000","last_unstale":"2024-09-21T08:26:05.519756+0000","last_undegraded":"2024-09-21T08:26:05.519756+0000","last_fullsized":"2024-09-21T08:26:05.519756+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:17.893561+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":296,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517019+0000","last_change":"2024-09-21T08:25:48.949361+0000","last_active":"2024-09-21T08:26:05.517019+0000","last_peered":"2024-09-21T08:26:05.517019+0000","last_clean":"2024-09-21T08:26:05.517019+0000","last_became_active":"2024-09-21T08:25:48.909222+0000","last_became_peered":"2024-09-21T08:25:48.909222+0000","last_unstale":"2024-09-21T08:26:05.517019+0000","last_undegraded":"2024-09-21T08:26:05.517019+0000","last_fullsized":"2024-09-21T08:26:05.517019+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:25:24.945807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:24.945807+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:43.015505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":83,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521155+0000","last_change":"2024-09-21T08:25:48.905281+0000","last_active":"2024-09-21T08:26:05.521155+0000","last_peered":"2024-09-21T08:26:05.521155+0000","last_clean":"2024-09-21T08:26:05.521155+0000","last_became_active":"2024-09-21T08:25:48.904956+0000","last_became_peered":"2024-09-21T08:25:48.904956+0000","last_unstale":"2024-09-21T08:26:05.521155+0000","last_undegraded":"2024-09-21T08:26:05.521155+0000","last_fullsized":"2024-09-21T08:26:05.521155+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":158,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530818+0000","last_change":"2024-09-21T08:25:48.914865+0000","last_active":"2024-09-21T08:26:05.530818+0000","last_peered":"2024-09-21T08:26:05.530818+0000","last_clean":"2024-09-21T08:26:05.530818+0000","last_became_active":"2024-09-21T08:25:48.912721+0000","last_became_peered":"2024-09-21T08:25:48.912721+0000","last_unstale":"2024-09-21T08:26:05.530818+0000","last_undegraded":"2024-09-21T08:26:05.530818+0000","last_fullsized":"2024-09-21T08:26:05.530818+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":48,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516629+0000","last_change":"2024-09-21T08:25:48.900151+0000","last_active":"2024-09-21T08:26:05.516629+0000","last_peered":"2024-09-21T08:26:05.516629+0000","last_clean":"2024-09-21T08:26:05.516629+0000","last_became_active":"2024-09-21T08:25:48.899796+0000","last_became_peered":"2024-09-21T08:25:48.899796+0000","last_unstale":"2024-09-21T08:26:05.516629+0000","last_undegraded":"2024-09-21T08:26:05.516629+0000","last_fullsized":"2024-09-21T08:26:05.516629+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":102,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530526+0000","last_change":"2024-09-21T08:25:48.964877+0000","last_active":"2024-09-21T08:26:05.530526+0000","last_peered":"2024-09-21T08:26:05.530526+0000","last_clean":"2024-09-21T08:26:05.530526+0000","last_became_active":"2024-09-21T08:25:48.913458+0000","last_became_peered":"2024-09-21T08:25:48.913458+0000","last_unstale":"2024-09-21T08:26:05.530526+0000","last_undegraded":"2024-09-21T08:26:05.530526+0000","last_fullsized":"2024-09-21T08:26:05.530526+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:21.424414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.424414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:21.922543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":182,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517258+0000","last_change":"2024-09-21T08:25:48.892022+0000","last_active":"2024-09-21T08:26:05.517258+0000","last_peered":"2024-09-21T08:26:05.517258+0000","last_clean":"2024-09-21T08:26:05.517258+0000","last_became_active":"2024-09-21T08:25:48.891706+0000","last_became_peered":"2024-09-21T08:25:48.891706+0000","last_unstale":"2024-09-21T08:26:05.517258+0000","last_undegraded":"2024-09-21T08:26:05.517258+0000","last_fullsized":"2024-09-21T08:26:05.517258+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.797084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:22.450359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":210,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517593+0000","last_change":"2024-09-21T08:23:36.584003+0000","last_active":"2024-09-21T08:26:05.517593+0000","last_peered":"2024-09-21T08:26:05.517593+0000","last_clean":"2024-09-21T08:26:05.517593+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:26:05.517593+0000","last_undegraded":"2024-09-21T08:26:05.517593+0000","last_fullsized":"2024-09-21T08:26:05.517593+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":126,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516921+0000","last_change":"2024-09-21T08:25:48.971328+0000","last_active":"2024-09-21T08:26:05.516921+0000","last_peered":"2024-09-21T08:26:05.516921+0000","last_clean":"2024-09-21T08:26:05.516921+0000","last_became_active":"2024-09-21T08:25:48.896963+0000","last_became_peered":"2024-09-21T08:25:48.896963+0000","last_unstale":"2024-09-21T08:26:05.516921+0000","last_undegraded":"2024-09-21T08:26:05.516921+0000","last_fullsized":"2024-09-21T08:26:05.516921+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":270,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521587+0000","last_change":"2024-09-21T08:25:48.989640+0000","last_active":"2024-09-21T08:26:05.521587+0000","last_peered":"2024-09-21T08:26:05.521587+0000","last_clean":"2024-09-21T08:26:05.521587+0000","last_became_active":"2024-09-21T08:25:48.908487+0000","last_became_peered":"2024-09-21T08:25:48.908487+0000","last_unstale":"2024-09-21T08:26:05.521587+0000","last_undegraded":"2024-09-21T08:26:05.521587+0000","last_fullsized":"2024-09-21T08:26:05.521587+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"156'12","reported_seq":290,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518367+0000","last_change":"2024-09-21T08:25:48.961434+0000","last_active":"2024-09-21T08:26:05.518367+0000","last_peered":"2024-09-21T08:26:05.518367+0000","last_clean":"2024-09-21T08:26:05.518367+0000","last_became_active":"2024-09-21T08:25:48.896649+0000","last_became_peered":"2024-09-21T08:25:48.896649+0000","last_unstale":"2024-09-21T08:26:05.518367+0000","last_undegraded":"2024-09-21T08:26:05.518367+0000","last_fullsized":"2024-09-21T08:26:05.518367+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.594212+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"172'90","reported_seq":566,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.568630+0000","last_change":"2024-09-21T08:26:05.521016+0000","last_active":"2024-09-21T08:26:05.568630+0000","last_peered":"2024-09-21T08:26:05.568630+0000","last_clean":"2024-09-21T08:26:05.568630+0000","last_became_active":"2024-09-21T08:24:56.503681+0000","last_became_peered":"2024-09-21T08:24:56.503681+0000","last_unstale":"2024-09-21T08:26:05.568630+0000","last_undegraded":"2024-09-21T08:26:05.568630+0000","last_fullsized":"2024-09-21T08:26:05.568630+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":90,"log_dups_size":0,"ondisk_log_size":90,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.0034713299999999999,"stat_sum":{"num_bytes":8773221,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":79914,"num_write":48,"num_write_kb":24758,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":12634927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.f","version":"73'15","reported_seq":128,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518154+0000","last_change":"2024-09-21T08:25:48.935201+0000","last_active":"2024-09-21T08:26:05.518154+0000","last_peered":"2024-09-21T08:26:05.518154+0000","last_clean":"2024-09-21T08:26:05.518154+0000","last_became_active":"2024-09-21T08:25:48.894852+0000","last_became_peered":"2024-09-21T08:25:48.894852+0000","last_unstale":"2024-09-21T08:26:05.518154+0000","last_undegraded":"2024-09-21T08:26:05.518154+0000","last_fullsized":"2024-09-21T08:26:05.518154+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":49,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516477+0000","last_change":"2024-09-21T08:25:48.897595+0000","last_active":"2024-09-21T08:26:05.516477+0000","last_peered":"2024-09-21T08:26:05.516477+0000","last_clean":"2024-09-21T08:26:05.516477+0000","last_became_active":"2024-09-21T08:25:48.897249+0000","last_became_peered":"2024-09-21T08:25:48.897249+0000","last_unstale":"2024-09-21T08:26:05.516477+0000","last_undegraded":"2024-09-21T08:26:05.516477+0000","last_fullsized":"2024-09-21T08:26:05.516477+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:45.434055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:45.434055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:59.844465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"85'9","reported_seq":242,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530521+0000","last_change":"2024-09-21T08:25:21.631370+0000","last_active":"2024-09-21T08:26:05.530521+0000","last_peered":"2024-09-21T08:26:05.530521+0000","last_clean":"2024-09-21T08:26:05.530521+0000","last_became_active":"2024-09-21T08:24:38.548789+0000","last_became_peered":"2024-09-21T08:24:38.548789+0000","last_unstale":"2024-09-21T08:26:05.530521+0000","last_undegraded":"2024-09-21T08:26:05.530521+0000","last_fullsized":"2024-09-21T08:26:05.530521+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:25:21.631254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.631254+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:44.705485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":218,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530424+0000","last_change":"2024-09-21T08:25:48.914809+0000","last_active":"2024-09-21T08:26:05.530424+0000","last_peered":"2024-09-21T08:26:05.530424+0000","last_clean":"2024-09-21T08:26:05.530424+0000","last_became_active":"2024-09-21T08:25:48.914453+0000","last_became_peered":"2024-09-21T08:25:48.914453+0000","last_unstale":"2024-09-21T08:26:05.530424+0000","last_undegraded":"2024-09-21T08:26:05.530424+0000","last_fullsized":"2024-09-21T08:26:05.530424+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"158'15","reported_seq":473,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516888+0000","last_change":"2024-09-21T08:26:05.517336+0000","last_active":"2024-09-21T08:26:05.516888+0000","last_peered":"2024-09-21T08:26:05.516888+0000","last_clean":"2024-09-21T08:26:05.516888+0000","last_became_active":"2024-09-21T08:25:48.901490+0000","last_became_peered":"2024-09-21T08:25:48.901490+0000","last_unstale":"2024-09-21T08:26:05.516888+0000","last_undegraded":"2024-09-21T08:26:05.516888+0000","last_fullsized":"2024-09-21T08:26:05.516888+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00068723899999999997,"stat_sum":{"num_bytes":3025260,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":10650,"num_write":11,"num_write_kb":2527,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":5130840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.0","version":"73'18","reported_seq":258,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516434+0000","last_change":"2024-09-21T08:25:48.942497+0000","last_active":"2024-09-21T08:26:05.516434+0000","last_peered":"2024-09-21T08:26:05.516434+0000","last_clean":"2024-09-21T08:26:05.516434+0000","last_became_active":"2024-09-21T08:25:48.895885+0000","last_became_peered":"2024-09-21T08:25:48.895885+0000","last_unstale":"2024-09-21T08:26:05.516434+0000","last_undegraded":"2024-09-21T08:26:05.516434+0000","last_fullsized":"2024-09-21T08:26:05.516434+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":174,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518447+0000","last_change":"2024-09-21T08:24:50.648706+0000","last_active":"2024-09-21T08:26:05.518447+0000","last_peered":"2024-09-21T08:26:05.518447+0000","last_clean":"2024-09-21T08:26:05.518447+0000","last_became_active":"2024-09-21T08:24:48.603740+0000","last_became_peered":"2024-09-21T08:24:48.603740+0000","last_unstale":"2024-09-21T08:26:05.518447+0000","last_undegraded":"2024-09-21T08:26:05.518447+0000","last_fullsized":"2024-09-21T08:26:05.518447+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.532278+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":167,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516870+0000","last_change":"2024-09-21T08:24:48.604125+0000","last_active":"2024-09-21T08:26:05.516870+0000","last_peered":"2024-09-21T08:26:05.516870+0000","last_clean":"2024-09-21T08:26:05.516870+0000","last_became_active":"2024-09-21T08:24:48.603825+0000","last_became_peered":"2024-09-21T08:24:48.603825+0000","last_unstale":"2024-09-21T08:26:05.516870+0000","last_undegraded":"2024-09-21T08:26:05.516870+0000","last_fullsized":"2024-09-21T08:26:05.516870+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":175,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518211+0000","last_change":"2024-09-21T08:24:56.501652+0000","last_active":"2024-09-21T08:26:05.518211+0000","last_peered":"2024-09-21T08:26:05.518211+0000","last_clean":"2024-09-21T08:26:05.518211+0000","last_became_active":"2024-09-21T08:24:56.501237+0000","last_became_peered":"2024-09-21T08:24:56.501237+0000","last_unstale":"2024-09-21T08:26:05.518211+0000","last_undegraded":"2024-09-21T08:26:05.518211+0000","last_fullsized":"2024-09-21T08:26:05.518211+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"172'183","reported_seq":722,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520620+0000","last_change":"2024-09-21T08:26:05.520620+0000","last_active":"2024-09-21T08:26:05.520620+0000","last_peered":"2024-09-21T08:26:05.520620+0000","last_clean":"2024-09-21T08:26:05.520620+0000","last_became_active":"2024-09-21T08:25:51.507328+0000","last_became_peered":"2024-09-21T08:25:51.507328+0000","last_unstale":"2024-09-21T08:26:05.520620+0000","last_undegraded":"2024-09-21T08:26:05.520620+0000","last_fullsized":"2024-09-21T08:26:05.520620+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":183,"log_dups_size":0,"ondisk_log_size":183,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00192713,"stat_sum":{"num_bytes":12874162,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":102,"num_read_kb":91865,"num_write":117,"num_write_kb":41346,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29,"num_bytes_recovered":22739695,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.1","version":"73'14","reported_seq":264,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529407+0000","last_change":"2024-09-21T08:24:56.561414+0000","last_active":"2024-09-21T08:26:05.529407+0000","last_peered":"2024-09-21T08:26:05.529407+0000","last_clean":"2024-09-21T08:26:05.529407+0000","last_became_active":"2024-09-21T08:24:56.509169+0000","last_became_peered":"2024-09-21T08:24:56.509169+0000","last_unstale":"2024-09-21T08:26:05.529407+0000","last_undegraded":"2024-09-21T08:26:05.529407+0000","last_fullsized":"2024-09-21T08:26:05.529407+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":252,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521192+0000","last_change":"2024-09-21T08:25:48.919957+0000","last_active":"2024-09-21T08:26:05.521192+0000","last_peered":"2024-09-21T08:26:05.521192+0000","last_clean":"2024-09-21T08:26:05.521192+0000","last_became_active":"2024-09-21T08:25:48.893688+0000","last_became_peered":"2024-09-21T08:25:48.893688+0000","last_unstale":"2024-09-21T08:26:05.521192+0000","last_undegraded":"2024-09-21T08:26:05.521192+0000","last_fullsized":"2024-09-21T08:26:05.521192+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.923056+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"156'11","reported_seq":114,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520354+0000","last_change":"2024-09-21T08:25:48.973195+0000","last_active":"2024-09-21T08:26:05.520354+0000","last_peered":"2024-09-21T08:26:05.520354+0000","last_clean":"2024-09-21T08:26:05.520354+0000","last_became_active":"2024-09-21T08:25:48.906808+0000","last_became_peered":"2024-09-21T08:25:48.906808+0000","last_unstale":"2024-09-21T08:26:05.520354+0000","last_undegraded":"2024-09-21T08:26:05.520354+0000","last_fullsized":"2024-09-21T08:26:05.520354+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":230,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517021+0000","last_change":"2024-09-21T08:25:48.900309+0000","last_active":"2024-09-21T08:26:05.517021+0000","last_peered":"2024-09-21T08:26:05.517021+0000","last_clean":"2024-09-21T08:26:05.517021+0000","last_became_active":"2024-09-21T08:25:48.899971+0000","last_became_peered":"2024-09-21T08:25:48.899971+0000","last_unstale":"2024-09-21T08:26:05.517021+0000","last_undegraded":"2024-09-21T08:26:05.517021+0000","last_fullsized":"2024-09-21T08:26:05.517021+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:21.561826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.561826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:44.050410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"164'42","reported_seq":449,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521266+0000","last_change":"2024-09-21T08:26:05.521441+0000","last_active":"2024-09-21T08:26:05.521266+0000","last_peered":"2024-09-21T08:26:05.521266+0000","last_clean":"2024-09-21T08:26:05.521266+0000","last_became_active":"2024-09-21T08:25:48.908454+0000","last_became_peered":"2024-09-21T08:25:48.908454+0000","last_unstale":"2024-09-21T08:26:05.521266+0000","last_undegraded":"2024-09-21T08:26:05.521266+0000","last_fullsized":"2024-09-21T08:26:05.521266+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":42,"log_dups_size":0,"ondisk_log_size":42,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00080784400000000005,"stat_sum":{"num_bytes":3210137,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":40,"num_read_kb":43376,"num_write":19,"num_write_kb":6321,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":5646358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.2","version":"73'10","reported_seq":92,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529285+0000","last_change":"2024-09-21T08:25:48.929418+0000","last_active":"2024-09-21T08:26:05.529285+0000","last_peered":"2024-09-21T08:26:05.529285+0000","last_clean":"2024-09-21T08:26:05.529285+0000","last_became_active":"2024-09-21T08:25:48.894952+0000","last_became_peered":"2024-09-21T08:25:48.894952+0000","last_unstale":"2024-09-21T08:26:05.529285+0000","last_undegraded":"2024-09-21T08:26:05.529285+0000","last_fullsized":"2024-09-21T08:26:05.529285+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":250,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529919+0000","last_change":"2024-09-21T08:24:56.519880+0000","last_active":"2024-09-21T08:26:05.529919+0000","last_peered":"2024-09-21T08:26:05.529919+0000","last_clean":"2024-09-21T08:26:05.529919+0000","last_became_active":"2024-09-21T08:24:56.500882+0000","last_became_peered":"2024-09-21T08:24:56.500882+0000","last_unstale":"2024-09-21T08:26:05.529919+0000","last_undegraded":"2024-09-21T08:26:05.529919+0000","last_fullsized":"2024-09-21T08:26:05.529919+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"156'10","reported_seq":275,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520047+0000","last_change":"2024-09-21T08:25:48.928203+0000","last_active":"2024-09-21T08:26:05.520047+0000","last_peered":"2024-09-21T08:26:05.520047+0000","last_clean":"2024-09-21T08:26:05.520047+0000","last_became_active":"2024-09-21T08:25:48.902651+0000","last_became_peered":"2024-09-21T08:25:48.902651+0000","last_unstale":"2024-09-21T08:26:05.520047+0000","last_undegraded":"2024-09-21T08:26:05.520047+0000","last_fullsized":"2024-09-21T08:26:05.520047+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":217,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.519948+0000","last_change":"2024-09-21T08:24:48.597514+0000","last_active":"2024-09-21T08:26:05.519948+0000","last_peered":"2024-09-21T08:26:05.519948+0000","last_clean":"2024-09-21T08:26:05.519948+0000","last_became_active":"2024-09-21T08:24:48.597048+0000","last_became_peered":"2024-09-21T08:24:48.597048+0000","last_unstale":"2024-09-21T08:26:05.519948+0000","last_undegraded":"2024-09-21T08:26:05.519948+0000","last_fullsized":"2024-09-21T08:26:05.519948+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"171'92","reported_seq":595,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:26:05.529140+0000","last_change":"2024-09-21T08:26:04.512663+0000","last_active":"2024-09-21T08:26:05.529140+0000","last_peered":"2024-09-21T08:26:05.529140+0000","last_clean":"2024-09-21T08:26:05.529140+0000","last_became_active":"2024-09-21T08:25:48.911864+0000","last_became_peered":"2024-09-21T08:25:48.911864+0000","last_unstale":"2024-09-21T08:26:05.529140+0000","last_undegraded":"2024-09-21T08:26:05.529140+0000","last_fullsized":"2024-09-21T08:26:05.529140+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":92,"log_dups_size":0,"ondisk_log_size":92,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.076674274000000001,"stat_sum":{"num_bytes":11233317,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":62,"num_read_kb":48499,"num_write":61,"num_write_kb":24487,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":8143226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"e"}]},{"pgid":"4.3","version":"73'19","reported_seq":255,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.519875+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:26:05.519875+0000","last_peered":"2024-09-21T08:26:05.519875+0000","last_clean":"2024-09-21T08:26:05.519875+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:26:05.519875+0000","last_undegraded":"2024-09-21T08:26:05.519875+0000","last_fullsized":"2024-09-21T08:26:05.519875+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":47,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516874+0000","last_change":"2024-09-21T08:25:48.906191+0000","last_active":"2024-09-21T08:26:05.516874+0000","last_peered":"2024-09-21T08:26:05.516874+0000","last_clean":"2024-09-21T08:26:05.516874+0000","last_became_active":"2024-09-21T08:25:48.905818+0000","last_became_peered":"2024-09-21T08:25:48.905818+0000","last_unstale":"2024-09-21T08:26:05.516874+0000","last_undegraded":"2024-09-21T08:26:05.516874+0000","last_fullsized":"2024-09-21T08:26:05.516874+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.529672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.529672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:10.620945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":157,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.519856+0000","last_change":"2024-09-21T08:25:39.339196+0000","last_active":"2024-09-21T08:26:05.519856+0000","last_peered":"2024-09-21T08:26:05.519856+0000","last_clean":"2024-09-21T08:26:05.519856+0000","last_became_active":"2024-09-21T08:24:56.499628+0000","last_became_peered":"2024-09-21T08:24:56.499628+0000","last_unstale":"2024-09-21T08:26:05.519856+0000","last_undegraded":"2024-09-21T08:26:05.519856+0000","last_fullsized":"2024-09-21T08:26:05.519856+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.339126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.339126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:57.963164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":115,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516262+0000","last_change":"2024-09-21T08:25:48.885420+0000","last_active":"2024-09-21T08:26:05.516262+0000","last_peered":"2024-09-21T08:26:05.516262+0000","last_clean":"2024-09-21T08:26:05.516262+0000","last_became_active":"2024-09-21T08:25:48.885120+0000","last_became_peered":"2024-09-21T08:25:48.885120+0000","last_unstale":"2024-09-21T08:26:05.516262+0000","last_undegraded":"2024-09-21T08:26:05.516262+0000","last_fullsized":"2024-09-21T08:26:05.516262+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.425389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:44.826050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"168'58","reported_seq":550,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.522762+0000","last_change":"2024-09-21T08:26:05.522919+0000","last_active":"2024-09-21T08:26:05.522762+0000","last_peered":"2024-09-21T08:26:05.522762+0000","last_clean":"2024-09-21T08:26:05.522762+0000","last_became_active":"2024-09-21T08:25:48.892621+0000","last_became_peered":"2024-09-21T08:25:48.892621+0000","last_unstale":"2024-09-21T08:26:05.522762+0000","last_undegraded":"2024-09-21T08:26:05.522762+0000","last_fullsized":"2024-09-21T08:26:05.522762+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":58,"log_dups_size":0,"ondisk_log_size":58,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00061225500000000003,"stat_sum":{"num_bytes":2100875,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":19,"num_read_kb":11108,"num_write":28,"num_write_kb":9639,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":10964761,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.4","version":"156'31","reported_seq":141,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517204+0000","last_change":"2024-09-21T08:25:48.954047+0000","last_active":"2024-09-21T08:26:05.517204+0000","last_peered":"2024-09-21T08:26:05.517204+0000","last_clean":"2024-09-21T08:26:05.517204+0000","last_became_active":"2024-09-21T08:25:48.910276+0000","last_became_peered":"2024-09-21T08:25:48.910276+0000","last_unstale":"2024-09-21T08:26:05.517204+0000","last_undegraded":"2024-09-21T08:26:05.517204+0000","last_fullsized":"2024-09-21T08:26:05.517204+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":197,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520222+0000","last_change":"2024-09-21T08:25:48.905376+0000","last_active":"2024-09-21T08:26:05.520222+0000","last_peered":"2024-09-21T08:26:05.520222+0000","last_clean":"2024-09-21T08:26:05.520222+0000","last_became_active":"2024-09-21T08:25:48.905056+0000","last_became_peered":"2024-09-21T08:25:48.905056+0000","last_unstale":"2024-09-21T08:26:05.520222+0000","last_undegraded":"2024-09-21T08:26:05.520222+0000","last_fullsized":"2024-09-21T08:26:05.520222+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":262,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529187+0000","last_change":"2024-09-21T08:26:02.760802+0000","last_active":"2024-09-21T08:26:05.529187+0000","last_peered":"2024-09-21T08:26:05.529187+0000","last_clean":"2024-09-21T08:26:05.529187+0000","last_became_active":"2024-09-21T08:25:48.894174+0000","last_became_peered":"2024-09-21T08:25:48.894174+0000","last_unstale":"2024-09-21T08:26:05.529187+0000","last_undegraded":"2024-09-21T08:26:05.529187+0000","last_fullsized":"2024-09-21T08:26:05.529187+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:21.445845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":233,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521297+0000","last_change":"2024-09-21T08:25:48.893035+0000","last_active":"2024-09-21T08:26:05.521297+0000","last_peered":"2024-09-21T08:26:05.521297+0000","last_clean":"2024-09-21T08:26:05.521297+0000","last_became_active":"2024-09-21T08:25:48.892728+0000","last_became_peered":"2024-09-21T08:25:48.892728+0000","last_unstale":"2024-09-21T08:26:05.521297+0000","last_undegraded":"2024-09-21T08:26:05.521297+0000","last_fullsized":"2024-09-21T08:26:05.521297+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":219,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518137+0000","last_change":"2024-09-21T08:25:48.897361+0000","last_active":"2024-09-21T08:26:05.518137+0000","last_peered":"2024-09-21T08:26:05.518137+0000","last_clean":"2024-09-21T08:26:05.518137+0000","last_became_active":"2024-09-21T08:25:48.896898+0000","last_became_peered":"2024-09-21T08:25:48.896898+0000","last_unstale":"2024-09-21T08:26:05.518137+0000","last_undegraded":"2024-09-21T08:26:05.518137+0000","last_fullsized":"2024-09-21T08:26:05.518137+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.602376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:43.388968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"168'46","reported_seq":541,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521817+0000","last_change":"2024-09-21T08:26:05.522274+0000","last_active":"2024-09-21T08:26:05.521817+0000","last_peered":"2024-09-21T08:26:05.521817+0000","last_clean":"2024-09-21T08:26:05.521817+0000","last_became_active":"2024-09-21T08:25:48.906238+0000","last_became_peered":"2024-09-21T08:25:48.906238+0000","last_unstale":"2024-09-21T08:26:05.521817+0000","last_undegraded":"2024-09-21T08:26:05.521817+0000","last_fullsized":"2024-09-21T08:26:05.521817+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":46,"log_dups_size":0,"ondisk_log_size":46,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014055789999999999,"stat_sum":{"num_bytes":2756348,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":24,"num_read_kb":18843,"num_write":25,"num_write_kb":6559,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4916498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.7","version":"73'13","reported_seq":104,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530492+0000","last_change":"2024-09-21T08:25:48.937399+0000","last_active":"2024-09-21T08:26:05.530492+0000","last_peered":"2024-09-21T08:26:05.530492+0000","last_clean":"2024-09-21T08:26:05.530492+0000","last_became_active":"2024-09-21T08:25:48.908351+0000","last_became_peered":"2024-09-21T08:25:48.908351+0000","last_unstale":"2024-09-21T08:26:05.530492+0000","last_undegraded":"2024-09-21T08:26:05.530492+0000","last_fullsized":"2024-09-21T08:26:05.530492+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":127,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.515880+0000","last_change":"2024-09-21T08:25:48.912816+0000","last_active":"2024-09-21T08:26:05.515880+0000","last_peered":"2024-09-21T08:26:05.515880+0000","last_clean":"2024-09-21T08:26:05.515880+0000","last_became_active":"2024-09-21T08:25:48.911245+0000","last_became_peered":"2024-09-21T08:25:48.911245+0000","last_unstale":"2024-09-21T08:26:05.515880+0000","last_undegraded":"2024-09-21T08:26:05.515880+0000","last_fullsized":"2024-09-21T08:26:05.515880+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":270,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516675+0000","last_change":"2024-09-21T08:25:48.926199+0000","last_active":"2024-09-21T08:26:05.516675+0000","last_peered":"2024-09-21T08:26:05.516675+0000","last_clean":"2024-09-21T08:26:05.516675+0000","last_became_active":"2024-09-21T08:25:48.905148+0000","last_became_peered":"2024-09-21T08:25:48.905148+0000","last_unstale":"2024-09-21T08:26:05.516675+0000","last_undegraded":"2024-09-21T08:26:05.516675+0000","last_fullsized":"2024-09-21T08:26:05.516675+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:50.675673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.675673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":223,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517886+0000","last_change":"2024-09-21T08:25:42.021037+0000","last_active":"2024-09-21T08:26:05.517886+0000","last_peered":"2024-09-21T08:26:05.517886+0000","last_clean":"2024-09-21T08:26:05.517886+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:26:05.517886+0000","last_undegraded":"2024-09-21T08:26:05.517886+0000","last_fullsized":"2024-09-21T08:26:05.517886+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:42.020928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.020928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:55.206204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":222,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518039+0000","last_change":"2024-09-21T08:24:56.501148+0000","last_active":"2024-09-21T08:26:05.518039+0000","last_peered":"2024-09-21T08:26:05.518039+0000","last_clean":"2024-09-21T08:26:05.518039+0000","last_became_active":"2024-09-21T08:24:56.498476+0000","last_became_peered":"2024-09-21T08:24:56.498476+0000","last_unstale":"2024-09-21T08:26:05.518039+0000","last_undegraded":"2024-09-21T08:26:05.518039+0000","last_fullsized":"2024-09-21T08:26:05.518039+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"168'241","reported_seq":712,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518587+0000","last_change":"2024-09-21T08:26:05.518793+0000","last_active":"2024-09-21T08:26:05.518587+0000","last_peered":"2024-09-21T08:26:05.518587+0000","last_clean":"2024-09-21T08:26:05.518587+0000","last_became_active":"2024-09-21T08:24:34.507710+0000","last_became_peered":"2024-09-21T08:24:34.507710+0000","last_unstale":"2024-09-21T08:26:05.518587+0000","last_undegraded":"2024-09-21T08:26:05.518587+0000","last_fullsized":"2024-09-21T08:26:05.518587+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":241,"log_dups_size":0,"ondisk_log_size":241,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000373884,"stat_sum":{"num_bytes":12622057,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":128,"num_read_kb":111772,"num_write":121,"num_write_kb":47832,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.6","version":"73'12","reported_seq":257,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521475+0000","last_change":"2024-09-21T08:25:48.924592+0000","last_active":"2024-09-21T08:26:05.521475+0000","last_peered":"2024-09-21T08:26:05.521475+0000","last_clean":"2024-09-21T08:26:05.521475+0000","last_became_active":"2024-09-21T08:25:48.888592+0000","last_became_peered":"2024-09-21T08:25:48.888592+0000","last_unstale":"2024-09-21T08:26:05.521475+0000","last_undegraded":"2024-09-21T08:26:05.521475+0000","last_fullsized":"2024-09-21T08:26:05.521475+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520426+0000","last_change":"2024-09-21T08:24:48.601343+0000","last_active":"2024-09-21T08:26:05.520426+0000","last_peered":"2024-09-21T08:26:05.520426+0000","last_clean":"2024-09-21T08:26:05.520426+0000","last_became_active":"2024-09-21T08:24:48.601044+0000","last_became_peered":"2024-09-21T08:24:48.601044+0000","last_unstale":"2024-09-21T08:26:05.520426+0000","last_undegraded":"2024-09-21T08:26:05.520426+0000","last_fullsized":"2024-09-21T08:26:05.520426+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":400,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516569+0000","last_change":"2024-09-21T08:25:48.914747+0000","last_active":"2024-09-21T08:26:05.516569+0000","last_peered":"2024-09-21T08:26:05.516569+0000","last_clean":"2024-09-21T08:26:05.516569+0000","last_became_active":"2024-09-21T08:25:48.894703+0000","last_became_peered":"2024-09-21T08:25:48.894703+0000","last_unstale":"2024-09-21T08:26:05.516569+0000","last_undegraded":"2024-09-21T08:26:05.516569+0000","last_fullsized":"2024-09-21T08:26:05.516569+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":148,"num_read_kb":143,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":230,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530461+0000","last_change":"2024-09-21T08:25:48.909001+0000","last_active":"2024-09-21T08:26:05.530461+0000","last_peered":"2024-09-21T08:26:05.530461+0000","last_clean":"2024-09-21T08:26:05.530461+0000","last_became_active":"2024-09-21T08:25:48.908492+0000","last_became_peered":"2024-09-21T08:25:48.908492+0000","last_unstale":"2024-09-21T08:26:05.530461+0000","last_undegraded":"2024-09-21T08:26:05.530461+0000","last_fullsized":"2024-09-21T08:26:05.530461+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.812789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.812789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:57.580731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":121,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530124+0000","last_change":"2024-09-21T08:25:48.914203+0000","last_active":"2024-09-21T08:26:05.530124+0000","last_peered":"2024-09-21T08:26:05.530124+0000","last_clean":"2024-09-21T08:26:05.530124+0000","last_became_active":"2024-09-21T08:25:48.912863+0000","last_became_peered":"2024-09-21T08:25:48.912863+0000","last_unstale":"2024-09-21T08:26:05.530124+0000","last_undegraded":"2024-09-21T08:26:05.530124+0000","last_fullsized":"2024-09-21T08:26:05.530124+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"169'83","reported_seq":567,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.522337+0000","last_change":"2024-09-21T08:26:05.522522+0000","last_active":"2024-09-21T08:26:05.522337+0000","last_peered":"2024-09-21T08:26:05.522337+0000","last_clean":"2024-09-21T08:26:05.522337+0000","last_became_active":"2024-09-21T08:24:48.593246+0000","last_became_peered":"2024-09-21T08:24:48.593246+0000","last_unstale":"2024-09-21T08:26:05.522337+0000","last_undegraded":"2024-09-21T08:26:05.522337+0000","last_fullsized":"2024-09-21T08:26:05.522337+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00037131600000000001,"stat_sum":{"num_bytes":5395047,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":68369,"num_write":40,"num_write_kb":13384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":4966638,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.5","version":"103'18","reported_seq":225,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517968+0000","last_change":"2024-09-21T08:24:56.564202+0000","last_active":"2024-09-21T08:26:05.517968+0000","last_peered":"2024-09-21T08:26:05.517968+0000","last_clean":"2024-09-21T08:26:05.517968+0000","last_became_active":"2024-09-21T08:24:56.503984+0000","last_became_peered":"2024-09-21T08:24:56.503984+0000","last_unstale":"2024-09-21T08:26:05.517968+0000","last_undegraded":"2024-09-21T08:26:05.517968+0000","last_fullsized":"2024-09-21T08:26:05.517968+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-09-21T08:24:48.583137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.583137+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.538958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":179,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529608+0000","last_change":"2024-09-21T08:25:48.914796+0000","last_active":"2024-09-21T08:26:05.529608+0000","last_peered":"2024-09-21T08:26:05.529608+0000","last_clean":"2024-09-21T08:26:05.529608+0000","last_became_active":"2024-09-21T08:25:48.912615+0000","last_became_peered":"2024-09-21T08:25:48.912615+0000","last_unstale":"2024-09-21T08:26:05.529608+0000","last_undegraded":"2024-09-21T08:26:05.529608+0000","last_fullsized":"2024-09-21T08:26:05.529608+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":516,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516697+0000","last_change":"2024-09-21T08:25:51.506619+0000","last_active":"2024-09-21T08:26:05.516697+0000","last_peered":"2024-09-21T08:26:05.516697+0000","last_clean":"2024-09-21T08:26:05.516697+0000","last_became_active":"2024-09-21T08:25:51.506298+0000","last_became_peered":"2024-09-21T08:25:51.506298+0000","last_unstale":"2024-09-21T08:26:05.516697+0000","last_undegraded":"2024-09-21T08:26:05.516697+0000","last_fullsized":"2024-09-21T08:26:05.516697+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:59.195279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":222,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517703+0000","last_change":"2024-09-21T08:25:41.700634+0000","last_active":"2024-09-21T08:26:05.517703+0000","last_peered":"2024-09-21T08:26:05.517703+0000","last_clean":"2024-09-21T08:26:05.517703+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:26:05.517703+0000","last_undegraded":"2024-09-21T08:26:05.517703+0000","last_fullsized":"2024-09-21T08:26:05.517703+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:56.978983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":238,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530650+0000","last_change":"2024-09-21T08:25:48.907243+0000","last_active":"2024-09-21T08:26:05.530650+0000","last_peered":"2024-09-21T08:26:05.530650+0000","last_clean":"2024-09-21T08:26:05.530650+0000","last_became_active":"2024-09-21T08:25:48.906778+0000","last_became_peered":"2024-09-21T08:25:48.906778+0000","last_unstale":"2024-09-21T08:26:05.530650+0000","last_undegraded":"2024-09-21T08:26:05.530650+0000","last_fullsized":"2024-09-21T08:26:05.530650+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"164'50","reported_seq":448,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521502+0000","last_change":"2024-09-21T08:26:05.521729+0000","last_active":"2024-09-21T08:26:05.521502+0000","last_peered":"2024-09-21T08:26:05.521502+0000","last_clean":"2024-09-21T08:26:05.521502+0000","last_became_active":"2024-09-21T08:25:48.906387+0000","last_became_peered":"2024-09-21T08:25:48.906387+0000","last_unstale":"2024-09-21T08:26:05.521502+0000","last_undegraded":"2024-09-21T08:26:05.521502+0000","last_fullsized":"2024-09-21T08:26:05.521502+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00059312600000000003,"stat_sum":{"num_bytes":5428661,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":39,"num_read_kb":34343,"num_write":36,"num_write_kb":14152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":6854523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.e","version":"73'11","reported_seq":137,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530626+0000","last_change":"2024-09-21T08:25:48.949515+0000","last_active":"2024-09-21T08:26:05.530626+0000","last_peered":"2024-09-21T08:26:05.530626+0000","last_clean":"2024-09-21T08:26:05.530626+0000","last_became_active":"2024-09-21T08:25:48.911473+0000","last_became_peered":"2024-09-21T08:25:48.911473+0000","last_unstale":"2024-09-21T08:26:05.530626+0000","last_undegraded":"2024-09-21T08:26:05.530626+0000","last_fullsized":"2024-09-21T08:26:05.530626+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":200,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516518+0000","last_change":"2024-09-21T08:25:48.907133+0000","last_active":"2024-09-21T08:26:05.516518+0000","last_peered":"2024-09-21T08:26:05.516518+0000","last_clean":"2024-09-21T08:26:05.516518+0000","last_became_active":"2024-09-21T08:25:48.906832+0000","last_became_peered":"2024-09-21T08:25:48.906832+0000","last_unstale":"2024-09-21T08:26:05.516518+0000","last_undegraded":"2024-09-21T08:26:05.516518+0000","last_fullsized":"2024-09-21T08:26:05.516518+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.756905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.756905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530670+0000","last_change":"2024-09-21T08:25:48.917071+0000","last_active":"2024-09-21T08:26:05.530670+0000","last_peered":"2024-09-21T08:26:05.530670+0000","last_clean":"2024-09-21T08:26:05.530670+0000","last_became_active":"2024-09-21T08:25:48.915146+0000","last_became_peered":"2024-09-21T08:25:48.915146+0000","last_unstale":"2024-09-21T08:26:05.530670+0000","last_undegraded":"2024-09-21T08:26:05.530670+0000","last_fullsized":"2024-09-21T08:26:05.530670+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":158,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529699+0000","last_change":"2024-09-21T08:25:48.914706+0000","last_active":"2024-09-21T08:26:05.529699+0000","last_peered":"2024-09-21T08:26:05.529699+0000","last_clean":"2024-09-21T08:26:05.529699+0000","last_became_active":"2024-09-21T08:25:48.912765+0000","last_became_peered":"2024-09-21T08:25:48.912765+0000","last_unstale":"2024-09-21T08:26:05.529699+0000","last_undegraded":"2024-09-21T08:26:05.529699+0000","last_fullsized":"2024-09-21T08:26:05.529699+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"166'77","reported_seq":532,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517803+0000","last_change":"2024-09-21T08:26:05.518113+0000","last_active":"2024-09-21T08:26:05.517803+0000","last_peered":"2024-09-21T08:26:05.517803+0000","last_clean":"2024-09-21T08:26:05.517803+0000","last_became_active":"2024-09-21T08:24:34.508974+0000","last_became_peered":"2024-09-21T08:24:34.508974+0000","last_unstale":"2024-09-21T08:26:05.517803+0000","last_undegraded":"2024-09-21T08:26:05.517803+0000","last_fullsized":"2024-09-21T08:26:05.517803+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00048605799999999999,"stat_sum":{"num_bytes":3109295,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":7783,"num_write":33,"num_write_kb":11824,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.d","version":"73'17","reported_seq":307,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517526+0000","last_change":"2024-09-21T08:25:48.934395+0000","last_active":"2024-09-21T08:26:05.517526+0000","last_peered":"2024-09-21T08:26:05.517526+0000","last_clean":"2024-09-21T08:26:05.517526+0000","last_became_active":"2024-09-21T08:25:48.904254+0000","last_became_peered":"2024-09-21T08:25:48.904254+0000","last_unstale":"2024-09-21T08:26:05.517526+0000","last_undegraded":"2024-09-21T08:26:05.517526+0000","last_fullsized":"2024-09-21T08:26:05.517526+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-09-21T08:25:42.978787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.978787+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:48.389318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":121,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529510+0000","last_change":"2024-09-21T08:25:48.907580+0000","last_active":"2024-09-21T08:26:05.529510+0000","last_peered":"2024-09-21T08:26:05.529510+0000","last_clean":"2024-09-21T08:26:05.529510+0000","last_became_active":"2024-09-21T08:25:48.907227+0000","last_became_peered":"2024-09-21T08:25:48.907227+0000","last_unstale":"2024-09-21T08:26:05.529510+0000","last_undegraded":"2024-09-21T08:26:05.529510+0000","last_fullsized":"2024-09-21T08:26:05.529510+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":125,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520806+0000","last_change":"2024-09-21T08:25:48.889493+0000","last_active":"2024-09-21T08:26:05.520806+0000","last_peered":"2024-09-21T08:26:05.520806+0000","last_clean":"2024-09-21T08:26:05.520806+0000","last_became_active":"2024-09-21T08:25:48.889206+0000","last_became_peered":"2024-09-21T08:25:48.889206+0000","last_unstale":"2024-09-21T08:26:05.520806+0000","last_undegraded":"2024-09-21T08:26:05.520806+0000","last_fullsized":"2024-09-21T08:26:05.520806+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":243,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518303+0000","last_change":"2024-09-21T08:25:48.909997+0000","last_active":"2024-09-21T08:26:05.518303+0000","last_peered":"2024-09-21T08:26:05.518303+0000","last_clean":"2024-09-21T08:26:05.518303+0000","last_became_active":"2024-09-21T08:25:48.908631+0000","last_became_peered":"2024-09-21T08:25:48.908631+0000","last_unstale":"2024-09-21T08:26:05.518303+0000","last_undegraded":"2024-09-21T08:26:05.518303+0000","last_fullsized":"2024-09-21T08:26:05.518303+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.766815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.605769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"100'7","reported_seq":474,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.522573+0000","last_change":"2024-09-21T08:26:05.522699+0000","last_active":"2024-09-21T08:26:05.522573+0000","last_peered":"2024-09-21T08:26:05.522573+0000","last_clean":"2024-09-21T08:26:05.522573+0000","last_became_active":"2024-09-21T08:24:48.586187+0000","last_became_peered":"2024-09-21T08:24:48.586187+0000","last_unstale":"2024-09-21T08:26:05.522573+0000","last_undegraded":"2024-09-21T08:26:05.522573+0000","last_fullsized":"2024-09-21T08:26:05.522573+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00041623000000000002,"stat_sum":{"num_bytes":2369754,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":20844,"num_write":5,"num_write_kb":3244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":949927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.c","version":"73'10","reported_seq":94,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517176+0000","last_change":"2024-09-21T08:25:48.987805+0000","last_active":"2024-09-21T08:26:05.517176+0000","last_peered":"2024-09-21T08:26:05.517176+0000","last_clean":"2024-09-21T08:26:05.517176+0000","last_became_active":"2024-09-21T08:25:48.899148+0000","last_became_peered":"2024-09-21T08:25:48.899148+0000","last_unstale":"2024-09-21T08:26:05.517176+0000","last_undegraded":"2024-09-21T08:26:05.517176+0000","last_fullsized":"2024-09-21T08:26:05.517176+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":217,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.519784+0000","last_change":"2024-09-21T08:25:48.891359+0000","last_active":"2024-09-21T08:26:05.519784+0000","last_peered":"2024-09-21T08:26:05.519784+0000","last_clean":"2024-09-21T08:26:05.519784+0000","last_became_active":"2024-09-21T08:25:48.890817+0000","last_became_peered":"2024-09-21T08:25:48.890817+0000","last_unstale":"2024-09-21T08:26:05.519784+0000","last_undegraded":"2024-09-21T08:26:05.519784+0000","last_fullsized":"2024-09-21T08:26:05.519784+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":263,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517414+0000","last_change":"2024-09-21T08:24:51.689976+0000","last_active":"2024-09-21T08:26:05.517414+0000","last_peered":"2024-09-21T08:26:05.517414+0000","last_clean":"2024-09-21T08:26:05.517414+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:26:05.517414+0000","last_undegraded":"2024-09-21T08:26:05.517414+0000","last_fullsized":"2024-09-21T08:26:05.517414+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:24:51.689901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.689901+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:09.850967+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":49,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517724+0000","last_change":"2024-09-21T08:25:48.903834+0000","last_active":"2024-09-21T08:26:05.517724+0000","last_peered":"2024-09-21T08:26:05.517724+0000","last_clean":"2024-09-21T08:26:05.517724+0000","last_became_active":"2024-09-21T08:25:48.903536+0000","last_became_peered":"2024-09-21T08:25:48.903536+0000","last_unstale":"2024-09-21T08:26:05.517724+0000","last_undegraded":"2024-09-21T08:26:05.517724+0000","last_fullsized":"2024-09-21T08:26:05.517724+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"172'154","reported_seq":612,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:26:05.541043+0000","last_change":"2024-09-21T08:26:03.720593+0000","last_active":"2024-09-21T08:26:05.541043+0000","last_peered":"2024-09-21T08:26:05.541043+0000","last_clean":"2024-09-21T08:26:05.541043+0000","last_became_active":"2024-09-21T08:24:48.590826+0000","last_became_peered":"2024-09-21T08:24:48.590826+0000","last_unstale":"2024-09-21T08:26:05.541043+0000","last_undegraded":"2024-09-21T08:26:05.541043+0000","last_fullsized":"2024-09-21T08:26:05.541043+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.19733205700000001,"stat_sum":{"num_bytes":13779997,"num_objects":5,"num_object_clones":1,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":61,"num_read_kb":34652,"num_write":94,"num_write_kb":33144,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":8754687,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"8"}]},{"pgid":"4.b","version":"73'9","reported_seq":291,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520002+0000","last_change":"2024-09-21T08:25:48.969652+0000","last_active":"2024-09-21T08:26:05.520002+0000","last_peered":"2024-09-21T08:26:05.520002+0000","last_clean":"2024-09-21T08:26:05.520002+0000","last_became_active":"2024-09-21T08:25:48.906741+0000","last_became_peered":"2024-09-21T08:25:48.906741+0000","last_unstale":"2024-09-21T08:26:05.520002+0000","last_undegraded":"2024-09-21T08:26:05.520002+0000","last_fullsized":"2024-09-21T08:26:05.520002+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:25:22.369811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.369811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:41.468044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":250,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530190+0000","last_change":"2024-09-21T08:25:48.916673+0000","last_active":"2024-09-21T08:26:05.530190+0000","last_peered":"2024-09-21T08:26:05.530190+0000","last_clean":"2024-09-21T08:26:05.530190+0000","last_became_active":"2024-09-21T08:25:48.914811+0000","last_became_peered":"2024-09-21T08:25:48.914811+0000","last_unstale":"2024-09-21T08:26:05.530190+0000","last_undegraded":"2024-09-21T08:26:05.530190+0000","last_fullsized":"2024-09-21T08:26:05.530190+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.665444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.665444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.303831+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":241,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518121+0000","last_change":"2024-09-21T08:25:48.900965+0000","last_active":"2024-09-21T08:26:05.518121+0000","last_peered":"2024-09-21T08:26:05.518121+0000","last_clean":"2024-09-21T08:26:05.518121+0000","last_became_active":"2024-09-21T08:25:48.900241+0000","last_became_peered":"2024-09-21T08:25:48.900241+0000","last_unstale":"2024-09-21T08:26:05.518121+0000","last_undegraded":"2024-09-21T08:26:05.518121+0000","last_fullsized":"2024-09-21T08:26:05.518121+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:48.797848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":211,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.519942+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:26:05.519942+0000","last_peered":"2024-09-21T08:26:05.519942+0000","last_clean":"2024-09-21T08:26:05.519942+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:26:05.519942+0000","last_undegraded":"2024-09-21T08:26:05.519942+0000","last_fullsized":"2024-09-21T08:26:05.519942+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"172'157","reported_seq":675,"reported_epoch":172,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:26:05.560129+0000","last_change":"2024-09-21T08:26:03.929408+0000","last_active":"2024-09-21T08:26:05.560129+0000","last_peered":"2024-09-21T08:26:05.560129+0000","last_clean":"2024-09-21T08:26:05.560129+0000","last_became_active":"2024-09-21T08:25:51.507296+0000","last_became_peered":"2024-09-21T08:25:51.507296+0000","last_unstale":"2024-09-21T08:26:05.560129+0000","last_undegraded":"2024-09-21T08:26:05.560129+0000","last_fullsized":"2024-09-21T08:26:05.560129+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42258957699999999,"stat_sum":{"num_bytes":13075813,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":134,"num_read_kb":135955,"num_write":89,"num_write_kb":39414,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":13898927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"1d","length":"c"}]},{"pgid":"4.a","version":"73'19","reported_seq":116,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517276+0000","last_change":"2024-09-21T08:25:48.964103+0000","last_active":"2024-09-21T08:26:05.517276+0000","last_peered":"2024-09-21T08:26:05.517276+0000","last_clean":"2024-09-21T08:26:05.517276+0000","last_became_active":"2024-09-21T08:25:48.895647+0000","last_became_peered":"2024-09-21T08:25:48.895647+0000","last_unstale":"2024-09-21T08:26:05.517276+0000","last_undegraded":"2024-09-21T08:26:05.517276+0000","last_fullsized":"2024-09-21T08:26:05.517276+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:43.390605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":236,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517756+0000","last_change":"2024-09-21T08:24:56.507593+0000","last_active":"2024-09-21T08:26:05.517756+0000","last_peered":"2024-09-21T08:26:05.517756+0000","last_clean":"2024-09-21T08:26:05.517756+0000","last_became_active":"2024-09-21T08:24:56.505054+0000","last_became_peered":"2024-09-21T08:24:56.505054+0000","last_unstale":"2024-09-21T08:26:05.517756+0000","last_undegraded":"2024-09-21T08:26:05.517756+0000","last_fullsized":"2024-09-21T08:26:05.517756+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.740674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.740674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":215,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516824+0000","last_change":"2024-09-21T08:23:34.564923+0000","last_active":"2024-09-21T08:26:05.516824+0000","last_peered":"2024-09-21T08:26:05.516824+0000","last_clean":"2024-09-21T08:26:05.516824+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:26:05.516824+0000","last_undegraded":"2024-09-21T08:26:05.516824+0000","last_fullsized":"2024-09-21T08:26:05.516824+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":230,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516711+0000","last_change":"2024-09-21T08:24:56.504692+0000","last_active":"2024-09-21T08:26:05.516711+0000","last_peered":"2024-09-21T08:26:05.516711+0000","last_clean":"2024-09-21T08:26:05.516711+0000","last_became_active":"2024-09-21T08:24:56.503564+0000","last_became_peered":"2024-09-21T08:24:56.503564+0000","last_unstale":"2024-09-21T08:26:05.516711+0000","last_undegraded":"2024-09-21T08:26:05.516711+0000","last_fullsized":"2024-09-21T08:26:05.516711+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"164'33","reported_seq":482,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518817+0000","last_change":"2024-09-21T08:26:05.519053+0000","last_active":"2024-09-21T08:26:05.518817+0000","last_peered":"2024-09-21T08:26:05.518817+0000","last_clean":"2024-09-21T08:26:05.518817+0000","last_became_active":"2024-09-21T08:25:48.910246+0000","last_became_peered":"2024-09-21T08:25:48.910246+0000","last_unstale":"2024-09-21T08:26:05.518817+0000","last_undegraded":"2024-09-21T08:26:05.518817+0000","last_fullsized":"2024-09-21T08:26:05.518817+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00038992999999999997,"stat_sum":{"num_bytes":3159683,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":17785,"num_write":14,"num_write_kb":2495,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2551608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.9","version":"73'12","reported_seq":141,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520691+0000","last_change":"2024-09-21T08:25:48.944363+0000","last_active":"2024-09-21T08:26:05.520691+0000","last_peered":"2024-09-21T08:26:05.520691+0000","last_clean":"2024-09-21T08:26:05.520691+0000","last_became_active":"2024-09-21T08:25:48.894334+0000","last_became_peered":"2024-09-21T08:25:48.894334+0000","last_unstale":"2024-09-21T08:26:05.520691+0000","last_undegraded":"2024-09-21T08:26:05.520691+0000","last_fullsized":"2024-09-21T08:26:05.520691+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:41.463820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.463820+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.215785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517744+0000","last_change":"2024-09-21T08:25:48.896592+0000","last_active":"2024-09-21T08:26:05.517744+0000","last_peered":"2024-09-21T08:26:05.517744+0000","last_clean":"2024-09-21T08:26:05.517744+0000","last_became_active":"2024-09-21T08:25:48.896222+0000","last_became_peered":"2024-09-21T08:25:48.896222+0000","last_unstale":"2024-09-21T08:26:05.517744+0000","last_undegraded":"2024-09-21T08:26:05.517744+0000","last_fullsized":"2024-09-21T08:26:05.517744+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":229,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.515972+0000","last_change":"2024-09-21T08:25:48.913293+0000","last_active":"2024-09-21T08:26:05.515972+0000","last_peered":"2024-09-21T08:26:05.515972+0000","last_clean":"2024-09-21T08:26:05.515972+0000","last_became_active":"2024-09-21T08:25:48.912993+0000","last_became_peered":"2024-09-21T08:25:48.912993+0000","last_unstale":"2024-09-21T08:26:05.515972+0000","last_undegraded":"2024-09-21T08:26:05.515972+0000","last_fullsized":"2024-09-21T08:26:05.515972+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:44.914394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:44.914394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:07.844366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":185,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517078+0000","last_change":"2024-09-21T08:25:48.901068+0000","last_active":"2024-09-21T08:26:05.517078+0000","last_peered":"2024-09-21T08:26:05.517078+0000","last_clean":"2024-09-21T08:26:05.517078+0000","last_became_active":"2024-09-21T08:25:48.900751+0000","last_became_peered":"2024-09-21T08:25:48.900751+0000","last_unstale":"2024-09-21T08:26:05.517078+0000","last_undegraded":"2024-09-21T08:26:05.517078+0000","last_fullsized":"2024-09-21T08:26:05.517078+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:38.201536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"172'220","reported_seq":831,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.559709+0000","last_change":"2024-09-21T08:26:05.530006+0000","last_active":"2024-09-21T08:26:05.559709+0000","last_peered":"2024-09-21T08:26:05.559709+0000","last_clean":"2024-09-21T08:26:05.559709+0000","last_became_active":"2024-09-21T08:25:51.507525+0000","last_became_peered":"2024-09-21T08:25:51.507525+0000","last_unstale":"2024-09-21T08:26:05.559709+0000","last_undegraded":"2024-09-21T08:26:05.559709+0000","last_fullsized":"2024-09-21T08:26:05.559709+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00042405600000000002,"stat_sum":{"num_bytes":16857388,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":190,"num_read_kb":171315,"num_write":140,"num_write_kb":43562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31871318,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"1d","length":"10"}]},{"pgid":"4.8","version":"73'15","reported_seq":297,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.528901+0000","last_change":"2024-09-21T08:25:48.957137+0000","last_active":"2024-09-21T08:26:05.528901+0000","last_peered":"2024-09-21T08:26:05.528901+0000","last_clean":"2024-09-21T08:26:05.528901+0000","last_became_active":"2024-09-21T08:25:48.911675+0000","last_became_peered":"2024-09-21T08:25:48.911675+0000","last_unstale":"2024-09-21T08:26:05.528901+0000","last_undegraded":"2024-09-21T08:26:05.528901+0000","last_fullsized":"2024-09-21T08:26:05.528901+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":266,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518506+0000","last_change":"2024-09-21T08:25:48.975855+0000","last_active":"2024-09-21T08:26:05.518506+0000","last_peered":"2024-09-21T08:26:05.518506+0000","last_clean":"2024-09-21T08:26:05.518506+0000","last_became_active":"2024-09-21T08:25:48.897974+0000","last_became_peered":"2024-09-21T08:25:48.897974+0000","last_unstale":"2024-09-21T08:26:05.518506+0000","last_undegraded":"2024-09-21T08:26:05.518506+0000","last_fullsized":"2024-09-21T08:26:05.518506+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"156'11","reported_seq":293,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516677+0000","last_change":"2024-09-21T08:25:48.977564+0000","last_active":"2024-09-21T08:26:05.516677+0000","last_peered":"2024-09-21T08:26:05.516677+0000","last_clean":"2024-09-21T08:26:05.516677+0000","last_became_active":"2024-09-21T08:25:48.897436+0000","last_became_peered":"2024-09-21T08:25:48.897436+0000","last_unstale":"2024-09-21T08:26:05.516677+0000","last_undegraded":"2024-09-21T08:26:05.516677+0000","last_fullsized":"2024-09-21T08:26:05.516677+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'10","last_scrub_stamp":"2024-09-21T08:25:42.708458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.708458+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:56.347110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":218,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529395+0000","last_change":"2024-09-21T08:24:56.500567+0000","last_active":"2024-09-21T08:26:05.529395+0000","last_peered":"2024-09-21T08:26:05.529395+0000","last_clean":"2024-09-21T08:26:05.529395+0000","last_became_active":"2024-09-21T08:24:56.500287+0000","last_became_peered":"2024-09-21T08:24:56.500287+0000","last_unstale":"2024-09-21T08:26:05.529395+0000","last_undegraded":"2024-09-21T08:26:05.529395+0000","last_fullsized":"2024-09-21T08:26:05.529395+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":149,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.519834+0000","last_change":"2024-09-21T08:24:56.500240+0000","last_active":"2024-09-21T08:26:05.519834+0000","last_peered":"2024-09-21T08:26:05.519834+0000","last_clean":"2024-09-21T08:26:05.519834+0000","last_became_active":"2024-09-21T08:24:56.499114+0000","last_became_peered":"2024-09-21T08:24:56.499114+0000","last_unstale":"2024-09-21T08:26:05.519834+0000","last_undegraded":"2024-09-21T08:26:05.519834+0000","last_fullsized":"2024-09-21T08:26:05.519834+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":215,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518427+0000","last_change":"2024-09-21T08:25:48.980718+0000","last_active":"2024-09-21T08:26:05.518427+0000","last_peered":"2024-09-21T08:26:05.518427+0000","last_clean":"2024-09-21T08:26:05.518427+0000","last_became_active":"2024-09-21T08:25:48.898422+0000","last_became_peered":"2024-09-21T08:25:48.898422+0000","last_unstale":"2024-09-21T08:26:05.518427+0000","last_undegraded":"2024-09-21T08:26:05.518427+0000","last_fullsized":"2024-09-21T08:26:05.518427+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530385+0000","last_change":"2024-09-21T08:25:48.908032+0000","last_active":"2024-09-21T08:26:05.530385+0000","last_peered":"2024-09-21T08:26:05.530385+0000","last_clean":"2024-09-21T08:26:05.530385+0000","last_became_active":"2024-09-21T08:25:48.907510+0000","last_became_peered":"2024-09-21T08:25:48.907510+0000","last_unstale":"2024-09-21T08:26:05.530385+0000","last_undegraded":"2024-09-21T08:26:05.530385+0000","last_fullsized":"2024-09-21T08:26:05.530385+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":222,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518135+0000","last_change":"2024-09-21T08:24:56.506870+0000","last_active":"2024-09-21T08:26:05.518135+0000","last_peered":"2024-09-21T08:26:05.518135+0000","last_clean":"2024-09-21T08:26:05.518135+0000","last_became_active":"2024-09-21T08:24:56.502937+0000","last_became_peered":"2024-09-21T08:24:56.502937+0000","last_unstale":"2024-09-21T08:26:05.518135+0000","last_undegraded":"2024-09-21T08:26:05.518135+0000","last_fullsized":"2024-09-21T08:26:05.518135+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"73'9","reported_seq":251,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521303+0000","last_change":"2024-09-21T08:24:48.629077+0000","last_active":"2024-09-21T08:26:05.521303+0000","last_peered":"2024-09-21T08:26:05.521303+0000","last_clean":"2024-09-21T08:26:05.521303+0000","last_became_active":"2024-09-21T08:24:48.602176+0000","last_became_peered":"2024-09-21T08:24:48.602176+0000","last_unstale":"2024-09-21T08:26:05.521303+0000","last_undegraded":"2024-09-21T08:26:05.521303+0000","last_fullsized":"2024-09-21T08:26:05.521303+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":242,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516531+0000","last_change":"2024-09-21T08:25:46.719639+0000","last_active":"2024-09-21T08:26:05.516531+0000","last_peered":"2024-09-21T08:26:05.516531+0000","last_clean":"2024-09-21T08:26:05.516531+0000","last_became_active":"2024-09-21T08:24:56.505134+0000","last_became_peered":"2024-09-21T08:24:56.505134+0000","last_unstale":"2024-09-21T08:26:05.516531+0000","last_undegraded":"2024-09-21T08:26:05.516531+0000","last_fullsized":"2024-09-21T08:26:05.516531+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:46.719545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:46.719545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:51.390395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":48,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518359+0000","last_change":"2024-09-21T08:25:48.896617+0000","last_active":"2024-09-21T08:26:05.518359+0000","last_peered":"2024-09-21T08:26:05.518359+0000","last_clean":"2024-09-21T08:26:05.518359+0000","last_became_active":"2024-09-21T08:25:48.896076+0000","last_became_peered":"2024-09-21T08:25:48.896076+0000","last_unstale":"2024-09-21T08:26:05.518359+0000","last_undegraded":"2024-09-21T08:26:05.518359+0000","last_fullsized":"2024-09-21T08:26:05.518359+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":225,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517351+0000","last_change":"2024-09-21T08:24:53.746385+0000","last_active":"2024-09-21T08:26:05.517351+0000","last_peered":"2024-09-21T08:26:05.517351+0000","last_clean":"2024-09-21T08:26:05.517351+0000","last_became_active":"2024-09-21T08:24:38.539845+0000","last_became_peered":"2024-09-21T08:24:38.539845+0000","last_unstale":"2024-09-21T08:26:05.517351+0000","last_undegraded":"2024-09-21T08:26:05.517351+0000","last_fullsized":"2024-09-21T08:26:05.517351+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.746295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.746295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.585736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":268,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530056+0000","last_change":"2024-09-21T08:24:56.529301+0000","last_active":"2024-09-21T08:26:05.530056+0000","last_peered":"2024-09-21T08:26:05.530056+0000","last_clean":"2024-09-21T08:26:05.530056+0000","last_became_active":"2024-09-21T08:24:56.499937+0000","last_became_peered":"2024-09-21T08:24:56.499937+0000","last_unstale":"2024-09-21T08:26:05.530056+0000","last_undegraded":"2024-09-21T08:26:05.530056+0000","last_fullsized":"2024-09-21T08:26:05.530056+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":235,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520091+0000","last_change":"2024-09-21T08:25:48.889025+0000","last_active":"2024-09-21T08:26:05.520091+0000","last_peered":"2024-09-21T08:26:05.520091+0000","last_clean":"2024-09-21T08:26:05.520091+0000","last_became_active":"2024-09-21T08:25:48.888733+0000","last_became_peered":"2024-09-21T08:25:48.888733+0000","last_unstale":"2024-09-21T08:26:05.520091+0000","last_undegraded":"2024-09-21T08:26:05.520091+0000","last_fullsized":"2024-09-21T08:26:05.520091+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"156'11","reported_seq":229,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517745+0000","last_change":"2024-09-21T08:25:48.958643+0000","last_active":"2024-09-21T08:26:05.517745+0000","last_peered":"2024-09-21T08:26:05.517745+0000","last_clean":"2024-09-21T08:26:05.517745+0000","last_became_active":"2024-09-21T08:25:48.896000+0000","last_became_peered":"2024-09-21T08:25:48.896000+0000","last_unstale":"2024-09-21T08:26:05.517745+0000","last_undegraded":"2024-09-21T08:26:05.517745+0000","last_fullsized":"2024-09-21T08:26:05.517745+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:16.416651+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":195,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518295+0000","last_change":"2024-09-21T08:25:39.989977+0000","last_active":"2024-09-21T08:26:05.518295+0000","last_peered":"2024-09-21T08:26:05.518295+0000","last_clean":"2024-09-21T08:26:05.518295+0000","last_became_active":"2024-09-21T08:24:38.542854+0000","last_became_peered":"2024-09-21T08:24:38.542854+0000","last_unstale":"2024-09-21T08:26:05.518295+0000","last_undegraded":"2024-09-21T08:26:05.518295+0000","last_fullsized":"2024-09-21T08:26:05.518295+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.989838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.989838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:49.118031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.14","version":"104'15","reported_seq":137,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517846+0000","last_change":"2024-09-21T08:25:48.935811+0000","last_active":"2024-09-21T08:26:05.517846+0000","last_peered":"2024-09-21T08:26:05.517846+0000","last_clean":"2024-09-21T08:26:05.517846+0000","last_became_active":"2024-09-21T08:25:48.894839+0000","last_became_peered":"2024-09-21T08:25:48.894839+0000","last_unstale":"2024-09-21T08:26:05.517846+0000","last_undegraded":"2024-09-21T08:26:05.517846+0000","last_fullsized":"2024-09-21T08:26:05.517846+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":256,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516803+0000","last_change":"2024-09-21T08:24:56.709528+0000","last_active":"2024-09-21T08:26:05.516803+0000","last_peered":"2024-09-21T08:26:05.516803+0000","last_clean":"2024-09-21T08:26:05.516803+0000","last_became_active":"2024-09-21T08:24:56.504665+0000","last_became_peered":"2024-09-21T08:24:56.504665+0000","last_unstale":"2024-09-21T08:26:05.516803+0000","last_undegraded":"2024-09-21T08:26:05.516803+0000","last_fullsized":"2024-09-21T08:26:05.516803+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:56.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.709437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:15.598671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"156'9","reported_seq":274,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530368+0000","last_change":"2024-09-21T08:25:48.931191+0000","last_active":"2024-09-21T08:26:05.530368+0000","last_peered":"2024-09-21T08:26:05.530368+0000","last_clean":"2024-09-21T08:26:05.530368+0000","last_became_active":"2024-09-21T08:25:48.908349+0000","last_became_peered":"2024-09-21T08:25:48.908349+0000","last_unstale":"2024-09-21T08:26:05.530368+0000","last_undegraded":"2024-09-21T08:26:05.530368+0000","last_fullsized":"2024-09-21T08:26:05.530368+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:31.979617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":217,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.522272+0000","last_change":"2024-09-21T08:24:48.592895+0000","last_active":"2024-09-21T08:26:05.522272+0000","last_peered":"2024-09-21T08:26:05.522272+0000","last_clean":"2024-09-21T08:26:05.522272+0000","last_became_active":"2024-09-21T08:24:48.592280+0000","last_became_peered":"2024-09-21T08:24:48.592280+0000","last_unstale":"2024-09-21T08:26:05.522272+0000","last_undegraded":"2024-09-21T08:26:05.522272+0000","last_fullsized":"2024-09-21T08:26:05.522272+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"73'11","reported_seq":98,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529499+0000","last_change":"2024-09-21T08:25:48.979008+0000","last_active":"2024-09-21T08:26:05.529499+0000","last_peered":"2024-09-21T08:26:05.529499+0000","last_clean":"2024-09-21T08:26:05.529499+0000","last_became_active":"2024-09-21T08:25:48.914638+0000","last_became_peered":"2024-09-21T08:25:48.914638+0000","last_unstale":"2024-09-21T08:26:05.529499+0000","last_undegraded":"2024-09-21T08:26:05.529499+0000","last_fullsized":"2024-09-21T08:26:05.529499+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":191,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517961+0000","last_change":"2024-09-21T08:24:56.504037+0000","last_active":"2024-09-21T08:26:05.517961+0000","last_peered":"2024-09-21T08:26:05.517961+0000","last_clean":"2024-09-21T08:26:05.517961+0000","last_became_active":"2024-09-21T08:24:56.503232+0000","last_became_peered":"2024-09-21T08:24:56.503232+0000","last_unstale":"2024-09-21T08:26:05.517961+0000","last_undegraded":"2024-09-21T08:26:05.517961+0000","last_fullsized":"2024-09-21T08:26:05.517961+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":103,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529358+0000","last_change":"2024-09-21T08:25:48.917229+0000","last_active":"2024-09-21T08:26:05.529358+0000","last_peered":"2024-09-21T08:26:05.529358+0000","last_clean":"2024-09-21T08:26:05.529358+0000","last_became_active":"2024-09-21T08:25:48.916802+0000","last_became_peered":"2024-09-21T08:25:48.916802+0000","last_unstale":"2024-09-21T08:26:05.529358+0000","last_undegraded":"2024-09-21T08:26:05.529358+0000","last_fullsized":"2024-09-21T08:26:05.529358+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":165,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520041+0000","last_change":"2024-09-21T08:24:48.600966+0000","last_active":"2024-09-21T08:26:05.520041+0000","last_peered":"2024-09-21T08:26:05.520041+0000","last_clean":"2024-09-21T08:26:05.520041+0000","last_became_active":"2024-09-21T08:24:48.600045+0000","last_became_peered":"2024-09-21T08:24:48.600045+0000","last_unstale":"2024-09-21T08:26:05.520041+0000","last_undegraded":"2024-09-21T08:26:05.520041+0000","last_fullsized":"2024-09-21T08:26:05.520041+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.12","version":"73'9","reported_seq":91,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.518030+0000","last_change":"2024-09-21T08:25:48.953827+0000","last_active":"2024-09-21T08:26:05.518030+0000","last_peered":"2024-09-21T08:26:05.518030+0000","last_clean":"2024-09-21T08:26:05.518030+0000","last_became_active":"2024-09-21T08:25:48.910041+0000","last_became_peered":"2024-09-21T08:25:48.910041+0000","last_unstale":"2024-09-21T08:26:05.518030+0000","last_undegraded":"2024-09-21T08:26:05.518030+0000","last_fullsized":"2024-09-21T08:26:05.518030+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":240,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517829+0000","last_change":"2024-09-21T08:24:56.505458+0000","last_active":"2024-09-21T08:26:05.517829+0000","last_peered":"2024-09-21T08:26:05.517829+0000","last_clean":"2024-09-21T08:26:05.517829+0000","last_became_active":"2024-09-21T08:24:56.504584+0000","last_became_peered":"2024-09-21T08:24:56.504584+0000","last_unstale":"2024-09-21T08:26:05.517829+0000","last_undegraded":"2024-09-21T08:26:05.517829+0000","last_fullsized":"2024-09-21T08:26:05.517829+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":171,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520115+0000","last_change":"2024-09-21T08:25:48.904228+0000","last_active":"2024-09-21T08:26:05.520115+0000","last_peered":"2024-09-21T08:26:05.520115+0000","last_clean":"2024-09-21T08:26:05.520115+0000","last_became_active":"2024-09-21T08:25:48.903934+0000","last_became_peered":"2024-09-21T08:25:48.903934+0000","last_unstale":"2024-09-21T08:26:05.520115+0000","last_undegraded":"2024-09-21T08:26:05.520115+0000","last_fullsized":"2024-09-21T08:26:05.520115+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":217,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.520259+0000","last_change":"2024-09-21T08:25:48.902778+0000","last_active":"2024-09-21T08:26:05.520259+0000","last_peered":"2024-09-21T08:26:05.520259+0000","last_clean":"2024-09-21T08:26:05.520259+0000","last_became_active":"2024-09-21T08:25:48.902435+0000","last_became_peered":"2024-09-21T08:25:48.902435+0000","last_unstale":"2024-09-21T08:26:05.520259+0000","last_undegraded":"2024-09-21T08:26:05.520259+0000","last_fullsized":"2024-09-21T08:26:05.520259+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.11","version":"73'11","reported_seq":241,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516612+0000","last_change":"2024-09-21T08:24:56.531655+0000","last_active":"2024-09-21T08:26:05.516612+0000","last_peered":"2024-09-21T08:26:05.516612+0000","last_clean":"2024-09-21T08:26:05.516612+0000","last_became_active":"2024-09-21T08:24:56.503223+0000","last_became_peered":"2024-09-21T08:24:56.503223+0000","last_unstale":"2024-09-21T08:26:05.516612+0000","last_undegraded":"2024-09-21T08:26:05.516612+0000","last_fullsized":"2024-09-21T08:26:05.516612+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:11.478113+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":238,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530253+0000","last_change":"2024-09-21T08:26:03.808138+0000","last_active":"2024-09-21T08:26:05.530253+0000","last_peered":"2024-09-21T08:26:05.530253+0000","last_clean":"2024-09-21T08:26:05.530253+0000","last_became_active":"2024-09-21T08:25:48.912523+0000","last_became_peered":"2024-09-21T08:25:48.912523+0000","last_unstale":"2024-09-21T08:26:05.530253+0000","last_undegraded":"2024-09-21T08:26:05.530253+0000","last_fullsized":"2024-09-21T08:26:05.530253+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:08.522124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":234,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516330+0000","last_change":"2024-09-21T08:24:58.688115+0000","last_active":"2024-09-21T08:26:05.516330+0000","last_peered":"2024-09-21T08:26:05.516330+0000","last_clean":"2024-09-21T08:26:05.516330+0000","last_became_active":"2024-09-21T08:24:56.503123+0000","last_became_peered":"2024-09-21T08:24:56.503123+0000","last_unstale":"2024-09-21T08:26:05.516330+0000","last_undegraded":"2024-09-21T08:26:05.516330+0000","last_fullsized":"2024-09-21T08:26:05.516330+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.688046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.688046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.364343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":171,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.519854+0000","last_change":"2024-09-21T08:24:56.508632+0000","last_active":"2024-09-21T08:26:05.519854+0000","last_peered":"2024-09-21T08:26:05.519854+0000","last_clean":"2024-09-21T08:26:05.519854+0000","last_became_active":"2024-09-21T08:24:56.508323+0000","last_became_peered":"2024-09-21T08:24:56.508323+0000","last_unstale":"2024-09-21T08:26:05.519854+0000","last_undegraded":"2024-09-21T08:26:05.519854+0000","last_fullsized":"2024-09-21T08:26:05.519854+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.10","version":"73'4","reported_seq":79,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.529022+0000","last_change":"2024-09-21T08:25:48.973819+0000","last_active":"2024-09-21T08:26:05.529022+0000","last_peered":"2024-09-21T08:26:05.529022+0000","last_clean":"2024-09-21T08:26:05.529022+0000","last_became_active":"2024-09-21T08:25:48.913344+0000","last_became_peered":"2024-09-21T08:25:48.913344+0000","last_unstale":"2024-09-21T08:26:05.529022+0000","last_undegraded":"2024-09-21T08:26:05.529022+0000","last_fullsized":"2024-09-21T08:26:05.529022+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":241,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521679+0000","last_change":"2024-09-21T08:25:48.907831+0000","last_active":"2024-09-21T08:26:05.521679+0000","last_peered":"2024-09-21T08:26:05.521679+0000","last_clean":"2024-09-21T08:26:05.521679+0000","last_became_active":"2024-09-21T08:25:48.907530+0000","last_became_peered":"2024-09-21T08:25:48.907530+0000","last_unstale":"2024-09-21T08:26:05.521679+0000","last_undegraded":"2024-09-21T08:26:05.521679+0000","last_fullsized":"2024-09-21T08:26:05.521679+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":193,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.516718+0000","last_change":"2024-09-21T08:25:48.899072+0000","last_active":"2024-09-21T08:26:05.516718+0000","last_peered":"2024-09-21T08:26:05.516718+0000","last_clean":"2024-09-21T08:26:05.516718+0000","last_became_active":"2024-09-21T08:25:48.898419+0000","last_became_peered":"2024-09-21T08:25:48.898419+0000","last_unstale":"2024-09-21T08:26:05.516718+0000","last_undegraded":"2024-09-21T08:26:05.516718+0000","last_fullsized":"2024-09-21T08:26:05.516718+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":228,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.517445+0000","last_change":"2024-09-21T08:25:48.896046+0000","last_active":"2024-09-21T08:26:05.517445+0000","last_peered":"2024-09-21T08:26:05.517445+0000","last_clean":"2024-09-21T08:26:05.517445+0000","last_became_active":"2024-09-21T08:25:48.895489+0000","last_became_peered":"2024-09-21T08:25:48.895489+0000","last_unstale":"2024-09-21T08:26:05.517445+0000","last_undegraded":"2024-09-21T08:26:05.517445+0000","last_fullsized":"2024-09-21T08:26:05.517445+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":79,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530166+0000","last_change":"2024-09-21T08:25:48.913489+0000","last_active":"2024-09-21T08:26:05.530166+0000","last_peered":"2024-09-21T08:26:05.530166+0000","last_clean":"2024-09-21T08:26:05.530166+0000","last_became_active":"2024-09-21T08:25:48.912542+0000","last_became_peered":"2024-09-21T08:25:48.912542+0000","last_unstale":"2024-09-21T08:26:05.530166+0000","last_undegraded":"2024-09-21T08:26:05.530166+0000","last_fullsized":"2024-09-21T08:26:05.530166+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":180,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.519616+0000","last_change":"2024-09-21T08:25:48.885761+0000","last_active":"2024-09-21T08:26:05.519616+0000","last_peered":"2024-09-21T08:26:05.519616+0000","last_clean":"2024-09-21T08:26:05.519616+0000","last_became_active":"2024-09-21T08:25:48.884347+0000","last_became_peered":"2024-09-21T08:25:48.884347+0000","last_unstale":"2024-09-21T08:26:05.519616+0000","last_undegraded":"2024-09-21T08:26:05.519616+0000","last_fullsized":"2024-09-21T08:26:05.519616+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":218,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.530555+0000","last_change":"2024-09-21T08:25:48.942921+0000","last_active":"2024-09-21T08:26:05.530555+0000","last_peered":"2024-09-21T08:26:05.530555+0000","last_clean":"2024-09-21T08:26:05.530555+0000","last_became_active":"2024-09-21T08:25:48.908941+0000","last_became_peered":"2024-09-21T08:25:48.908941+0000","last_unstale":"2024-09-21T08:26:05.530555+0000","last_undegraded":"2024-09-21T08:26:05.530555+0000","last_fullsized":"2024-09-21T08:26:05.530555+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":223,"reported_epoch":172,"state":"active+clean","last_fresh":"2024-09-21T08:26:05.521709+0000","last_change":"2024-09-21T08:25:25.350265+0000","last_active":"2024-09-21T08:26:05.521709+0000","last_peered":"2024-09-21T08:26:05.521709+0000","last_clean":"2024-09-21T08:26:05.521709+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:26:05.521709+0000","last_undegraded":"2024-09-21T08:26:05.521709+0000","last_fullsized":"2024-09-21T08:26:05.521709+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.350197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.350197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:48.284282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":16,"stat_sum":{"num_bytes":119771015,"num_objects":41,"num_object_clones":2,"num_object_copies":123,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":41,"num_whiteouts":0,"num_read":1023,"num_read_kb":907073,"num_write":881,"num_write_kb":324688,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":168,"num_bytes_recovered":140023933,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":375517184,"data_stored":387619595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1548,"ondisk_log_size":1548,"up":48,"acting":48,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":81,"ondisk_log_size":81,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":675,"num_read_kb":432,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":5132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":19638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":8157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":148,"num_read_kb":143,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070500,"num_pgs":113,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":102312,"kb_used_data":66964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93642840,"statfs":{"total":95995035648,"available":95890268160,"internally_reserved":0,"allocated":68571136,"data_stored":68995448,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397546,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659181,"num_pgs":109,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":86332,"kb_used_data":50984,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93658820,"statfs":{"total":95995035648,"available":95906631680,"internally_reserved":0,"allocated":52207616,"data_stored":53314706,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986227,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182455,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411389,"num_pgs":107,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":112832,"kb_used_data":77480,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93632320,"statfs":{"total":95995035648,"available":95879495680,"internally_reserved":0,"allocated":79339520,"data_stored":81421861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574913,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738438,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116592,"kb_used_data":81244,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93628560,"statfs":{"total":95995035648,"available":95875645440,"internally_reserved":0,"allocated":83193856,"data_stored":84544007,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":3262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":78553088,"data_stored":79677694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45531136,"data_stored":53470237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":75399168,"data_stored":77013938,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32387072,"data_stored":32188068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":50970624,"data_stored":52813175,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":64581632,"data_stored":64586068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:26:08.921 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T08:26:08.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_0 pg_num 26 2024-09-21T08:26:08.947 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:08.963 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:09.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:09.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:09.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:09.039 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:09.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:09.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:09.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:09.163 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:09.226 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:08 smithi165 podman[74659]: 2024-09-21 08:26:08.874890831 +0000 UTC m=+0.649515406 container remove ae6d54d2591339eeab1590759b85b64aa6cbaa47f70b4754ac4fc11a30658232 (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:09.227 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@prometheus.a.service: Deactivated successfully. 2024-09-21T08:26:09.227 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 systemd[1]: Stopped Ceph prometheus.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:26:09.227 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 systemd[1]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@prometheus.a.service: Consumed 1.311s CPU time. 2024-09-21T08:26:09.227 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 systemd[1]: Starting Ceph prometheus.a for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:26:09.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:09.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:09.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:09.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:09.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:09.410 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:09.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:09.411 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:09.478 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 podman[74826]: 2024-09-21 08:26:09.225727409 +0000 UTC m=+0.087816052 container create 1700e997ea3d83a76b6add08d6219d5be2cf5479dbf22f7c2b8b2f8256d8184b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:09.478 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 podman[74826]: 2024-09-21 08:26:09.154823003 +0000 UTC m=+0.016911654 image pull a07b618ecd1dce142bce4c52f0e80982eaf1f14265a2415c2d35978ccaa0a464 quay.io/prometheus/prometheus:v2.43.0 2024-09-21T08:26:09.478 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 podman[74826]: 2024-09-21 08:26:09.444803614 +0000 UTC m=+0.306892267 container init 1700e997ea3d83a76b6add08d6219d5be2cf5479dbf22f7c2b8b2f8256d8184b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:09.478 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 podman[74826]: 2024-09-21 08:26:09.449492582 +0000 UTC m=+0.311581225 container start 1700e997ea3d83a76b6add08d6219d5be2cf5479dbf22f7c2b8b2f8256d8184b (image=quay.io/prometheus/prometheus:v2.43.0, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a, maintainer=The Prometheus Authors ) 2024-09-21T08:26:09.478 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.473Z caller=main.go:564 level=info msg="Starting Prometheus Server" mode=server version="(version=2.43.0, branch=HEAD, revision=edfc3bcd025dd6fe296c167a14a216cab1e552ee)" 2024-09-21T08:26:09.478 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.473Z caller=main.go:569 level=info build_context="(go=go1.19.7, platform=linux/amd64, user=root@8a0ee342e522, date=20230321-12:56:07, tags=netgo,builtinassets)" 2024-09-21T08:26:09.478 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.473Z caller=main.go:570 level=info host_details="(Linux 5.14.0-508.el9.x86_64 #1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024 x86_64 smithi165 (none))" 2024-09-21T08:26:09.479 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.473Z caller=main.go:571 level=info fd_limits="(soft=1048576, hard=1048576)" 2024-09-21T08:26:09.479 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.473Z caller=main.go:572 level=info vm_limits="(soft=unlimited, hard=unlimited)" 2024-09-21T08:26:09.479 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.474Z caller=web.go:561 level=info component=web msg="Start listening for connections" address=:9095 2024-09-21T08:26:09.479 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.475Z caller=main.go:1005 level=info msg="Starting TSDB ..." 2024-09-21T08:26:09.479 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.476Z caller=tls_config.go:232 level=info component=web msg="Listening on" address=[::]:9095 2024-09-21T08:26:09.479 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.476Z caller=tls_config.go:235 level=info component=web msg="TLS is disabled." http2=false address=[::]:9095 2024-09-21T08:26:09.479 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.477Z caller=head.go:587 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any" 2024-09-21T08:26:09.479 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.477Z caller=head.go:658 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=1.495µs 2024-09-21T08:26:09.479 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.477Z caller=head.go:664 level=info component=tsdb msg="Replaying WAL, this may take a while" 2024-09-21T08:26:09.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:09.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:09.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:09.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:09.543 INFO:tasks.rados.rados.0.smithi137.stdout:526: done (3 left) 2024-09-21T08:26:09.544 INFO:tasks.rados.rados.0.smithi137.stdout:527: done (2 left) 2024-09-21T08:26:09.544 INFO:tasks.rados.rados.0.smithi137.stdout:528: done (1 left) 2024-09-21T08:26:09.544 INFO:tasks.rados.rados.0.smithi137.stdout:529: done (0 left) 2024-09-21T08:26:09.544 INFO:tasks.rados.rados.0.smithi137.stdout:530: read oid 42 snap -1 2024-09-21T08:26:09.544 INFO:tasks.rados.rados.0.smithi137.stdout:530: expect deleted 2024-09-21T08:26:09.544 INFO:tasks.rados.rados.0.smithi137.stdout:531: write oid 3 current snap is 46 2024-09-21T08:26:09.544 INFO:tasks.rados.rados.0.smithi137.stdout:531: seq_num 220 ranges {497540=612664,1840862=440060,3058823=464784} 2024-09-21T08:26:09.549 INFO:tasks.rados.rados.0.smithi137.stdout:531: writing smithi13779221-3 from 497540 to 1110204 tid 1 2024-09-21T08:26:09.556 INFO:tasks.rados.rados.0.smithi137.stdout:531: writing smithi13779221-3 from 1840862 to 2280922 tid 2 2024-09-21T08:26:09.563 INFO:tasks.rados.rados.0.smithi137.stdout:531: writing smithi13779221-3 from 3058823 to 3523607 tid 3 2024-09-21T08:26:09.563 INFO:tasks.rados.rados.0.smithi137.stdout:532: snap_create 2024-09-21T08:26:09.563 INFO:tasks.rados.rados.0.smithi137.stdout:531: finishing write tid 1 to smithi13779221-3 2024-09-21T08:26:09.563 INFO:tasks.rados.rados.0.smithi137.stdout:531: finishing write tid 2 to smithi13779221-3 2024-09-21T08:26:09.565 INFO:tasks.rados.rados.0.smithi137.stdout:531: finishing write tid 3 to smithi13779221-3 2024-09-21T08:26:09.565 INFO:tasks.rados.rados.0.smithi137.stdout:531: finishing write tid 4 to smithi13779221-3 2024-09-21T08:26:09.566 INFO:tasks.rados.rados.0.smithi137.stdout:531: finishing write tid 5 to smithi13779221-3 2024-09-21T08:26:09.566 INFO:tasks.rados.rados.0.smithi137.stdout:531: oid 3 updating version 0 to 142 2024-09-21T08:26:09.566 INFO:tasks.rados.rados.0.smithi137.stdout:531: oid 3 version 142 is already newer than 140 2024-09-21T08:26:09.566 INFO:tasks.rados.rados.0.smithi137.stdout:531: oid 3 version 142 is already newer than 141 2024-09-21T08:26:09.566 INFO:tasks.rados.rados.0.smithi137.stdout:531: oid 3 updating version 142 to 143 2024-09-21T08:26:09.566 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 143 (ObjNum 220 snap 46 seq_num 220) dirty exists 2024-09-21T08:26:09.566 INFO:tasks.rados.rados.0.smithi137.stdout:531: left oid 3 (ObjNum 220 snap 46 seq_num 220) 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.24394.0:1 2.1 2:c4c92e5a:::rbd_trash:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e66)", 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:30.537490+0000", 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 159.03618004099999, 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010371768999999999, 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:09.576 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.24394", 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.165:0/2482038767", 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537490+0000", 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537490+0000", 2024-09-21T08:26:09.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537499+0000", 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8969999999999997e-06 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537514+0000", 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4524e-05 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537527+0000", 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3788000000000001e-05 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537537+0000", 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.007e-05 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537597+0000", 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9914000000000003e-05 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:09.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.537603+0000", 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1480000000000003e-06 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.544050+0000", 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064466499999999999 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.544174+0000", 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012360800000000001 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,3", 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.544298+0000", 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124428 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.545421+0000", 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001123162 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.547715+0000", 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022933649999999999 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.547754+0000", 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9532e-05 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.547813+0000", 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8746999999999998e-05 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:30.547862+0000", 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8936e-05 2024-09-21T08:26:09.582 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:260 7.8 7:1fd29771:::smithi13779221-22:head [copy-from ver 15] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 83.019347940000003, 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025065418999999999, 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.583 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 260 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554325+0000", 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.667e-06 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554332+0000", 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1600000000000001e-06 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554346+0000", 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4011e-05 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554355+0000", 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9199999999999993e-06 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554524+0000", 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000169469 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554534+0000", 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5489999999999995e-06 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557746+0000", 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003212164 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557822+0000", 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6108999999999999e-05 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568552+0000", 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01072972 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574773+0000", 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062208760000000002 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576999+0000", 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022266059999999999 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579275+0000", 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002276054 2024-09-21T08:26:09.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:09.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579335+0000", 2024-09-21T08:26:09.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9775000000000003e-05 2024-09-21T08:26:09.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579388+0000", 2024-09-21T08:26:09.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2339000000000002e-05 2024-09-21T08:26:09.590 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.590 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.590 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.590 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:261 7.8 7:1fd29771:::smithi13779221-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:09.590 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:26:09.590 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 83.019203379000004, 2024-09-21T08:26:09.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020259856, 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 261 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554468+0000", 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0219999999999999e-06 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554475+0000", 2024-09-21T08:26:09.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.81e-06 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554484+0000", 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0550000000000005e-06 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554491+0000", 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2080000000000003e-06 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554607+0000", 2024-09-21T08:26:09.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011585400000000001 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554612+0000", 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0130000000000001e-06 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557969+0000", 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033576320000000002 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:09.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558007+0000", 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.735e-05 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574538+0000", 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016531694 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574630+0000", 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1902000000000002e-05 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574727+0000", 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6316000000000004e-05 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:332 7.3 7:c851e2c3:::smithi13779221-25:head [read 0~2403368,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 81.887526558000005, 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90930888700000001, 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.596 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 332 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686147+0000", 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5370000000000002e-06 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686160+0000", 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2542e-05 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7927999999999998e-05 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686187+0000", 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3610000000000007e-06 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686261+0000", 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4197999999999997e-05 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:26:09.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686271+0000", 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5259999999999999e-06 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.590700+0000", 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90442903500000005 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.590838+0000", 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137863 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595452+0000", 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046148969999999997 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:333 7.3 7:c851e2c3:::smithi13779221-25:head [read 0~2403368,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 81.887463029000003, 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91060290700000002, 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.601 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 333 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686208+0000", 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4900000000000005e-07 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686212+0000", 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9940000000000002e-06 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686221+0000", 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8789999999999992e-06 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686225+0000", 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5059999999999997e-06 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686303+0000", 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7855000000000006e-05 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686310+0000", 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0829999999999998e-06 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595485+0000", 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90917505300000001 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595590+0000", 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105059 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.596810+0000", 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012197289999999999 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:334 7.3 7:c851e2c3:::smithi13779221-25:head [sparse-read 0~1282710] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 81.887426023000003, 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91097041300000003, 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.606 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 334 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686245+0000", 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2600000000000002e-07 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686247+0000", 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7429999999999998e-06 2024-09-21T08:26:09.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686255+0000", 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8369999999999997e-06 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686259+0000", 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0509999999999998e-06 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686326+0000", 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6845000000000001e-05 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686331+0000", 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.13e-06 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.596911+0000", 2024-09-21T08:26:09.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.910579845 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.597015+0000", 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103538 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.597215+0000", 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000199798 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.611 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:569 7.e 7:78d05d33:::smithi13779221-41:head [copy-from ver 7] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.026977297999998, 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033085196999999997, 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 569 2024-09-21T08:26:09.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546695+0000", 2024-09-21T08:26:09.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3920000000000001e-06 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546704+0000", 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9330000000000007e-06 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546720+0000", 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5651999999999999e-05 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546731+0000", 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1147e-05 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546794+0000", 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3214999999999997e-05 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546801+0000", 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3350000000000001e-06 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556379+0000", 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095778679999999998 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556505+0000", 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012529699999999999 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564765+0000", 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082601020000000001 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576097+0000", 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011331938999999999 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576715+0000", 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00061843800000000002 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579665+0000", 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029497 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579729+0000", 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4337000000000004e-05 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579778+0000", 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8841999999999998e-05 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:570 7.e 7:78d05d33:::smithi13779221-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:26:09.618 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.026836207000002, 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024747969000000002, 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 570 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:26:09.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546835+0000", 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4e-07 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546838+0000", 2024-09-21T08:26:09.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4829999999999999e-06 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546845+0000", 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5810000000000002e-06 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546857+0000", 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2034e-05 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546900+0000", 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2945000000000003e-05 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546905+0000", 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2700000000000004e-06 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556676+0000", 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097709930000000004 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556717+0000", 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.104e-05 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571353+0000", 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014636287 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571465+0000", 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111943 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571582+0000", 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011655299999999999 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:579 7.8 7:144c2ec5:::smithi13779221-45:head [write 430279~695389,stat] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.555701+0000", 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.017968871000001, 2024-09-21T08:26:09.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013969973, 2024-09-21T08:26:09.625 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.625 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.625 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:09.625 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.625 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.625 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.625 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 579 2024-09-21T08:26:09.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.625 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.555701+0000", 2024-09-21T08:26:09.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.555701+0000", 2024-09-21T08:26:09.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.555705+0000", 2024-09-21T08:26:09.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9079999999999997e-06 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556188+0000", 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000483191 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556205+0000", 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6733000000000001e-05 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556219+0000", 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4174e-05 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.558737+0000", 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025182009999999999 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.558873+0000", 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013586100000000001 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.559091+0000", 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021720299999999999 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.562602+0000", 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035114299999999998 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.563008+0000", 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040637600000000001 2024-09-21T08:26:09.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.569474+0000", 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0064653660000000002 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.569603+0000", 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012898599999999999 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.569671+0000", 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8544e-05 2024-09-21T08:26:09.631 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.632 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.632 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.632 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:580 7.8 7:144c2ec5:::smithi13779221-45:head [write 1876518~654358] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:26:09.632 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.565488+0000", 2024-09-21T08:26:09.632 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.008182017999999, 2024-09-21T08:26:09.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011725635, 2024-09-21T08:26:09.632 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.633 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.633 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:09.633 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.633 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 580 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565488+0000", 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565488+0000", 2024-09-21T08:26:09.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565492+0000", 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8709999999999999e-06 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565881+0000", 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00038888699999999998 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565899+0000", 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7978999999999998e-05 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565910+0000", 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0926e-05 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566036+0000", 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012639300000000001 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566147+0000", 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110665 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566310+0000", 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016324699999999999 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.568318+0000", 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002007988 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571597+0000", 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032789519999999999 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576888+0000", 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052913480000000004 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.577141+0000", 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025245299999999998 2024-09-21T08:26:09.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.577214+0000", 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2925999999999999e-05 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:583 7.8 7:144c2ec5:::smithi13779221-45:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.575211+0000", 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 58.998459355000001, 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094641029999999998, 2024-09-21T08:26:09.639 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 583 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.575211+0000", 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.575211+0000", 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.575211+0000", 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2900000000000004e-07 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.575214+0000", 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.469e-06 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.575219+0000", 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4480000000000002e-06 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.575224+0000", 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8189999999999999e-06 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.577752+0000", 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025276500000000002 2024-09-21T08:26:09.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.577832+0000", 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0647000000000002e-05 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584423+0000", 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065904129999999998 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584519+0000", 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6589000000000006e-05 2024-09-21T08:26:09.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584675+0000", 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155539 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:827 7.a 7:53537bd3:::smithi13779221-39:1b [read 0~3159683,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.705976+0000", 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 39.867693868000003, 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011099859, 2024-09-21T08:26:09.644 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 827 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705976+0000", 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705976+0000", 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705980+0000", 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.602e-06 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705991+0000", 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1117e-05 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706012+0000", 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1107999999999998e-05 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706023+0000", 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0895e-05 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706127+0000", 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104374 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706134+0000", 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7950000000000002e-06 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.713152+0000", 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0070179229999999997 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.713256+0000", 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103609 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.717076+0000", 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003820436 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:828 7.a 7:53537bd3:::smithi13779221-39:1b [sparse-read 0~1389729] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.706054+0000", 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 39.867615897, 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011281029, 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 828 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.650 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706054+0000", 2024-09-21T08:26:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706054+0000", 2024-09-21T08:26:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706055+0000", 2024-09-21T08:26:09.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4499999999999996e-07 2024-09-21T08:26:09.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706059+0000", 2024-09-21T08:26:09.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5499999999999999e-06 2024-09-21T08:26:09.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706070+0000", 2024-09-21T08:26:09.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1276e-05 2024-09-21T08:26:09.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706075+0000", 2024-09-21T08:26:09.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6169999999999998e-06 2024-09-21T08:26:09.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706176+0000", 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100511 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706182+0000", 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6550000000000001e-06 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.717114+0000", 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010932860000000001 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.717211+0000", 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6737999999999994e-05 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.717335+0000", 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000124077 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.656 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:829 7.a 7:53537bd3:::smithi13779221-39:1b [sparse-read 0~1389729] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.706147+0000", 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 39.867523204000001, 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011548141, 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 829 2024-09-21T08:26:09.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706147+0000", 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706147+0000", 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706148+0000", 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6000000000000002e-07 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706151+0000", 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3069999999999998e-06 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706158+0000", 2024-09-21T08:26:09.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8199999999999999e-06 2024-09-21T08:26:09.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706164+0000", 2024-09-21T08:26:09.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8089999999999998e-06 2024-09-21T08:26:09.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706204+0000", 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0027000000000001e-05 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706208+0000", 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4610000000000001e-06 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.717489+0000", 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011281253999999999 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.717582+0000", 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2375999999999998e-05 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.717695+0000", 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011322700000000001 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.662 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1051 7.8 7:1fd29771:::smithi13779221-22:head [copy-from ver 34] snapc 22=[22,21,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:41.117869+0000", 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 28.455801043000001, 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016292628, 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1051 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.663 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117869+0000", 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117869+0000", 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117872+0000", 2024-09-21T08:26:09.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.061e-06 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117883+0000", 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0559e-05 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117902+0000", 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9099000000000001e-05 2024-09-21T08:26:09.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117911+0000", 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7819999999999999e-06 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117980+0000", 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9146999999999994e-05 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.118082+0000", 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102656 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.124659+0000", 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065771120000000004 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.128416+0000", 2024-09-21T08:26:09.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0037568300000000001 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.133983+0000", 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055663359999999999 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.134089+0000", 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010618199999999999 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.134127+0000", 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8631000000000003e-05 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.134162+0000", 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4233000000000002e-05 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.669 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1052 7.8 7:1fd29771:::smithi13779221-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e147)", 2024-09-21T08:26:09.670 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:41.117932+0000", 2024-09-21T08:26:09.670 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 28.455737765999999, 2024-09-21T08:26:09.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097151149999999999, 2024-09-21T08:26:09.670 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.670 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.670 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.670 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.670 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.670 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.670 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1052 2024-09-21T08:26:09.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.671 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117932+0000", 2024-09-21T08:26:09.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117932+0000", 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117933+0000", 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7000000000000003e-07 2024-09-21T08:26:09.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117937+0000", 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3510000000000002e-06 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117944+0000", 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9840000000000004e-06 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.117949+0000", 2024-09-21T08:26:09.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8649999999999998e-06 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.118244+0000", 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029480100000000003 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.118288+0000", 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.354e-05 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.127488+0000", 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0092002809999999994 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.127565+0000", 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6958000000000004e-05 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:41.127647+0000", 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2465000000000004e-05 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.675 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:1392 7.c e172/102 7:3cae00e7:::smithi13779221-28:head v 172'90, mlcod=171'87)", 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:05.565981+0000", 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 4.0076887650000002, 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0099145220000000003, 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.565981+0000", 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.565981+0000", 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.565984+0000", 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.356e-06 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.570731+0000", 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047470029999999996 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.570749+0000", 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8448999999999999e-05 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.570759+0000", 2024-09-21T08:26:09.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.764e-06 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.570840+0000", 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1187999999999996e-05 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.570871+0000", 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1068999999999997e-05 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:09.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.575822+0000", 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0049504989999999997 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.575896+0000", 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4194000000000001e-05 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1420 7.e 7:795bb4fc:::smithi13779221-26:head [read 0~3630312,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e174)", 2024-09-21T08:26:09.680 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:07.824011+0000", 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.749659579, 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011032350999999999, 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1420 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824011+0000", 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824011+0000", 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824011+0000", 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8100000000000003e-07 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824015+0000", 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4510000000000001e-06 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824020+0000", 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.49e-06 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824036+0000", 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5557999999999999e-05 2024-09-21T08:26:09.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.829601+0000", 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055650370000000001 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.829726+0000", 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012548400000000001 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.835043+0000", 2024-09-21T08:26:09.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0053167500000000003 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1421 7.e 7:795bb4fc:::smithi13779221-26:head [sparse-read 0~1964086] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e174)", 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:07.824054+0000", 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.749615693, 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011337691, 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.685 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1421 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824054+0000", 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824054+0000", 2024-09-21T08:26:09.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824056+0000", 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1039999999999999e-06 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824059+0000", 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6370000000000001e-06 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824066+0000", 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5139999999999998e-06 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824071+0000", 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2800000000000003e-06 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.835114+0000", 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011043116 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.835223+0000", 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108718 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.835392+0000", 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016932199999999999 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1422 7.e 7:795bb4fc:::smithi13779221-26:head [read 0~3630312,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e174)", 2024-09-21T08:26:09.689 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:07.824077+0000", 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.749592893, 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012821325, 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1422 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:09.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824077+0000", 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824077+0000", 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824078+0000", 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0100000000000005e-07 2024-09-21T08:26:09.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:09.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824081+0000", 2024-09-21T08:26:09.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4800000000000001e-06 2024-09-21T08:26:09.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:09.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824088+0000", 2024-09-21T08:26:09.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9430000000000003e-06 2024-09-21T08:26:09.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:09.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.824095+0000", 2024-09-21T08:26:09.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3010000000000003e-06 2024-09-21T08:26:09.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:09.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.835418+0000", 2024-09-21T08:26:09.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011323229000000001 2024-09-21T08:26:09.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:09.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.835462+0000", 2024-09-21T08:26:09.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4094999999999997e-05 2024-09-21T08:26:09.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:09.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:09.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:09.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:07.836899+0000", 2024-09-21T08:26:09.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001436676 2024-09-21T08:26:09.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.695 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:09.695 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:09.695 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:26:09.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:09.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:09.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:09.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:09.721 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-mon[29634]: pgmap v167: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 115 MiB data, 408 MiB used, 357 GiB / 358 GiB avail; 2.2 KiB/s rd, 2.8 MiB/s wr, 9 op/s 2024-09-21T08:26:09.721 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-mon[29634]: osdmap e175: 8 total, 8 up, 4 in 2024-09-21T08:26:09.721 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-mon[41044]: pgmap v167: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 115 MiB data, 408 MiB used, 357 GiB / 358 GiB avail; 2.2 KiB/s rd, 2.8 MiB/s wr, 9 op/s 2024-09-21T08:26:09.721 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-mon[41044]: osdmap e175: 8 total, 8 up, 4 in 2024-09-21T08:26:09.722 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:09] ENGINE Bus STOPPING 2024-09-21T08:26:09.722 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:09] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T08:26:09.722 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:09] ENGINE Bus STOPPED 2024-09-21T08:26:09.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-mon[32893]: pgmap v167: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 115 MiB data, 408 MiB used, 357 GiB / 358 GiB avail; 2.2 KiB/s rd, 2.8 MiB/s wr, 9 op/s 2024-09-21T08:26:09.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-mon[32893]: osdmap e175: 8 total, 8 up, 4 in 2024-09-21T08:26:09.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.478Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1 2024-09-21T08:26:09.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.479Z caller=head.go:735 level=info component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1 2024-09-21T08:26:09.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.479Z caller=head.go:772 level=info component=tsdb msg="WAL replay completed" checkpoint_replay_duration=18.168µs wal_replay_duration=1.944025ms wbl_replay_duration=86ns total_replay_duration=1.979698ms 2024-09-21T08:26:09.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.479Z caller=main.go:1026 level=info fs_type=XFS_SUPER_MAGIC 2024-09-21T08:26:09.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.479Z caller=main.go:1029 level=info msg="TSDB started" 2024-09-21T08:26:09.763 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.479Z caller=main.go:1209 level=info msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml 2024-09-21T08:26:09.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.501Z caller=main.go:1246 level=info msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=21.3537ms db_storage=741ns remote_storage=1.044µs web_handler=395ns query_engine=843ns scrape=394.439µs scrape_sd=85.018µs notify=10.156µs notify_sd=8.737µs rules=20.555387ms tracing=6.472µs 2024-09-21T08:26:09.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.501Z caller=main.go:990 level=info msg="Server is ready to receive web requests." 2024-09-21T08:26:09.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-prometheus-a[74841]: ts=2024-09-21T08:26:09.501Z caller=manager.go:974 level=info component="rule manager" msg="Starting rule manager..." 2024-09-21T08:26:09.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 bash[74826]: 1700e997ea3d83a76b6add08d6219d5be2cf5479dbf22f7c2b8b2f8256d8184b 2024-09-21T08:26:09.764 INFO:journalctl@ceph.prometheus.a.smithi165.stdout:Sep 21 08:26:09 smithi165 systemd[1]: Started Ceph prometheus.a for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:26:09.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:09.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:09.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:09.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:09.843 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:09.843 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:09.843 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:09.843 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:09.843 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:09.843 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:09.843 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:09.843 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:09.848 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:09.848 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:09.865 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:09.866 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:09.881 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:09.882 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:09.898 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:09.898 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:09.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:09.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:09.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:09.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:10.017 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:09] ENGINE Bus STARTING 2024-09-21T08:26:10.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:09] ENGINE Serving on http://:::9283 2024-09-21T08:26:10.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:09] ENGINE Bus STARTED 2024-09-21T08:26:10.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:09 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:09] ENGINE Bus STOPPING 2024-09-21T08:26:10.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:10.095 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:10.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:10.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:10.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:10.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:26:10.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:10.222 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:10.222 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:10.323 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:10.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:10.353 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:10.353 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:10.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:10.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:10.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:10.481 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:10.518 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T08:26:10.518 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:10] ENGINE Bus STOPPED 2024-09-21T08:26:10.518 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:10] ENGINE Bus STARTING 2024-09-21T08:26:10.519 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:10] ENGINE Serving on http://:::9283 2024-09-21T08:26:10.519 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:10] ENGINE Bus STARTED 2024-09-21T08:26:10.519 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:10] ENGINE Bus STOPPING 2024-09-21T08:26:10.550 INFO:tasks.rados.rados.0.smithi137.stdout:530: done (2 left) 2024-09-21T08:26:10.550 INFO:tasks.rados.rados.0.smithi137.stdout:531: done (1 left) 2024-09-21T08:26:10.551 INFO:tasks.rados.rados.0.smithi137.stdout:532: done (0 left) 2024-09-21T08:26:10.551 INFO:tasks.rados.rados.0.smithi137.stdout:533: snap_remove snap 45 2024-09-21T08:26:10.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:10.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:10.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:10.605 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:10.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:10.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:10.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:10.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:10.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:10.855 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:10.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:10.856 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:10.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: 6.d scrub starts 2024-09-21T08:26:10.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: 6.d scrub ok 2024-09-21T08:26:10.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: osdmap e176: 8 total, 8 up, 4 in 2024-09-21T08:26:10.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:10.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:10.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T08:26:10.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T08:26:10.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T08:26:10.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T08:26:10.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:26:10.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/450646705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: 6.d scrub starts 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: 6.d scrub ok 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: osdmap e176: 8 total, 8 up, 4 in 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:10.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T08:26:10.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T08:26:10.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T08:26:10.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T08:26:10.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:10.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T08:26:10.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T08:26:10.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T08:26:10.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T08:26:10.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:10.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:26:10.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/450646705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:10.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T08:26:10.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:10.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:10.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:10.983 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: 6.d scrub starts 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: 6.d scrub ok 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: osdmap e176: 8 total, 8 up, 4 in 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi137.front.sepia.ceph.com:9093"}]: dispatch 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-09-21T08:26:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T08:26:11.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi165.front.sepia.ceph.com:9095"}]: dispatch 2024-09-21T08:26:11.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:11.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T08:26:11.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-09-21T08:26:11.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T08:26:11.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mon.? v1:172.21.15.137:0/987000754' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi165.front.sepia.ceph.com:3000"}]: dispatch 2024-09-21T08:26:11.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:11.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:26:11.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:10 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/450646705' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:11.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:11.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:11.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:11.106 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:11.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:11.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:11.230 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:11.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:11.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:10] ENGINE HTTP Server cherrypy._cpwsgi_server.CPWSGIServer(('::', 9283)) shut down 2024-09-21T08:26:11.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:10] ENGINE Bus STOPPED 2024-09-21T08:26:11.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:10] ENGINE Bus STARTING 2024-09-21T08:26:11.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:11] ENGINE Serving on http://:::9283 2024-09-21T08:26:11.268 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: [21/Sep/2024:08:26:11] ENGINE Bus STARTED 2024-09-21T08:26:11.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:11.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:11.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:11.355 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:11.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:11.478 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:11.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:11.479 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:11.505 INFO:tasks.rados.rados.0.smithi137.stdout:533: done (0 left) 2024-09-21T08:26:11.505 INFO:tasks.rados.rados.0.smithi137.stdout:534: write (excl) oid 28 current snap is 47 2024-09-21T08:26:11.505 INFO:tasks.rados.rados.0.smithi137.stdout:534: seq_num 221 ranges {514347=404266,1499344=700071,2686246=676056} 2024-09-21T08:26:11.510 INFO:tasks.rados.rados.0.smithi137.stdout:534: writing smithi13779221-28 from 514347 to 918613 tid 1 2024-09-21T08:26:11.519 INFO:tasks.rados.rados.0.smithi137.stdout:534: writing smithi13779221-28 from 1499344 to 2199415 tid 2 2024-09-21T08:26:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:534: writing smithi13779221-28 from 2686246 to 3362302 tid 3 2024-09-21T08:26:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:535: delete oid 25 current snap is 47 2024-09-21T08:26:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:534: finishing write tid 1 to smithi13779221-28 2024-09-21T08:26:11.533 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:11.535 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:11.541 INFO:tasks.rados.rados.0.smithi137.stdout:535: done (1 left) 2024-09-21T08:26:11.541 INFO:tasks.rados.rados.0.smithi137.stdout:536: snap_remove snap 46 2024-09-21T08:26:11.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:11.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:11.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:11.603 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:11.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:11.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:11.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:11.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:11.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[41044]: 5.1a deep-scrub starts 2024-09-21T08:26:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[41044]: 5.1a deep-scrub ok 2024-09-21T08:26:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[41044]: pgmap v170: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 115 MiB data, 408 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[41044]: 5.3 deep-scrub starts 2024-09-21T08:26:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[41044]: 5.3 deep-scrub ok 2024-09-21T08:26:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[41044]: osdmap e177: 8 total, 8 up, 4 in 2024-09-21T08:26:11.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[41044]: osdmap e178: 8 total, 8 up, 4 in 2024-09-21T08:26:11.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[29634]: 5.1a deep-scrub starts 2024-09-21T08:26:11.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[29634]: 5.1a deep-scrub ok 2024-09-21T08:26:11.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[29634]: pgmap v170: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 115 MiB data, 408 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:11.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[29634]: 5.3 deep-scrub starts 2024-09-21T08:26:11.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[29634]: 5.3 deep-scrub ok 2024-09-21T08:26:11.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[29634]: osdmap e177: 8 total, 8 up, 4 in 2024-09-21T08:26:11.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:11 smithi137 ceph-mon[29634]: osdmap e178: 8 total, 8 up, 4 in 2024-09-21T08:26:11.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:11.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:11.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:11.849 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:11.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:11 smithi165 ceph-mon[32893]: 5.1a deep-scrub starts 2024-09-21T08:26:11.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:11 smithi165 ceph-mon[32893]: 5.1a deep-scrub ok 2024-09-21T08:26:11.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:11 smithi165 ceph-mon[32893]: pgmap v170: 148 pgs: 3 active+clean+snaptrim, 145 active+clean; 115 MiB data, 408 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:11.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:11 smithi165 ceph-mon[32893]: 5.3 deep-scrub starts 2024-09-21T08:26:11.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:11 smithi165 ceph-mon[32893]: 5.3 deep-scrub ok 2024-09-21T08:26:11.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:11 smithi165 ceph-mon[32893]: osdmap e177: 8 total, 8 up, 4 in 2024-09-21T08:26:11.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:11 smithi165 ceph-mon[32893]: osdmap e178: 8 total, 8 up, 4 in 2024-09-21T08:26:11.915 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:26:11 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:26:11.950 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:11.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:11.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:11.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:12.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:12.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:12.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:12.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:12.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:12.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:12.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:12.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:12.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:12.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:12.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:12.341 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:12.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:12.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:12.465 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:12.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:12.465 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:12.507 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:26:12.513 INFO:tasks.rados.rados.0.smithi137.stdout:536: done (1 left) 2024-09-21T08:26:12.513 INFO:tasks.rados.rados.0.smithi137.stdout:537: write oid 11 current snap is 47 2024-09-21T08:26:12.513 INFO:tasks.rados.rados.0.smithi137.stdout:534: finishing write tid 2 to smithi13779221-28 2024-09-21T08:26:12.513 INFO:tasks.rados.rados.0.smithi137.stdout:537: seq_num 223 ranges {547961=554764,1616308=703740,2741092=698600} 2024-09-21T08:26:12.520 INFO:tasks.rados.rados.0.smithi137.stdout:537: writing smithi13779221-11 from 547961 to 1102725 tid 1 2024-09-21T08:26:12.528 INFO:tasks.rados.rados.0.smithi137.stdout:537: writing smithi13779221-11 from 1616308 to 2320048 tid 2 2024-09-21T08:26:12.539 INFO:tasks.rados.rados.0.smithi137.stdout:537: writing smithi13779221-11 from 2741092 to 3439692 tid 3 2024-09-21T08:26:12.540 INFO:tasks.rados.rados.0.smithi137.stdout:534: finishing write tid 3 to smithi13779221-28 2024-09-21T08:26:12.540 INFO:tasks.rados.rados.0.smithi137.stdout:534: finishing write tid 4 to smithi13779221-28 2024-09-21T08:26:12.540 INFO:tasks.rados.rados.0.smithi137.stdout:534: finishing write tid 5 to smithi13779221-28 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:534: oid 28 updating version 0 to 95 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:534: oid 28 version 95 is already newer than 94 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:534: oid 28 version 95 is already newer than 93 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:534: oid 28 version 95 is already newer than 92 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 95 (ObjNum 221 snap 47 seq_num 221) dirty exists 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:534: left oid 28 (ObjNum 221 snap 47 seq_num 221) 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:537: finishing write tid 1 to smithi13779221-11 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:537: finishing write tid 2 to smithi13779221-11 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:534: done (1 left) 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:538: rollback oid 36 current snap is 47 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 36 to 44 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:539: copy_from oid 23 from oid 27 current snap is 47 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:540: write oid 1 current snap is 47 2024-09-21T08:26:12.541 INFO:tasks.rados.rados.0.smithi137.stdout:540: seq_num 224 ranges {564768=630013,1674790=447398,2768515=626225} 2024-09-21T08:26:12.548 INFO:tasks.rados.rados.0.smithi137.stdout:540: writing smithi13779221-1 from 564768 to 1194781 tid 1 2024-09-21T08:26:12.555 INFO:tasks.rados.rados.0.smithi137.stdout:540: writing smithi13779221-1 from 1674790 to 2122188 tid 2 2024-09-21T08:26:12.562 INFO:tasks.rados.rados.0.smithi137.stdout:540: writing smithi13779221-1 from 2768515 to 3394740 tid 3 2024-09-21T08:26:12.562 INFO:tasks.rados.rados.0.smithi137.stdout:541: write (excl) oid 41 current snap is 47 2024-09-21T08:26:12.562 INFO:tasks.rados.rados.0.smithi137.stdout:541: seq_num 225 ranges {581575=705262,1733272=707409,3195938=437497} 2024-09-21T08:26:12.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:12.569 INFO:tasks.rados.rados.0.smithi137.stdout:541: writing smithi13779221-41 from 581575 to 1286837 tid 1 2024-09-21T08:26:12.575 INFO:tasks.rados.rados.0.smithi137.stdout:541: writing smithi13779221-41 from 1733272 to 2440681 tid 2 2024-09-21T08:26:12.582 INFO:tasks.rados.rados.0.smithi137.stdout:541: writing smithi13779221-41 from 3195938 to 3633435 tid 3 2024-09-21T08:26:12.582 INFO:tasks.rados.rados.0.smithi137.stdout:542: write (excl) oid 22 current snap is 47 2024-09-21T08:26:12.583 INFO:tasks.rados.rados.0.smithi137.stdout:537: finishing write tid 3 to smithi13779221-11 2024-09-21T08:26:12.583 INFO:tasks.rados.rados.0.smithi137.stdout:542: seq_num 226 ranges {598382=780511,2075401=451067,3107008=648769} 2024-09-21T08:26:12.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:12.590 INFO:tasks.rados.rados.0.smithi137.stdout:542: writing smithi13779221-22 from 598382 to 1378893 tid 1 2024-09-21T08:26:12.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:12.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:12.596 INFO:tasks.rados.rados.0.smithi137.stdout:542: writing smithi13779221-22 from 2075401 to 2526468 tid 2 2024-09-21T08:26:12.604 INFO:tasks.rados.rados.0.smithi137.stdout:542: writing smithi13779221-22 from 3107008 to 3755777 tid 3 2024-09-21T08:26:12.604 INFO:tasks.rados.rados.0.smithi137.stdout:543: delete oid 14 current snap is 47 2024-09-21T08:26:12.604 INFO:tasks.rados.rados.0.smithi137.stdout:537: finishing write tid 4 to smithi13779221-11 2024-09-21T08:26:12.604 INFO:tasks.rados.rados.0.smithi137.stdout:537: finishing write tid 5 to smithi13779221-11 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:537: oid 11 updating version 0 to 74 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:537: oid 11 version 74 is already newer than 73 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:537: oid 11 updating version 74 to 75 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:537: oid 11 updating version 75 to 76 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 76 (ObjNum 223 snap 47 seq_num 223) dirty exists 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:537: left oid 11 (ObjNum 223 snap 47 seq_num 223) 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:538: finishing rollback tid 0 to smithi13779221-36 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:538: finishing rollback tid 1 to smithi13779221-36 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:538: finishing rollback tid 2 to smithi13779221-36 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 232 (ObjNum 181 snap 36 seq_num 181) dirty exists 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:539: finishing copy_from racing read to smithi13779221-23 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:540: finishing write tid 1 to smithi13779221-1 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:539: finishing copy_from to smithi13779221-23 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 164 (ObjNum 213 snap 43 seq_num 213) dirty exists 2024-09-21T08:26:12.605 INFO:tasks.rados.rados.0.smithi137.stdout:540: finishing write tid 2 to smithi13779221-1 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:540: finishing write tid 3 to smithi13779221-1 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:540: finishing write tid 4 to smithi13779221-1 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:540: finishing write tid 5 to smithi13779221-1 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:540: oid 1 updating version 0 to 10 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:540: oid 1 updating version 10 to 12 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:540: oid 1 version 12 is already newer than 9 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:540: oid 1 version 12 is already newer than 11 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 12 (ObjNum 224 snap 47 seq_num 224) dirty exists 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:540: left oid 1 (ObjNum 224 snap 47 seq_num 224) 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:541: finishing write tid 1 to smithi13779221-41 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:541: finishing write tid 2 to smithi13779221-41 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:541: finishing write tid 3 to smithi13779221-41 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:541: finishing write tid 4 to smithi13779221-41 2024-09-21T08:26:12.606 INFO:tasks.rados.rados.0.smithi137.stdout:541: finishing write tid 5 to smithi13779221-41 2024-09-21T08:26:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:541: oid 41 updating version 0 to 87 2024-09-21T08:26:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:541: oid 41 version 87 is already newer than 84 2024-09-21T08:26:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:541: oid 41 version 87 is already newer than 86 2024-09-21T08:26:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:541: oid 41 version 87 is already newer than 85 2024-09-21T08:26:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 87 (ObjNum 225 snap 47 seq_num 225) dirty exists 2024-09-21T08:26:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:541: left oid 41 (ObjNum 225 snap 47 seq_num 225) 2024-09-21T08:26:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:542: finishing write tid 1 to smithi13779221-22 2024-09-21T08:26:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:542: finishing write tid 2 to smithi13779221-22 2024-09-21T08:26:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:537: done (6 left) 2024-09-21T08:26:12.608 INFO:tasks.rados.rados.0.smithi137.stdout:538: done (5 left) 2024-09-21T08:26:12.608 INFO:tasks.rados.rados.0.smithi137.stdout:539: done (4 left) 2024-09-21T08:26:12.608 INFO:tasks.rados.rados.0.smithi137.stdout:540: done (3 left) 2024-09-21T08:26:12.608 INFO:tasks.rados.rados.0.smithi137.stdout:541: done (2 left) 2024-09-21T08:26:12.608 INFO:tasks.rados.rados.0.smithi137.stdout:543: done (1 left) 2024-09-21T08:26:12.608 INFO:tasks.rados.rados.0.smithi137.stdout:544: copy_from oid 27 from oid 46 current snap is 47 2024-09-21T08:26:12.608 INFO:tasks.rados.rados.0.smithi137.stdout:545: snap_remove snap 44 2024-09-21T08:26:12.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:12.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:12.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:12.714 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:12.732 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:12 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4082107601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:12.732 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:12 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:12.732 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:12 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4082107601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:26:12.732 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:12 smithi137 ceph-mon[29634]: osdmap e179: 8 total, 8 up, 4 in 2024-09-21T08:26:12.732 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:12 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4082107601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:12.732 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:12 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:12.732 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:12 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4082107601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:26:12.732 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:12 smithi137 ceph-mon[41044]: osdmap e179: 8 total, 8 up, 4 in 2024-09-21T08:26:12.814 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:12.835 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:12.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:12.836 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:12.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:12 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4082107601' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:12.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:12 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:12.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:12 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4082107601' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:26:12.844 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:12 smithi165 ceph-mon[32893]: osdmap e179: 8 total, 8 up, 4 in 2024-09-21T08:26:12.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:12.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:12.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:12.958 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:13.059 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:13.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:13.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:13.080 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:13.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:13.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:13.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:13.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:13.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:13.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:13.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:13.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:13.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:13.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:13.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:13.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:13.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:26:13.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:13.575 INFO:tasks.rados.rados.0.smithi137.stdout:545: done (2 left) 2024-09-21T08:26:13.576 INFO:tasks.rados.rados.0.smithi137.stdout:546: read oid 43 snap -1 2024-09-21T08:26:13.576 INFO:tasks.rados.rados.0.smithi137.stdout:546: expect deleted 2024-09-21T08:26:13.576 INFO:tasks.rados.rados.0.smithi137.stdout:547: write oid 33 current snap is 47 2024-09-21T08:26:13.576 INFO:tasks.rados.rados.0.smithi137.stdout:547: seq_num 228 ranges {631996=531009,1792365=454736,2761854=671313} 2024-09-21T08:26:13.576 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 7 pg_num to 26 2024-09-21T08:26:13.579 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:13.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:13.580 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:13.581 INFO:tasks.rados.rados.0.smithi137.stdout:547: writing smithi13779221-33 from 631996 to 1163005 tid 1 2024-09-21T08:26:13.587 INFO:tasks.rados.rados.0.smithi137.stdout:547: writing smithi13779221-33 from 1792365 to 2247101 tid 2 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:547: writing smithi13779221-33 from 2761854 to 3433167 tid 3 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:548: read oid 35 snap -1 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:548: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:549: rollback oid 39 current snap is 47 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:549: done (5 left) 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:550: read oid 40 snap -1 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:550: expect (ObjNum 211 snap 43 seq_num 211) 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:551: rollback oid 49 current snap is 47 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:551: done (6 left) 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:552: copy_from oid 2 from oid 32 current snap is 47 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:542: finishing write tid 3 to smithi13779221-22 2024-09-21T08:26:13.595 INFO:tasks.rados.rados.0.smithi137.stdout:542: finishing write tid 4 to smithi13779221-22 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:542: finishing write tid 5 to smithi13779221-22 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:542: oid 22 updating version 0 to 159 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:542: oid 22 updating version 159 to 161 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:542: oid 22 version 161 is already newer than 158 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:542: oid 22 version 161 is already newer than 160 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 161 (ObjNum 226 snap 47 seq_num 226) dirty exists 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:542: left oid 22 (ObjNum 226 snap 47 seq_num 226) 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:544: finishing copy_from racing read to smithi13779221-27 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:544: finishing copy_from to smithi13779221-27 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 225 (ObjNum 218 snap 45 seq_num 218) dirty exists 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:547: finishing write tid 1 to smithi13779221-33 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:547: finishing write tid 2 to smithi13779221-33 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:542: done (6 left) 2024-09-21T08:26:13.596 INFO:tasks.rados.rados.0.smithi137.stdout:544: done (5 left) 2024-09-21T08:26:13.597 INFO:tasks.rados.rados.0.smithi137.stdout:546: done (4 left) 2024-09-21T08:26:13.597 INFO:tasks.rados.rados.0.smithi137.stdout:553: write (excl) oid 39 current snap is 47 2024-09-21T08:26:13.597 INFO:tasks.rados.rados.0.smithi137.stdout:553: seq_num 229 ranges {648803=722611,1967200=714747,3421983=426820} 2024-09-21T08:26:13.602 INFO:tasks.rados.rados.0.smithi137.stdout:553: writing smithi13779221-39 from 648803 to 1371414 tid 1 2024-09-21T08:26:13.610 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:13.610 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:13.611 INFO:tasks.rados.rados.0.smithi137.stdout:553: writing smithi13779221-39 from 1967200 to 2681947 tid 2 2024-09-21T08:26:13.617 INFO:tasks.rados.rados.0.smithi137.stdout:553: writing smithi13779221-39 from 3421983 to 3848803 tid 3 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:552: finishing copy_from to smithi13779221-2 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:552: got expected ENOENT (src dne) 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:552: finishing copy_from racing read to smithi13779221-2 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:547: finishing write tid 3 to smithi13779221-33 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:547: finishing write tid 4 to smithi13779221-33 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:547: finishing write tid 5 to smithi13779221-33 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:547: oid 33 updating version 0 to 47 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:547: oid 33 updating version 47 to 49 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:547: oid 33 updating version 49 to 50 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:547: oid 33 version 50 is already newer than 48 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 50 (ObjNum 228 snap 47 seq_num 228) dirty exists 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:547: left oid 33 (ObjNum 228 snap 47 seq_num 228) 2024-09-21T08:26:13.618 INFO:tasks.rados.rados.0.smithi137.stdout:548: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:26:13.681 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:13.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:13.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:13.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:13.747 INFO:tasks.rados.rados.0.smithi137.stdout:553: finishing write tid 1 to smithi13779221-39 2024-09-21T08:26:13.748 INFO:tasks.rados.rados.0.smithi137.stdout:550: expect (ObjNum 211 snap 43 seq_num 211) 2024-09-21T08:26:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:13 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:13 smithi165 ceph-mon[32893]: pgmap v173: 148 pgs: 148 active+clean; 118 MiB data, 409 MiB used, 357 GiB / 358 GiB avail; 27 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2024-09-21T08:26:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:13 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3921506749' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-21T08:26:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:13 smithi165 ceph-mon[32893]: from='client.44993 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-21T08:26:13.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:13 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:13.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:13 smithi137 ceph-mon[29634]: pgmap v173: 148 pgs: 148 active+clean; 118 MiB data, 409 MiB used, 357 GiB / 358 GiB avail; 27 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2024-09-21T08:26:13.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:13 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3921506749' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-21T08:26:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:13 smithi137 ceph-mon[29634]: from='client.44993 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-21T08:26:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:13 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:13 smithi137 ceph-mon[41044]: pgmap v173: 148 pgs: 148 active+clean; 118 MiB data, 409 MiB used, 357 GiB / 358 GiB avail; 27 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2024-09-21T08:26:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:13 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3921506749' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-21T08:26:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:13 smithi137 ceph-mon[41044]: from='client.44993 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]: dispatch 2024-09-21T08:26:13.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:13.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:13.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:13.827 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:13.899 INFO:tasks.rados.rados.0.smithi137.stdout:553: finishing write tid 2 to smithi13779221-39 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:547: done (4 left) 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:548: done (3 left) 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:550: done (2 left) 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:552: done (1 left) 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:554: copy_from oid 31 from oid 34 current snap is 47 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:553: finishing write tid 3 to smithi13779221-39 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:553: finishing write tid 4 to smithi13779221-39 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:553: finishing write tid 5 to smithi13779221-39 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:553: oid 39 updating version 0 to 34 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:553: oid 39 updating version 34 to 36 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:553: oid 39 version 36 is already newer than 35 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:553: oid 39 updating version 36 to 37 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 37 (ObjNum 229 snap 47 seq_num 229) dirty exists 2024-09-21T08:26:13.900 INFO:tasks.rados.rados.0.smithi137.stdout:553: left oid 39 (ObjNum 229 snap 47 seq_num 229) 2024-09-21T08:26:13.901 INFO:tasks.rados.rados.0.smithi137.stdout:553: done (1 left) 2024-09-21T08:26:13.901 INFO:tasks.rados.rados.0.smithi137.stdout:555: delete oid 21 current snap is 47 2024-09-21T08:26:13.903 INFO:tasks.rados.rados.0.smithi137.stdout:555: done (1 left) 2024-09-21T08:26:13.903 INFO:tasks.rados.rados.0.smithi137.stdout:556: read oid 13 snap -1 2024-09-21T08:26:13.903 INFO:tasks.rados.rados.0.smithi137.stdout:556: expect (ObjNum 179 snap 35 seq_num 179) 2024-09-21T08:26:13.903 INFO:tasks.rados.rados.0.smithi137.stdout:557: write oid 20 current snap is 47 2024-09-21T08:26:13.903 INFO:tasks.rados.rados.0.smithi137.stdout:557: seq_num 231 ranges {682417=473109,1567811=602063,2844123=621482} 2024-09-21T08:26:13.909 INFO:tasks.rados.rados.0.smithi137.stdout:557: writing smithi13779221-20 from 682417 to 1155526 tid 1 2024-09-21T08:26:13.915 INFO:tasks.rados.rados.0.smithi137.stdout:557: writing smithi13779221-20 from 1567811 to 2169874 tid 2 2024-09-21T08:26:13.922 INFO:tasks.rados.rados.0.smithi137.stdout:557: writing smithi13779221-20 from 2844123 to 3465605 tid 3 2024-09-21T08:26:13.923 INFO:tasks.rados.rados.0.smithi137.stdout:558: write (excl) oid 41 current snap is 47 2024-09-21T08:26:13.923 INFO:tasks.rados.rados.0.smithi137.stdout:558: seq_num 232 ranges {699224=548358,2026293=462074,3271546=432754} 2024-09-21T08:26:13.928 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:13.929 INFO:tasks.rados.rados.0.smithi137.stdout:558: writing smithi13779221-41 from 699224 to 1247582 tid 1 2024-09-21T08:26:13.936 INFO:tasks.rados.rados.0.smithi137.stdout:558: writing smithi13779221-41 from 2026293 to 2488367 tid 2 2024-09-21T08:26:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:558: writing smithi13779221-41 from 3271546 to 3704300 tid 3 2024-09-21T08:26:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:559: delete oid 10 current snap is 47 2024-09-21T08:26:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:554: finishing copy_from racing read to smithi13779221-31 2024-09-21T08:26:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:556: expect (ObjNum 179 snap 35 seq_num 179) 2024-09-21T08:26:13.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:13.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:13.951 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:14.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:14.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:14.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:14.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:14.145 INFO:tasks.rados.rados.0.smithi137.stdout:557: finishing write tid 1 to smithi13779221-20 2024-09-21T08:26:14.145 INFO:tasks.rados.rados.0.smithi137.stdout:557: finishing write tid 2 to smithi13779221-20 2024-09-21T08:26:14.145 INFO:tasks.rados.rados.0.smithi137.stdout:554: finishing copy_from to smithi13779221-31 2024-09-21T08:26:14.145 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 239 (ObjNum 203 snap 40 seq_num 203) dirty exists 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:557: finishing write tid 3 to smithi13779221-20 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:557: finishing write tid 4 to smithi13779221-20 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:554: done (4 left) 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:556: done (3 left) 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:559: done (2 left) 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:560: write oid 43 current snap is 47 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:557: finishing write tid 5 to smithi13779221-20 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:557: oid 20 updating version 0 to 244 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:557: oid 20 version 244 is already newer than 243 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:557: oid 20 version 244 is already newer than 242 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:557: oid 20 updating version 244 to 245 2024-09-21T08:26:14.146 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 245 (ObjNum 231 snap 47 seq_num 231) dirty exists 2024-09-21T08:26:14.147 INFO:tasks.rados.rados.0.smithi137.stdout:557: left oid 20 (ObjNum 231 snap 47 seq_num 231) 2024-09-21T08:26:14.147 INFO:tasks.rados.rados.0.smithi137.stdout:558: finishing write tid 1 to smithi13779221-41 2024-09-21T08:26:14.147 INFO:tasks.rados.rados.0.smithi137.stdout:560: seq_num 234 ranges {732838=698856,2026904=465743,3210039=571651} 2024-09-21T08:26:14.152 INFO:tasks.rados.rados.0.smithi137.stdout:560: writing smithi13779221-43 from 732838 to 1431694 tid 1 2024-09-21T08:26:14.158 INFO:tasks.rados.rados.0.smithi137.stdout:560: writing smithi13779221-43 from 2026904 to 2492647 tid 2 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:560: writing smithi13779221-43 from 3210039 to 3781690 tid 3 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:557: done (2 left) 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:561: snap_create 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:558: finishing write tid 2 to smithi13779221-41 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:558: finishing write tid 3 to smithi13779221-41 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:558: finishing write tid 4 to smithi13779221-41 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:558: finishing write tid 5 to smithi13779221-41 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:558: oid 41 updating version 0 to 88 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:558: oid 41 updating version 88 to 91 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:558: oid 41 version 91 is already newer than 89 2024-09-21T08:26:14.166 INFO:tasks.rados.rados.0.smithi137.stdout:558: oid 41 version 91 is already newer than 90 2024-09-21T08:26:14.167 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 91 (ObjNum 232 snap 47 seq_num 232) dirty exists 2024-09-21T08:26:14.167 INFO:tasks.rados.rados.0.smithi137.stdout:558: left oid 41 (ObjNum 232 snap 47 seq_num 232) 2024-09-21T08:26:14.167 INFO:tasks.rados.rados.0.smithi137.stdout:560: finishing write tid 1 to smithi13779221-43 2024-09-21T08:26:14.167 INFO:tasks.rados.rados.0.smithi137.stdout:560: finishing write tid 2 to smithi13779221-43 2024-09-21T08:26:14.167 INFO:tasks.rados.rados.0.smithi137.stdout:560: finishing write tid 3 to smithi13779221-43 2024-09-21T08:26:14.168 INFO:tasks.rados.rados.0.smithi137.stdout:560: finishing write tid 4 to smithi13779221-43 2024-09-21T08:26:14.168 INFO:tasks.rados.rados.0.smithi137.stdout:560: finishing write tid 5 to smithi13779221-43 2024-09-21T08:26:14.168 INFO:tasks.rados.rados.0.smithi137.stdout:560: oid 43 updating version 0 to 95 2024-09-21T08:26:14.168 INFO:tasks.rados.rados.0.smithi137.stdout:560: oid 43 version 95 is already newer than 94 2024-09-21T08:26:14.168 INFO:tasks.rados.rados.0.smithi137.stdout:560: oid 43 version 95 is already newer than 92 2024-09-21T08:26:14.168 INFO:tasks.rados.rados.0.smithi137.stdout:560: oid 43 version 95 is already newer than 93 2024-09-21T08:26:14.168 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 95 (ObjNum 234 snap 47 seq_num 234) dirty exists 2024-09-21T08:26:14.168 INFO:tasks.rados.rados.0.smithi137.stdout:560: left oid 43 (ObjNum 234 snap 47 seq_num 234) 2024-09-21T08:26:14.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:14.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:14.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:14.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:14.297 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:14.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:14.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:14.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:14.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:14.442 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:14.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:14.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:14.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:14.565 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:14.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:14.566 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:14.667 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:14.688 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:14.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:14.689 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:14.790 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:14.811 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:14.812 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:14.812 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:14.903 INFO:tasks.rados.rados.0.smithi137.stdout:558: done (2 left) 2024-09-21T08:26:14.903 INFO:tasks.rados.rados.0.smithi137.stdout:560: done (1 left) 2024-09-21T08:26:14.903 INFO:tasks.rados.rados.0.smithi137.stdout:561: done (0 left) 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:562: read oid 22 snap 47 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:562: expect (ObjNum 226 snap 47 seq_num 226) 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:563: rollback oid 7 current snap is 48 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 7 to 47 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:564: copy_from oid 28 from oid 40 current snap is 48 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:565: copy_from oid 26 from oid 2 current snap is 48 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:566: read oid 17 snap -1 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:566: expect (ObjNum 192 snap 39 seq_num 192) 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:567: rollback oid 50 current snap is 48 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 50 to 47 2024-09-21T08:26:14.904 INFO:tasks.rados.rados.0.smithi137.stdout:568: write (excl) oid 49 current snap is 48 2024-09-21T08:26:14.905 INFO:tasks.rados.rados.0.smithi137.stdout:568: seq_num 235 ranges {749645=774105,2085386=609401,3237462=712183} 2024-09-21T08:26:14.912 INFO:tasks.rados.rados.0.smithi137.stdout:568: writing smithi13779221-49 from 749645 to 1523750 tid 1 2024-09-21T08:26:14.912 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:14.919 INFO:tasks.rados.rados.0.smithi137.stdout:568: writing smithi13779221-49 from 2085386 to 2694787 tid 2 2024-09-21T08:26:14.927 INFO:tasks.rados.rados.0.smithi137.stdout:568: writing smithi13779221-49 from 3237462 to 3949645 tid 3 2024-09-21T08:26:14.927 INFO:tasks.rados.rados.0.smithi137.stdout:569: snap_create 2024-09-21T08:26:14.927 INFO:tasks.rados.rados.0.smithi137.stdout:563: finishing rollback tid 0 to smithi13779221-7 2024-09-21T08:26:14.927 INFO:tasks.rados.rados.0.smithi137.stdout:563: finishing rollback tid 1 to smithi13779221-7 2024-09-21T08:26:14.927 INFO:tasks.rados.rados.0.smithi137.stdout:564: finishing copy_from racing read to smithi13779221-28 2024-09-21T08:26:14.927 INFO:tasks.rados.rados.0.smithi137.stdout:563: finishing rollback tid 2 to smithi13779221-7 2024-09-21T08:26:14.927 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 240 (ObjNum 156 snap 31 seq_num 156) dirty exists 2024-09-21T08:26:14.927 INFO:tasks.rados.rados.0.smithi137.stdout:564: finishing copy_from to smithi13779221-28 2024-09-21T08:26:14.928 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 85 (ObjNum 211 snap 43 seq_num 211) dirty exists 2024-09-21T08:26:14.935 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:14.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:14.936 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:14 smithi165 ceph-mon[32893]: from='client.44993 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-21T08:26:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:14 smithi165 ceph-mon[32893]: osdmap e180: 8 total, 8 up, 4 in 2024-09-21T08:26:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:14 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:14 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:14 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:14 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:15.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[29634]: from='client.44993 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-21T08:26:15.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[29634]: osdmap e180: 8 total, 8 up, 4 in 2024-09-21T08:26:15.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:15.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:15.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:15.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:15.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[41044]: from='client.44993 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "26"}]': finished 2024-09-21T08:26:15.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[41044]: osdmap e180: 8 total, 8 up, 4 in 2024-09-21T08:26:15.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:15.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:15.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:15.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:14 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:15.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:15.059 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:15.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:15.060 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:15.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T08:26:15.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:15.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:15.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:15.184 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:15.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:15.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:15.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:15.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:15.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:15.419 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:26:15.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:15.429 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:15.430 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:15.517 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:15.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:15.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:15.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:15.553 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:15.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:15.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:15.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:15.676 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:15.776 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:15.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:15.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:15.799 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:15.900 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:15.906 INFO:tasks.rados.rados.0.smithi137.stdout:563: done (7 left) 2024-09-21T08:26:15.906 INFO:tasks.rados.rados.0.smithi137.stdout:564: done (6 left) 2024-09-21T08:26:15.907 INFO:tasks.rados.rados.0.smithi137.stdout:569: done (5 left) 2024-09-21T08:26:15.907 INFO:tasks.rados.rados.0.smithi137.stdout:570: rollback oid 12 current snap is 49 2024-09-21T08:26:15.907 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 12 to 47 2024-09-21T08:26:15.907 INFO:tasks.rados.rados.0.smithi137.stdout:571: copy_from oid 32 from oid 10 current snap is 49 2024-09-21T08:26:15.907 INFO:tasks.rados.rados.0.smithi137.stdout:572: delete oid 9 current snap is 49 2024-09-21T08:26:15.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:15 smithi137 ceph-mon[29634]: pgmap v176: 148 pgs: 148 active+clean; 118 MiB data, 409 MiB used, 357 GiB / 358 GiB avail; 27 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2024-09-21T08:26:15.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:15 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:15.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:15 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T08:26:15.920 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:15 smithi137 ceph-mon[29634]: osdmap e181: 8 total, 8 up, 4 in 2024-09-21T08:26:15.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:15 smithi137 ceph-mon[41044]: pgmap v176: 148 pgs: 148 active+clean; 118 MiB data, 409 MiB used, 357 GiB / 358 GiB avail; 27 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2024-09-21T08:26:15.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:15 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:15.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:15 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T08:26:15.921 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:15 smithi137 ceph-mon[41044]: osdmap e181: 8 total, 8 up, 4 in 2024-09-21T08:26:15.923 INFO:tasks.rados.rados.0.smithi137.stdout:571: finishing copy_from to smithi13779221-32 2024-09-21T08:26:15.923 INFO:tasks.rados.rados.0.smithi137.stdout:571: got expected ENOENT (src dne) 2024-09-21T08:26:15.925 INFO:tasks.rados.rados.0.smithi137.stdout:571: finishing copy_from racing read to smithi13779221-32 2024-09-21T08:26:15.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:15.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:15.927 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:15.939 INFO:tasks.rados.rados.0.smithi137.stdout:562: expect (ObjNum 226 snap 47 seq_num 226) 2024-09-21T08:26:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:15 smithi165 ceph-mon[32893]: pgmap v176: 148 pgs: 148 active+clean; 118 MiB data, 409 MiB used, 357 GiB / 358 GiB avail; 27 MiB/s rd, 1.2 MiB/s wr, 25 op/s 2024-09-21T08:26:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:15 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:15 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "26"}]': finished 2024-09-21T08:26:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:15 smithi165 ceph-mon[32893]: osdmap e181: 8 total, 8 up, 4 in 2024-09-21T08:26:16.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:16.042 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:16.053 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:16.054 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:16.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:16.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:16.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:16.083 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:16.084 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:16.094 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:16.114 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:16.114 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:16.142 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:16.142 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:16.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:16.173 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:16.193 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:16.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:16.194 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:16.200 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:16.217 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:16.247 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:16.247 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:16.248 INFO:tasks.rados.rados.0.smithi137.stdout:562: done (7 left) 2024-09-21T08:26:16.248 INFO:tasks.rados.rados.0.smithi137.stdout:571: done (6 left) 2024-09-21T08:26:16.248 INFO:tasks.rados.rados.0.smithi137.stdout:572: done (5 left) 2024-09-21T08:26:16.248 INFO:tasks.rados.rados.0.smithi137.stdout:573: write oid 30 current snap is 49 2024-09-21T08:26:16.248 INFO:tasks.rados.rados.0.smithi137.stdout:565: finishing copy_from racing read to smithi13779221-26 2024-09-21T08:26:16.248 INFO:tasks.rados.rados.0.smithi137.stdout:566: expect (ObjNum 192 snap 39 seq_num 192) 2024-09-21T08:26:16.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:16.316 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:16.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:16.317 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:16.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:16.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:16.440 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:16.440 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:16.478 INFO:tasks.rados.rados.0.smithi137.stdout:568: finishing write tid 1 to smithi13779221-49 2024-09-21T08:26:16.478 INFO:tasks.rados.rados.0.smithi137.stdout:568: finishing write tid 2 to smithi13779221-49 2024-09-21T08:26:16.478 INFO:tasks.rados.rados.0.smithi137.stdout:573: seq_num 237 ranges {783259=640956,1918703=613070,3008661=405467,3821939=161320} 2024-09-21T08:26:16.485 INFO:tasks.rados.rados.0.smithi137.stdout:573: writing smithi13779221-30 from 783259 to 1424215 tid 1 2024-09-21T08:26:16.491 INFO:tasks.rados.rados.0.smithi137.stdout:573: writing smithi13779221-30 from 1918703 to 2531773 tid 2 2024-09-21T08:26:16.496 INFO:tasks.rados.rados.0.smithi137.stdout:573: writing smithi13779221-30 from 3008661 to 3414128 tid 3 2024-09-21T08:26:16.500 INFO:tasks.rados.rados.0.smithi137.stdout:573: writing smithi13779221-30 from 3821939 to 3983259 tid 4 2024-09-21T08:26:16.500 INFO:tasks.rados.rados.0.smithi137.stdout:566: done (5 left) 2024-09-21T08:26:16.500 INFO:tasks.rados.rados.0.smithi137.stdout:574: copy_from oid 22 from oid 3 current snap is 49 2024-09-21T08:26:16.500 INFO:tasks.rados.rados.0.smithi137.stdout:568: finishing write tid 3 to smithi13779221-49 2024-09-21T08:26:16.500 INFO:tasks.rados.rados.0.smithi137.stdout:568: finishing write tid 4 to smithi13779221-49 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:568: finishing write tid 5 to smithi13779221-49 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:568: oid 49 updating version 0 to 249 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:568: oid 49 version 249 is already newer than 248 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:568: oid 49 version 249 is already newer than 247 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:568: oid 49 updating version 249 to 250 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 250 (ObjNum 235 snap 48 seq_num 235) dirty exists 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:568: left oid 49 (ObjNum 235 snap 48 seq_num 235) 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:565: finishing copy_from to smithi13779221-26 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 99 (ObjNum 200 snap 40 seq_num 200) dirty exists 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:567: finishing rollback tid 0 to smithi13779221-50 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:567: finishing rollback tid 1 to smithi13779221-50 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:567: finishing rollback tid 2 to smithi13779221-50 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 189 (ObjNum 171 snap 34 seq_num 171) dirty exists 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:570: finishing rollback tid 0 to smithi13779221-12 2024-09-21T08:26:16.501 INFO:tasks.rados.rados.0.smithi137.stdout:570: finishing rollback tid 1 to smithi13779221-12 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:570: finishing rollback tid 2 to smithi13779221-12 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 193 (ObjNum 201 snap 40 seq_num 201) dirty exists 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:565: done (5 left) 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:567: done (4 left) 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:568: done (3 left) 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:570: done (2 left) 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:575: rollback oid 24 current snap is 49 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 24 to 48 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:573: finishing write tid 1 to smithi13779221-30 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:573: finishing write tid 2 to smithi13779221-30 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:573: finishing write tid 3 to smithi13779221-30 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:576: read oid 20 snap 48 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:576: expect (ObjNum 231 snap 47 seq_num 231) 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:577: read oid 27 snap -1 2024-09-21T08:26:16.502 INFO:tasks.rados.rados.0.smithi137.stdout:577: expect (ObjNum 218 snap 45 seq_num 218) 2024-09-21T08:26:16.503 INFO:tasks.rados.rados.0.smithi137.stdout:578: read oid 48 snap -1 2024-09-21T08:26:16.503 INFO:tasks.rados.rados.0.smithi137.stdout:578: expect (ObjNum 189 snap 38 seq_num 189) 2024-09-21T08:26:16.503 INFO:tasks.rados.rados.0.smithi137.stdout:579: delete oid 39 current snap is 49 2024-09-21T08:26:16.505 INFO:tasks.rados.rados.0.smithi137.stdout:573: finishing write tid 4 to smithi13779221-30 2024-09-21T08:26:16.505 INFO:tasks.rados.rados.0.smithi137.stdout:573: finishing write tid 5 to smithi13779221-30 2024-09-21T08:26:16.505 INFO:tasks.rados.rados.0.smithi137.stdout:578: expect (ObjNum 189 snap 38 seq_num 189) 2024-09-21T08:26:16.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:16.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:16.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:16.563 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:16.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:16.686 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:16.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:16.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:573: finishing write tid 6 to smithi13779221-30 2024-09-21T08:26:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:573: oid 30 updating version 0 to 245 2024-09-21T08:26:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:573: oid 30 version 245 is already newer than 241 2024-09-21T08:26:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:573: oid 30 version 245 is already newer than 242 2024-09-21T08:26:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:573: oid 30 version 245 is already newer than 244 2024-09-21T08:26:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:573: oid 30 version 245 is already newer than 243 2024-09-21T08:26:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 245 (ObjNum 237 snap 49 seq_num 237) dirty exists 2024-09-21T08:26:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:573: left oid 30 (ObjNum 237 snap 49 seq_num 237) 2024-09-21T08:26:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:573: done (6 left) 2024-09-21T08:26:16.700 INFO:tasks.rados.rados.0.smithi137.stdout:578: done (5 left) 2024-09-21T08:26:16.701 INFO:tasks.rados.rados.0.smithi137.stdout:579: done (4 left) 2024-09-21T08:26:16.701 INFO:tasks.rados.rados.0.smithi137.stdout:580: read oid 30 snap 48 2024-09-21T08:26:16.701 INFO:tasks.rados.rados.0.smithi137.stdout:580: expect deleted 2024-09-21T08:26:16.701 INFO:tasks.rados.rados.0.smithi137.stdout:581: snap_remove snap 47 2024-09-21T08:26:16.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:16.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:16.810 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:16.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:16.906 INFO:tasks.rados.rados.0.smithi137.stdout:581: done (5 left) 2024-09-21T08:26:16.907 INFO:tasks.rados.rados.0.smithi137.stdout:582: copy_from oid 6 from oid 9 current snap is 49 2024-09-21T08:26:16.907 INFO:tasks.rados.rados.0.smithi137.stdout:577: expect (ObjNum 218 snap 45 seq_num 218) 2024-09-21T08:26:16.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:16.935 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:16.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:16.936 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:17.037 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:17.058 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:17.058 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:17.059 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:17.131 INFO:tasks.rados.rados.0.smithi137.stdout:575: finishing rollback tid 1 to smithi13779221-24 2024-09-21T08:26:17.131 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 0 (ObjNum 97 snap 0 seq_num 402191712) dirty dne 2024-09-21T08:26:17.131 INFO:tasks.rados.rados.0.smithi137.stdout:574: finishing copy_from racing read to smithi13779221-22 2024-09-21T08:26:17.131 INFO:tasks.rados.rados.0.smithi137.stdout:576: expect (ObjNum 231 snap 47 seq_num 231) 2024-09-21T08:26:17.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:17.180 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:17.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:17.181 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:17.200 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:16 smithi137 ceph-mon[29634]: osdmap e182: 8 total, 8 up, 4 in 2024-09-21T08:26:17.201 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:16 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/645494728' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:17.201 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:16 smithi137 ceph-mon[29634]: pgmap v179: 158 pgs: 2 active+clean+snaptrim, 10 peering, 10 unknown, 136 active+clean; 132 MiB data, 455 MiB used, 357 GiB / 358 GiB avail; 7.5 MiB/s rd, 7.6 MiB/s wr, 24 op/s 2024-09-21T08:26:17.201 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:16 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:17.201 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:16 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:17.201 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:16 smithi137 ceph-mon[41044]: osdmap e182: 8 total, 8 up, 4 in 2024-09-21T08:26:17.201 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:16 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/645494728' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:17.201 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:16 smithi137 ceph-mon[41044]: pgmap v179: 158 pgs: 2 active+clean+snaptrim, 10 peering, 10 unknown, 136 active+clean; 132 MiB data, 455 MiB used, 357 GiB / 358 GiB avail; 7.5 MiB/s rd, 7.6 MiB/s wr, 24 op/s 2024-09-21T08:26:17.201 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:16 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:17.201 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:16 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:17.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:16 smithi165 ceph-mon[32893]: osdmap e182: 8 total, 8 up, 4 in 2024-09-21T08:26:17.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:16 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/645494728' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:17.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:16 smithi165 ceph-mon[32893]: pgmap v179: 158 pgs: 2 active+clean+snaptrim, 10 peering, 10 unknown, 136 active+clean; 132 MiB data, 455 MiB used, 357 GiB / 358 GiB avail; 7.5 MiB/s rd, 7.6 MiB/s wr, 24 op/s 2024-09-21T08:26:17.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:16 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:17.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:16 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]: dispatch 2024-09-21T08:26:17.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:17.306 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:17.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:17.307 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:17.310 INFO:tasks.rados.rados.0.smithi137.stdout:574: finishing copy_from to smithi13779221-22 2024-09-21T08:26:17.310 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 143 (ObjNum 220 snap 46 seq_num 220) dirty exists 2024-09-21T08:26:17.310 INFO:tasks.rados.rados.0.smithi137.stdout:574: done (5 left) 2024-09-21T08:26:17.311 INFO:tasks.rados.rados.0.smithi137.stdout:575: done (4 left) 2024-09-21T08:26:17.311 INFO:tasks.rados.rados.0.smithi137.stdout:576: done (3 left) 2024-09-21T08:26:17.311 INFO:tasks.rados.rados.0.smithi137.stdout:577: done (2 left) 2024-09-21T08:26:17.311 INFO:tasks.rados.rados.0.smithi137.stdout:583: write oid 46 current snap is 49 2024-09-21T08:26:17.311 INFO:tasks.rados.rados.0.smithi137.stdout:583: seq_num 239 ranges {416873=791454,1919314=97559} 2024-09-21T08:26:17.317 INFO:tasks.rados.rados.0.smithi137.stdout:583: writing smithi13779221-46 from 416873 to 1208327 tid 1 2024-09-21T08:26:17.321 INFO:tasks.rados.rados.0.smithi137.stdout:583: writing smithi13779221-46 from 1919314 to 2016873 tid 2 2024-09-21T08:26:17.322 INFO:tasks.rados.rados.0.smithi137.stdout:584: write oid 23 current snap is 49 2024-09-21T08:26:17.322 INFO:tasks.rados.rados.0.smithi137.stdout:584: seq_num 240 ranges {433680=466703,1577796=455884} 2024-09-21T08:26:17.326 INFO:tasks.rados.rados.0.smithi137.stdout:584: writing smithi13779221-23 from 433680 to 900383 tid 1 2024-09-21T08:26:17.333 INFO:tasks.rados.rados.0.smithi137.stdout:584: writing smithi13779221-23 from 1577796 to 2033680 tid 2 2024-09-21T08:26:17.333 INFO:tasks.rados.rados.0.smithi137.stdout:580: done (3 left) 2024-09-21T08:26:17.333 INFO:tasks.rados.rados.0.smithi137.stdout:585: snap_create 2024-09-21T08:26:17.333 INFO:tasks.rados.rados.0.smithi137.stdout:583: finishing write tid 1 to smithi13779221-46 2024-09-21T08:26:17.333 INFO:tasks.rados.rados.0.smithi137.stdout:583: finishing write tid 2 to smithi13779221-46 2024-09-21T08:26:17.333 INFO:tasks.rados.rados.0.smithi137.stdout:583: finishing write tid 3 to smithi13779221-46 2024-09-21T08:26:17.333 INFO:tasks.rados.rados.0.smithi137.stdout:583: finishing write tid 4 to smithi13779221-46 2024-09-21T08:26:17.333 INFO:tasks.rados.rados.0.smithi137.stdout:583: oid 46 updating version 0 to 251 2024-09-21T08:26:17.333 INFO:tasks.rados.rados.0.smithi137.stdout:583: oid 46 version 251 is already newer than 249 2024-09-21T08:26:17.333 INFO:tasks.rados.rados.0.smithi137.stdout:583: oid 46 version 251 is already newer than 250 2024-09-21T08:26:17.334 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 251 (ObjNum 239 snap 49 seq_num 239) dirty exists 2024-09-21T08:26:17.334 INFO:tasks.rados.rados.0.smithi137.stdout:583: left oid 46 (ObjNum 239 snap 49 seq_num 239) 2024-09-21T08:26:17.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:17.430 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:17.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:17.431 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:17.474 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:26:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:26:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:26:17.474 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:26:17.474 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:26:17.532 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:17.553 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:17.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:17.554 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:17.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:17.676 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:17.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:17.677 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:17.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:17.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:17.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:17.800 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:17.901 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:17.912 INFO:tasks.rados.rados.0.smithi137.stdout:583: done (3 left) 2024-09-21T08:26:17.912 INFO:tasks.rados.rados.0.smithi137.stdout:585: done (2 left) 2024-09-21T08:26:17.912 INFO:tasks.rados.rados.0.smithi137.stdout:586: rollback oid 39 current snap is 50 2024-09-21T08:26:17.912 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 39 to 48 2024-09-21T08:26:17.913 INFO:tasks.rados.rados.0.smithi137.stdout:587: copy_from oid 34 from oid 40 current snap is 50 2024-09-21T08:26:17.913 INFO:tasks.rados.rados.0.smithi137.stdout:588: read oid 32 snap -1 2024-09-21T08:26:17.913 INFO:tasks.rados.rados.0.smithi137.stdout:588: expect deleted 2024-09-21T08:26:17.913 INFO:tasks.rados.rados.0.smithi137.stdout:589: write (excl) oid 48 current snap is 50 2024-09-21T08:26:17.913 INFO:tasks.rados.rados.0.smithi137.stdout:589: seq_num 241 ranges {450487=541952,1636278=414209} 2024-09-21T08:26:17.918 INFO:tasks.rados.rados.0.smithi137.stdout:589: writing smithi13779221-48 from 450487 to 992439 tid 1 2024-09-21T08:26:17.925 INFO:tasks.rados.rados.0.smithi137.stdout:589: writing smithi13779221-48 from 1636278 to 2050487 tid 2 2024-09-21T08:26:17.925 INFO:tasks.rados.rados.0.smithi137.stdout:586: finishing rollback tid 1 to smithi13779221-39 2024-09-21T08:26:17.925 INFO:tasks.rados.rados.0.smithi137.stdout:586: finishing rollback tid 2 to smithi13779221-39 2024-09-21T08:26:17.925 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 43 (ObjNum 229 snap 47 seq_num 229) dirty exists 2024-09-21T08:26:17.925 INFO:tasks.rados.rados.0.smithi137.stdout:589: finishing write tid 1 to smithi13779221-48 2024-09-21T08:26:17.925 INFO:tasks.rados.rados.0.smithi137.stdout:586: done (5 left) 2024-09-21T08:26:17.925 INFO:tasks.rados.rados.0.smithi137.stdout:588: done (4 left) 2024-09-21T08:26:17.925 INFO:tasks.rados.rados.0.smithi137.stdout:590: delete oid 18 current snap is 50 2024-09-21T08:26:17.928 INFO:tasks.rados.rados.0.smithi137.stdout:590: done (4 left) 2024-09-21T08:26:17.928 INFO:tasks.rados.rados.0.smithi137.stdout:591: write oid 37 current snap is 50 2024-09-21T08:26:17.928 INFO:tasks.rados.rados.0.smithi137.stdout:591: seq_num 243 ranges {484101=692450,1636889=447212} 2024-09-21T08:26:17.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:17.930 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:17.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:17.935 INFO:tasks.rados.rados.0.smithi137.stdout:591: writing smithi13779221-37 from 484101 to 1176551 tid 1 2024-09-21T08:26:17.941 INFO:tasks.rados.rados.0.smithi137.stdout:591: writing smithi13779221-37 from 1636889 to 2084101 tid 2 2024-09-21T08:26:17.941 INFO:tasks.rados.rados.0.smithi137.stdout:592: rollback oid 26 current snap is 50 2024-09-21T08:26:17.941 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 26 to 48 2024-09-21T08:26:17.941 INFO:tasks.rados.rados.0.smithi137.stdout:589: finishing write tid 2 to smithi13779221-48 2024-09-21T08:26:17.941 INFO:tasks.rados.rados.0.smithi137.stdout:589: finishing write tid 3 to smithi13779221-48 2024-09-21T08:26:17.941 INFO:tasks.rados.rados.0.smithi137.stdout:589: finishing write tid 4 to smithi13779221-48 2024-09-21T08:26:17.941 INFO:tasks.rados.rados.0.smithi137.stdout:589: oid 48 updating version 0 to 188 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:589: oid 48 version 188 is already newer than 187 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:589: oid 48 updating version 188 to 189 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 189 (ObjNum 241 snap 50 seq_num 241) dirty exists 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:589: left oid 48 (ObjNum 241 snap 50 seq_num 241) 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:582: finishing copy_from to smithi13779221-6 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:582: got expected ENOENT (src dne) 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:582: finishing copy_from racing read to smithi13779221-6 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:582: done (5 left) 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:589: done (4 left) 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:593: snap_create 2024-09-21T08:26:17.942 INFO:tasks.rados.rados.0.smithi137.stdout:591: finishing write tid 1 to smithi13779221-37 2024-09-21T08:26:17.944 INFO:tasks.rados.rados.0.smithi137.stdout:591: finishing write tid 2 to smithi13779221-37 2024-09-21T08:26:17.944 INFO:tasks.rados.rados.0.smithi137.stdout:592: finishing rollback tid 0 to smithi13779221-26 2024-09-21T08:26:17.944 INFO:tasks.rados.rados.0.smithi137.stdout:592: finishing rollback tid 1 to smithi13779221-26 2024-09-21T08:26:17.944 INFO:tasks.rados.rados.0.smithi137.stdout:591: finishing write tid 3 to smithi13779221-37 2024-09-21T08:26:17.944 INFO:tasks.rados.rados.0.smithi137.stdout:591: finishing write tid 4 to smithi13779221-37 2024-09-21T08:26:17.944 INFO:tasks.rados.rados.0.smithi137.stdout:591: oid 37 updating version 0 to 255 2024-09-21T08:26:17.945 INFO:tasks.rados.rados.0.smithi137.stdout:591: oid 37 version 255 is already newer than 254 2024-09-21T08:26:17.945 INFO:tasks.rados.rados.0.smithi137.stdout:591: oid 37 version 255 is already newer than 253 2024-09-21T08:26:17.945 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 255 (ObjNum 243 snap 50 seq_num 243) dirty exists 2024-09-21T08:26:17.945 INFO:tasks.rados.rados.0.smithi137.stdout:591: left oid 37 (ObjNum 243 snap 50 seq_num 243) 2024-09-21T08:26:17.945 INFO:tasks.rados.rados.0.smithi137.stdout:592: finishing rollback tid 2 to smithi13779221-26 2024-09-21T08:26:17.945 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 105 (ObjNum 200 snap 40 seq_num 200) dirty exists 2024-09-21T08:26:17.948 INFO:tasks.rados.rados.0.smithi137.stdout:584: finishing write tid 1 to smithi13779221-23 2024-09-21T08:26:17.948 INFO:tasks.rados.rados.0.smithi137.stdout:584: finishing write tid 2 to smithi13779221-23 2024-09-21T08:26:17.948 INFO:tasks.rados.rados.0.smithi137.stdout:584: finishing write tid 3 to smithi13779221-23 2024-09-21T08:26:17.948 INFO:tasks.rados.rados.0.smithi137.stdout:584: finishing write tid 4 to smithi13779221-23 2024-09-21T08:26:17.948 INFO:tasks.rados.rados.0.smithi137.stdout:584: oid 23 updating version 0 to 165 2024-09-21T08:26:17.948 INFO:tasks.rados.rados.0.smithi137.stdout:584: oid 23 updating version 165 to 166 2024-09-21T08:26:17.948 INFO:tasks.rados.rados.0.smithi137.stdout:584: oid 23 updating version 166 to 167 2024-09-21T08:26:17.948 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 167 (ObjNum 240 snap 49 seq_num 240) dirty exists 2024-09-21T08:26:17.948 INFO:tasks.rados.rados.0.smithi137.stdout:584: left oid 23 (ObjNum 240 snap 49 seq_num 240) 2024-09-21T08:26:18.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:18.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:18.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:18.054 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:18.108 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:17 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T08:26:18.108 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:17 smithi137 ceph-mon[29634]: osdmap e183: 8 total, 8 up, 4 in 2024-09-21T08:26:18.108 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:17 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T08:26:18.108 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:17 smithi137 ceph-mon[41044]: osdmap e183: 8 total, 8 up, 4 in 2024-09-21T08:26:18.155 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:18.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:18.178 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:18.178 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:18.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:17 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "26"}]': finished 2024-09-21T08:26:18.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:17 smithi165 ceph-mon[32893]: osdmap e183: 8 total, 8 up, 4 in 2024-09-21T08:26:18.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:18.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:18.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:18.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:18.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T08:26:18.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:18.422 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:18.423 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:18.423 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:18.524 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:18.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:18.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:18.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:18.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:18.668 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:18.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:18.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:18.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:18.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:18.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:18.794 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:18.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:18.915 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:18.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:18.916 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:18.922 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:26:18.927 INFO:tasks.rados.rados.0.smithi137.stdout:584: done (4 left) 2024-09-21T08:26:18.927 INFO:tasks.rados.rados.0.smithi137.stdout:591: done (3 left) 2024-09-21T08:26:18.927 INFO:tasks.rados.rados.0.smithi137.stdout:592: done (2 left) 2024-09-21T08:26:18.927 INFO:tasks.rados.rados.0.smithi137.stdout:593: done (1 left) 2024-09-21T08:26:18.927 INFO:tasks.rados.rados.0.smithi137.stdout:594: snap_create 2024-09-21T08:26:18.945 INFO:tasks.rados.rados.0.smithi137.stdout:587: finishing copy_from racing read to smithi13779221-34 2024-09-21T08:26:18.950 INFO:tasks.rados.rados.0.smithi137.stdout:587: finishing copy_from to smithi13779221-34 2024-09-21T08:26:18.950 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 85 (ObjNum 211 snap 43 seq_num 211) dirty exists 2024-09-21T08:26:19.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:19.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:18 smithi137 ceph-mon[29634]: osdmap e184: 8 total, 8 up, 4 in 2024-09-21T08:26:19.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:18 smithi137 ceph-mon[29634]: pgmap v182: 158 pgs: 2 active+clean+snaptrim, 10 peering, 10 unknown, 136 active+clean; 132 MiB data, 455 MiB used, 357 GiB / 358 GiB avail; 7.5 MiB/s rd, 7.6 MiB/s wr, 24 op/s 2024-09-21T08:26:19.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:18 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3974088961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:19.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:18 smithi137 ceph-mon[41044]: osdmap e184: 8 total, 8 up, 4 in 2024-09-21T08:26:19.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:18 smithi137 ceph-mon[41044]: pgmap v182: 158 pgs: 2 active+clean+snaptrim, 10 peering, 10 unknown, 136 active+clean; 132 MiB data, 455 MiB used, 357 GiB / 358 GiB avail; 7.5 MiB/s rd, 7.6 MiB/s wr, 24 op/s 2024-09-21T08:26:19.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:18 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3974088961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:19.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:19.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:19.040 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:19.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:19.162 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:19.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:19.163 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:19.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:26:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:18 smithi165 ceph-mon[32893]: osdmap e184: 8 total, 8 up, 4 in 2024-09-21T08:26:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:18 smithi165 ceph-mon[32893]: pgmap v182: 158 pgs: 2 active+clean+snaptrim, 10 peering, 10 unknown, 136 active+clean; 132 MiB data, 455 MiB used, 357 GiB / 358 GiB avail; 7.5 MiB/s rd, 7.6 MiB/s wr, 24 op/s 2024-09-21T08:26:19.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:18 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3974088961' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:19.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:19.292 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:19.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:19.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:19.329 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:19.393 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:19.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:19.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:19.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:19.517 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:19.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:19.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:19.540 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:19.633 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 3, 6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:26:19.633 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:26:19.633 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T08:26:19.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T08:26:19.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:19.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:19.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:19.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:19.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:19.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:19.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:19.824 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:19.924 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:19.929 INFO:tasks.rados.rados.0.smithi137.stdout:587: done (1 left) 2024-09-21T08:26:19.929 INFO:tasks.rados.rados.0.smithi137.stdout:594: done (0 left) 2024-09-21T08:26:19.929 INFO:tasks.rados.rados.0.smithi137.stdout:595: rollback oid 7 current snap is 52 2024-09-21T08:26:19.929 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 7 to 50 2024-09-21T08:26:19.930 INFO:tasks.rados.rados.0.smithi137.stdout:596: write oid 4 current snap is 52 2024-09-21T08:26:19.930 INFO:tasks.rados.rados.0.smithi137.stdout:596: seq_num 244 ranges {500908=484052,1411724=689184} 2024-09-21T08:26:19.936 INFO:tasks.rados.rados.0.smithi137.stdout:596: writing smithi13779221-4 from 500908 to 984960 tid 1 2024-09-21T08:26:19.943 INFO:tasks.rados.rados.0.smithi137.stdout:596: writing smithi13779221-4 from 1411724 to 2100908 tid 2 2024-09-21T08:26:19.943 INFO:tasks.rados.rados.0.smithi137.stdout:597: rollback oid 25 current snap is 52 2024-09-21T08:26:19.943 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 25 to 50 2024-09-21T08:26:19.943 INFO:tasks.rados.rados.0.smithi137.stdout:595: finishing rollback tid 0 to smithi13779221-7 2024-09-21T08:26:19.943 INFO:tasks.rados.rados.0.smithi137.stdout:595: finishing rollback tid 1 to smithi13779221-7 2024-09-21T08:26:19.943 INFO:tasks.rados.rados.0.smithi137.stdout:595: finishing rollback tid 2 to smithi13779221-7 2024-09-21T08:26:19.943 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 259 (ObjNum 156 snap 31 seq_num 156) dirty exists 2024-09-21T08:26:19.943 INFO:tasks.rados.rados.0.smithi137.stdout:596: finishing write tid 1 to smithi13779221-4 2024-09-21T08:26:19.943 INFO:tasks.rados.rados.0.smithi137.stdout:595: done (2 left) 2024-09-21T08:26:19.943 INFO:tasks.rados.rados.0.smithi137.stdout:598: snap_remove snap 49 2024-09-21T08:26:19.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:19.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:19.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:20.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:20.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:20.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:20.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:20.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:20.198 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:20.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:20.199 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:19 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:19 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3974088961' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:19 smithi165 ceph-mon[32893]: osdmap e185: 8 total, 8 up, 4 in 2024-09-21T08:26:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:19 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:26:20.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:19 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:26:20.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:19 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:20.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:19 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3974088961' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:19 smithi137 ceph-mon[29634]: osdmap e185: 8 total, 8 up, 4 in 2024-09-21T08:26:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:19 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:26:20.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:19 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:26:20.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:19 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:20.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:19 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3974088961' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:20.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:19 smithi137 ceph-mon[41044]: osdmap e185: 8 total, 8 up, 4 in 2024-09-21T08:26:20.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:19 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:26:20.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:19 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:26:20.300 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:20.322 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:20.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:20.323 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:20.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:20.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:20.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:20.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:20.540 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:26:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:26:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:26:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:26:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.540 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44417.0:49 5.0 5:05bf5b68:::notify.1:head [watch watch cookie 94074441549248] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:34.575056+0000", 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 165.96316759499999, 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020510799999999998, 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44417", 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 49 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.541 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575056+0000", 2024-09-21T08:26:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575056+0000", 2024-09-21T08:26:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575057+0000", 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0869999999999999e-06 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575060+0000", 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3189999999999999e-06 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575072+0000", 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1497e-05 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575076+0000", 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.437e-06 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575414+0000", 2024-09-21T08:26:20.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033762100000000003 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.575490+0000", 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6458000000000006e-05 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.576135+0000", 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064504000000000004 2024-09-21T08:26:20.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.576216+0000", 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1156999999999996e-05 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,4", 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.576330+0000", 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113461 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:20.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.576666+0000", 2024-09-21T08:26:20.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00033670100000000001 2024-09-21T08:26:20.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:20.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.576964+0000", 2024-09-21T08:26:20.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029736500000000001 2024-09-21T08:26:20.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:20.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.577005+0000", 2024-09-21T08:26:20.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1458999999999997e-05 2024-09-21T08:26:20.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:20.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.577049+0000", 2024-09-21T08:26:20.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.401e-05 2024-09-21T08:26:20.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.549 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.549 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.577107+0000", 2024-09-21T08:26:20.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7467999999999999e-05 2024-09-21T08:26:20.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.549 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.549 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44417.0:59 5.14 5:2b04a3e9:::notify.6:head [watch watch cookie 94074441582864] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:26:20.549 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:34.582635+0000", 2024-09-21T08:26:20.550 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 165.95558799200001, 2024-09-21T08:26:20.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001918236, 2024-09-21T08:26:20.550 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.550 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.550 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:20.550 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.550 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44417", 2024-09-21T08:26:20.550 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:26:20.550 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 59 2024-09-21T08:26:20.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582635+0000", 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582635+0000", 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582637+0000", 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.393e-06 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582641+0000", 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5170000000000001e-06 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582649+0000", 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1200000000000002e-06 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582656+0000", 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.917e-06 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.582973+0000", 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00031646400000000002 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.583053+0000", 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0791999999999995e-05 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.583691+0000", 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00063707800000000001 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.583750+0000", 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9629000000000001e-05 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.583840+0000", 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0261999999999997e-05 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.584190+0000", 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035001200000000002 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.584384+0000", 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000193637 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.584483+0000", 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9339999999999994e-05 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.555 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.556 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:20.556 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.584515+0000", 2024-09-21T08:26:20.556 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1498000000000002e-05 2024-09-21T08:26:20.556 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.556 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.556 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:20.557 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.557 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:34.584553+0000", 2024-09-21T08:26:20.557 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8577e-05 2024-09-21T08:26:20.557 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.557 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:359 4.19 4:9ed7f5b7:::obj_delete_at_hint.0000000004:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:37.648931+0000", 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 162.88929211300001, 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025273050000000001, 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:26:20.558 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 359 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648931+0000", 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648931+0000", 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.559 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648932+0000", 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.164e-06 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648937+0000", 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9529999999999998e-06 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648948+0000", 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0589e-05 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.560 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.648955+0000", 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0230000000000004e-06 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.649000+0000", 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4855999999999999e-05 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.649071+0000", 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1392999999999999e-05 2024-09-21T08:26:20.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.649224+0000", 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015272800000000001 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.649703+0000", 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00047908 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.650827+0000", 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011241090000000001 2024-09-21T08:26:20.562 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.651380+0000", 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00055312099999999995 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.651424+0000", 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3677999999999998e-05 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:37.651458+0000", 2024-09-21T08:26:20.563 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4610999999999998e-05 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:993 5.0 5:05bf5b68:::notify.1:head [watch ping cookie 94837804043520] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:55.596150+0000", 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 144.94207288199999, 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034626729999999999, 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.564 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 993 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596150+0000", 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596150+0000", 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.565 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596152+0000", 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.186e-06 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596156+0000", 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8359999999999999e-06 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596164+0000", 2024-09-21T08:26:20.566 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4460000000000004e-06 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596172+0000", 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9610000000000005e-06 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596241+0000", 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9238000000000004e-05 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:20.567 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596247+0000", 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6210000000000001e-06 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599458+0000", 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003211546 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599525+0000", 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6511000000000001e-05 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.568 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599613+0000", 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8327999999999999e-05 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:994 5.14 5:2b04a3e9:::notify.6:head [watch ping cookie 94837804033152] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e75)", 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:23:55.596282+0000", 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 144.94194136300001, 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029709879999999999, 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.569 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 994 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596282+0000", 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.570 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596282+0000", 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596283+0000", 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9599999999999998e-07 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596287+0000", 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7799999999999998e-06 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.571 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596296+0000", 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1360000000000003e-06 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596302+0000", 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1890000000000004e-06 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596377+0000", 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5512999999999993e-05 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.572 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.596384+0000", 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1619999999999996e-06 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599087+0000", 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002703488 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599154+0000", 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7207999999999998e-05 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.573 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:23:55.599253+0000", 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8616000000000006e-05 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:1082 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94837804043520 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 102.378697845, 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.39751146399999998, 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.574 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1082 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.575 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159528+0000", 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2749999999999998e-06 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159533+0000", 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2480000000000004e-06 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.576 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159547+0000", 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4516e-05 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159556+0000", 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9460000000000004e-06 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159617+0000", 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0999000000000003e-05 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.577 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159625+0000", 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240000000000007e-06 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544062+0000", 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.38443652099999998 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544359+0000", 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000297378 2024-09-21T08:26:20.578 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548773+0000", 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044136440000000004 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548858+0000", 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5638999999999998e-05 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.579 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.557037+0000", 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081783740000000004 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(osd.0.0:103 7.6 e91/80 7:611b1c4b:::smithi13779221-3:3 v 91'13, mlcod=89'10)", 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:44.494057+0000", 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 96.044166004999994, 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043783829999999996, 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.580 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494057+0000", 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494057+0000", 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494058+0000", 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6600000000000005e-07 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.581 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494070+0000", 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2245e-05 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494080+0000", 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4299999999999995e-06 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494089+0000", 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.662e-06 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.582 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494146+0000", 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6526000000000001e-05 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.494167+0000", 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0690999999999999e-05 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.498394+0000", 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042278339999999998 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.583 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:44.498436+0000", 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1128999999999999e-05 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:255 7.2 7:4031afd1:::smithi13779221-48:head [copy-from ver 3] snapc 4=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 95.030387450999996, 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015259864, 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.584 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 255 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.585 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507839+0000", 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0089999999999999e-06 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507852+0000", 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3617e-05 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.586 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507872+0000", 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9482000000000001e-05 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507883+0000", 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1036000000000001e-05 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507941+0000", 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7819999999999999e-05 2024-09-21T08:26:20.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507946+0000", 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9080000000000003e-06 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510282+0000", 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023368170000000001 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510376+0000", 2024-09-21T08:26:20.588 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3846999999999998e-05 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.516270+0000", 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005893934 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522776+0000", 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065057179999999997 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522902+0000", 2024-09-21T08:26:20.589 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012641 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522990+0000", 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7491000000000006e-05 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523050+0000", 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0075999999999999e-05 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.590 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523096+0000", 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5698999999999997e-05 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:256 7.2 7:4031afd1:::smithi13779221-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 95.030320153000005, 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013379356, 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.591 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 256 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507904+0000", 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8000000000000005e-07 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507907+0000", 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.258e-06 2024-09-21T08:26:20.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507915+0000", 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.571e-06 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507920+0000", 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1189999999999996e-06 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507998+0000", 2024-09-21T08:26:20.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8181000000000002e-05 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.508004+0000", 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3960000000000003e-06 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510529+0000", 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025244989999999999 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511053+0000", 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052380199999999997 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521102+0000", 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010049361999999999 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521183+0000", 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0495999999999996e-05 2024-09-21T08:26:20.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521282+0000", 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9891999999999998e-05 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(osd.3.25:1 7.3 7:c851e2c3:::smithi13779221-25:head [assert-version v3,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.511125+0000", 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 95.027098378000005, 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032274180000000001, 2024-09-21T08:26:20.597 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511125+0000", 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511125+0000", 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511127+0000", 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.689e-06 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511133+0000", 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0599999999999996e-06 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511143+0000", 2024-09-21T08:26:20.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.039e-05 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511151+0000", 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3569999999999998e-06 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511209+0000", 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7954000000000001e-05 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511273+0000", 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3498999999999996e-05 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.514352+0000", 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003079469 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:262 7.a 7:53537bd3:::smithi13779221-39:5 [sparse-read 0~1161879,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554307+0000", 2024-09-21T08:26:20.601 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.983916042000004, 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030460629999999999, 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 262 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554307+0000", 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554307+0000", 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554310+0000", 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4269999999999998e-06 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554319+0000", 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0149999999999992e-06 2024-09-21T08:26:20.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554332+0000", 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3229e-05 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554343+0000", 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0669999999999999e-05 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554401+0000", 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8100000000000003e-05 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554575+0000", 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000174461 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557353+0000", 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027781609999999999 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.605 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:263 7.a 7:53537bd3:::smithi13779221-39:5 [sparse-read 0~1161879] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554362+0000", 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.983860953000004, 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032289990000000002, 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 263 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.606 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554362+0000", 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554362+0000", 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554363+0000", 2024-09-21T08:26:20.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3200000000000004e-07 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554367+0000", 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0980000000000004e-06 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554374+0000", 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6699999999999997e-06 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554380+0000", 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6339999999999998e-06 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557390+0000", 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030107010000000002 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557479+0000", 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8479000000000004e-05 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557591+0000", 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011258499999999999 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:264 7.a 7:53537bd3:::smithi13779221-39:5 [sparse-read 0~1161879] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554387+0000", 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.983835976999998, 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0045927440000000002, 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.610 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 264 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554387+0000", 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554387+0000", 2024-09-21T08:26:20.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.612 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554388+0000", 2024-09-21T08:26:20.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.37e-07 2024-09-21T08:26:20.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554392+0000", 2024-09-21T08:26:20.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7120000000000002e-06 2024-09-21T08:26:20.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554397+0000", 2024-09-21T08:26:20.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6810000000000003e-06 2024-09-21T08:26:20.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554402+0000", 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6990000000000002e-06 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558684+0000", 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042816920000000001 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558800+0000", 2024-09-21T08:26:20.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116122 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558980+0000", 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180101 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:260 7.8 e93/84 7:1fd29771:::smithi13779221-22:head v 93'20, mlcod=86'17)", 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:26:20.615 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.969273819999998, 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078947310000000003, 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568951+0000", 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1510000000000001e-06 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571221+0000", 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022708279999999999 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571244+0000", 2024-09-21T08:26:20.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2759999999999999e-05 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571257+0000", 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2669e-05 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571601+0000", 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034448900000000001 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571645+0000", 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3498999999999998e-05 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576787+0000", 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051417010000000003 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576844+0000", 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7633999999999997e-05 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.619 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:283 7.2 7:43ab705f:::smithi13779221-10:head [read 0~2151263,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762012+0000", 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.776211081, 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044470589999999997, 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:20.620 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 283 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762012+0000", 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762012+0000", 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762013+0000", 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9599999999999998e-07 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762028+0000", 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5381000000000001e-05 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762036+0000", 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5290000000000003e-06 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762042+0000", 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0399999999999998e-06 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.763797+0000", 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001754898 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.763899+0000", 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102248 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766459+0000", 2024-09-21T08:26:20.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025601669999999999 2024-09-21T08:26:20.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.624 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.625 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:284 7.2 7:43ab705f:::smithi13779221-10:head [sparse-read 0~1051225] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:20.625 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762051+0000", 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.776171947999998, 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0047848099999999996, 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 284 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762051+0000", 2024-09-21T08:26:20.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762051+0000", 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762052+0000", 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.85e-07 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762055+0000", 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.723e-06 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762060+0000", 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4890000000000003e-06 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762065+0000", 2024-09-21T08:26:20.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0019999999999997e-06 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766502+0000", 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044364219999999998 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766636+0000", 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133831 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766836+0000", 2024-09-21T08:26:20.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020055800000000001 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:294 7.2 7:4d793d31:::smithi13779221-9:head [copy-from ver 3] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.775557023999994, 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017337353, 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 294 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762668+0000", 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.353e-06 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762673+0000", 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1569999999999999e-06 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762683+0000", 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0331e-05 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762688+0000", 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1410000000000004e-06 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766904+0000", 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042154899999999997 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767004+0000", 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100135 2024-09-21T08:26:20.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.771465+0000", 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044613170000000002 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779379+0000", 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079137649999999997 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779883+0000", 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050370000000000005 2024-09-21T08:26:20.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779896+0000", 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3458999999999999e-05 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779950+0000", 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3851000000000003e-05 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.780004+0000", 2024-09-21T08:26:20.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3653999999999998e-05 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:295 7.2 7:4d793d31:::smithi13779221-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.775526306000003, 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012332286, 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:20.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 295 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4400000000000001e-07 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762699+0000", 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0099999999999998e-06 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762705+0000", 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0279999999999997e-06 2024-09-21T08:26:20.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762710+0000", 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8210000000000001e-06 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767340+0000", 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046298770000000001 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.768012+0000", 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067169199999999999 2024-09-21T08:26:20.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774900+0000", 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068879680000000004 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774961+0000", 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0980999999999998e-05 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.775029+0000", 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8465000000000003e-05 2024-09-21T08:26:20.640 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.641 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.641 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.641 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(osd.3.25:2 7.2 7:48ea7305:::smithi13779221-5:head [assert-version v3,copy-get max 8388608] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:20.641 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.767305+0000", 2024-09-21T08:26:20.641 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.770918397000003, 2024-09-21T08:26:20.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002516744, 2024-09-21T08:26:20.641 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.641 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767305+0000", 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767305+0000", 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767311+0000", 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2809999999999997e-06 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767331+0000", 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9417000000000001e-05 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767352+0000", 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1790999999999999e-05 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767366+0000", 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.376e-05 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.768057+0000", 2024-09-21T08:26:20.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069047800000000001 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.768156+0000", 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9221000000000004e-05 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.769822+0000", 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001665796 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.645 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:321 7.c e93/80 7:3cae00e7:::smithi13779221-28:head v 93'11, mlcod=86'8)", 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.815621+0000", 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.722601849, 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054018499999999997, 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:20.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.815621+0000", 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.815621+0000", 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:20.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.815622+0000", 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9900000000000001e-07 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.818864+0000", 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032423980000000001 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.818878+0000", 2024-09-21T08:26:20.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3699e-05 2024-09-21T08:26:20.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:20.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.818886+0000", 2024-09-21T08:26:20.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0960000000000001e-06 2024-09-21T08:26:20.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:20.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.818961+0000", 2024-09-21T08:26:20.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5345000000000005e-05 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.818989+0000", 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.811e-05 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.820957+0000", 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001967856 2024-09-21T08:26:20.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:20.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:20.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:20.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.821023+0000", 2024-09-21T08:26:20.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5946999999999997e-05 2024-09-21T08:26:20.651 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.651 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.651 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.651 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:20.651 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:20.651 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:26:20.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:20.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:20.653 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:20.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:20.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:20.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:20.776 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:20.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:20.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:20.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:20.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:20.930 INFO:tasks.rados.rados.0.smithi137.stdout:598: done (2 left) 2024-09-21T08:26:20.930 INFO:tasks.rados.rados.0.smithi137.stdout:599: snap_create 2024-09-21T08:26:20.930 INFO:tasks.rados.rados.0.smithi137.stdout:597: finishing rollback tid 1 to smithi13779221-25 2024-09-21T08:26:20.930 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 0 (ObjNum 129 snap 0 seq_num 3760704968) dirty dne 2024-09-21T08:26:20.930 INFO:tasks.rados.rados.0.smithi137.stdout:596: finishing write tid 2 to smithi13779221-4 2024-09-21T08:26:20.930 INFO:tasks.rados.rados.0.smithi137.stdout:596: finishing write tid 3 to smithi13779221-4 2024-09-21T08:26:20.930 INFO:tasks.rados.rados.0.smithi137.stdout:596: finishing write tid 4 to smithi13779221-4 2024-09-21T08:26:20.930 INFO:tasks.rados.rados.0.smithi137.stdout:596: oid 4 updating version 0 to 240 2024-09-21T08:26:20.930 INFO:tasks.rados.rados.0.smithi137.stdout:596: oid 4 updating version 240 to 241 2024-09-21T08:26:20.931 INFO:tasks.rados.rados.0.smithi137.stdout:596: oid 4 updating version 241 to 242 2024-09-21T08:26:20.931 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 242 (ObjNum 244 snap 52 seq_num 244) dirty exists 2024-09-21T08:26:20.931 INFO:tasks.rados.rados.0.smithi137.stdout:596: left oid 4 (ObjNum 244 snap 52 seq_num 244) 2024-09-21T08:26:20.932 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:26:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:26:21.006 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:21.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:21.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:21.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:21.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:21.153 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:21.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:21.154 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:21.255 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:20 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:26:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:20 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:26:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:20 smithi165 ceph-mon[32893]: osdmap e186: 8 total, 8 up, 4 in 2024-09-21T08:26:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:20 smithi165 ceph-mon[32893]: pgmap v185: 158 pgs: 2 active+clean+snaptrim, 10 peering, 10 unknown, 136 active+clean; 132 MiB data, 455 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:21.263 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:20 smithi165 ceph-mon[32893]: osdmap e187: 8 total, 8 up, 4 in 2024-09-21T08:26:21.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:26:21.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:26:21.267 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-mon[29634]: osdmap e186: 8 total, 8 up, 4 in 2024-09-21T08:26:21.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-mon[29634]: pgmap v185: 158 pgs: 2 active+clean+snaptrim, 10 peering, 10 unknown, 136 active+clean; 132 MiB data, 455 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:21.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-mon[29634]: osdmap e187: 8 total, 8 up, 4 in 2024-09-21T08:26:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:26:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:26:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-mon[41044]: osdmap e186: 8 total, 8 up, 4 in 2024-09-21T08:26:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-mon[41044]: pgmap v185: 158 pgs: 2 active+clean+snaptrim, 10 peering, 10 unknown, 136 active+clean; 132 MiB data, 455 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:21.268 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:20 smithi137 ceph-mon[41044]: osdmap e187: 8 total, 8 up, 4 in 2024-09-21T08:26:21.279 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:21.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:21.279 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:21.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:21.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:21.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:21.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:21.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:21.513 INFO:tasks.rados.rados.0.smithi137.stdout:596: done (2 left) 2024-09-21T08:26:21.513 INFO:tasks.rados.rados.0.smithi137.stdout:597: done (1 left) 2024-09-21T08:26:21.513 INFO:tasks.rados.rados.0.smithi137.stdout:599: done (0 left) 2024-09-21T08:26:21.514 INFO:tasks.rados.rados.0.smithi137.stdout:600: read oid 21 snap -1 2024-09-21T08:26:21.514 INFO:tasks.rados.rados.0.smithi137.stdout:600: expect deleted 2024-09-21T08:26:21.514 INFO:tasks.rados.rados.0.smithi137.stdout:601: write (excl) oid 29 current snap is 53 2024-09-21T08:26:21.514 INFO:tasks.rados.rados.0.smithi137.stdout:601: seq_num 245 ranges {517715=559301,1870206=247509} 2024-09-21T08:26:21.520 INFO:tasks.rados.rados.0.smithi137.stdout:601: writing smithi13779221-29 from 517715 to 1077016 tid 1 2024-09-21T08:26:21.526 INFO:tasks.rados.rados.0.smithi137.stdout:601: writing smithi13779221-29 from 1870206 to 2117715 tid 2 2024-09-21T08:26:21.526 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:21.527 INFO:tasks.rados.rados.0.smithi137.stdout:600: done (1 left) 2024-09-21T08:26:21.527 INFO:tasks.rados.rados.0.smithi137.stdout:602: snap_remove snap 48 2024-09-21T08:26:21.527 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:21.527 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:21.628 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:21.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:21.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:21.650 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:21.751 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:21.773 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:21.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:21.773 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:21.786 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:21.786 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:21.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T08:26:21.875 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:21.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:21.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:21.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:21.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:22.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:26:21 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:26:22.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:22.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:22.022 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:22.105 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:22.122 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:22.143 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:22.145 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:22.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:22.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:22.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:22.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:22.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:26:22.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:22.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:22.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:22.390 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:22.434 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:22.434 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:22.434 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[29634]: osdmap e188: 8 total, 8 up, 4 in 2024-09-21T08:26:22.434 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:26:22.434 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:26:22.434 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:22.435 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:22.435 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:22.435 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[41044]: osdmap e188: 8 total, 8 up, 4 in 2024-09-21T08:26:22.435 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:26:22.435 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:26:22.435 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:22 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:22.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:22.514 INFO:tasks.rados.rados.0.smithi137.stdout:602: done (1 left) 2024-09-21T08:26:22.514 INFO:tasks.rados.rados.0.smithi137.stdout:603: delete oid 11 current snap is 53 2024-09-21T08:26:22.514 INFO:tasks.rados.rados.0.smithi137.stdout:601: finishing write tid 1 to smithi13779221-29 2024-09-21T08:26:22.514 INFO:tasks.rados.rados.0.smithi137.stdout:601: finishing write tid 2 to smithi13779221-29 2024-09-21T08:26:22.514 INFO:tasks.rados.rados.0.smithi137.stdout:601: finishing write tid 3 to smithi13779221-29 2024-09-21T08:26:22.514 INFO:tasks.rados.rados.0.smithi137.stdout:601: finishing write tid 4 to smithi13779221-29 2024-09-21T08:26:22.515 INFO:tasks.rados.rados.0.smithi137.stdout:601: oid 29 updating version 0 to 266 2024-09-21T08:26:22.515 INFO:tasks.rados.rados.0.smithi137.stdout:601: oid 29 version 266 is already newer than 265 2024-09-21T08:26:22.515 INFO:tasks.rados.rados.0.smithi137.stdout:601: oid 29 version 266 is already newer than 264 2024-09-21T08:26:22.515 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 266 (ObjNum 245 snap 53 seq_num 245) dirty exists 2024-09-21T08:26:22.515 INFO:tasks.rados.rados.0.smithi137.stdout:601: left oid 29 (ObjNum 245 snap 53 seq_num 245) 2024-09-21T08:26:22.515 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:22.516 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:22.516 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:22.522 INFO:tasks.rados.rados.0.smithi137.stdout:601: done (1 left) 2024-09-21T08:26:22.522 INFO:tasks.rados.rados.0.smithi137.stdout:603: done (0 left) 2024-09-21T08:26:22.522 INFO:tasks.rados.rados.0.smithi137.stdout:604: rollback oid 28 current snap is 53 2024-09-21T08:26:22.522 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 28 to 52 2024-09-21T08:26:22.522 INFO:tasks.rados.rados.0.smithi137.stdout:605: read oid 21 snap 52 2024-09-21T08:26:22.522 INFO:tasks.rados.rados.0.smithi137.stdout:605: expect deleted 2024-09-21T08:26:22.522 INFO:tasks.rados.rados.0.smithi137.stdout:606: write oid 39 current snap is 53 2024-09-21T08:26:22.522 INFO:tasks.rados.rados.0.smithi137.stdout:606: seq_num 247 ranges {551329=709799,1870817=280512} 2024-09-21T08:26:22.529 INFO:tasks.rados.rados.0.smithi137.stdout:606: writing smithi13779221-39 from 551329 to 1261128 tid 1 2024-09-21T08:26:22.534 INFO:tasks.rados.rados.0.smithi137.stdout:606: writing smithi13779221-39 from 1870817 to 2151329 tid 2 2024-09-21T08:26:22.534 INFO:tasks.rados.rados.0.smithi137.stdout:607: snap_remove snap 51 2024-09-21T08:26:22.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:22 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:22.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:22 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:22.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:22 smithi165 ceph-mon[32893]: osdmap e188: 8 total, 8 up, 4 in 2024-09-21T08:26:22.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:22 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:26:22.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:22 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:26:22.587 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:22 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:26:22.598 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:22.599 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:26:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:26:22.599 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:26:22.599 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:26:22.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:22.639 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:22.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:22.640 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 189, 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "ec8566d6-77f1-11ef-baf6-efdc52797490", 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-21T08:18:21.794737+0000", 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-21T08:26:22.504030+0000", 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-21T08:23:19.473450+0000", 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-21T08:25:47.856639+0000", 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5806080, 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "nodeep-scrub", 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "noscrub", 2024-09-21T08:26:22.720 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 7, 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-21T08:26:22.721 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:21:15.037508+0000", 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:22.722 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 1, 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 1, 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 1, 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 1, 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 1, 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:22.723 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "122", 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:22.724 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:22.725 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:22.726 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 7.8899998664855957, 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 7.8899998664855957, 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3, 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3, 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:22.727 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:28.246186+0000", 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:22.728 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:22.729 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 68, 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:22.730 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:22.731 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:22.732 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.6600000858306885, 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.6600000858306885, 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.3300000429153442, 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.3300000429153442, 2024-09-21T08:26:22.733 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:29.209879+0000", 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:26:22.734 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:26:22.735 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:26:22.736 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:26:22.736 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:26:22.736 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:26:22.736 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:22.736 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:22.736 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:22.736 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:22.737 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:22.737 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:22.737 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:22.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.737 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "67", 2024-09-21T08:26:22.737 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:22.737 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:22.737 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:22.737 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:22.738 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:22.739 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:22.740 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.5, 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.5, 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:22.741 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:30.851833+0000", 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:22.742 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:22.743 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "69", 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:22.744 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:22.745 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:22.746 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.5, 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.5, 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:22.747 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:32.798630+0000", 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:22.748 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:22.749 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "71", 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:22.750 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:22.751 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:26:22.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.2599999904632568, 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.2599999904632568, 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.1299999952316284, 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.1299999952316284, 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.753 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:34.858938+0000", 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:22.754 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "73", 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:22.755 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:22.756 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:22.757 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.5, 2024-09-21T08:26:22.758 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.5, 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_0", 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:24:33.441067+0000", 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T08:26:22.759 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 26, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 26, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 26, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 26, 2024-09-21T08:26:22.760 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 26, 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "189", 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "181", 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 54, 2024-09-21T08:26:22.761 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 189, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:22.762 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:22.763 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.380000114440918, 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.380000114440918, 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.690000057220459, 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.690000057220459, 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.764 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977", 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 186, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.766 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.767 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/3938795530", 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/3938795530", 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/3938795530", 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/3938795530", 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d", 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:22.768 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 102, 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:26:22.769 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.770 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.771 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/2743268361", 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/2743268361", 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/2743268361", 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/2743268361", 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d", 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:22.772 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 185, 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.773 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:26:22.774 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1500397557", 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1500397557", 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1500397557", 2024-09-21T08:26:22.775 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1500397557", 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "73f21606-38be-4578-a86b-bb417217c0e0", 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:22.776 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 25, 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 84, 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.778 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.779 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/617155021", 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/617155021", 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/617155021", 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/617155021", 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-21T08:26:22.780 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae", 2024-09-21T08:26:22.781 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:22.781 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:26:22.781 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:26:22.781 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:22.781 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:22.781 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:22.781 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 31, 2024-09-21T08:26:22.781 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 80, 2024-09-21T08:26:22.781 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:22.781 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6800", 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.782 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6801", 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6803", 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6802", 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6800/70360958", 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6801/70360958", 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6803/70360958", 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6802/70360958", 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:22.784 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd", 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 41, 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 181, 2024-09-21T08:26:22.785 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6804", 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.786 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6805", 2024-09-21T08:26:22.787 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:26:22.787 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.787 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6807", 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6806", 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6804/1835779380", 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6805/1835779380", 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6807/1835779380", 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6806/1835779380", 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:22.789 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b", 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 49, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 94, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:22.790 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6808", 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6809", 2024-09-21T08:26:22.791 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6811", 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.792 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6810", 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6808/3820354208", 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6809/3820354208", 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6811/3820354208", 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6810/3820354208", 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:22.793 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f", 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 59, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 186, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:22.794 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6812", 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6813", 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:26:22.795 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6815", 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:22.796 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6814", 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6812/1933524273", 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6813/1933524273", 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6815/1933524273", 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6814/1933524273", 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:22.797 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:25:46.489431+0000", 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:22.798 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:26:00.210653+0000", 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:26:02.707590+0000", 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:22.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:25:25.742806+0000", 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:22.800 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:26:18.932348+0000", 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:26:19.772946+0000", 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:25:15.918748+0000", 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.802 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:25:57.391802+0000", 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.803 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "2.1", 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 0, 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1478868457": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1738156348": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2643910317": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:26:22.804 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/705906958": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3936900881": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/223603733": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1718301516": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2580000250": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3166961425": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/700535732": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1900990534": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1825289826": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3010106803": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3442439903": "2024-09-22T08:23:59.682278+0000" 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.805 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [ 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.806 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 37, 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 15 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [ 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:22.807 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 50, 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-21T08:26:22.808 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:22.809 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:26:22.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:22.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:22.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:22.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:22.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:22.955 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:22.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:23.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:23.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:23.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:23.077 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:23.178 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:23.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:23.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:23.201 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:23.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T08:26:23.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:23.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:23.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:23.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:23.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:23.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:23.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:23.452 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:23.514 INFO:tasks.rados.rados.0.smithi137.stdout:607: done (3 left) 2024-09-21T08:26:23.514 INFO:tasks.rados.rados.0.smithi137.stdout:608: write (excl) oid 11 current snap is 53 2024-09-21T08:26:23.514 INFO:tasks.rados.rados.0.smithi137.stdout:608: seq_num 248 ranges {568136=785048,1929299=238837} 2024-09-21T08:26:23.516 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:23.521 INFO:tasks.rados.rados.0.smithi137.stdout:608: writing smithi13779221-11 from 568136 to 1353184 tid 1 2024-09-21T08:26:23.526 INFO:tasks.rados.rados.0.smithi137.stdout:608: writing smithi13779221-11 from 1929299 to 2168136 tid 2 2024-09-21T08:26:23.526 INFO:tasks.rados.rados.0.smithi137.stdout:609: snap_remove snap 50 2024-09-21T08:26:23.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:23.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:23.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:23.574 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:23 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:23 smithi137 ceph-mon[29634]: pgmap v188: 158 pgs: 3 peering, 155 active+clean; 148 MiB data, 506 MiB used, 357 GiB / 358 GiB avail; 9.9 MiB/s rd, 4.7 MiB/s wr, 20 op/s; 9.3 MiB/s, 13 objects/s recovering 2024-09-21T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:23 smithi137 ceph-mon[29634]: osdmap e189: 8 total, 8 up, 4 in 2024-09-21T08:26:23.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:23 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/434979922' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:26:23.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:23 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:23.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:23 smithi137 ceph-mon[41044]: pgmap v188: 158 pgs: 3 peering, 155 active+clean; 148 MiB data, 506 MiB used, 357 GiB / 358 GiB avail; 9.9 MiB/s rd, 4.7 MiB/s wr, 20 op/s; 9.3 MiB/s, 13 objects/s recovering 2024-09-21T08:26:23.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:23 smithi137 ceph-mon[41044]: osdmap e189: 8 total, 8 up, 4 in 2024-09-21T08:26:23.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:23 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/434979922' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:26:23.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:23.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:23.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:23.697 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:23.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:23 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:23.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:23 smithi165 ceph-mon[32893]: pgmap v188: 158 pgs: 3 peering, 155 active+clean; 148 MiB data, 506 MiB used, 357 GiB / 358 GiB avail; 9.9 MiB/s rd, 4.7 MiB/s wr, 20 op/s; 9.3 MiB/s, 13 objects/s recovering 2024-09-21T08:26:23.789 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:23 smithi165 ceph-mon[32893]: osdmap e189: 8 total, 8 up, 4 in 2024-09-21T08:26:23.790 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:23 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/434979922' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:26:23.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:23.820 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:23.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:23.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:23.900 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:23.900 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:23.917 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:23.917 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:23.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:23.933 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:23.934 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:23.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:23.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:23.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:23.949 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:23.950 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:23.965 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:23.966 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:23.982 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:23.982 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:23.998 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:23.998 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:24.014 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:24.014 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:24.025 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:24.025 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:24.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:24.065 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:24.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:24.066 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:24.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:24.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:24.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:24.190 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:24.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:24.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:24.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:24.312 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:24.340 INFO:tasks.thrashosds.thrasher:j is {'epoch': 189, 'fsid': 'ec8566d6-77f1-11ef-baf6-efdc52797490', 'created': '2024-09-21T08:18:21.794737+0000', 'modified': '2024-09-21T08:26:22.504030+0000', 'last_up_change': '2024-09-21T08:23:19.473450+0000', 'last_in_change': '2024-09-21T08:25:47.856639+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T08:21:15.037508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '122', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T08:23:28.246186+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6600000858306885, 'score_stable': 2.6600000858306885, 'optimal_score': 0.5, 'raw_score_acting': 1.3300000429153442, 'raw_score_stable': 1.3300000429153442, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T08:23:29.209879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 0.5, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T08:23:30.851833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 0.5, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T08:23:32.798630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.259999990463257, 'score_stable': 2.259999990463257, 'optimal_score': 0.5, 'raw_score_acting': 1.1299999952316284, 'raw_score_stable': 1.1299999952316284, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T08:23:34.858938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.5, 'score_stable': 2.5, 'optimal_score': 0.5, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-21T08:24:33.441067+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '189', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '181', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 54, 'snap_epoch': 189, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.380000114440918, 'score_stable': 3.380000114440918, 'optimal_score': 0.5, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '1b007d1d-6c6b-4a68-9e68-2c5fd7d26977', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 186, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 3938795530}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 3938795530}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 3938795530}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 3938795530}]}, 'public_addr': '172.21.15.137:6801/3938795530', 'cluster_addr': '172.21.15.137:6802/3938795530', 'heartbeat_back_addr': '172.21.15.137:6804/3938795530', 'heartbeat_front_addr': '172.21.15.137:6803/3938795530', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8651068e-694f-4bd4-9917-5c55670aca5d', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 2743268361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 2743268361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 2743268361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 2743268361}]}, 'public_addr': '172.21.15.137:6805/2743268361', 'cluster_addr': '172.21.15.137:6806/2743268361', 'heartbeat_back_addr': '172.21.15.137:6808/2743268361', 'heartbeat_front_addr': '172.21.15.137:6807/2743268361', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2a22f0f7-c610-4995-98a7-d4785efb1e5d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 185, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1500397557}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1500397557}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1500397557}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1500397557}]}, 'public_addr': '172.21.15.137:6809/1500397557', 'cluster_addr': '172.21.15.137:6810/1500397557', 'heartbeat_back_addr': '172.21.15.137:6812/1500397557', 'heartbeat_front_addr': '172.21.15.137:6811/1500397557', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '73f21606-38be-4578-a86b-bb417217c0e0', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 84, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 617155021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 617155021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 617155021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 617155021}]}, 'public_addr': '172.21.15.137:6813/617155021', 'cluster_addr': '172.21.15.137:6814/617155021', 'heartbeat_back_addr': '172.21.15.137:6816/617155021', 'heartbeat_front_addr': '172.21.15.137:6815/617155021', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e5f6762b-65f5-4dc7-8e54-9087149ae6ae', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 70360958}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 70360958}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 70360958}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 70360958}]}, 'public_addr': '172.21.15.165:6800/70360958', 'cluster_addr': '172.21.15.165:6801/70360958', 'heartbeat_back_addr': '172.21.15.165:6803/70360958', 'heartbeat_front_addr': '172.21.15.165:6802/70360958', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f7f8c456-b77f-4435-a860-202e64dda8dd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 181, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 1835779380}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 1835779380}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 1835779380}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 1835779380}]}, 'public_addr': '172.21.15.165:6804/1835779380', 'cluster_addr': '172.21.15.165:6805/1835779380', 'heartbeat_back_addr': '172.21.15.165:6807/1835779380', 'heartbeat_front_addr': '172.21.15.165:6806/1835779380', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b8b3df16-73fa-41a5-a93f-117c599d488b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 3820354208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 3820354208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 3820354208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 3820354208}]}, 'public_addr': '172.21.15.165:6808/3820354208', 'cluster_addr': '172.21.15.165:6809/3820354208', 'heartbeat_back_addr': '172.21.15.165:6811/3820354208', 'heartbeat_front_addr': '172.21.15.165:6810/3820354208', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ddf234e4-1494-40bd-a31b-333b01717c3f', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 186, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 1933524273}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 1933524273}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 1933524273}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 1933524273}]}, 'public_addr': '172.21.15.165:6812/1933524273', 'cluster_addr': '172.21.15.165:6813/1933524273', 'heartbeat_back_addr': '172.21.15.165:6815/1933524273', 'heartbeat_front_addr': '172.21.15.165:6814/1933524273', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:25:46.489431+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:26:00.210653+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:26:02.707590+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:25:25.742806+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:26:18.932348+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:26:19.772946+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:25:15.918748+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:25:57.391802+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1', 'mappings': [{'from': 0, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:0/1478868457': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/1738156348': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/2643910317': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/705906958': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/3936900881': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/223603733': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/1718301516': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:6800/2580000250': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/3166961425': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/700535732': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:6800/1900990534': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:6800/1825289826': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/3010106803': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/3442439903': '2024-09-22T08:23:59.682278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 37, 'length': 15}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 50, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T08:26:24.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:26:24.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:24.445 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:24.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:24.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:609: done (4 left) 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:610: delete oid 45 current snap is 53 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:604: finishing rollback tid 0 to smithi13779221-28 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:604: finishing rollback tid 1 to smithi13779221-28 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:604: finishing rollback tid 2 to smithi13779221-28 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 107 (ObjNum 211 snap 43 seq_num 211) dirty exists 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:606: finishing write tid 1 to smithi13779221-39 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:606: finishing write tid 2 to smithi13779221-39 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:606: finishing write tid 3 to smithi13779221-39 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:606: finishing write tid 4 to smithi13779221-39 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:606: oid 39 updating version 0 to 47 2024-09-21T08:26:24.524 INFO:tasks.rados.rados.0.smithi137.stdout:606: oid 39 version 47 is already newer than 45 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:606: oid 39 version 47 is already newer than 46 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 47 (ObjNum 247 snap 53 seq_num 247) dirty exists 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:606: left oid 39 (ObjNum 247 snap 53 seq_num 247) 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:608: finishing write tid 1 to smithi13779221-11 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:608: finishing write tid 2 to smithi13779221-11 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:608: finishing write tid 3 to smithi13779221-11 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:608: finishing write tid 4 to smithi13779221-11 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:608: oid 11 updating version 0 to 78 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:608: oid 11 updating version 78 to 79 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:608: oid 11 updating version 79 to 80 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 80 (ObjNum 248 snap 53 seq_num 248) dirty exists 2024-09-21T08:26:24.525 INFO:tasks.rados.rados.0.smithi137.stdout:608: left oid 11 (ObjNum 248 snap 53 seq_num 248) 2024-09-21T08:26:24.531 INFO:tasks.rados.rados.0.smithi137.stdout:604: done (4 left) 2024-09-21T08:26:24.531 INFO:tasks.rados.rados.0.smithi137.stdout:605: done (3 left) 2024-09-21T08:26:24.531 INFO:tasks.rados.rados.0.smithi137.stdout:606: done (2 left) 2024-09-21T08:26:24.531 INFO:tasks.rados.rados.0.smithi137.stdout:608: done (1 left) 2024-09-21T08:26:24.531 INFO:tasks.rados.rados.0.smithi137.stdout:610: done (0 left) 2024-09-21T08:26:24.531 INFO:tasks.rados.rados.0.smithi137.stdout:611: write oid 29 current snap is 53 2024-09-21T08:26:24.531 INFO:tasks.rados.rados.0.smithi137.stdout:611: seq_num 250 ranges {601750=535546,1929910=271840} 2024-09-21T08:26:24.537 INFO:tasks.rados.rados.0.smithi137.stdout:611: writing smithi13779221-29 from 601750 to 1137296 tid 1 2024-09-21T08:26:24.543 INFO:tasks.rados.rados.0.smithi137.stdout:611: writing smithi13779221-29 from 1929910 to 2201750 tid 2 2024-09-21T08:26:24.543 INFO:tasks.rados.rados.0.smithi137.stdout:612: read oid 28 snap -1 2024-09-21T08:26:24.543 INFO:tasks.rados.rados.0.smithi137.stdout:612: expect (ObjNum 211 snap 43 seq_num 211) 2024-09-21T08:26:24.544 INFO:tasks.rados.rados.0.smithi137.stdout:613: delete oid 34 current snap is 53 2024-09-21T08:26:24.544 INFO:tasks.rados.rados.0.smithi137.stdout:611: finishing write tid 1 to smithi13779221-29 2024-09-21T08:26:24.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:24.547 INFO:tasks.rados.rados.0.smithi137.stdout:613: done (2 left) 2024-09-21T08:26:24.547 INFO:tasks.rados.rados.0.smithi137.stdout:614: read oid 4 snap -1 2024-09-21T08:26:24.547 INFO:tasks.rados.rados.0.smithi137.stdout:614: expect (ObjNum 244 snap 52 seq_num 244) 2024-09-21T08:26:24.547 INFO:tasks.rados.rados.0.smithi137.stdout:615: delete oid 11 current snap is 53 2024-09-21T08:26:24.547 INFO:tasks.rados.rados.0.smithi137.stdout:611: finishing write tid 2 to smithi13779221-29 2024-09-21T08:26:24.547 INFO:tasks.rados.rados.0.smithi137.stdout:611: finishing write tid 3 to smithi13779221-29 2024-09-21T08:26:24.547 INFO:tasks.rados.rados.0.smithi137.stdout:611: finishing write tid 4 to smithi13779221-29 2024-09-21T08:26:24.548 INFO:tasks.rados.rados.0.smithi137.stdout:611: oid 29 updating version 0 to 273 2024-09-21T08:26:24.548 INFO:tasks.rados.rados.0.smithi137.stdout:611: oid 29 version 273 is already newer than 271 2024-09-21T08:26:24.548 INFO:tasks.rados.rados.0.smithi137.stdout:611: oid 29 version 273 is already newer than 272 2024-09-21T08:26:24.548 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 273 (ObjNum 250 snap 53 seq_num 250) dirty exists 2024-09-21T08:26:24.548 INFO:tasks.rados.rados.0.smithi137.stdout:611: left oid 29 (ObjNum 250 snap 53 seq_num 250) 2024-09-21T08:26:24.549 INFO:tasks.rados.rados.0.smithi137.stdout:611: done (3 left) 2024-09-21T08:26:24.549 INFO:tasks.rados.rados.0.smithi137.stdout:615: done (2 left) 2024-09-21T08:26:24.549 INFO:tasks.rados.rados.0.smithi137.stdout:616: copy_from oid 26 from oid 14 current snap is 53 2024-09-21T08:26:24.549 INFO:tasks.rados.rados.0.smithi137.stdout:617: copy_from oid 43 from oid 7 current snap is 53 2024-09-21T08:26:24.549 INFO:tasks.rados.rados.0.smithi137.stdout:618: snap_remove snap 52 2024-09-21T08:26:24.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:24.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:24.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:24.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:24 smithi165 ceph-mon[32893]: osdmap e190: 8 total, 8 up, 4 in 2024-09-21T08:26:24.617 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:24 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3009610106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:24.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T08:26:24.672 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:24.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:24.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:24.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:24.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:24 smithi137 ceph-mon[29634]: osdmap e190: 8 total, 8 up, 4 in 2024-09-21T08:26:24.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:24 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3009610106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:24.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:24 smithi137 ceph-mon[41044]: osdmap e190: 8 total, 8 up, 4 in 2024-09-21T08:26:24.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:24 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3009610106' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:24.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:24.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:24.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:24.833 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:24.889 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:24.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:24.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:24.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:24.960 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:24.983 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:25.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:25.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:25.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:25.083 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:25.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:25.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:25.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:25.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:25.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:25.328 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:25.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:25.329 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:25.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:25.453 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:25.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:25.454 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:25.528 INFO:tasks.rados.rados.0.smithi137.stdout:618: done (4 left) 2024-09-21T08:26:25.528 INFO:tasks.rados.rados.0.smithi137.stdout:619: snap_create 2024-09-21T08:26:25.528 INFO:tasks.rados.rados.0.smithi137.stdout:616: finishing copy_from to smithi13779221-26 2024-09-21T08:26:25.528 INFO:tasks.rados.rados.0.smithi137.stdout:616: got expected ENOENT (src dne) 2024-09-21T08:26:25.528 INFO:tasks.rados.rados.0.smithi137.stdout:616: finishing copy_from racing read to smithi13779221-26 2024-09-21T08:26:25.528 INFO:tasks.rados.rados.0.smithi137.stdout:614: expect (ObjNum 244 snap 52 seq_num 244) 2024-09-21T08:26:25.529 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:26:25.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:25.576 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:25.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:25.577 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:25.634 INFO:tasks.rados.rados.0.smithi137.stdout:617: finishing copy_from racing read to smithi13779221-43 2024-09-21T08:26:25.635 INFO:tasks.rados.rados.0.smithi137.stdout:612: expect (ObjNum 211 snap 43 seq_num 211) 2024-09-21T08:26:25.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:25.688 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:25 smithi137 ceph-mon[29634]: pgmap v191: 158 pgs: 3 peering, 155 active+clean; 148 MiB data, 506 MiB used, 357 GiB / 358 GiB avail; 9.9 MiB/s rd, 4.7 MiB/s wr, 20 op/s; 9.3 MiB/s, 13 objects/s recovering 2024-09-21T08:26:25.688 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:25 smithi137 ceph-mon[29634]: osdmap e191: 8 total, 8 up, 4 in 2024-09-21T08:26:25.688 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:25 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/645888079' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:25.688 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:25 smithi137 ceph-mon[29634]: from='client.45023 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:25.688 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:25 smithi137 ceph-mon[41044]: pgmap v191: 158 pgs: 3 peering, 155 active+clean; 148 MiB data, 506 MiB used, 357 GiB / 358 GiB avail; 9.9 MiB/s rd, 4.7 MiB/s wr, 20 op/s; 9.3 MiB/s, 13 objects/s recovering 2024-09-21T08:26:25.688 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:25 smithi137 ceph-mon[41044]: osdmap e191: 8 total, 8 up, 4 in 2024-09-21T08:26:25.688 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:25 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/645888079' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:25.689 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:25 smithi137 ceph-mon[41044]: from='client.45023 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:25.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:25.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:25.701 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:25.742 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:25.742 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:25.786 INFO:tasks.rados.rados.0.smithi137.stdout:617: finishing copy_from to smithi13779221-43 2024-09-21T08:26:25.786 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 259 (ObjNum 156 snap 31 seq_num 156) dirty exists 2024-09-21T08:26:25.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:25.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:25.824 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:25.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:25.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:25.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:25.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:25.946 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:25 smithi165 ceph-mon[32893]: pgmap v191: 158 pgs: 3 peering, 155 active+clean; 148 MiB data, 506 MiB used, 357 GiB / 358 GiB avail; 9.9 MiB/s rd, 4.7 MiB/s wr, 20 op/s; 9.3 MiB/s, 13 objects/s recovering 2024-09-21T08:26:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:25 smithi165 ceph-mon[32893]: osdmap e191: 8 total, 8 up, 4 in 2024-09-21T08:26:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:25 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/645888079' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:25 smithi165 ceph-mon[32893]: from='client.45023 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:26.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:26.069 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:26.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:26.070 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:26.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:26.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:26.192 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:26.193 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:26.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:26.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:26.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:26.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:26.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T08:26:26.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:26.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:26.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:26.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:26.511 INFO:tasks.rados.rados.0.smithi137.stdout:612: done (4 left) 2024-09-21T08:26:26.511 INFO:tasks.rados.rados.0.smithi137.stdout:614: done (3 left) 2024-09-21T08:26:26.511 INFO:tasks.rados.rados.0.smithi137.stdout:616: done (2 left) 2024-09-21T08:26:26.511 INFO:tasks.rados.rados.0.smithi137.stdout:617: done (1 left) 2024-09-21T08:26:26.511 INFO:tasks.rados.rados.0.smithi137.stdout:619: done (0 left) 2024-09-21T08:26:26.511 INFO:tasks.rados.rados.0.smithi137.stdout:620: delete oid 39 current snap is 54 2024-09-21T08:26:26.514 INFO:tasks.rados.rados.0.smithi137.stdout:620: done (0 left) 2024-09-21T08:26:26.515 INFO:tasks.rados.rados.0.smithi137.stdout:621: snap_create 2024-09-21T08:26:26.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:26.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:26.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:26.601 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:26.621 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:26.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:26.722 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:26:26.724 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:26:26.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:26.726 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:26.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:26.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:26 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:26.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:26 smithi165 ceph-mon[32893]: from='client.45023 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:26:26.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:26 smithi165 ceph-mon[32893]: osdmap e192: 8 total, 8 up, 4 in 2024-09-21T08:26:26.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:26 smithi165 ceph-mon[32893]: osdmap e193: 8 total, 8 up, 4 in 2024-09-21T08:26:26.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:26.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:26.850 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:26.850 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:26.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:26.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:26.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:26.973 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:27.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:26 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:27.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:26 smithi137 ceph-mon[29634]: from='client.45023 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:26:27.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:26 smithi137 ceph-mon[29634]: osdmap e192: 8 total, 8 up, 4 in 2024-09-21T08:26:27.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:26 smithi137 ceph-mon[29634]: osdmap e193: 8 total, 8 up, 4 in 2024-09-21T08:26:27.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:26 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:27.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:26 smithi137 ceph-mon[41044]: from='client.45023 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:26:27.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:26 smithi137 ceph-mon[41044]: osdmap e192: 8 total, 8 up, 4 in 2024-09-21T08:26:27.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:26 smithi137 ceph-mon[41044]: osdmap e193: 8 total, 8 up, 4 in 2024-09-21T08:26:27.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:27.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:27.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:27.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:27.137 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:27.137 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:27.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:27.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:27.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:27.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:27.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:27.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:27.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:27.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:27.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:27.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:26:27.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:27.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:27.517 INFO:tasks.rados.rados.0.smithi137.stdout:621: done (0 left) 2024-09-21T08:26:27.517 INFO:tasks.rados.rados.0.smithi137.stdout:622: copy_from oid 1 from oid 20 current snap is 55 2024-09-21T08:26:27.517 INFO:tasks.rados.rados.0.smithi137.stdout:623: read oid 48 snap 54 2024-09-21T08:26:27.517 INFO:tasks.rados.rados.0.smithi137.stdout:623: expect (ObjNum 241 snap 50 seq_num 241) 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:624: copy_from oid 45 from oid 30 current snap is 55 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:625: rollback oid 14 current snap is 55 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 14 to 53 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:626: copy_from oid 27 from oid 19 current snap is 55 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:627: read oid 2 snap -1 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:627: expect (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:628: copy_from oid 24 from oid 17 current snap is 55 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:629: rollback oid 5 current snap is 55 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 5 to 53 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:630: write oid 8 current snap is 55 2024-09-21T08:26:27.518 INFO:tasks.rados.rados.0.smithi137.stdout:630: seq_num 254 ranges {668978=552895,1763838=505140} 2024-09-21T08:26:27.524 INFO:tasks.rados.rados.0.smithi137.stdout:630: writing smithi13779221-8 from 668978 to 1221873 tid 1 2024-09-21T08:26:27.531 INFO:tasks.rados.rados.0.smithi137.stdout:630: writing smithi13779221-8 from 1763838 to 2268978 tid 2 2024-09-21T08:26:27.531 INFO:tasks.rados.rados.0.smithi137.stdout:631: read oid 41 snap -1 2024-09-21T08:26:27.532 INFO:tasks.rados.rados.0.smithi137.stdout:631: expect (ObjNum 232 snap 47 seq_num 232) 2024-09-21T08:26:27.532 INFO:tasks.rados.rados.0.smithi137.stdout:632: write oid 13 current snap is 55 2024-09-21T08:26:27.532 INFO:tasks.rados.rados.0.smithi137.stdout:632: seq_num 255 ranges {685785=628144,1822320=463465} 2024-09-21T08:26:27.540 INFO:tasks.rados.rados.0.smithi137.stdout:632: writing smithi13779221-13 from 685785 to 1313929 tid 1 2024-09-21T08:26:27.545 INFO:tasks.rados.rados.0.smithi137.stdout:632: writing smithi13779221-13 from 1822320 to 2285785 tid 2 2024-09-21T08:26:27.546 INFO:tasks.rados.rados.0.smithi137.stdout:625: finishing rollback tid 1 to smithi13779221-14 2024-09-21T08:26:27.546 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 0 (ObjNum 4113 snap 0 seq_num 4294967295) dirty dne 2024-09-21T08:26:27.546 INFO:tasks.rados.rados.0.smithi137.stdout:623: expect (ObjNum 241 snap 50 seq_num 241) 2024-09-21T08:26:27.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:27.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:27.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:27.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:27.664 INFO:tasks.rados.rados.0.smithi137.stdout:629: finishing rollback tid 0 to smithi13779221-5 2024-09-21T08:26:27.664 INFO:tasks.rados.rados.0.smithi137.stdout:629: finishing rollback tid 1 to smithi13779221-5 2024-09-21T08:26:27.664 INFO:tasks.rados.rados.0.smithi137.stdout:628: finishing copy_from racing read to smithi13779221-24 2024-09-21T08:26:27.664 INFO:tasks.rados.rados.0.smithi137.stdout:627: expect (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:26:27.671 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":195,"stamp":"2024-09-21T08:26:26.506855+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":134916914,"num_objects":246,"num_object_clones":6,"num_object_copies":738,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":246,"num_whiteouts":2,"num_read":2273,"num_read_kb":1111211,"num_write":1645,"num_write_kb":388441,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":535,"num_bytes_recovered":181986796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3568,"ondisk_log_size":3568,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":468020,"kb_used_data":316988,"kb_used_omap":6,"kb_used_meta":145465,"kb_avail":374512588,"statfs":{"total":383980142592,"available":383500890112,"internally_reserved":0,"allocated":324595712,"data_stored":340052461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6375,"internal_metadata":148956953},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-20700765,"num_objects":-7,"num_object_clones":-7,"num_object_copies":-21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-7,"num_whiteouts":1,"num_read":30,"num_read_kb":20003,"num_write":22,"num_write_kb":6108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.995260"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":269,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527802+0000","last_change":"2024-09-21T08:25:38.378517+0000","last_active":"2024-09-21T08:26:24.527802+0000","last_peered":"2024-09-21T08:26:24.527802+0000","last_clean":"2024-09-21T08:26:24.527802+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:26:24.527802+0000","last_undegraded":"2024-09-21T08:26:24.527802+0000","last_fullsized":"2024-09-21T08:26:24.527802+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:52.783162+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":168,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526256+0000","last_change":"2024-09-21T08:25:48.982638+0000","last_active":"2024-09-21T08:26:24.526256+0000","last_peered":"2024-09-21T08:26:24.526256+0000","last_clean":"2024-09-21T08:26:24.526256+0000","last_became_active":"2024-09-21T08:25:48.910812+0000","last_became_peered":"2024-09-21T08:25:48.910812+0000","last_unstale":"2024-09-21T08:26:24.526256+0000","last_undegraded":"2024-09-21T08:26:24.526256+0000","last_fullsized":"2024-09-21T08:26:24.526256+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:24:50.572709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.572709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215939+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":84,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525041+0000","last_change":"2024-09-21T08:25:48.912078+0000","last_active":"2024-09-21T08:26:24.525041+0000","last_peered":"2024-09-21T08:26:24.525041+0000","last_clean":"2024-09-21T08:26:24.525041+0000","last_became_active":"2024-09-21T08:25:48.911674+0000","last_became_peered":"2024-09-21T08:25:48.911674+0000","last_unstale":"2024-09-21T08:26:24.525041+0000","last_undegraded":"2024-09-21T08:26:24.525041+0000","last_fullsized":"2024-09-21T08:26:24.525041+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808630+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":231,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526363+0000","last_change":"2024-09-21T08:25:48.914509+0000","last_active":"2024-09-21T08:26:24.526363+0000","last_peered":"2024-09-21T08:26:24.526363+0000","last_clean":"2024-09-21T08:26:24.526363+0000","last_became_active":"2024-09-21T08:25:48.911853+0000","last_became_peered":"2024-09-21T08:25:48.911853+0000","last_unstale":"2024-09-21T08:26:24.526363+0000","last_undegraded":"2024-09-21T08:26:24.526363+0000","last_fullsized":"2024-09-21T08:26:24.526363+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808675+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":214,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525751+0000","last_change":"2024-09-21T08:25:48.900959+0000","last_active":"2024-09-21T08:26:24.525751+0000","last_peered":"2024-09-21T08:26:24.525751+0000","last_clean":"2024-09-21T08:26:24.525751+0000","last_became_active":"2024-09-21T08:25:48.900647+0000","last_became_peered":"2024-09-21T08:25:48.900647+0000","last_unstale":"2024-09-21T08:26:24.525751+0000","last_undegraded":"2024-09-21T08:26:24.525751+0000","last_fullsized":"2024-09-21T08:26:24.525751+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":205,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526577+0000","last_change":"2024-09-21T08:25:48.884959+0000","last_active":"2024-09-21T08:26:24.526577+0000","last_peered":"2024-09-21T08:26:24.526577+0000","last_clean":"2024-09-21T08:26:24.526577+0000","last_became_active":"2024-09-21T08:25:48.884673+0000","last_became_peered":"2024-09-21T08:25:48.884673+0000","last_unstale":"2024-09-21T08:26:24.526577+0000","last_undegraded":"2024-09-21T08:26:24.526577+0000","last_fullsized":"2024-09-21T08:26:24.526577+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:19.906906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:19.906906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:28.227332+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":272,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525166+0000","last_change":"2024-09-21T08:25:20.942089+0000","last_active":"2024-09-21T08:26:24.525166+0000","last_peered":"2024-09-21T08:26:24.525166+0000","last_clean":"2024-09-21T08:26:24.525166+0000","last_became_active":"2024-09-21T08:24:48.594855+0000","last_became_peered":"2024-09-21T08:24:48.594855+0000","last_unstale":"2024-09-21T08:26:24.525166+0000","last_undegraded":"2024-09-21T08:26:24.525166+0000","last_fullsized":"2024-09-21T08:26:24.525166+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:25:20.941990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:20.941990+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:29.954407+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":89,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525298+0000","last_change":"2024-09-21T08:25:48.912528+0000","last_active":"2024-09-21T08:26:24.525298+0000","last_peered":"2024-09-21T08:26:24.525298+0000","last_clean":"2024-09-21T08:26:24.525298+0000","last_became_active":"2024-09-21T08:25:48.911583+0000","last_became_peered":"2024-09-21T08:25:48.911583+0000","last_unstale":"2024-09-21T08:26:24.525298+0000","last_undegraded":"2024-09-21T08:26:24.525298+0000","last_fullsized":"2024-09-21T08:26:24.525298+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.640831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.640831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"180'167","reported_seq":801,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525373+0000","last_change":"2024-09-21T08:26:24.527114+0000","last_active":"2024-09-21T08:26:24.525373+0000","last_peered":"2024-09-21T08:26:24.525373+0000","last_clean":"2024-09-21T08:26:24.525373+0000","last_became_active":"2024-09-21T08:26:20.935479+0000","last_became_peered":"2024-09-21T08:26:20.935479+0000","last_unstale":"2024-09-21T08:26:24.525373+0000","last_undegraded":"2024-09-21T08:26:24.525373+0000","last_fullsized":"2024-09-21T08:26:24.525373+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":167,"log_dups_size":0,"ondisk_log_size":167,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0019019989999999999,"stat_sum":{"num_bytes":6579924,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":67978,"num_write":47,"num_write_kb":21497,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":10613390,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"11"}]},{"pgid":"3.1d","version":"0'0","reported_seq":280,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528260+0000","last_change":"2024-09-21T08:25:48.907786+0000","last_active":"2024-09-21T08:26:24.528260+0000","last_peered":"2024-09-21T08:26:24.528260+0000","last_clean":"2024-09-21T08:26:24.528260+0000","last_became_active":"2024-09-21T08:25:48.906922+0000","last_became_peered":"2024-09-21T08:25:48.906922+0000","last_unstale":"2024-09-21T08:26:24.528260+0000","last_undegraded":"2024-09-21T08:26:24.528260+0000","last_fullsized":"2024-09-21T08:26:24.528260+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808477+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":264,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526545+0000","last_change":"2024-09-21T08:24:56.523774+0000","last_active":"2024-09-21T08:26:24.526545+0000","last_peered":"2024-09-21T08:26:24.526545+0000","last_clean":"2024-09-21T08:26:24.526545+0000","last_became_active":"2024-09-21T08:24:56.500171+0000","last_became_peered":"2024-09-21T08:24:56.500171+0000","last_unstale":"2024-09-21T08:26:24.526545+0000","last_undegraded":"2024-09-21T08:26:24.526545+0000","last_fullsized":"2024-09-21T08:26:24.526545+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":262,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527848+0000","last_change":"2024-09-21T08:25:22.581281+0000","last_active":"2024-09-21T08:26:24.527848+0000","last_peered":"2024-09-21T08:26:24.527848+0000","last_clean":"2024-09-21T08:26:24.527848+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:26:24.527848+0000","last_undegraded":"2024-09-21T08:26:24.527848+0000","last_fullsized":"2024-09-21T08:26:24.527848+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:33.300785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":255,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526478+0000","last_change":"2024-09-21T08:25:48.885985+0000","last_active":"2024-09-21T08:26:24.526478+0000","last_peered":"2024-09-21T08:26:24.526478+0000","last_clean":"2024-09-21T08:26:24.526478+0000","last_became_active":"2024-09-21T08:25:48.885136+0000","last_became_peered":"2024-09-21T08:25:48.885136+0000","last_unstale":"2024-09-21T08:26:24.526478+0000","last_undegraded":"2024-09-21T08:26:24.526478+0000","last_fullsized":"2024-09-21T08:26:24.526478+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434627+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"190'170","reported_seq":754,"reported_epoch":191,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:26:24.526203+0000","last_change":"2024-09-21T08:26:24.524682+0000","last_active":"2024-09-21T08:26:24.526203+0000","last_peered":"2024-09-21T08:26:24.526203+0000","last_clean":"2024-09-21T08:26:24.526203+0000","last_became_active":"2024-09-21T08:26:20.935965+0000","last_became_peered":"2024-09-21T08:26:20.935965+0000","last_unstale":"2024-09-21T08:26:24.526203+0000","last_undegraded":"2024-09-21T08:26:24.526203+0000","last_fullsized":"2024-09-21T08:26:24.526203+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42670619900000001,"stat_sum":{"num_bytes":6537923,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":48,"num_read_kb":42337,"num_write":51,"num_write_kb":22768,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":14891822,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"d"}]},{"pgid":"3.1c","version":"0'0","reported_seq":288,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526589+0000","last_change":"2024-09-21T08:25:48.912437+0000","last_active":"2024-09-21T08:26:24.526589+0000","last_peered":"2024-09-21T08:26:24.526589+0000","last_clean":"2024-09-21T08:26:24.526589+0000","last_became_active":"2024-09-21T08:25:48.912043+0000","last_became_peered":"2024-09-21T08:25:48.912043+0000","last_unstale":"2024-09-21T08:26:24.526589+0000","last_undegraded":"2024-09-21T08:26:24.526589+0000","last_fullsized":"2024-09-21T08:26:24.526589+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808889+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":121,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525528+0000","last_change":"2024-09-21T08:25:48.937429+0000","last_active":"2024-09-21T08:26:24.525528+0000","last_peered":"2024-09-21T08:26:24.525528+0000","last_clean":"2024-09-21T08:26:24.525528+0000","last_became_active":"2024-09-21T08:25:48.907461+0000","last_became_peered":"2024-09-21T08:25:48.907461+0000","last_unstale":"2024-09-21T08:26:24.525528+0000","last_undegraded":"2024-09-21T08:26:24.525528+0000","last_fullsized":"2024-09-21T08:26:24.525528+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:24:32.569846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:32.569846+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526855+0000","last_change":"2024-09-21T08:26:09.782427+0000","last_active":"2024-09-21T08:26:24.526855+0000","last_peered":"2024-09-21T08:26:24.526855+0000","last_clean":"2024-09-21T08:26:24.526855+0000","last_became_active":"2024-09-21T08:25:48.885803+0000","last_became_peered":"2024-09-21T08:25:48.885803+0000","last_unstale":"2024-09-21T08:26:24.526855+0000","last_undegraded":"2024-09-21T08:26:24.526855+0000","last_fullsized":"2024-09-21T08:26:24.526855+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_clean_scrub_stamp":"2024-09-21T08:26:09.782307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:16.118388+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525053+0000","last_change":"2024-09-21T08:25:48.966469+0000","last_active":"2024-09-21T08:26:24.525053+0000","last_peered":"2024-09-21T08:26:24.525053+0000","last_clean":"2024-09-21T08:26:24.525053+0000","last_became_active":"2024-09-21T08:25:48.911981+0000","last_became_peered":"2024-09-21T08:25:48.911981+0000","last_unstale":"2024-09-21T08:26:24.525053+0000","last_undegraded":"2024-09-21T08:26:24.525053+0000","last_fullsized":"2024-09-21T08:26:24.525053+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":226,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527239+0000","last_change":"2024-09-21T08:25:39.855515+0000","last_active":"2024-09-21T08:26:24.527239+0000","last_peered":"2024-09-21T08:26:24.527239+0000","last_clean":"2024-09-21T08:26:24.527239+0000","last_became_active":"2024-09-21T08:24:56.499884+0000","last_became_peered":"2024-09-21T08:24:56.499884+0000","last_unstale":"2024-09-21T08:26:24.527239+0000","last_undegraded":"2024-09-21T08:26:24.527239+0000","last_fullsized":"2024-09-21T08:26:24.527239+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.855448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.855448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:04.367456+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":281,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527803+0000","last_change":"2024-09-21T08:25:48.891123+0000","last_active":"2024-09-21T08:26:24.527803+0000","last_peered":"2024-09-21T08:26:24.527803+0000","last_clean":"2024-09-21T08:26:24.527803+0000","last_became_active":"2024-09-21T08:25:48.890600+0000","last_became_peered":"2024-09-21T08:25:48.890600+0000","last_unstale":"2024-09-21T08:26:24.527803+0000","last_undegraded":"2024-09-21T08:26:24.527803+0000","last_fullsized":"2024-09-21T08:26:24.527803+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.435179+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":334,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529162+0000","last_change":"2024-09-21T08:25:48.949361+0000","last_active":"2024-09-21T08:26:24.529162+0000","last_peered":"2024-09-21T08:26:24.529162+0000","last_clean":"2024-09-21T08:26:24.529162+0000","last_became_active":"2024-09-21T08:25:48.909222+0000","last_became_peered":"2024-09-21T08:25:48.909222+0000","last_unstale":"2024-09-21T08:26:24.529162+0000","last_undegraded":"2024-09-21T08:26:24.529162+0000","last_fullsized":"2024-09-21T08:26:24.529162+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:25:24.945807+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:24.945807+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:43.015505+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":121,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527323+0000","last_change":"2024-09-21T08:25:48.905281+0000","last_active":"2024-09-21T08:26:24.527323+0000","last_peered":"2024-09-21T08:26:24.527323+0000","last_clean":"2024-09-21T08:26:24.527323+0000","last_became_active":"2024-09-21T08:25:48.904956+0000","last_became_peered":"2024-09-21T08:25:48.904956+0000","last_unstale":"2024-09-21T08:26:24.527323+0000","last_undegraded":"2024-09-21T08:26:24.527323+0000","last_fullsized":"2024-09-21T08:26:24.527323+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434487+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":196,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527659+0000","last_change":"2024-09-21T08:25:48.914865+0000","last_active":"2024-09-21T08:26:24.527659+0000","last_peered":"2024-09-21T08:26:24.527659+0000","last_clean":"2024-09-21T08:26:24.527659+0000","last_became_active":"2024-09-21T08:25:48.912721+0000","last_became_peered":"2024-09-21T08:25:48.912721+0000","last_unstale":"2024-09-21T08:26:24.527659+0000","last_undegraded":"2024-09-21T08:26:24.527659+0000","last_fullsized":"2024-09-21T08:26:24.527659+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808543+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":86,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529683+0000","last_change":"2024-09-21T08:25:48.900151+0000","last_active":"2024-09-21T08:26:24.529683+0000","last_peered":"2024-09-21T08:26:24.529683+0000","last_clean":"2024-09-21T08:26:24.529683+0000","last_became_active":"2024-09-21T08:25:48.899796+0000","last_became_peered":"2024-09-21T08:25:48.899796+0000","last_unstale":"2024-09-21T08:26:24.529683+0000","last_undegraded":"2024-09-21T08:26:24.529683+0000","last_fullsized":"2024-09-21T08:26:24.529683+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":140,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528549+0000","last_change":"2024-09-21T08:25:48.964877+0000","last_active":"2024-09-21T08:26:24.528549+0000","last_peered":"2024-09-21T08:26:24.528549+0000","last_clean":"2024-09-21T08:26:24.528549+0000","last_became_active":"2024-09-21T08:25:48.913458+0000","last_became_peered":"2024-09-21T08:25:48.913458+0000","last_unstale":"2024-09-21T08:26:24.528549+0000","last_undegraded":"2024-09-21T08:26:24.528549+0000","last_fullsized":"2024-09-21T08:26:24.528549+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:21.424414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.424414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":220,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529514+0000","last_change":"2024-09-21T08:25:48.892022+0000","last_active":"2024-09-21T08:26:24.529514+0000","last_peered":"2024-09-21T08:26:24.529514+0000","last_clean":"2024-09-21T08:26:24.529514+0000","last_became_active":"2024-09-21T08:25:48.891706+0000","last_became_peered":"2024-09-21T08:25:48.891706+0000","last_unstale":"2024-09-21T08:26:24.529514+0000","last_undegraded":"2024-09-21T08:26:24.529514+0000","last_fullsized":"2024-09-21T08:26:24.529514+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.797084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:28.227318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":248,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527694+0000","last_change":"2024-09-21T08:23:36.584003+0000","last_active":"2024-09-21T08:26:24.527694+0000","last_peered":"2024-09-21T08:26:24.527694+0000","last_clean":"2024-09-21T08:26:24.527694+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:26:24.527694+0000","last_undegraded":"2024-09-21T08:26:24.527694+0000","last_fullsized":"2024-09-21T08:26:24.527694+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778253+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"66'1","reported_seq":163,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526692+0000","last_change":"2024-09-21T08:25:48.971328+0000","last_active":"2024-09-21T08:26:24.526692+0000","last_peered":"2024-09-21T08:26:24.526692+0000","last_clean":"2024-09-21T08:26:24.526692+0000","last_became_active":"2024-09-21T08:25:48.896963+0000","last_became_peered":"2024-09-21T08:25:48.896963+0000","last_unstale":"2024-09-21T08:26:24.526692+0000","last_undegraded":"2024-09-21T08:26:24.526692+0000","last_fullsized":"2024-09-21T08:26:24.526692+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778046+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":308,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528340+0000","last_change":"2024-09-21T08:25:48.989640+0000","last_active":"2024-09-21T08:26:24.528340+0000","last_peered":"2024-09-21T08:26:24.528340+0000","last_clean":"2024-09-21T08:26:24.528340+0000","last_became_active":"2024-09-21T08:25:48.908487+0000","last_became_peered":"2024-09-21T08:25:48.908487+0000","last_unstale":"2024-09-21T08:26:24.528340+0000","last_undegraded":"2024-09-21T08:26:24.528340+0000","last_fullsized":"2024-09-21T08:26:24.528340+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"156'12","reported_seq":331,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529186+0000","last_change":"2024-09-21T08:25:48.961434+0000","last_active":"2024-09-21T08:26:24.529186+0000","last_peered":"2024-09-21T08:26:24.529186+0000","last_clean":"2024-09-21T08:26:24.529186+0000","last_became_active":"2024-09-21T08:25:48.896649+0000","last_became_peered":"2024-09-21T08:25:48.896649+0000","last_unstale":"2024-09-21T08:26:24.529186+0000","last_undegraded":"2024-09-21T08:26:24.529186+0000","last_fullsized":"2024-09-21T08:26:24.529186+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.594212+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:29.875977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"191'111","reported_seq":685,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.548716+0000","last_change":"2024-09-21T08:26:24.531442+0000","last_active":"2024-09-21T08:26:24.548716+0000","last_peered":"2024-09-21T08:26:24.548716+0000","last_clean":"2024-09-21T08:26:24.548716+0000","last_became_active":"2024-09-21T08:24:56.503681+0000","last_became_peered":"2024-09-21T08:24:56.503681+0000","last_unstale":"2024-09-21T08:26:24.548716+0000","last_undegraded":"2024-09-21T08:26:24.548716+0000","last_fullsized":"2024-09-21T08:26:24.548716+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0040995980000000003,"stat_sum":{"num_bytes":9714446,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":101,"num_read_kb":107229,"num_write":56,"num_write_kb":29962,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":12634927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"11"}]},{"pgid":"4.f","version":"73'15","reported_seq":165,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528932+0000","last_change":"2024-09-21T08:25:48.935201+0000","last_active":"2024-09-21T08:26:24.528932+0000","last_peered":"2024-09-21T08:26:24.528932+0000","last_clean":"2024-09-21T08:26:24.528932+0000","last_became_active":"2024-09-21T08:25:48.894852+0000","last_became_peered":"2024-09-21T08:25:48.894852+0000","last_unstale":"2024-09-21T08:26:24.528932+0000","last_undegraded":"2024-09-21T08:26:24.528932+0000","last_fullsized":"2024-09-21T08:26:24.528932+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778079+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":86,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527783+0000","last_change":"2024-09-21T08:25:48.897595+0000","last_active":"2024-09-21T08:26:24.527783+0000","last_peered":"2024-09-21T08:26:24.527783+0000","last_clean":"2024-09-21T08:26:24.527783+0000","last_became_active":"2024-09-21T08:25:48.897249+0000","last_became_peered":"2024-09-21T08:25:48.897249+0000","last_unstale":"2024-09-21T08:26:24.527783+0000","last_undegraded":"2024-09-21T08:26:24.527783+0000","last_fullsized":"2024-09-21T08:26:24.527783+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:45.434055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:45.434055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:59.844465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"85'9","reported_seq":284,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527181+0000","last_change":"2024-09-21T08:25:21.631370+0000","last_active":"2024-09-21T08:26:24.527181+0000","last_peered":"2024-09-21T08:26:24.527181+0000","last_clean":"2024-09-21T08:26:24.527181+0000","last_became_active":"2024-09-21T08:24:38.548789+0000","last_became_peered":"2024-09-21T08:24:38.548789+0000","last_unstale":"2024-09-21T08:26:24.527181+0000","last_undegraded":"2024-09-21T08:26:24.527181+0000","last_fullsized":"2024-09-21T08:26:24.527181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:25:21.631254+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.631254+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:44.705485+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":264,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528447+0000","last_change":"2024-09-21T08:26:08.864503+0000","last_active":"2024-09-21T08:26:24.528447+0000","last_peered":"2024-09-21T08:26:24.528447+0000","last_clean":"2024-09-21T08:26:24.528447+0000","last_became_active":"2024-09-21T08:25:48.914453+0000","last_became_peered":"2024-09-21T08:25:48.914453+0000","last_unstale":"2024-09-21T08:26:24.528447+0000","last_undegraded":"2024-09-21T08:26:24.528447+0000","last_fullsized":"2024-09-21T08:26:24.528447+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:08.864399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:08.864399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:29.676635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"179'19","reported_seq":570,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.529176+0000","last_change":"2024-09-21T08:26:23.525412+0000","last_active":"2024-09-21T08:26:24.529176+0000","last_peered":"2024-09-21T08:26:24.529176+0000","last_clean":"2024-09-21T08:26:24.529176+0000","last_became_active":"2024-09-21T08:26:15.917324+0000","last_became_peered":"2024-09-21T08:26:15.917324+0000","last_unstale":"2024-09-21T08:26:24.529176+0000","last_undegraded":"2024-09-21T08:26:24.529176+0000","last_fullsized":"2024-09-21T08:26:24.529176+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42164727499999999,"stat_sum":{"num_bytes":1512630,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":5325,"num_write":6,"num_write_kb":1263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2565420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25","length":"f"}]},{"pgid":"4.0","version":"73'18","reported_seq":295,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527870+0000","last_change":"2024-09-21T08:25:48.942497+0000","last_active":"2024-09-21T08:26:24.527870+0000","last_peered":"2024-09-21T08:26:24.527870+0000","last_clean":"2024-09-21T08:26:24.527870+0000","last_became_active":"2024-09-21T08:25:48.895885+0000","last_became_peered":"2024-09-21T08:25:48.895885+0000","last_unstale":"2024-09-21T08:26:24.527870+0000","last_undegraded":"2024-09-21T08:26:24.527870+0000","last_fullsized":"2024-09-21T08:26:24.527870+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778173+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":211,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529319+0000","last_change":"2024-09-21T08:24:50.648706+0000","last_active":"2024-09-21T08:26:24.529319+0000","last_peered":"2024-09-21T08:26:24.529319+0000","last_clean":"2024-09-21T08:26:24.529319+0000","last_became_active":"2024-09-21T08:24:48.603740+0000","last_became_peered":"2024-09-21T08:24:48.603740+0000","last_unstale":"2024-09-21T08:26:24.529319+0000","last_undegraded":"2024-09-21T08:26:24.529319+0000","last_fullsized":"2024-09-21T08:26:24.529319+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778415+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":204,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526515+0000","last_change":"2024-09-21T08:24:48.604125+0000","last_active":"2024-09-21T08:26:24.526515+0000","last_peered":"2024-09-21T08:26:24.526515+0000","last_clean":"2024-09-21T08:26:24.526515+0000","last_became_active":"2024-09-21T08:24:48.603825+0000","last_became_peered":"2024-09-21T08:24:48.603825+0000","last_unstale":"2024-09-21T08:26:24.526515+0000","last_undegraded":"2024-09-21T08:26:24.526515+0000","last_fullsized":"2024-09-21T08:26:24.526515+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":213,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529955+0000","last_change":"2024-09-21T08:24:56.501652+0000","last_active":"2024-09-21T08:26:24.529955+0000","last_peered":"2024-09-21T08:26:24.529955+0000","last_clean":"2024-09-21T08:26:24.529955+0000","last_became_active":"2024-09-21T08:24:56.501237+0000","last_became_peered":"2024-09-21T08:24:56.501237+0000","last_unstale":"2024-09-21T08:26:24.529955+0000","last_undegraded":"2024-09-21T08:26:24.529955+0000","last_fullsized":"2024-09-21T08:26:24.529955+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"191'193","reported_seq":823,"reported_epoch":191,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:26:24.943083+0000","last_change":"2024-09-21T08:26:24.943083+0000","last_active":"2024-09-21T08:26:24.943083+0000","last_peered":"2024-09-21T08:26:24.943083+0000","last_clean":"2024-09-21T08:26:24.943083+0000","last_became_active":"2024-09-21T08:26:15.917361+0000","last_became_peered":"2024-09-21T08:26:15.917361+0000","last_unstale":"2024-09-21T08:26:24.943083+0000","last_undegraded":"2024-09-21T08:26:24.943083+0000","last_fullsized":"2024-09-21T08:26:24.943083+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.42002327099999998,"stat_sum":{"num_bytes":5311045,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":62,"num_read_kb":61320,"num_write":62,"num_write_kb":21608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":11369847,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25","length":"d"},{"start":"33","length":"1"}]},{"pgid":"4.1","version":"73'14","reported_seq":302,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.524910+0000","last_change":"2024-09-21T08:24:56.561414+0000","last_active":"2024-09-21T08:26:24.524910+0000","last_peered":"2024-09-21T08:26:24.524910+0000","last_clean":"2024-09-21T08:26:24.524910+0000","last_became_active":"2024-09-21T08:24:56.509169+0000","last_became_peered":"2024-09-21T08:24:56.509169+0000","last_unstale":"2024-09-21T08:26:24.524910+0000","last_undegraded":"2024-09-21T08:26:24.524910+0000","last_fullsized":"2024-09-21T08:26:24.524910+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808527+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":290,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527678+0000","last_change":"2024-09-21T08:25:48.919957+0000","last_active":"2024-09-21T08:26:24.527678+0000","last_peered":"2024-09-21T08:26:24.527678+0000","last_clean":"2024-09-21T08:26:24.527678+0000","last_became_active":"2024-09-21T08:25:48.893688+0000","last_became_peered":"2024-09-21T08:25:48.893688+0000","last_unstale":"2024-09-21T08:26:24.527678+0000","last_undegraded":"2024-09-21T08:26:24.527678+0000","last_fullsized":"2024-09-21T08:26:24.527678+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"156'11","reported_seq":156,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526633+0000","last_change":"2024-09-21T08:25:48.973195+0000","last_active":"2024-09-21T08:26:24.526633+0000","last_peered":"2024-09-21T08:26:24.526633+0000","last_clean":"2024-09-21T08:26:24.526633+0000","last_became_active":"2024-09-21T08:25:48.906808+0000","last_became_peered":"2024-09-21T08:25:48.906808+0000","last_unstale":"2024-09-21T08:26:24.526633+0000","last_undegraded":"2024-09-21T08:26:24.526633+0000","last_fullsized":"2024-09-21T08:26:24.526633+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434915+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":267,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525988+0000","last_change":"2024-09-21T08:25:48.900309+0000","last_active":"2024-09-21T08:26:24.525988+0000","last_peered":"2024-09-21T08:26:24.525988+0000","last_clean":"2024-09-21T08:26:24.525988+0000","last_became_active":"2024-09-21T08:25:48.899971+0000","last_became_peered":"2024-09-21T08:25:48.899971+0000","last_unstale":"2024-09-21T08:26:24.525988+0000","last_undegraded":"2024-09-21T08:26:24.525988+0000","last_fullsized":"2024-09-21T08:26:24.525988+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:21.561826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.561826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:44.050410+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"179'43","reported_seq":544,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.949230+0000","last_change":"2024-09-21T08:26:24.949273+0000","last_active":"2024-09-21T08:26:24.949230+0000","last_peered":"2024-09-21T08:26:24.949230+0000","last_clean":"2024-09-21T08:26:24.949230+0000","last_became_active":"2024-09-21T08:26:15.915554+0000","last_became_peered":"2024-09-21T08:26:15.915554+0000","last_unstale":"2024-09-21T08:26:24.949230+0000","last_undegraded":"2024-09-21T08:26:24.949230+0000","last_fullsized":"2024-09-21T08:26:24.949230+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.422254042,"stat_sum":{"num_bytes":1605068,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":21688,"num_write":9,"num_write_kb":3160,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2823179,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"10"}]},{"pgid":"4.2","version":"73'10","reported_seq":138,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526089+0000","last_change":"2024-09-21T08:26:05.809673+0000","last_active":"2024-09-21T08:26:24.526089+0000","last_peered":"2024-09-21T08:26:24.526089+0000","last_clean":"2024-09-21T08:26:24.526089+0000","last_became_active":"2024-09-21T08:25:48.894952+0000","last_became_peered":"2024-09-21T08:25:48.894952+0000","last_unstale":"2024-09-21T08:26:24.526089+0000","last_undegraded":"2024-09-21T08:26:24.526089+0000","last_fullsized":"2024-09-21T08:26:24.526089+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:26:05.809597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.809597+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:07.999081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":288,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528084+0000","last_change":"2024-09-21T08:24:56.519880+0000","last_active":"2024-09-21T08:26:24.528084+0000","last_peered":"2024-09-21T08:26:24.528084+0000","last_clean":"2024-09-21T08:26:24.528084+0000","last_became_active":"2024-09-21T08:24:56.500882+0000","last_became_peered":"2024-09-21T08:24:56.500882+0000","last_unstale":"2024-09-21T08:26:24.528084+0000","last_undegraded":"2024-09-21T08:26:24.528084+0000","last_fullsized":"2024-09-21T08:26:24.528084+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"176'11","reported_seq":326,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528825+0000","last_change":"2024-09-21T08:26:10.440643+0000","last_active":"2024-09-21T08:26:24.528825+0000","last_peered":"2024-09-21T08:26:24.528825+0000","last_clean":"2024-09-21T08:26:24.528825+0000","last_became_active":"2024-09-21T08:25:48.902651+0000","last_became_peered":"2024-09-21T08:25:48.902651+0000","last_unstale":"2024-09-21T08:26:24.528825+0000","last_undegraded":"2024-09-21T08:26:24.528825+0000","last_fullsized":"2024-09-21T08:26:24.528825+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'11","last_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_deep_scrub":"176'11","last_deep_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_clean_scrub_stamp":"2024-09-21T08:26:10.440547+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:15.364032+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":255,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526357+0000","last_change":"2024-09-21T08:24:48.597514+0000","last_active":"2024-09-21T08:26:24.526357+0000","last_peered":"2024-09-21T08:26:24.526357+0000","last_clean":"2024-09-21T08:26:24.526357+0000","last_became_active":"2024-09-21T08:24:48.597048+0000","last_became_peered":"2024-09-21T08:24:48.597048+0000","last_unstale":"2024-09-21T08:26:24.526357+0000","last_undegraded":"2024-09-21T08:26:24.526357+0000","last_fullsized":"2024-09-21T08:26:24.526357+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.435052+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"189'100","reported_seq":708,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.940307+0000","last_change":"2024-09-21T08:26:24.940563+0000","last_active":"2024-09-21T08:26:24.940307+0000","last_peered":"2024-09-21T08:26:24.940307+0000","last_clean":"2024-09-21T08:26:24.940307+0000","last_became_active":"2024-09-21T08:26:15.913945+0000","last_became_peered":"2024-09-21T08:26:15.913945+0000","last_unstale":"2024-09-21T08:26:24.940307+0000","last_undegraded":"2024-09-21T08:26:24.940307+0000","last_fullsized":"2024-09-21T08:26:24.940307+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":100,"log_dups_size":0,"ondisk_log_size":100,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:29.811097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.414697172,"stat_sum":{"num_bytes":1815156,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":24249,"num_write":31,"num_write_kb":12243,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":4071613,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"25","length":"11"}]},{"pgid":"4.3","version":"73'19","reported_seq":293,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528599+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:26:24.528599+0000","last_peered":"2024-09-21T08:26:24.528599+0000","last_clean":"2024-09-21T08:26:24.528599+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:26:24.528599+0000","last_undegraded":"2024-09-21T08:26:24.528599+0000","last_fullsized":"2024-09-21T08:26:24.528599+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":85,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529164+0000","last_change":"2024-09-21T08:25:48.906191+0000","last_active":"2024-09-21T08:26:24.529164+0000","last_peered":"2024-09-21T08:26:24.529164+0000","last_clean":"2024-09-21T08:26:24.529164+0000","last_became_active":"2024-09-21T08:25:48.905818+0000","last_became_peered":"2024-09-21T08:25:48.905818+0000","last_unstale":"2024-09-21T08:26:24.529164+0000","last_undegraded":"2024-09-21T08:26:24.529164+0000","last_fullsized":"2024-09-21T08:26:24.529164+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.529672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.529672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":195,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526636+0000","last_change":"2024-09-21T08:25:39.339196+0000","last_active":"2024-09-21T08:26:24.526636+0000","last_peered":"2024-09-21T08:26:24.526636+0000","last_clean":"2024-09-21T08:26:24.526636+0000","last_became_active":"2024-09-21T08:24:56.499628+0000","last_became_peered":"2024-09-21T08:24:56.499628+0000","last_unstale":"2024-09-21T08:26:24.526636+0000","last_undegraded":"2024-09-21T08:26:24.526636+0000","last_fullsized":"2024-09-21T08:26:24.526636+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.339126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.339126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:57.963164+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":153,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525375+0000","last_change":"2024-09-21T08:25:48.885420+0000","last_active":"2024-09-21T08:26:24.525375+0000","last_peered":"2024-09-21T08:26:24.525375+0000","last_clean":"2024-09-21T08:26:24.525375+0000","last_became_active":"2024-09-21T08:25:48.885120+0000","last_became_peered":"2024-09-21T08:25:48.885120+0000","last_unstale":"2024-09-21T08:26:24.525375+0000","last_undegraded":"2024-09-21T08:26:24.525375+0000","last_fullsized":"2024-09-21T08:26:24.525375+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.425389+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:44.826050+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"191'71","reported_seq":659,"reported_epoch":191,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:26:24.950451+0000","last_change":"2024-09-21T08:26:24.950451+0000","last_active":"2024-09-21T08:26:24.950451+0000","last_peered":"2024-09-21T08:26:24.950451+0000","last_clean":"2024-09-21T08:26:24.950451+0000","last_became_active":"2024-09-21T08:26:15.921432+0000","last_became_peered":"2024-09-21T08:26:15.921432+0000","last_unstale":"2024-09-21T08:26:24.950451+0000","last_undegraded":"2024-09-21T08:26:24.950451+0000","last_fullsized":"2024-09-21T08:26:24.950451+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":71,"log_dups_size":0,"ondisk_log_size":71,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0092620139999999,"stat_sum":{"num_bytes":1873980,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":1,"num_read":13,"num_read_kb":5555,"num_write":23,"num_write_kb":6777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":5482380,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"f"}]},{"pgid":"4.4","version":"156'31","reported_seq":183,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529346+0000","last_change":"2024-09-21T08:25:48.954047+0000","last_active":"2024-09-21T08:26:24.529346+0000","last_peered":"2024-09-21T08:26:24.529346+0000","last_clean":"2024-09-21T08:26:24.529346+0000","last_became_active":"2024-09-21T08:25:48.910276+0000","last_became_peered":"2024-09-21T08:25:48.910276+0000","last_unstale":"2024-09-21T08:26:24.529346+0000","last_undegraded":"2024-09-21T08:26:24.529346+0000","last_fullsized":"2024-09-21T08:26:24.529346+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":358,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":235,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528952+0000","last_change":"2024-09-21T08:25:48.905376+0000","last_active":"2024-09-21T08:26:24.528952+0000","last_peered":"2024-09-21T08:26:24.528952+0000","last_clean":"2024-09-21T08:26:24.528952+0000","last_became_active":"2024-09-21T08:25:48.905056+0000","last_became_peered":"2024-09-21T08:25:48.905056+0000","last_unstale":"2024-09-21T08:26:24.528952+0000","last_undegraded":"2024-09-21T08:26:24.528952+0000","last_fullsized":"2024-09-21T08:26:24.528952+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434942+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":300,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525882+0000","last_change":"2024-09-21T08:26:02.760802+0000","last_active":"2024-09-21T08:26:24.525882+0000","last_peered":"2024-09-21T08:26:24.525882+0000","last_clean":"2024-09-21T08:26:24.525882+0000","last_became_active":"2024-09-21T08:25:48.894174+0000","last_became_peered":"2024-09-21T08:25:48.894174+0000","last_unstale":"2024-09-21T08:26:24.525882+0000","last_undegraded":"2024-09-21T08:26:24.525882+0000","last_fullsized":"2024-09-21T08:26:24.525882+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:21.445845+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":38,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527161+0000","last_change":"2024-09-21T08:25:48.893035+0000","last_active":"2024-09-21T08:26:24.527161+0000","last_peered":"2024-09-21T08:26:24.527161+0000","last_clean":"2024-09-21T08:26:24.527161+0000","last_became_active":"2024-09-21T08:25:48.892728+0000","last_became_peered":"2024-09-21T08:25:48.892728+0000","last_unstale":"2024-09-21T08:26:24.527161+0000","last_undegraded":"2024-09-21T08:26:24.527161+0000","last_fullsized":"2024-09-21T08:26:24.527161+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":256,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529181+0000","last_change":"2024-09-21T08:25:48.897361+0000","last_active":"2024-09-21T08:26:24.529181+0000","last_peered":"2024-09-21T08:26:24.529181+0000","last_clean":"2024-09-21T08:26:24.529181+0000","last_became_active":"2024-09-21T08:25:48.896898+0000","last_became_peered":"2024-09-21T08:25:48.896898+0000","last_unstale":"2024-09-21T08:26:24.529181+0000","last_undegraded":"2024-09-21T08:26:24.529181+0000","last_fullsized":"2024-09-21T08:26:24.529181+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.602376+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:43.388968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"180'50","reported_seq":641,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.948456+0000","last_change":"2024-09-21T08:26:24.948491+0000","last_active":"2024-09-21T08:26:24.948456+0000","last_peered":"2024-09-21T08:26:24.948456+0000","last_clean":"2024-09-21T08:26:24.948456+0000","last_became_active":"2024-09-21T08:26:15.916805+0000","last_became_peered":"2024-09-21T08:26:15.916805+0000","last_unstale":"2024-09-21T08:26:24.948456+0000","last_undegraded":"2024-09-21T08:26:24.948456+0000","last_fullsized":"2024-09-21T08:26:24.948456+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42010332099999997,"stat_sum":{"num_bytes":1915998,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":12,"num_read_kb":9421,"num_write":15,"num_write_kb":4089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2458249,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"10"}]},{"pgid":"4.7","version":"73'13","reported_seq":142,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527093+0000","last_change":"2024-09-21T08:25:48.937399+0000","last_active":"2024-09-21T08:26:24.527093+0000","last_peered":"2024-09-21T08:26:24.527093+0000","last_clean":"2024-09-21T08:26:24.527093+0000","last_became_active":"2024-09-21T08:25:48.908351+0000","last_became_peered":"2024-09-21T08:25:48.908351+0000","last_unstale":"2024-09-21T08:26:24.527093+0000","last_undegraded":"2024-09-21T08:26:24.527093+0000","last_fullsized":"2024-09-21T08:26:24.527093+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":165,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525192+0000","last_change":"2024-09-21T08:25:48.912816+0000","last_active":"2024-09-21T08:26:24.525192+0000","last_peered":"2024-09-21T08:26:24.525192+0000","last_clean":"2024-09-21T08:26:24.525192+0000","last_became_active":"2024-09-21T08:25:48.911245+0000","last_became_peered":"2024-09-21T08:25:48.911245+0000","last_unstale":"2024-09-21T08:26:24.525192+0000","last_undegraded":"2024-09-21T08:26:24.525192+0000","last_fullsized":"2024-09-21T08:26:24.525192+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215606+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":308,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525270+0000","last_change":"2024-09-21T08:25:48.926199+0000","last_active":"2024-09-21T08:26:24.525270+0000","last_peered":"2024-09-21T08:26:24.525270+0000","last_clean":"2024-09-21T08:26:24.525270+0000","last_became_active":"2024-09-21T08:25:48.905148+0000","last_became_peered":"2024-09-21T08:25:48.905148+0000","last_unstale":"2024-09-21T08:26:24.525270+0000","last_undegraded":"2024-09-21T08:26:24.525270+0000","last_fullsized":"2024-09-21T08:26:24.525270+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:50.675673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.675673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215787+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":261,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526478+0000","last_change":"2024-09-21T08:25:42.021037+0000","last_active":"2024-09-21T08:26:24.526478+0000","last_peered":"2024-09-21T08:26:24.526478+0000","last_clean":"2024-09-21T08:26:24.526478+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:26:24.526478+0000","last_undegraded":"2024-09-21T08:26:24.526478+0000","last_fullsized":"2024-09-21T08:26:24.526478+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:42.020928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.020928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:55.206204+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":259,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529604+0000","last_change":"2024-09-21T08:24:56.501148+0000","last_active":"2024-09-21T08:26:24.529604+0000","last_peered":"2024-09-21T08:26:24.529604+0000","last_clean":"2024-09-21T08:26:24.529604+0000","last_became_active":"2024-09-21T08:24:56.498476+0000","last_became_peered":"2024-09-21T08:24:56.498476+0000","last_unstale":"2024-09-21T08:26:24.529604+0000","last_undegraded":"2024-09-21T08:26:24.529604+0000","last_fullsized":"2024-09-21T08:26:24.529604+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"180'245","reported_seq":817,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.530160+0000","last_change":"2024-09-21T08:26:24.530370+0000","last_active":"2024-09-21T08:26:24.530160+0000","last_peered":"2024-09-21T08:26:24.530160+0000","last_clean":"2024-09-21T08:26:24.530160+0000","last_became_active":"2024-09-21T08:26:15.915988+0000","last_became_peered":"2024-09-21T08:26:15.915988+0000","last_unstale":"2024-09-21T08:26:24.530160+0000","last_undegraded":"2024-09-21T08:26:24.530160+0000","last_fullsized":"2024-09-21T08:26:24.530160+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":245,"log_dups_size":0,"ondisk_log_size":245,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00049722899999999997,"stat_sum":{"num_bytes":6613554,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":69,"num_read_kb":65491,"num_write":63,"num_write_kb":24745,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"11"}]},{"pgid":"4.6","version":"73'12","reported_seq":303,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527884+0000","last_change":"2024-09-21T08:26:07.384721+0000","last_active":"2024-09-21T08:26:24.527884+0000","last_peered":"2024-09-21T08:26:24.527884+0000","last_clean":"2024-09-21T08:26:24.527884+0000","last_became_active":"2024-09-21T08:25:48.888592+0000","last_became_peered":"2024-09-21T08:25:48.888592+0000","last_unstale":"2024-09-21T08:26:24.527884+0000","last_undegraded":"2024-09-21T08:26:24.527884+0000","last_fullsized":"2024-09-21T08:26:24.527884+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:26:07.384640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:07.384640+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:17.751790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":273,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526824+0000","last_change":"2024-09-21T08:24:48.601343+0000","last_active":"2024-09-21T08:26:24.526824+0000","last_peered":"2024-09-21T08:26:24.526824+0000","last_clean":"2024-09-21T08:26:24.526824+0000","last_became_active":"2024-09-21T08:24:48.601044+0000","last_became_peered":"2024-09-21T08:24:48.601044+0000","last_unstale":"2024-09-21T08:26:24.526824+0000","last_undegraded":"2024-09-21T08:26:24.526824+0000","last_fullsized":"2024-09-21T08:26:24.526824+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.435025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":457,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528503+0000","last_change":"2024-09-21T08:25:48.914747+0000","last_active":"2024-09-21T08:26:24.528503+0000","last_peered":"2024-09-21T08:26:24.528503+0000","last_clean":"2024-09-21T08:26:24.528503+0000","last_became_active":"2024-09-21T08:25:48.894703+0000","last_became_peered":"2024-09-21T08:25:48.894703+0000","last_unstale":"2024-09-21T08:26:24.528503+0000","last_undegraded":"2024-09-21T08:26:24.528503+0000","last_fullsized":"2024-09-21T08:26:24.528503+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:29.875888+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":168,"num_read_kb":163,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":389,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":268,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526936+0000","last_change":"2024-09-21T08:25:48.909001+0000","last_active":"2024-09-21T08:26:24.526936+0000","last_peered":"2024-09-21T08:26:24.526936+0000","last_clean":"2024-09-21T08:26:24.526936+0000","last_became_active":"2024-09-21T08:25:48.908492+0000","last_became_peered":"2024-09-21T08:25:48.908492+0000","last_unstale":"2024-09-21T08:26:24.526936+0000","last_undegraded":"2024-09-21T08:26:24.526936+0000","last_fullsized":"2024-09-21T08:26:24.526936+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.812789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.812789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:57.580731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":159,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526694+0000","last_change":"2024-09-21T08:25:48.914203+0000","last_active":"2024-09-21T08:26:24.526694+0000","last_peered":"2024-09-21T08:26:24.526694+0000","last_clean":"2024-09-21T08:26:24.526694+0000","last_became_active":"2024-09-21T08:25:48.912863+0000","last_became_peered":"2024-09-21T08:25:48.912863+0000","last_unstale":"2024-09-21T08:26:24.526694+0000","last_undegraded":"2024-09-21T08:26:24.526694+0000","last_fullsized":"2024-09-21T08:26:24.526694+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808769+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"180'96","reported_seq":674,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.528689+0000","last_change":"2024-09-21T08:26:24.524650+0000","last_active":"2024-09-21T08:26:24.528689+0000","last_peered":"2024-09-21T08:26:24.528689+0000","last_clean":"2024-09-21T08:26:24.528689+0000","last_became_active":"2024-09-21T08:26:15.922712+0000","last_became_peered":"2024-09-21T08:26:15.922712+0000","last_unstale":"2024-09-21T08:26:24.528689+0000","last_undegraded":"2024-09-21T08:26:24.528689+0000","last_fullsized":"2024-09-21T08:26:24.528689+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":96,"log_dups_size":0,"ondisk_log_size":96,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0100685220000001,"stat_sum":{"num_bytes":3638715,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":35,"num_read_kb":34185,"num_write":23,"num_write_kb":9181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2483319,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"f"}]},{"pgid":"4.5","version":"103'18","reported_seq":266,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529542+0000","last_change":"2024-09-21T08:24:56.564202+0000","last_active":"2024-09-21T08:26:24.529542+0000","last_peered":"2024-09-21T08:26:24.529542+0000","last_clean":"2024-09-21T08:26:24.529542+0000","last_became_active":"2024-09-21T08:24:56.503984+0000","last_became_peered":"2024-09-21T08:24:56.503984+0000","last_unstale":"2024-09-21T08:26:24.529542+0000","last_undegraded":"2024-09-21T08:26:24.529542+0000","last_fullsized":"2024-09-21T08:26:24.529542+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-09-21T08:24:48.583137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.583137+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:29.876102+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":154,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":217,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526540+0000","last_change":"2024-09-21T08:25:48.914796+0000","last_active":"2024-09-21T08:26:24.526540+0000","last_peered":"2024-09-21T08:26:24.526540+0000","last_clean":"2024-09-21T08:26:24.526540+0000","last_became_active":"2024-09-21T08:25:48.912615+0000","last_became_peered":"2024-09-21T08:25:48.912615+0000","last_unstale":"2024-09-21T08:26:24.526540+0000","last_undegraded":"2024-09-21T08:26:24.526540+0000","last_fullsized":"2024-09-21T08:26:24.526540+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808493+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":553,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527965+0000","last_change":"2024-09-21T08:25:51.506619+0000","last_active":"2024-09-21T08:26:24.527965+0000","last_peered":"2024-09-21T08:26:24.527965+0000","last_clean":"2024-09-21T08:26:24.527965+0000","last_became_active":"2024-09-21T08:25:51.506298+0000","last_became_peered":"2024-09-21T08:25:51.506298+0000","last_unstale":"2024-09-21T08:26:24.527965+0000","last_undegraded":"2024-09-21T08:26:24.527965+0000","last_fullsized":"2024-09-21T08:26:24.527965+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:59.195279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":259,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527887+0000","last_change":"2024-09-21T08:25:41.700634+0000","last_active":"2024-09-21T08:26:24.527887+0000","last_peered":"2024-09-21T08:26:24.527887+0000","last_clean":"2024-09-21T08:26:24.527887+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:26:24.527887+0000","last_undegraded":"2024-09-21T08:26:24.527887+0000","last_fullsized":"2024-09-21T08:26:24.527887+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:56.978983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":276,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527341+0000","last_change":"2024-09-21T08:25:48.907243+0000","last_active":"2024-09-21T08:26:24.527341+0000","last_peered":"2024-09-21T08:26:24.527341+0000","last_clean":"2024-09-21T08:26:24.527341+0000","last_became_active":"2024-09-21T08:25:48.906778+0000","last_became_peered":"2024-09-21T08:25:48.906778+0000","last_unstale":"2024-09-21T08:26:24.527341+0000","last_undegraded":"2024-09-21T08:26:24.527341+0000","last_fullsized":"2024-09-21T08:26:24.527341+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808858+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"180'51","reported_seq":539,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.950086+0000","last_change":"2024-09-21T08:26:24.950124+0000","last_active":"2024-09-21T08:26:24.950086+0000","last_peered":"2024-09-21T08:26:24.950086+0000","last_clean":"2024-09-21T08:26:24.950086+0000","last_became_active":"2024-09-21T08:25:48.906387+0000","last_became_peered":"2024-09-21T08:25:48.906387+0000","last_unstale":"2024-09-21T08:26:24.950086+0000","last_undegraded":"2024-09-21T08:26:24.950086+0000","last_fullsized":"2024-09-21T08:26:24.950086+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":51,"log_dups_size":0,"ondisk_log_size":51,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42244750800000003,"stat_sum":{"num_bytes":3243751,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":39,"num_read_kb":34343,"num_write":37,"num_write_kb":14152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":6854523,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"10"}]},{"pgid":"4.e","version":"73'11","reported_seq":175,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527475+0000","last_change":"2024-09-21T08:25:48.949515+0000","last_active":"2024-09-21T08:26:24.527475+0000","last_peered":"2024-09-21T08:26:24.527475+0000","last_clean":"2024-09-21T08:26:24.527475+0000","last_became_active":"2024-09-21T08:25:48.911473+0000","last_became_peered":"2024-09-21T08:25:48.911473+0000","last_unstale":"2024-09-21T08:26:24.527475+0000","last_undegraded":"2024-09-21T08:26:24.527475+0000","last_fullsized":"2024-09-21T08:26:24.527475+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808560+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":238,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525274+0000","last_change":"2024-09-21T08:25:48.907133+0000","last_active":"2024-09-21T08:26:24.525274+0000","last_peered":"2024-09-21T08:26:24.525274+0000","last_clean":"2024-09-21T08:26:24.525274+0000","last_became_active":"2024-09-21T08:25:48.906832+0000","last_became_peered":"2024-09-21T08:25:48.906832+0000","last_unstale":"2024-09-21T08:26:24.525274+0000","last_undegraded":"2024-09-21T08:26:24.525274+0000","last_fullsized":"2024-09-21T08:26:24.525274+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.756905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.756905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215756+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":272,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527387+0000","last_change":"2024-09-21T08:25:48.917071+0000","last_active":"2024-09-21T08:26:24.527387+0000","last_peered":"2024-09-21T08:26:24.527387+0000","last_clean":"2024-09-21T08:26:24.527387+0000","last_became_active":"2024-09-21T08:25:48.915146+0000","last_became_peered":"2024-09-21T08:25:48.915146+0000","last_unstale":"2024-09-21T08:26:24.527387+0000","last_undegraded":"2024-09-21T08:26:24.527387+0000","last_fullsized":"2024-09-21T08:26:24.527387+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808738+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":196,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526274+0000","last_change":"2024-09-21T08:25:48.914706+0000","last_active":"2024-09-21T08:26:24.526274+0000","last_peered":"2024-09-21T08:26:24.526274+0000","last_clean":"2024-09-21T08:26:24.526274+0000","last_became_active":"2024-09-21T08:25:48.912765+0000","last_became_peered":"2024-09-21T08:25:48.912765+0000","last_unstale":"2024-09-21T08:26:24.526274+0000","last_undegraded":"2024-09-21T08:26:24.526274+0000","last_fullsized":"2024-09-21T08:26:24.526274+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808660+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"191'109","reported_seq":658,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.943011+0000","last_change":"2024-09-21T08:26:24.943053+0000","last_active":"2024-09-21T08:26:24.943011+0000","last_peered":"2024-09-21T08:26:24.943011+0000","last_clean":"2024-09-21T08:26:24.943011+0000","last_became_active":"2024-09-21T08:24:34.508974+0000","last_became_peered":"2024-09-21T08:24:34.508974+0000","last_unstale":"2024-09-21T08:26:24.943011+0000","last_undegraded":"2024-09-21T08:26:24.943011+0000","last_fullsized":"2024-09-21T08:26:24.943011+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41626115000000002,"stat_sum":{"num_bytes":13815354,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":57,"num_read_kb":50251,"num_write":57,"num_write_kb":24503,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25","length":"e"}]},{"pgid":"4.d","version":"73'17","reported_seq":345,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529526+0000","last_change":"2024-09-21T08:25:48.934395+0000","last_active":"2024-09-21T08:26:24.529526+0000","last_peered":"2024-09-21T08:26:24.529526+0000","last_clean":"2024-09-21T08:26:24.529526+0000","last_became_active":"2024-09-21T08:25:48.904254+0000","last_became_peered":"2024-09-21T08:25:48.904254+0000","last_unstale":"2024-09-21T08:26:24.529526+0000","last_undegraded":"2024-09-21T08:26:24.529526+0000","last_fullsized":"2024-09-21T08:26:24.529526+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-09-21T08:25:42.978787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.978787+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:48.389318+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":159,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526453+0000","last_change":"2024-09-21T08:25:48.907580+0000","last_active":"2024-09-21T08:26:24.526453+0000","last_peered":"2024-09-21T08:26:24.526453+0000","last_clean":"2024-09-21T08:26:24.526453+0000","last_became_active":"2024-09-21T08:25:48.907227+0000","last_became_peered":"2024-09-21T08:25:48.907227+0000","last_unstale":"2024-09-21T08:26:24.526453+0000","last_undegraded":"2024-09-21T08:26:24.526453+0000","last_fullsized":"2024-09-21T08:26:24.526453+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":163,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528447+0000","last_change":"2024-09-21T08:25:48.889493+0000","last_active":"2024-09-21T08:26:24.528447+0000","last_peered":"2024-09-21T08:26:24.528447+0000","last_clean":"2024-09-21T08:26:24.528447+0000","last_became_active":"2024-09-21T08:25:48.889206+0000","last_became_peered":"2024-09-21T08:25:48.889206+0000","last_unstale":"2024-09-21T08:26:24.528447+0000","last_undegraded":"2024-09-21T08:26:24.528447+0000","last_fullsized":"2024-09-21T08:26:24.528447+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":281,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526588+0000","last_change":"2024-09-21T08:25:48.909997+0000","last_active":"2024-09-21T08:26:24.526588+0000","last_peered":"2024-09-21T08:26:24.526588+0000","last_clean":"2024-09-21T08:26:24.526588+0000","last_became_active":"2024-09-21T08:25:48.908631+0000","last_became_peered":"2024-09-21T08:25:48.908631+0000","last_unstale":"2024-09-21T08:26:24.526588+0000","last_undegraded":"2024-09-21T08:26:24.526588+0000","last_fullsized":"2024-09-21T08:26:24.526588+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.766815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215880+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"180'14","reported_seq":571,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.950390+0000","last_change":"2024-09-21T08:26:24.950444+0000","last_active":"2024-09-21T08:26:24.950390+0000","last_peered":"2024-09-21T08:26:24.950390+0000","last_clean":"2024-09-21T08:26:24.950390+0000","last_became_active":"2024-09-21T08:24:48.586187+0000","last_became_peered":"2024-09-21T08:24:48.586187+0000","last_unstale":"2024-09-21T08:26:24.950390+0000","last_undegraded":"2024-09-21T08:26:24.950390+0000","last_fullsized":"2024-09-21T08:26:24.950390+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.421885656,"stat_sum":{"num_bytes":3764768,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":21,"num_read_kb":20845,"num_write":10,"num_write_kb":4909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":949927,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"10"}]},{"pgid":"4.c","version":"73'10","reported_seq":131,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527113+0000","last_change":"2024-09-21T08:25:48.987805+0000","last_active":"2024-09-21T08:26:24.527113+0000","last_peered":"2024-09-21T08:26:24.527113+0000","last_clean":"2024-09-21T08:26:24.527113+0000","last_became_active":"2024-09-21T08:25:48.899148+0000","last_became_peered":"2024-09-21T08:25:48.899148+0000","last_unstale":"2024-09-21T08:26:24.527113+0000","last_undegraded":"2024-09-21T08:26:24.527113+0000","last_fullsized":"2024-09-21T08:26:24.527113+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778064+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":255,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527922+0000","last_change":"2024-09-21T08:25:48.891359+0000","last_active":"2024-09-21T08:26:24.527922+0000","last_peered":"2024-09-21T08:26:24.527922+0000","last_clean":"2024-09-21T08:26:24.527922+0000","last_became_active":"2024-09-21T08:25:48.890817+0000","last_became_peered":"2024-09-21T08:25:48.890817+0000","last_unstale":"2024-09-21T08:26:24.527922+0000","last_undegraded":"2024-09-21T08:26:24.527922+0000","last_fullsized":"2024-09-21T08:26:24.527922+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":305,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529229+0000","last_change":"2024-09-21T08:24:51.689976+0000","last_active":"2024-09-21T08:26:24.529229+0000","last_peered":"2024-09-21T08:26:24.529229+0000","last_clean":"2024-09-21T08:26:24.529229+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:26:24.529229+0000","last_undegraded":"2024-09-21T08:26:24.529229+0000","last_fullsized":"2024-09-21T08:26:24.529229+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:24:51.689901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.689901+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":87,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526255+0000","last_change":"2024-09-21T08:25:48.903834+0000","last_active":"2024-09-21T08:26:24.526255+0000","last_peered":"2024-09-21T08:26:24.526255+0000","last_clean":"2024-09-21T08:26:24.526255+0000","last_became_active":"2024-09-21T08:25:48.903536+0000","last_became_peered":"2024-09-21T08:25:48.903536+0000","last_unstale":"2024-09-21T08:26:24.526255+0000","last_undegraded":"2024-09-21T08:26:24.526255+0000","last_fullsized":"2024-09-21T08:26:24.526255+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"191'173","reported_seq":762,"reported_epoch":191,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:26:24.944580+0000","last_change":"2024-09-21T08:26:24.944580+0000","last_active":"2024-09-21T08:26:24.944580+0000","last_peered":"2024-09-21T08:26:24.944580+0000","last_clean":"2024-09-21T08:26:24.944580+0000","last_became_active":"2024-09-21T08:26:15.914229+0000","last_became_peered":"2024-09-21T08:26:15.914229+0000","last_unstale":"2024-09-21T08:26:24.944580+0000","last_undegraded":"2024-09-21T08:26:24.944580+0000","last_fullsized":"2024-09-21T08:26:24.944580+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":173,"log_dups_size":0,"ondisk_log_size":173,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.42054993899999998,"stat_sum":{"num_bytes":5748027,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":49,"num_read_kb":37859,"num_write":54,"num_write_kb":20304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":4377343,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25","length":"d"},{"start":"33","length":"1"}]},{"pgid":"4.b","version":"73'9","reported_seq":329,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528704+0000","last_change":"2024-09-21T08:25:48.969652+0000","last_active":"2024-09-21T08:26:24.528704+0000","last_peered":"2024-09-21T08:26:24.528704+0000","last_clean":"2024-09-21T08:26:24.528704+0000","last_became_active":"2024-09-21T08:25:48.906741+0000","last_became_peered":"2024-09-21T08:25:48.906741+0000","last_unstale":"2024-09-21T08:26:24.528704+0000","last_undegraded":"2024-09-21T08:26:24.528704+0000","last_fullsized":"2024-09-21T08:26:24.528704+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:25:22.369811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.369811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:41.468044+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":288,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526792+0000","last_change":"2024-09-21T08:25:48.916673+0000","last_active":"2024-09-21T08:26:24.526792+0000","last_peered":"2024-09-21T08:26:24.526792+0000","last_clean":"2024-09-21T08:26:24.526792+0000","last_became_active":"2024-09-21T08:25:48.914811+0000","last_became_peered":"2024-09-21T08:25:48.914811+0000","last_unstale":"2024-09-21T08:26:24.526792+0000","last_undegraded":"2024-09-21T08:26:24.526792+0000","last_fullsized":"2024-09-21T08:26:24.526792+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.665444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.665444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":279,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529893+0000","last_change":"2024-09-21T08:25:48.900965+0000","last_active":"2024-09-21T08:26:24.529893+0000","last_peered":"2024-09-21T08:26:24.529893+0000","last_clean":"2024-09-21T08:26:24.529893+0000","last_became_active":"2024-09-21T08:25:48.900241+0000","last_became_peered":"2024-09-21T08:25:48.900241+0000","last_unstale":"2024-09-21T08:26:24.529893+0000","last_undegraded":"2024-09-21T08:26:24.529893+0000","last_fullsized":"2024-09-21T08:26:24.529893+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:48.797848+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":249,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526312+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:26:24.526312+0000","last_peered":"2024-09-21T08:26:24.526312+0000","last_clean":"2024-09-21T08:26:24.526312+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:26:24.526312+0000","last_undegraded":"2024-09-21T08:26:24.526312+0000","last_fullsized":"2024-09-21T08:26:24.526312+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434881+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"182'168","reported_seq":805,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.947709+0000","last_change":"2024-09-21T08:26:24.947764+0000","last_active":"2024-09-21T08:26:24.947709+0000","last_peered":"2024-09-21T08:26:24.947709+0000","last_clean":"2024-09-21T08:26:24.947709+0000","last_became_active":"2024-09-21T08:26:15.917755+0000","last_became_peered":"2024-09-21T08:26:15.917755+0000","last_unstale":"2024-09-21T08:26:24.947709+0000","last_undegraded":"2024-09-21T08:26:24.947709+0000","last_fullsized":"2024-09-21T08:26:24.947709+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42097654099999998,"stat_sum":{"num_bytes":6579924,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":76,"num_read_kb":83741,"num_write":47,"num_write_kb":21496,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":6949463,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"10"}]},{"pgid":"4.a","version":"73'19","reported_seq":153,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526694+0000","last_change":"2024-09-21T08:25:48.964103+0000","last_active":"2024-09-21T08:26:24.526694+0000","last_peered":"2024-09-21T08:26:24.526694+0000","last_clean":"2024-09-21T08:26:24.526694+0000","last_became_active":"2024-09-21T08:25:48.895647+0000","last_became_peered":"2024-09-21T08:25:48.895647+0000","last_unstale":"2024-09-21T08:26:24.526694+0000","last_undegraded":"2024-09-21T08:26:24.526694+0000","last_fullsized":"2024-09-21T08:26:24.526694+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:43.390605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":273,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529374+0000","last_change":"2024-09-21T08:24:56.507593+0000","last_active":"2024-09-21T08:26:24.529374+0000","last_peered":"2024-09-21T08:26:24.529374+0000","last_clean":"2024-09-21T08:26:24.529374+0000","last_became_active":"2024-09-21T08:24:56.505054+0000","last_became_peered":"2024-09-21T08:24:56.505054+0000","last_unstale":"2024-09-21T08:26:24.529374+0000","last_undegraded":"2024-09-21T08:26:24.529374+0000","last_fullsized":"2024-09-21T08:26:24.529374+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.740674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.740674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":253,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525424+0000","last_change":"2024-09-21T08:23:34.564923+0000","last_active":"2024-09-21T08:26:24.525424+0000","last_peered":"2024-09-21T08:26:24.525424+0000","last_clean":"2024-09-21T08:26:24.525424+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:26:24.525424+0000","last_undegraded":"2024-09-21T08:26:24.525424+0000","last_fullsized":"2024-09-21T08:26:24.525424+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":267,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527532+0000","last_change":"2024-09-21T08:24:56.504692+0000","last_active":"2024-09-21T08:26:24.527532+0000","last_peered":"2024-09-21T08:26:24.527532+0000","last_clean":"2024-09-21T08:26:24.527532+0000","last_became_active":"2024-09-21T08:24:56.503564+0000","last_became_peered":"2024-09-21T08:24:56.503564+0000","last_unstale":"2024-09-21T08:26:24.527532+0000","last_undegraded":"2024-09-21T08:26:24.527532+0000","last_fullsized":"2024-09-21T08:26:24.527532+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778320+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"189'49","reported_seq":584,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.944529+0000","last_change":"2024-09-21T08:26:24.944571+0000","last_active":"2024-09-21T08:26:24.944529+0000","last_peered":"2024-09-21T08:26:24.944529+0000","last_clean":"2024-09-21T08:26:24.944529+0000","last_became_active":"2024-09-21T08:25:48.910246+0000","last_became_peered":"2024-09-21T08:25:48.910246+0000","last_unstale":"2024-09-21T08:26:24.944529+0000","last_undegraded":"2024-09-21T08:26:24.944529+0000","last_fullsized":"2024-09-21T08:26:24.944529+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":49,"log_dups_size":0,"ondisk_log_size":49,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41433005099999998,"stat_sum":{"num_bytes":4839114,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":17787,"num_write":25,"num_write_kb":5284,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":2551608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25","length":"f"}]},{"pgid":"4.9","version":"73'12","reported_seq":179,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526930+0000","last_change":"2024-09-21T08:25:48.944363+0000","last_active":"2024-09-21T08:26:24.526930+0000","last_peered":"2024-09-21T08:26:24.526930+0000","last_clean":"2024-09-21T08:26:24.526930+0000","last_became_active":"2024-09-21T08:25:48.894334+0000","last_became_peered":"2024-09-21T08:25:48.894334+0000","last_unstale":"2024-09-21T08:26:24.526930+0000","last_undegraded":"2024-09-21T08:26:24.526930+0000","last_fullsized":"2024-09-21T08:26:24.526930+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:41.463820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.463820+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.215785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528708+0000","last_change":"2024-09-21T08:25:48.896592+0000","last_active":"2024-09-21T08:26:24.528708+0000","last_peered":"2024-09-21T08:26:24.528708+0000","last_clean":"2024-09-21T08:26:24.528708+0000","last_became_active":"2024-09-21T08:25:48.896222+0000","last_became_peered":"2024-09-21T08:25:48.896222+0000","last_unstale":"2024-09-21T08:26:24.528708+0000","last_undegraded":"2024-09-21T08:26:24.528708+0000","last_fullsized":"2024-09-21T08:26:24.528708+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":267,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525397+0000","last_change":"2024-09-21T08:25:48.913293+0000","last_active":"2024-09-21T08:26:24.525397+0000","last_peered":"2024-09-21T08:26:24.525397+0000","last_clean":"2024-09-21T08:26:24.525397+0000","last_became_active":"2024-09-21T08:25:48.912993+0000","last_became_peered":"2024-09-21T08:25:48.912993+0000","last_unstale":"2024-09-21T08:26:24.525397+0000","last_undegraded":"2024-09-21T08:26:24.525397+0000","last_fullsized":"2024-09-21T08:26:24.525397+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:44.914394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:44.914394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:07.844366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":222,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526088+0000","last_change":"2024-09-21T08:25:48.901068+0000","last_active":"2024-09-21T08:26:24.526088+0000","last_peered":"2024-09-21T08:26:24.526088+0000","last_clean":"2024-09-21T08:26:24.526088+0000","last_became_active":"2024-09-21T08:25:48.900751+0000","last_became_peered":"2024-09-21T08:25:48.900751+0000","last_unstale":"2024-09-21T08:26:24.526088+0000","last_undegraded":"2024-09-21T08:26:24.526088+0000","last_fullsized":"2024-09-21T08:26:24.526088+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:38.201536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"191'275","reported_seq":977,"reported_epoch":191,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:26:24.941618+0000","last_change":"2024-09-21T08:26:22.939714+0000","last_active":"2024-09-21T08:26:24.941618+0000","last_peered":"2024-09-21T08:26:24.941618+0000","last_clean":"2024-09-21T08:26:24.941618+0000","last_became_active":"2024-09-21T08:25:51.507525+0000","last_became_peered":"2024-09-21T08:25:51.507525+0000","last_unstale":"2024-09-21T08:26:24.941618+0000","last_undegraded":"2024-09-21T08:26:24.941618+0000","last_fullsized":"2024-09-21T08:26:24.941618+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":275,"log_dups_size":0,"ondisk_log_size":275,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:29.811109+0000","scrub_duration":0,"objects_trimmed":1,"snaptrim_duration":0.41102610699999997,"stat_sum":{"num_bytes":22353409,"num_objects":9,"num_object_clones":1,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":226,"num_read_kb":213538,"num_write":175,"num_write_kb":50490,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31871318,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"25","length":"d"},{"start":"33","length":"1"}]},{"pgid":"4.8","version":"73'15","reported_seq":335,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525476+0000","last_change":"2024-09-21T08:25:48.957137+0000","last_active":"2024-09-21T08:26:24.525476+0000","last_peered":"2024-09-21T08:26:24.525476+0000","last_clean":"2024-09-21T08:26:24.525476+0000","last_became_active":"2024-09-21T08:25:48.911675+0000","last_became_peered":"2024-09-21T08:25:48.911675+0000","last_unstale":"2024-09-21T08:26:24.525476+0000","last_undegraded":"2024-09-21T08:26:24.525476+0000","last_fullsized":"2024-09-21T08:26:24.525476+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":303,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529367+0000","last_change":"2024-09-21T08:25:48.975855+0000","last_active":"2024-09-21T08:26:24.529367+0000","last_peered":"2024-09-21T08:26:24.529367+0000","last_clean":"2024-09-21T08:26:24.529367+0000","last_became_active":"2024-09-21T08:25:48.897974+0000","last_became_peered":"2024-09-21T08:25:48.897974+0000","last_unstale":"2024-09-21T08:26:24.529367+0000","last_undegraded":"2024-09-21T08:26:24.529367+0000","last_fullsized":"2024-09-21T08:26:24.529367+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778352+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"156'11","reported_seq":334,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526208+0000","last_change":"2024-09-21T08:25:48.977564+0000","last_active":"2024-09-21T08:26:24.526208+0000","last_peered":"2024-09-21T08:26:24.526208+0000","last_clean":"2024-09-21T08:26:24.526208+0000","last_became_active":"2024-09-21T08:25:48.897436+0000","last_became_peered":"2024-09-21T08:25:48.897436+0000","last_unstale":"2024-09-21T08:26:24.526208+0000","last_undegraded":"2024-09-21T08:26:24.526208+0000","last_fullsized":"2024-09-21T08:26:24.526208+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'10","last_scrub_stamp":"2024-09-21T08:25:42.708458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.708458+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:56.347110+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":256,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526271+0000","last_change":"2024-09-21T08:24:56.500567+0000","last_active":"2024-09-21T08:26:24.526271+0000","last_peered":"2024-09-21T08:26:24.526271+0000","last_clean":"2024-09-21T08:26:24.526271+0000","last_became_active":"2024-09-21T08:24:56.500287+0000","last_became_peered":"2024-09-21T08:24:56.500287+0000","last_unstale":"2024-09-21T08:26:24.526271+0000","last_undegraded":"2024-09-21T08:26:24.526271+0000","last_fullsized":"2024-09-21T08:26:24.526271+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"180'50","reported_seq":628,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.525907+0000","last_change":"2024-09-21T08:26:23.525369+0000","last_active":"2024-09-21T08:26:24.525907+0000","last_peered":"2024-09-21T08:26:24.525907+0000","last_clean":"2024-09-21T08:26:24.525907+0000","last_became_active":"2024-09-21T08:26:17.924327+0000","last_became_peered":"2024-09-21T08:26:17.924327+0000","last_unstale":"2024-09-21T08:26:24.525907+0000","last_undegraded":"2024-09-21T08:26:24.525907+0000","last_fullsized":"2024-09-21T08:26:24.525907+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42197120999999999,"stat_sum":{"num_bytes":1915998,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":12,"num_read_kb":9422,"num_write":15,"num_write_kb":4090,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":2458249,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25","length":"f"}]},{"pgid":"3.10","version":"0'0","reported_seq":187,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526307+0000","last_change":"2024-09-21T08:24:56.500240+0000","last_active":"2024-09-21T08:26:24.526307+0000","last_peered":"2024-09-21T08:26:24.526307+0000","last_clean":"2024-09-21T08:26:24.526307+0000","last_became_active":"2024-09-21T08:24:56.499114+0000","last_became_peered":"2024-09-21T08:24:56.499114+0000","last_unstale":"2024-09-21T08:26:24.526307+0000","last_undegraded":"2024-09-21T08:26:24.526307+0000","last_fullsized":"2024-09-21T08:26:24.526307+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434832+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":252,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529250+0000","last_change":"2024-09-21T08:25:48.980718+0000","last_active":"2024-09-21T08:26:24.529250+0000","last_peered":"2024-09-21T08:26:24.529250+0000","last_clean":"2024-09-21T08:26:24.529250+0000","last_became_active":"2024-09-21T08:25:48.898422+0000","last_became_peered":"2024-09-21T08:25:48.898422+0000","last_unstale":"2024-09-21T08:26:24.529250+0000","last_undegraded":"2024-09-21T08:26:24.529250+0000","last_fullsized":"2024-09-21T08:26:24.529250+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":272,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527015+0000","last_change":"2024-09-21T08:25:48.908032+0000","last_active":"2024-09-21T08:26:24.527015+0000","last_peered":"2024-09-21T08:26:24.527015+0000","last_clean":"2024-09-21T08:26:24.527015+0000","last_became_active":"2024-09-21T08:25:48.907510+0000","last_became_peered":"2024-09-21T08:25:48.907510+0000","last_unstale":"2024-09-21T08:26:24.527015+0000","last_undegraded":"2024-09-21T08:26:24.527015+0000","last_fullsized":"2024-09-21T08:26:24.527015+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":259,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529701+0000","last_change":"2024-09-21T08:24:56.506870+0000","last_active":"2024-09-21T08:26:24.529701+0000","last_peered":"2024-09-21T08:26:24.529701+0000","last_clean":"2024-09-21T08:26:24.529701+0000","last_became_active":"2024-09-21T08:24:56.502937+0000","last_became_peered":"2024-09-21T08:24:56.502937+0000","last_unstale":"2024-09-21T08:26:24.529701+0000","last_undegraded":"2024-09-21T08:26:24.529701+0000","last_fullsized":"2024-09-21T08:26:24.529701+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"191'260","reported_seq":888,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.530113+0000","last_change":"2024-09-21T08:26:24.530113+0000","last_active":"2024-09-21T08:26:24.530113+0000","last_peered":"2024-09-21T08:26:24.530113+0000","last_clean":"2024-09-21T08:26:24.530113+0000","last_became_active":"2024-09-21T08:26:20.938955+0000","last_became_peered":"2024-09-21T08:26:20.938955+0000","last_unstale":"2024-09-21T08:26:24.530113+0000","last_undegraded":"2024-09-21T08:26:24.530113+0000","last_fullsized":"2024-09-21T08:26:24.530113+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":260,"log_dups_size":0,"ondisk_log_size":260,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001326168,"stat_sum":{"num_bytes":7857273,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":71,"num_read_kb":65491,"num_write":70,"num_write_kb":30257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":14874427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"11"}]},{"pgid":"4.16","version":"73'9","reported_seq":289,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527786+0000","last_change":"2024-09-21T08:24:48.629077+0000","last_active":"2024-09-21T08:26:24.527786+0000","last_peered":"2024-09-21T08:26:24.527786+0000","last_clean":"2024-09-21T08:26:24.527786+0000","last_became_active":"2024-09-21T08:24:48.602176+0000","last_became_peered":"2024-09-21T08:24:48.602176+0000","last_unstale":"2024-09-21T08:26:24.527786+0000","last_undegraded":"2024-09-21T08:26:24.527786+0000","last_fullsized":"2024-09-21T08:26:24.527786+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":279,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525909+0000","last_change":"2024-09-21T08:25:46.719639+0000","last_active":"2024-09-21T08:26:24.525909+0000","last_peered":"2024-09-21T08:26:24.525909+0000","last_clean":"2024-09-21T08:26:24.525909+0000","last_became_active":"2024-09-21T08:24:56.505134+0000","last_became_peered":"2024-09-21T08:24:56.505134+0000","last_unstale":"2024-09-21T08:26:24.525909+0000","last_undegraded":"2024-09-21T08:26:24.525909+0000","last_fullsized":"2024-09-21T08:26:24.525909+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:46.719545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:46.719545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:51.390395+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":93,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529216+0000","last_change":"2024-09-21T08:26:05.814845+0000","last_active":"2024-09-21T08:26:24.529216+0000","last_peered":"2024-09-21T08:26:24.529216+0000","last_clean":"2024-09-21T08:26:24.529216+0000","last_became_active":"2024-09-21T08:25:48.896076+0000","last_became_peered":"2024-09-21T08:25:48.896076+0000","last_unstale":"2024-09-21T08:26:24.529216+0000","last_undegraded":"2024-09-21T08:26:24.529216+0000","last_fullsized":"2024-09-21T08:26:24.529216+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:05.814753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.814753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:09.836805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":263,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529638+0000","last_change":"2024-09-21T08:24:53.746385+0000","last_active":"2024-09-21T08:26:24.529638+0000","last_peered":"2024-09-21T08:26:24.529638+0000","last_clean":"2024-09-21T08:26:24.529638+0000","last_became_active":"2024-09-21T08:24:38.539845+0000","last_became_peered":"2024-09-21T08:24:38.539845+0000","last_unstale":"2024-09-21T08:26:24.529638+0000","last_undegraded":"2024-09-21T08:26:24.529638+0000","last_fullsized":"2024-09-21T08:26:24.529638+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.746295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.746295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:28.227304+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"189'102","reported_seq":693,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.946115+0000","last_change":"2024-09-21T08:26:24.946160+0000","last_active":"2024-09-21T08:26:24.946115+0000","last_peered":"2024-09-21T08:26:24.946115+0000","last_clean":"2024-09-21T08:26:24.946115+0000","last_became_active":"2024-09-21T08:26:17.924805+0000","last_became_peered":"2024-09-21T08:26:17.924805+0000","last_unstale":"2024-09-21T08:26:24.946115+0000","last_undegraded":"2024-09-21T08:26:24.946115+0000","last_fullsized":"2024-09-21T08:26:24.946115+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":102,"log_dups_size":0,"ondisk_log_size":102,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.41702267199999998,"stat_sum":{"num_bytes":2092505,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":37,"num_read_kb":34186,"num_write":27,"num_write_kb":10084,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":6063210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"f"}]},{"pgid":"4.15","version":"73'9","reported_seq":306,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528164+0000","last_change":"2024-09-21T08:24:56.529301+0000","last_active":"2024-09-21T08:26:24.528164+0000","last_peered":"2024-09-21T08:26:24.528164+0000","last_clean":"2024-09-21T08:26:24.528164+0000","last_became_active":"2024-09-21T08:24:56.499937+0000","last_became_peered":"2024-09-21T08:24:56.499937+0000","last_unstale":"2024-09-21T08:26:24.528164+0000","last_undegraded":"2024-09-21T08:26:24.528164+0000","last_fullsized":"2024-09-21T08:26:24.528164+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808577+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":273,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526457+0000","last_change":"2024-09-21T08:25:48.889025+0000","last_active":"2024-09-21T08:26:24.526457+0000","last_peered":"2024-09-21T08:26:24.526457+0000","last_clean":"2024-09-21T08:26:24.526457+0000","last_became_active":"2024-09-21T08:25:48.888733+0000","last_became_peered":"2024-09-21T08:25:48.888733+0000","last_unstale":"2024-09-21T08:26:24.526457+0000","last_undegraded":"2024-09-21T08:26:24.526457+0000","last_fullsized":"2024-09-21T08:26:24.526457+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434651+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"156'11","reported_seq":270,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527836+0000","last_change":"2024-09-21T08:25:48.958643+0000","last_active":"2024-09-21T08:26:24.527836+0000","last_peered":"2024-09-21T08:26:24.527836+0000","last_clean":"2024-09-21T08:26:24.527836+0000","last_became_active":"2024-09-21T08:25:48.896000+0000","last_became_peered":"2024-09-21T08:25:48.896000+0000","last_unstale":"2024-09-21T08:26:24.527836+0000","last_undegraded":"2024-09-21T08:26:24.527836+0000","last_fullsized":"2024-09-21T08:26:24.527836+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:27.824550+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":233,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526496+0000","last_change":"2024-09-21T08:25:39.989977+0000","last_active":"2024-09-21T08:26:24.526496+0000","last_peered":"2024-09-21T08:26:24.526496+0000","last_clean":"2024-09-21T08:26:24.526496+0000","last_became_active":"2024-09-21T08:24:38.542854+0000","last_became_peered":"2024-09-21T08:24:38.542854+0000","last_unstale":"2024-09-21T08:26:24.526496+0000","last_undegraded":"2024-09-21T08:26:24.526496+0000","last_fullsized":"2024-09-21T08:26:24.526496+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.989838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.989838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:49.118031+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"179'63","reported_seq":636,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529350+0000","last_change":"2024-09-21T08:26:24.529760+0000","last_active":"2024-09-21T08:26:24.529350+0000","last_peered":"2024-09-21T08:26:24.529350+0000","last_clean":"2024-09-21T08:26:24.529350+0000","last_became_active":"2024-09-21T08:26:17.919628+0000","last_became_peered":"2024-09-21T08:26:17.919628+0000","last_unstale":"2024-09-21T08:26:24.529350+0000","last_undegraded":"2024-09-21T08:26:24.529350+0000","last_fullsized":"2024-09-21T08:26:24.529350+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001334383,"stat_sum":{"num_bytes":1873981,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":10,"num_read_kb":5554,"num_write":17,"num_write_kb":5776,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":5482381,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"11"}]},{"pgid":"4.14","version":"104'15","reported_seq":174,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527974+0000","last_change":"2024-09-21T08:25:48.935811+0000","last_active":"2024-09-21T08:26:24.527974+0000","last_peered":"2024-09-21T08:26:24.527974+0000","last_clean":"2024-09-21T08:26:24.527974+0000","last_became_active":"2024-09-21T08:25:48.894839+0000","last_became_peered":"2024-09-21T08:25:48.894839+0000","last_unstale":"2024-09-21T08:26:24.527974+0000","last_undegraded":"2024-09-21T08:26:24.527974+0000","last_fullsized":"2024-09-21T08:26:24.527974+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778400+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":293,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525721+0000","last_change":"2024-09-21T08:24:56.709528+0000","last_active":"2024-09-21T08:26:24.525721+0000","last_peered":"2024-09-21T08:26:24.525721+0000","last_clean":"2024-09-21T08:26:24.525721+0000","last_became_active":"2024-09-21T08:24:56.504665+0000","last_became_peered":"2024-09-21T08:24:56.504665+0000","last_unstale":"2024-09-21T08:26:24.525721+0000","last_undegraded":"2024-09-21T08:26:24.525721+0000","last_fullsized":"2024-09-21T08:26:24.525721+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:56.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.709437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:27.824533+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"156'9","reported_seq":316,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528389+0000","last_change":"2024-09-21T08:25:48.931191+0000","last_active":"2024-09-21T08:26:24.528389+0000","last_peered":"2024-09-21T08:26:24.528389+0000","last_clean":"2024-09-21T08:26:24.528389+0000","last_became_active":"2024-09-21T08:25:48.908349+0000","last_became_peered":"2024-09-21T08:25:48.908349+0000","last_unstale":"2024-09-21T08:26:24.528389+0000","last_undegraded":"2024-09-21T08:26:24.528389+0000","last_fullsized":"2024-09-21T08:26:24.528389+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:31.979617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":255,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528927+0000","last_change":"2024-09-21T08:24:48.592895+0000","last_active":"2024-09-21T08:26:24.528927+0000","last_peered":"2024-09-21T08:26:24.528927+0000","last_clean":"2024-09-21T08:26:24.528927+0000","last_became_active":"2024-09-21T08:24:48.592280+0000","last_became_peered":"2024-09-21T08:24:48.592280+0000","last_unstale":"2024-09-21T08:26:24.528927+0000","last_undegraded":"2024-09-21T08:26:24.528927+0000","last_fullsized":"2024-09-21T08:26:24.528927+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"172'94","reported_seq":655,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.524998+0000","last_change":"2024-09-21T08:26:23.524400+0000","last_active":"2024-09-21T08:26:24.524998+0000","last_peered":"2024-09-21T08:26:24.524998+0000","last_clean":"2024-09-21T08:26:24.524998+0000","last_became_active":"2024-09-21T08:26:17.923502+0000","last_became_peered":"2024-09-21T08:26:17.923502+0000","last_unstale":"2024-09-21T08:26:24.524998+0000","last_undegraded":"2024-09-21T08:26:24.524998+0000","last_fullsized":"2024-09-21T08:26:24.524998+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":94,"log_dups_size":0,"ondisk_log_size":94,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42235843899999997,"stat_sum":{"num_bytes":5075714,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":31,"num_read_kb":24250,"num_write":31,"num_write_kb":12244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":7230966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"25","length":"f"}]},{"pgid":"4.13","version":"73'11","reported_seq":136,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525917+0000","last_change":"2024-09-21T08:25:48.979008+0000","last_active":"2024-09-21T08:26:24.525917+0000","last_peered":"2024-09-21T08:26:24.525917+0000","last_clean":"2024-09-21T08:26:24.525917+0000","last_became_active":"2024-09-21T08:25:48.914638+0000","last_became_peered":"2024-09-21T08:25:48.914638+0000","last_unstale":"2024-09-21T08:26:24.525917+0000","last_undegraded":"2024-09-21T08:26:24.525917+0000","last_fullsized":"2024-09-21T08:26:24.525917+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":228,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528156+0000","last_change":"2024-09-21T08:24:56.504037+0000","last_active":"2024-09-21T08:26:24.528156+0000","last_peered":"2024-09-21T08:26:24.528156+0000","last_clean":"2024-09-21T08:26:24.528156+0000","last_became_active":"2024-09-21T08:24:56.503232+0000","last_became_peered":"2024-09-21T08:24:56.503232+0000","last_unstale":"2024-09-21T08:26:24.528156+0000","last_undegraded":"2024-09-21T08:26:24.528156+0000","last_fullsized":"2024-09-21T08:26:24.528156+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778203+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":141,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526188+0000","last_change":"2024-09-21T08:25:48.917229+0000","last_active":"2024-09-21T08:26:24.526188+0000","last_peered":"2024-09-21T08:26:24.526188+0000","last_clean":"2024-09-21T08:26:24.526188+0000","last_became_active":"2024-09-21T08:25:48.916802+0000","last_became_peered":"2024-09-21T08:25:48.916802+0000","last_unstale":"2024-09-21T08:26:24.526188+0000","last_undegraded":"2024-09-21T08:26:24.526188+0000","last_fullsized":"2024-09-21T08:26:24.526188+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808754+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":203,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526459+0000","last_change":"2024-09-21T08:24:48.600966+0000","last_active":"2024-09-21T08:26:24.526459+0000","last_peered":"2024-09-21T08:26:24.526459+0000","last_clean":"2024-09-21T08:26:24.526459+0000","last_became_active":"2024-09-21T08:24:48.600045+0000","last_became_peered":"2024-09-21T08:24:48.600045+0000","last_unstale":"2024-09-21T08:26:24.526459+0000","last_undegraded":"2024-09-21T08:26:24.526459+0000","last_fullsized":"2024-09-21T08:26:24.526459+0000","mapping_epoch":94,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":95,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434706+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"179'43","reported_seq":468,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.525586+0000","last_change":"2024-09-21T08:26:24.529267+0000","last_active":"2024-09-21T08:26:24.525586+0000","last_peered":"2024-09-21T08:26:24.525586+0000","last_clean":"2024-09-21T08:26:24.525586+0000","last_became_active":"2024-09-21T08:26:17.921717+0000","last_became_peered":"2024-09-21T08:26:17.921717+0000","last_unstale":"2024-09-21T08:26:24.525586+0000","last_undegraded":"2024-09-21T08:26:24.525586+0000","last_fullsized":"2024-09-21T08:26:24.525586+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0038224550000000002,"stat_sum":{"num_bytes":1605069,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":17,"num_read_kb":16983,"num_write":10,"num_write_kb":3161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":6033316,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"25","length":"11"}]},{"pgid":"4.12","version":"73'9","reported_seq":129,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529842+0000","last_change":"2024-09-21T08:25:48.953827+0000","last_active":"2024-09-21T08:26:24.529842+0000","last_peered":"2024-09-21T08:26:24.529842+0000","last_clean":"2024-09-21T08:26:24.529842+0000","last_became_active":"2024-09-21T08:25:48.910041+0000","last_became_peered":"2024-09-21T08:25:48.910041+0000","last_unstale":"2024-09-21T08:26:24.529842+0000","last_undegraded":"2024-09-21T08:26:24.529842+0000","last_fullsized":"2024-09-21T08:26:24.529842+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.215637+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":277,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.529402+0000","last_change":"2024-09-21T08:24:56.505458+0000","last_active":"2024-09-21T08:26:24.529402+0000","last_peered":"2024-09-21T08:26:24.529402+0000","last_clean":"2024-09-21T08:26:24.529402+0000","last_became_active":"2024-09-21T08:24:56.504584+0000","last_became_peered":"2024-09-21T08:24:56.504584+0000","last_unstale":"2024-09-21T08:26:24.529402+0000","last_undegraded":"2024-09-21T08:26:24.529402+0000","last_fullsized":"2024-09-21T08:26:24.529402+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":209,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528857+0000","last_change":"2024-09-21T08:25:48.904228+0000","last_active":"2024-09-21T08:26:24.528857+0000","last_peered":"2024-09-21T08:26:24.528857+0000","last_clean":"2024-09-21T08:26:24.528857+0000","last_became_active":"2024-09-21T08:25:48.903934+0000","last_became_peered":"2024-09-21T08:25:48.903934+0000","last_unstale":"2024-09-21T08:26:24.528857+0000","last_undegraded":"2024-09-21T08:26:24.528857+0000","last_fullsized":"2024-09-21T08:26:24.528857+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434999+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":255,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526557+0000","last_change":"2024-09-21T08:25:48.902778+0000","last_active":"2024-09-21T08:26:24.526557+0000","last_peered":"2024-09-21T08:26:24.526557+0000","last_clean":"2024-09-21T08:26:24.526557+0000","last_became_active":"2024-09-21T08:25:48.902435+0000","last_became_peered":"2024-09-21T08:25:48.902435+0000","last_unstale":"2024-09-21T08:26:24.526557+0000","last_undegraded":"2024-09-21T08:26:24.526557+0000","last_fullsized":"2024-09-21T08:26:24.526557+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434731+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"189'200","reported_seq":833,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.949774+0000","last_change":"2024-09-21T08:26:24.949813+0000","last_active":"2024-09-21T08:26:24.949774+0000","last_peered":"2024-09-21T08:26:24.949774+0000","last_clean":"2024-09-21T08:26:24.949774+0000","last_became_active":"2024-09-21T08:26:17.923264+0000","last_became_peered":"2024-09-21T08:26:17.923264+0000","last_unstale":"2024-09-21T08:26:24.949774+0000","last_undegraded":"2024-09-21T08:26:24.949774+0000","last_fullsized":"2024-09-21T08:26:24.949774+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.42339930599999998,"stat_sum":{"num_bytes":6437081,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":51,"num_read_kb":45933,"num_write":63,"num_write_kb":20673,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":11369848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"10"}]},{"pgid":"4.11","version":"73'11","reported_seq":278,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526061+0000","last_change":"2024-09-21T08:24:56.531655+0000","last_active":"2024-09-21T08:26:24.526061+0000","last_peered":"2024-09-21T08:26:24.526061+0000","last_clean":"2024-09-21T08:26:24.526061+0000","last_became_active":"2024-09-21T08:24:56.503223+0000","last_became_peered":"2024-09-21T08:24:56.503223+0000","last_unstale":"2024-09-21T08:26:24.526061+0000","last_undegraded":"2024-09-21T08:26:24.526061+0000","last_fullsized":"2024-09-21T08:26:24.526061+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778430+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":276,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526851+0000","last_change":"2024-09-21T08:26:03.808138+0000","last_active":"2024-09-21T08:26:24.526851+0000","last_peered":"2024-09-21T08:26:24.526851+0000","last_clean":"2024-09-21T08:26:24.526851+0000","last_became_active":"2024-09-21T08:25:48.912523+0000","last_became_peered":"2024-09-21T08:25:48.912523+0000","last_unstale":"2024-09-21T08:26:24.526851+0000","last_undegraded":"2024-09-21T08:26:24.526851+0000","last_fullsized":"2024-09-21T08:26:24.526851+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:08.522124+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":271,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528223+0000","last_change":"2024-09-21T08:24:58.688115+0000","last_active":"2024-09-21T08:26:24.528223+0000","last_peered":"2024-09-21T08:26:24.528223+0000","last_clean":"2024-09-21T08:26:24.528223+0000","last_became_active":"2024-09-21T08:24:56.503123+0000","last_became_peered":"2024-09-21T08:24:56.503123+0000","last_unstale":"2024-09-21T08:26:24.528223+0000","last_undegraded":"2024-09-21T08:26:24.528223+0000","last_fullsized":"2024-09-21T08:26:24.528223+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.688046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.688046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.364343+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":209,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528499+0000","last_change":"2024-09-21T08:24:56.508632+0000","last_active":"2024-09-21T08:26:24.528499+0000","last_peered":"2024-09-21T08:26:24.528499+0000","last_clean":"2024-09-21T08:26:24.528499+0000","last_became_active":"2024-09-21T08:24:56.508323+0000","last_became_peered":"2024-09-21T08:24:56.508323+0000","last_unstale":"2024-09-21T08:26:24.528499+0000","last_undegraded":"2024-09-21T08:26:24.528499+0000","last_fullsized":"2024-09-21T08:26:24.528499+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"186'22","reported_seq":584,"reported_epoch":191,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:26:24.946775+0000","last_change":"2024-09-21T08:26:24.946816+0000","last_active":"2024-09-21T08:26:24.946775+0000","last_peered":"2024-09-21T08:26:24.946775+0000","last_clean":"2024-09-21T08:26:24.946775+0000","last_became_active":"2024-09-21T08:26:17.925392+0000","last_became_peered":"2024-09-21T08:26:17.925392+0000","last_unstale":"2024-09-21T08:26:24.946775+0000","last_undegraded":"2024-09-21T08:26:24.946775+0000","last_fullsized":"2024-09-21T08:26:24.946775+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.419373985,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":7,"num_read_kb":5325,"num_write":6,"num_write_kb":1264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":2565420,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"25","length":"10"}]},{"pgid":"4.10","version":"73'4","reported_seq":117,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527647+0000","last_change":"2024-09-21T08:25:48.973819+0000","last_active":"2024-09-21T08:26:24.527647+0000","last_peered":"2024-09-21T08:26:24.527647+0000","last_clean":"2024-09-21T08:26:24.527647+0000","last_became_active":"2024-09-21T08:25:48.913344+0000","last_became_peered":"2024-09-21T08:25:48.913344+0000","last_unstale":"2024-09-21T08:26:24.527647+0000","last_undegraded":"2024-09-21T08:26:24.527647+0000","last_fullsized":"2024-09-21T08:26:24.527647+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808444+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":279,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527705+0000","last_change":"2024-09-21T08:25:48.907831+0000","last_active":"2024-09-21T08:26:24.527705+0000","last_peered":"2024-09-21T08:26:24.527705+0000","last_clean":"2024-09-21T08:26:24.527705+0000","last_became_active":"2024-09-21T08:25:48.907530+0000","last_became_peered":"2024-09-21T08:25:48.907530+0000","last_unstale":"2024-09-21T08:26:24.527705+0000","last_undegraded":"2024-09-21T08:26:24.527705+0000","last_fullsized":"2024-09-21T08:26:24.527705+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.434602+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":230,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526269+0000","last_change":"2024-09-21T08:25:48.899072+0000","last_active":"2024-09-21T08:26:24.526269+0000","last_peered":"2024-09-21T08:26:24.526269+0000","last_clean":"2024-09-21T08:26:24.526269+0000","last_became_active":"2024-09-21T08:25:48.898419+0000","last_became_peered":"2024-09-21T08:25:48.898419+0000","last_unstale":"2024-09-21T08:26:24.526269+0000","last_undegraded":"2024-09-21T08:26:24.526269+0000","last_fullsized":"2024-09-21T08:26:24.526269+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":265,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528964+0000","last_change":"2024-09-21T08:25:48.896046+0000","last_active":"2024-09-21T08:26:24.528964+0000","last_peered":"2024-09-21T08:26:24.528964+0000","last_clean":"2024-09-21T08:26:24.528964+0000","last_became_active":"2024-09-21T08:25:48.895489+0000","last_became_peered":"2024-09-21T08:25:48.895489+0000","last_unstale":"2024-09-21T08:26:24.528964+0000","last_undegraded":"2024-09-21T08:26:24.528964+0000","last_fullsized":"2024-09-21T08:26:24.528964+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:24.778095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":117,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528253+0000","last_change":"2024-09-21T08:25:48.913489+0000","last_active":"2024-09-21T08:26:24.528253+0000","last_peered":"2024-09-21T08:26:24.528253+0000","last_clean":"2024-09-21T08:26:24.528253+0000","last_became_active":"2024-09-21T08:25:48.912542+0000","last_became_peered":"2024-09-21T08:25:48.912542+0000","last_unstale":"2024-09-21T08:26:24.528253+0000","last_undegraded":"2024-09-21T08:26:24.528253+0000","last_fullsized":"2024-09-21T08:26:24.528253+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":218,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.526686+0000","last_change":"2024-09-21T08:25:48.885761+0000","last_active":"2024-09-21T08:26:24.526686+0000","last_peered":"2024-09-21T08:26:24.526686+0000","last_clean":"2024-09-21T08:26:24.526686+0000","last_became_active":"2024-09-21T08:25:48.884347+0000","last_became_peered":"2024-09-21T08:25:48.884347+0000","last_unstale":"2024-09-21T08:26:24.526686+0000","last_undegraded":"2024-09-21T08:26:24.526686+0000","last_fullsized":"2024-09-21T08:26:24.526686+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:26.435116+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":256,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.527215+0000","last_change":"2024-09-21T08:25:48.942921+0000","last_active":"2024-09-21T08:26:24.527215+0000","last_peered":"2024-09-21T08:26:24.527215+0000","last_clean":"2024-09-21T08:26:24.527215+0000","last_became_active":"2024-09-21T08:25:48.908941+0000","last_became_peered":"2024-09-21T08:25:48.908941+0000","last_unstale":"2024-09-21T08:26:24.527215+0000","last_undegraded":"2024-09-21T08:26:24.527215+0000","last_fullsized":"2024-09-21T08:26:24.527215+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:25.808843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":261,"reported_epoch":191,"state":"active+clean","last_fresh":"2024-09-21T08:26:24.528518+0000","last_change":"2024-09-21T08:25:25.350265+0000","last_active":"2024-09-21T08:26:24.528518+0000","last_peered":"2024-09-21T08:26:24.528518+0000","last_clean":"2024-09-21T08:26:24.528518+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:26:24.528518+0000","last_undegraded":"2024-09-21T08:26:24.528518+0000","last_fullsized":"2024-09-21T08:26:24.528518+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.350197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.350197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:26:48.284282+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":134320407,"num_objects":47,"num_object_clones":6,"num_object_copies":141,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":47,"num_whiteouts":2,"num_read":1194,"num_read_kb":1110286,"num_write":1030,"num_write_kb":385980,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":224,"num_bytes_recovered":179026145,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":413855744,"data_stored":438306059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2938,"ondisk_log_size":2938,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":26,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1274,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":82,"ondisk_log_size":82,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":675,"num_read_kb":432,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":268,"num_bytes_recovered":5132,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":741376,"data_stored":19638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":403,"ondisk_log_size":403,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3226,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":8157,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":6,"ondisk_log_size":6,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":168,"num_read_kb":163,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":427,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070504,"num_pgs":122,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116488,"kb_used_data":76972,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93628664,"statfs":{"total":95995035648,"available":95875751936,"internally_reserved":0,"allocated":78819328,"data_stored":82312678,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397550,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659185,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":110580,"kb_used_data":74576,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93634572,"statfs":{"total":95995035648,"available":95881801728,"internally_reserved":0,"allocated":76365824,"data_stored":78975358,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986231,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182459,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":124708,"kb_used_data":86956,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93620444,"statfs":{"total":95995035648,"available":95867334656,"internally_reserved":0,"allocated":89042944,"data_stored":93756026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574917,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":121,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":116244,"kb_used_data":78484,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93628908,"statfs":{"total":95995035648,"available":95876001792,"internally_reserved":0,"allocated":80367616,"data_stored":85008399,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2228,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":3262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":77647872,"data_stored":83314883,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45531136,"data_stored":53470237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86347776,"data_stored":92062737,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32387072,"data_stored":32188068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":71069696,"data_stored":73392382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":72777728,"data_stored":76007337,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:26:27.677 INFO:tasks.thrashosds.thrasher:Setting 7.f to [5, 7, 3, 4, 1, 2] 2024-09-21T08:26:27.677 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '7.f', '5', '7', '3', '4', '1', '2'] 2024-09-21T08:26:27.678 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 7.f 5 7 3 4 1 2 2024-09-21T08:26:27.690 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:27.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:27 smithi165 ceph-mon[32893]: pgmap v194: 158 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 139 active+clean; 129 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 4.9 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-21T08:26:27.696 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:27 smithi165 ceph-mon[32893]: osdmap e194: 8 total, 8 up, 4 in 2024-09-21T08:26:27.712 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:27.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:27.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:27.725 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T08:26:27.743 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:27.814 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:27.817 INFO:tasks.rados.rados.0.smithi137.stdout:629: finishing rollback tid 2 to smithi13779221-5 2024-09-21T08:26:27.817 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 204 (ObjNum 205 snap 41 seq_num 205) dirty exists 2024-09-21T08:26:27.817 INFO:tasks.rados.rados.0.smithi137.stdout:626: finishing copy_from racing read to smithi13779221-27 2024-09-21T08:26:27.817 INFO:tasks.rados.rados.0.smithi137.stdout:624: finishing copy_from racing read to smithi13779221-45 2024-09-21T08:26:27.817 INFO:tasks.rados.rados.0.smithi137.stdout:628: finishing copy_from to smithi13779221-24 2024-09-21T08:26:27.817 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 135 (ObjNum 192 snap 39 seq_num 192) dirty exists 2024-09-21T08:26:27.817 INFO:tasks.rados.rados.0.smithi137.stdout:623: done (10 left) 2024-09-21T08:26:27.817 INFO:tasks.rados.rados.0.smithi137.stdout:625: done (9 left) 2024-09-21T08:26:27.817 INFO:tasks.rados.rados.0.smithi137.stdout:627: done (8 left) 2024-09-21T08:26:27.818 INFO:tasks.rados.rados.0.smithi137.stdout:628: done (7 left) 2024-09-21T08:26:27.818 INFO:tasks.rados.rados.0.smithi137.stdout:629: done (6 left) 2024-09-21T08:26:27.818 INFO:tasks.rados.rados.0.smithi137.stdout:633: delete oid 7 current snap is 55 2024-09-21T08:26:27.818 INFO:tasks.rados.rados.0.smithi137.stdout:631: expect (ObjNum 232 snap 47 seq_num 232) 2024-09-21T08:26:27.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:27.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:27.836 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:27.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:27 smithi137 ceph-mon[29634]: pgmap v194: 158 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 139 active+clean; 129 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 4.9 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-21T08:26:27.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:27 smithi137 ceph-mon[29634]: osdmap e194: 8 total, 8 up, 4 in 2024-09-21T08:26:27.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:27 smithi137 ceph-mon[41044]: pgmap v194: 158 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 139 active+clean; 129 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 4.9 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-21T08:26:27.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:27 smithi137 ceph-mon[41044]: osdmap e194: 8 total, 8 up, 4 in 2024-09-21T08:26:27.937 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:27.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:27.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:27.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:28.010 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:28.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:28.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:28.083 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:28.083 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:28.152 INFO:tasks.rados.rados.0.smithi137.stdout:624: finishing copy_from to smithi13779221-45 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 245 (ObjNum 237 snap 49 seq_num 237) dirty exists 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:626: finishing copy_from to smithi13779221-27 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 54 (ObjNum 193 snap 39 seq_num 193) dirty exists 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:630: finishing write tid 1 to smithi13779221-8 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:630: finishing write tid 2 to smithi13779221-8 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:630: finishing write tid 3 to smithi13779221-8 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:622: finishing copy_from racing read to smithi13779221-1 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:630: finishing write tid 4 to smithi13779221-8 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:630: oid 8 updating version 0 to 248 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:630: oid 8 version 248 is already newer than 246 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:630: oid 8 version 248 is already newer than 247 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 248 (ObjNum 254 snap 55 seq_num 254) dirty exists 2024-09-21T08:26:28.153 INFO:tasks.rados.rados.0.smithi137.stdout:630: left oid 8 (ObjNum 254 snap 55 seq_num 254) 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:632: finishing write tid 1 to smithi13779221-13 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:632: finishing write tid 2 to smithi13779221-13 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:624: done (6 left) 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:626: done (5 left) 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:630: done (4 left) 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:631: done (3 left) 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:633: done (2 left) 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:634: copy_from oid 32 from oid 50 current snap is 55 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:632: finishing write tid 3 to smithi13779221-13 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:632: finishing write tid 4 to smithi13779221-13 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:632: oid 13 updating version 0 to 279 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:632: oid 13 updating version 279 to 281 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:632: oid 13 version 281 is already newer than 280 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 281 (ObjNum 255 snap 55 seq_num 255) dirty exists 2024-09-21T08:26:28.154 INFO:tasks.rados.rados.0.smithi137.stdout:632: left oid 13 (ObjNum 255 snap 55 seq_num 255) 2024-09-21T08:26:28.155 INFO:tasks.rados.rados.0.smithi137.stdout:622: finishing copy_from to smithi13779221-1 2024-09-21T08:26:28.155 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 245 (ObjNum 231 snap 47 seq_num 231) dirty exists 2024-09-21T08:26:28.155 INFO:tasks.rados.rados.0.smithi137.stdout:622: done (2 left) 2024-09-21T08:26:28.155 INFO:tasks.rados.rados.0.smithi137.stdout:632: done (1 left) 2024-09-21T08:26:28.155 INFO:tasks.rados.rados.0.smithi137.stdout:635: copy_from oid 11 from oid 3 current snap is 55 2024-09-21T08:26:28.155 INFO:tasks.rados.rados.0.smithi137.stdout:636: read oid 9 snap 53 2024-09-21T08:26:28.155 INFO:tasks.rados.rados.0.smithi137.stdout:636: expect deleted 2024-09-21T08:26:28.155 INFO:tasks.rados.rados.0.smithi137.stdout:637: delete oid 15 current snap is 55 2024-09-21T08:26:28.156 INFO:tasks.rados.rados.0.smithi137.stdout:636: done (3 left) 2024-09-21T08:26:28.156 INFO:tasks.rados.rados.0.smithi137.stdout:637: done (2 left) 2024-09-21T08:26:28.156 INFO:tasks.rados.rados.0.smithi137.stdout:638: snap_remove snap 54 2024-09-21T08:26:28.184 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:28.206 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:28.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:28.207 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:28.295 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:26:28.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:28.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:28.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:28.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:28.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:28.396 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:28.415 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:28.428 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:28.429 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:28.446 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:28.446 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:28.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:28.465 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:28.465 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:28.474 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:28.475 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:28.481 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:28.481 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:28.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:28.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:28.485 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:28.498 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:28.498 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:28.515 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:28.515 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:28.531 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:28.532 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:28.543 INFO:tasks.rados.rados.0.smithi137.stdout:638: done (2 left) 2024-09-21T08:26:28.543 INFO:tasks.rados.rados.0.smithi137.stdout:639: delete oid 10 current snap is 55 2024-09-21T08:26:28.543 INFO:tasks.rados.rados.0.smithi137.stdout:634: finishing copy_from racing read to smithi13779221-32 2024-09-21T08:26:28.543 INFO:tasks.rados.rados.0.smithi137.stdout:635: finishing copy_from racing read to smithi13779221-11 2024-09-21T08:26:28.543 INFO:tasks.rados.rados.0.smithi137.stdout:635: finishing copy_from to smithi13779221-11 2024-09-21T08:26:28.543 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 143 (ObjNum 220 snap 46 seq_num 220) dirty exists 2024-09-21T08:26:28.543 INFO:tasks.rados.rados.0.smithi137.stdout:634: finishing copy_from to smithi13779221-32 2024-09-21T08:26:28.544 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 189 (ObjNum 171 snap 34 seq_num 171) dirty exists 2024-09-21T08:26:28.544 INFO:tasks.rados.rados.0.smithi137.stdout:634: done (2 left) 2024-09-21T08:26:28.544 INFO:tasks.rados.rados.0.smithi137.stdout:635: done (1 left) 2024-09-21T08:26:28.545 INFO:tasks.rados.rados.0.smithi137.stdout:639: done (0 left) 2024-09-21T08:26:28.545 INFO:tasks.rados.rados.0.smithi137.stdout:640: write oid 48 current snap is 55 2024-09-21T08:26:28.545 INFO:tasks.rados.rados.0.smithi137.stdout:640: seq_num 259 ranges {753013=645493,2056248=296765} 2024-09-21T08:26:28.552 INFO:tasks.rados.rados.0.smithi137.stdout:640: writing smithi13779221-48 from 753013 to 1398506 tid 1 2024-09-21T08:26:28.558 INFO:tasks.rados.rados.0.smithi137.stdout:640: writing smithi13779221-48 from 2056248 to 2353013 tid 2 2024-09-21T08:26:28.558 INFO:tasks.rados.rados.0.smithi137.stdout:641: write oid 19 current snap is 55 2024-09-21T08:26:28.558 INFO:tasks.rados.rados.0.smithi137.stdout:640: finishing write tid 1 to smithi13779221-48 2024-09-21T08:26:28.558 INFO:tasks.rados.rados.0.smithi137.stdout:641: seq_num 260 ranges {769820=720742,2114730=255090} 2024-09-21T08:26:28.566 INFO:tasks.rados.rados.0.smithi137.stdout:641: writing smithi13779221-19 from 769820 to 1490562 tid 1 2024-09-21T08:26:28.571 INFO:tasks.rados.rados.0.smithi137.stdout:641: writing smithi13779221-19 from 2114730 to 2369820 tid 2 2024-09-21T08:26:28.571 INFO:tasks.rados.rados.0.smithi137.stdout:642: snap_remove snap 53 2024-09-21T08:26:28.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:28.607 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:28.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:28.608 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:28.709 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:28.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:28.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:28.735 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:28 smithi165 ceph-mon[32893]: from='client.15138 v1:172.21.15.137:0/837277948' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:26:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:28 smithi165 ceph-mon[32893]: 7.0 scrub starts 2024-09-21T08:26:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:28 smithi165 ceph-mon[32893]: 7.0 scrub ok 2024-09-21T08:26:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:28 smithi165 ceph-mon[32893]: 7.19 scrub starts 2024-09-21T08:26:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:28 smithi165 ceph-mon[32893]: 7.19 scrub ok 2024-09-21T08:26:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:28 smithi165 ceph-mon[32893]: 7.10 scrub starts 2024-09-21T08:26:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:28 smithi165 ceph-mon[32893]: 7.10 scrub ok 2024-09-21T08:26:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:28 smithi165 ceph-mon[32893]: 7.12 scrub starts 2024-09-21T08:26:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:28 smithi165 ceph-mon[32893]: 7.12 scrub ok 2024-09-21T08:26:28.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:28 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2123884892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:28.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:28.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:28.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:28.857 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:28.958 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:28.980 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:28.981 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:28.981 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:29.010 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[29634]: from='client.15138 v1:172.21.15.137:0/837277948' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:26:29.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[29634]: 7.0 scrub starts 2024-09-21T08:26:29.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[29634]: 7.0 scrub ok 2024-09-21T08:26:29.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[29634]: 7.19 scrub starts 2024-09-21T08:26:29.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[29634]: 7.19 scrub ok 2024-09-21T08:26:29.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[29634]: 7.10 scrub starts 2024-09-21T08:26:29.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[29634]: 7.10 scrub ok 2024-09-21T08:26:29.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[29634]: 7.12 scrub starts 2024-09-21T08:26:29.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[29634]: 7.12 scrub ok 2024-09-21T08:26:29.011 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2123884892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:29.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[41044]: from='client.15138 v1:172.21.15.137:0/837277948' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:26:29.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[41044]: 7.0 scrub starts 2024-09-21T08:26:29.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[41044]: 7.0 scrub ok 2024-09-21T08:26:29.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[41044]: 7.19 scrub starts 2024-09-21T08:26:29.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[41044]: 7.19 scrub ok 2024-09-21T08:26:29.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[41044]: 7.10 scrub starts 2024-09-21T08:26:29.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[41044]: 7.10 scrub ok 2024-09-21T08:26:29.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[41044]: 7.12 scrub starts 2024-09-21T08:26:29.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[41044]: 7.12 scrub ok 2024-09-21T08:26:29.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:28 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2123884892' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:29.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:29.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:29.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:29.104 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:29.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T08:26:29.205 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:29.245 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:29.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:29.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:29.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:29.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:29.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:29.370 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:29.427 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:29.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:29.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:29.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:29.493 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:29.544 INFO:tasks.rados.rados.0.smithi137.stdout:642: done (2 left) 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:643: snap_create 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:640: finishing write tid 2 to smithi13779221-48 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:640: finishing write tid 3 to smithi13779221-48 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:640: finishing write tid 4 to smithi13779221-48 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:640: oid 48 updating version 0 to 198 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:640: oid 48 version 198 is already newer than 197 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:640: oid 48 version 198 is already newer than 196 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 198 (ObjNum 259 snap 55 seq_num 259) dirty exists 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:640: left oid 48 (ObjNum 259 snap 55 seq_num 259) 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:641: finishing write tid 1 to smithi13779221-19 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:641: finishing write tid 2 to smithi13779221-19 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:641: finishing write tid 3 to smithi13779221-19 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:641: finishing write tid 4 to smithi13779221-19 2024-09-21T08:26:29.545 INFO:tasks.rados.rados.0.smithi137.stdout:641: oid 19 updating version 0 to 56 2024-09-21T08:26:29.546 INFO:tasks.rados.rados.0.smithi137.stdout:641: oid 19 updating version 56 to 57 2024-09-21T08:26:29.546 INFO:tasks.rados.rados.0.smithi137.stdout:641: oid 19 updating version 57 to 58 2024-09-21T08:26:29.546 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 58 (ObjNum 260 snap 55 seq_num 260) dirty exists 2024-09-21T08:26:29.546 INFO:tasks.rados.rados.0.smithi137.stdout:641: left oid 19 (ObjNum 260 snap 55 seq_num 260) 2024-09-21T08:26:29.593 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:29.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:29.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:29.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:29.704 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[29634]: 7.11 scrub starts 2024-09-21T08:26:29.704 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[29634]: 7.11 scrub ok 2024-09-21T08:26:29.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[29634]: pgmap v197: 158 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 139 active+clean; 129 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 4.9 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-21T08:26:29.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[29634]: 7.8 scrub starts 2024-09-21T08:26:29.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[29634]: 7.8 scrub ok 2024-09-21T08:26:29.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[29634]: 7.18 scrub starts 2024-09-21T08:26:29.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[29634]: 7.18 scrub ok 2024-09-21T08:26:29.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[29634]: osdmap e195: 8 total, 8 up, 4 in 2024-09-21T08:26:29.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[41044]: 7.11 scrub starts 2024-09-21T08:26:29.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[41044]: 7.11 scrub ok 2024-09-21T08:26:29.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[41044]: pgmap v197: 158 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 139 active+clean; 129 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 4.9 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-21T08:26:29.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[41044]: 7.8 scrub starts 2024-09-21T08:26:29.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[41044]: 7.8 scrub ok 2024-09-21T08:26:29.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[41044]: 7.18 scrub starts 2024-09-21T08:26:29.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[41044]: 7.18 scrub ok 2024-09-21T08:26:29.706 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:29 smithi137 ceph-mon[41044]: osdmap e195: 8 total, 8 up, 4 in 2024-09-21T08:26:29.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:29.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:29.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:29.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:29.740 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:29.835 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:29 smithi165 ceph-mon[32893]: 7.11 scrub starts 2024-09-21T08:26:29.835 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:29 smithi165 ceph-mon[32893]: 7.11 scrub ok 2024-09-21T08:26:29.835 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:29 smithi165 ceph-mon[32893]: pgmap v197: 158 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 139 active+clean; 129 MiB data, 457 MiB used, 357 GiB / 358 GiB avail; 4.9 MiB/s rd, 1.5 MiB/s wr, 12 op/s 2024-09-21T08:26:29.835 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:29 smithi165 ceph-mon[32893]: 7.8 scrub starts 2024-09-21T08:26:29.835 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:29 smithi165 ceph-mon[32893]: 7.8 scrub ok 2024-09-21T08:26:29.835 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:29 smithi165 ceph-mon[32893]: 7.18 scrub starts 2024-09-21T08:26:29.835 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:29 smithi165 ceph-mon[32893]: 7.18 scrub ok 2024-09-21T08:26:29.836 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:29 smithi165 ceph-mon[32893]: osdmap e195: 8 total, 8 up, 4 in 2024-09-21T08:26:29.839 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:29.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:29.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:29.860 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:29.940 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:26:29.940 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:26:29.940 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:26:29.940 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:26:29.940 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.940 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.24394.0:7 2.2 2:5cea7035:::rbd_info:head [write 0~19] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e68)", 2024-09-21T08:26:29.940 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:32.548847+0000", 2024-09-21T08:26:29.941 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 177.388393436, 2024-09-21T08:26:29.941 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010300768, 2024-09-21T08:26:29.941 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:29.941 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:29.941 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:29.941 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:29.941 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.24394", 2024-09-21T08:26:29.941 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.165:0/2482038767", 2024-09-21T08:26:29.941 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 7 2024-09-21T08:26:29.941 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.942 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:29.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:29.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.548847+0000", 2024-09-21T08:26:29.942 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.942 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.942 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.942 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:29.942 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.548847+0000", 2024-09-21T08:26:29.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:29.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.548850+0000", 2024-09-21T08:26:29.943 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0060000000000001e-06 2024-09-21T08:26:29.943 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.943 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.943 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:29.943 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.548860+0000", 2024-09-21T08:26:29.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6069999999999997e-06 2024-09-21T08:26:29.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:29.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.548876+0000", 2024-09-21T08:26:29.944 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6110999999999999e-05 2024-09-21T08:26:29.944 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.944 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.944 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:29.944 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.548889+0000", 2024-09-21T08:26:29.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3309000000000001e-05 2024-09-21T08:26:29.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.945 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.548953+0000", 2024-09-21T08:26:29.945 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4358999999999998e-05 2024-09-21T08:26:29.945 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.945 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.945 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:29.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.549113+0000", 2024-09-21T08:26:29.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015924099999999999 2024-09-21T08:26:29.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:26:29.946 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.549257+0000", 2024-09-21T08:26:29.946 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014384700000000001 2024-09-21T08:26:29.946 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.946 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.946 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:29.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.549836+0000", 2024-09-21T08:26:29.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00057959699999999995 2024-09-21T08:26:29.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:29.947 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.550921+0000", 2024-09-21T08:26:29.947 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001084638 2024-09-21T08:26:29.947 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.947 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.947 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:29.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.559006+0000", 2024-09-21T08:26:29.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0080851900000000008 2024-09-21T08:26:29.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.948 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:29.948 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.559081+0000", 2024-09-21T08:26:29.948 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4802999999999995e-05 2024-09-21T08:26:29.948 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.948 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.949 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:29.949 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:32.559148+0000", 2024-09-21T08:26:29.949 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7059999999999998e-05 2024-09-21T08:26:29.949 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.949 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:29.949 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.949 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.949 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.949 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1083 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94837804037760 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:26:29.949 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:26:29.950 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 111.777660208, 2024-09-21T08:26:29.950 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.395092834, 2024-09-21T08:26:29.950 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:29.950 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:29.950 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:29.950 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:29.950 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:26:29.950 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:26:29.950 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1083 2024-09-21T08:26:29.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.951 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:29.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:29.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:26:29.951 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.951 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.951 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.951 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:29.951 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:26:29.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.952 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:29.952 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159586+0000", 2024-09-21T08:26:29.952 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.48e-06 2024-09-21T08:26:29.952 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.952 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:29.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159599+0000", 2024-09-21T08:26:29.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3699e-05 2024-09-21T08:26:29.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.953 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:29.953 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159630+0000", 2024-09-21T08:26:29.953 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0629000000000001e-05 2024-09-21T08:26:29.953 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.953 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:29.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159647+0000", 2024-09-21T08:26:29.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7030000000000001e-05 2024-09-21T08:26:29.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.954 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.954 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159767+0000", 2024-09-21T08:26:29.954 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119571 2024-09-21T08:26:29.954 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.954 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159781+0000", 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3844e-05 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.549248+0000", 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.38946733 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.955 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.549559+0000", 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000311057 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552031+0000", 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024720720000000001 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552119+0000", 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8261000000000001e-05 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.956 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.554673+0000", 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025538610000000002 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:569 7.e e117/80 7:78d05d33:::smithi13779221-41:head v 117'23, mlcod=100'19)", 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:10.568420+0000", 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 79.368820415000002, 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073491440000000002, 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:29.957 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.568420+0000", 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.568420+0000", 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.568422+0000", 2024-09-21T08:26:29.958 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.658e-06 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.571669+0000", 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003247504 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.571680+0000", 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0417999999999999e-05 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:29.959 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.571685+0000", 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3539999999999999e-06 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.571741+0000", 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5764999999999999e-05 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.571759+0000", 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7944999999999998e-05 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.960 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.575734+0000", 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039748869999999999 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.575769+0000", 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5612999999999999e-05 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:682 7.0 7:00f4c0d4:::smithi13779221-18:head [write 699191~532079] snapc 18=[18,17,16,15,11,d] ondisk+write+known_if_redirected+supports_pool_eio e126)", 2024-09-21T08:26:29.961 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:19.661470+0000", 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 70.275770131000002, 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010816673000000001, 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 682 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:29.962 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.661470+0000", 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.661470+0000", 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.661472+0000", 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1299999999999999e-06 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.963 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.662249+0000", 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077630699999999995 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.662259+0000", 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0529e-05 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.662267+0000", 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9990000000000008e-06 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.964 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.662324+0000", 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6193000000000002e-05 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.662328+0000", 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8199999999999996e-06 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.663726+0000", 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001398065 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.965 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.663808+0000", 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1170999999999999e-05 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.664184+0000", 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037603700000000002 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.668575+0000", 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043918109999999998 2024-09-21T08:26:29.966 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.671417+0000", 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028415110000000001 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.672204+0000", 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00078694099999999999 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.672244+0000", 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9690000000000001e-05 2024-09-21T08:26:29.967 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:19.672287+0000", 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3469e-05 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1031 7.0 7:00f4c0d4:::smithi13779221-18:head [copy-from ver 6] snapc 22=[22,21,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:40.814163+0000", 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 49.123077314, 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012690369, 2024-09-21T08:26:29.968 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1031 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814163+0000", 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.969 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814163+0000", 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814165+0000", 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8679999999999999e-06 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814173+0000", 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7030000000000006e-06 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.970 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814184+0000", 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1423999999999999e-05 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814190+0000", 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4709999999999998e-06 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814237+0000", 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7290000000000003e-05 2024-09-21T08:26:29.971 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.972 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.972 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:29.972 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814301+0000", 2024-09-21T08:26:29.972 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3720000000000007e-05 2024-09-21T08:26:29.972 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:29.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:26:29.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.818021+0000", 2024-09-21T08:26:29.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003720105 2024-09-21T08:26:29.974 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.974 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.974 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:29.974 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.821257+0000", 2024-09-21T08:26:29.974 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032359979999999999 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.826736+0000", 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054788179999999999 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.826774+0000", 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8127000000000003e-05 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:29.975 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.826811+0000", 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7144999999999998e-05 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.826854+0000", 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2700000000000001e-05 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:1051 7.8 e147/94 7:1fd29771:::smithi13779221-22:head v 147'111, mlcod=147'107)", 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:41.124818+0000", 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 48.812422193000003, 2024-09-21T08:26:29.976 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089655589999999997, 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.124818+0000", 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.124818+0000", 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.977 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.124820+0000", 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0669999999999999e-06 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.128774+0000", 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039534469999999997 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.128791+0000", 2024-09-21T08:26:29.978 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7325000000000001e-05 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.128798+0000", 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1359999999999999e-06 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.128897+0000", 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8833999999999996e-05 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.979 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.128924+0000", 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7013999999999999e-05 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.133708+0000", 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047841000000000003 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.133784+0000", 2024-09-21T08:26:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5636000000000006e-05 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1159 7.b 7:df2b42d9:::smithi13779221-36:head [write 642067~696188,stat] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.047121828000002, 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62427366799999995, 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:29.981 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1159 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.982 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890120+0000", 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6950000000000001e-06 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890849+0000", 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072891299999999998 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890861+0000", 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1899000000000001e-05 2024-09-21T08:26:29.983 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890868+0000", 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2110000000000001e-06 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890947+0000", 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8598999999999997e-05 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890954+0000", 2024-09-21T08:26:29.984 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1180000000000003e-06 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508089+0000", 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61713446299999997 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508263+0000", 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174671 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:29.985 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508366+0000", 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102786 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511111+0000", 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027452710000000001 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.513304+0000", 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021930560000000001 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.986 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514317+0000", 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010129959999999999 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514355+0000", 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7404000000000003e-05 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514392+0000", 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7586000000000002e-05 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.987 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1160 7.b 7:df2b42d9:::smithi13779221-36:head [write 2025476~459397] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.040673939000001, 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61820804500000004, 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:29.988 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1160 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.989 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896568+0000", 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2610000000000001e-06 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896958+0000", 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039041199999999997 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896984+0000", 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6115e-05 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.990 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896989+0000", 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0379999999999999e-06 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.897069+0000", 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9196000000000005e-05 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.897075+0000", 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.28e-06 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.991 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509576+0000", 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61250132700000004 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509636+0000", 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9753999999999997e-05 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509725+0000", 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9524000000000005e-05 2024-09-21T08:26:29.992 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511333+0000", 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001607619 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.513767+0000", 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024336819999999999 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514703+0000", 2024-09-21T08:26:29.993 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093633599999999996 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514736+0000", 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2438999999999999e-05 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514775+0000", 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9062e-05 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:29.994 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1161 7.b 7:df2b42d9:::smithi13779221-36:head [write 2971091~70976] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.037795013999997, 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61638660599999995, 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1161 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.995 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899446+0000", 2024-09-21T08:26:29.996 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0800000000000004e-07 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899537+0000", 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0414999999999994e-05 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899544+0000", 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8580000000000002e-06 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:29.997 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899547+0000", 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8970000000000001e-06 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899597+0000", 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9058999999999999e-05 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899605+0000", 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3990000000000004e-06 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.998 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510251+0000", 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61064600599999996 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510324+0000", 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2961999999999997e-05 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510438+0000", 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113793 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:29.999 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511337+0000", 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089906399999999998 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514615+0000", 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00327853 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515770+0000", 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001154715 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.000 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515802+0000", 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1956999999999999e-05 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515832+0000", 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0142999999999999e-05 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.001 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1162 7.b 7:df2b42d9:::smithi13779221-36:head [setxattr _header (57) in=7b,truncate 3042067] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.037685946000003, 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61641711799999999, 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1162 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.002 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9100000000000004e-07 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.003 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899557+0000", 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3989999999999998e-06 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899561+0000", 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9349999999999996e-06 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899564+0000", 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7700000000000002e-06 2024-09-21T08:26:30.004 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899633+0000", 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9268000000000002e-05 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899637+0000", 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9650000000000002e-06 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510758+0000", 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61112027999999996 2024-09-21T08:26:30.005 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510819+0000", 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0866999999999998e-05 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510929+0000", 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110914 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:30.006 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511340+0000", 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041042100000000001 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514641+0000", 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003301511 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515865+0000", 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012240319999999999 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.007 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515898+0000", 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3036e-05 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515972+0000", 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3229e-05 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1163 7.b 7:df2b42d9:::smithi13779221-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:26:30.008 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.037654533000001, 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61652938899999998, 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1163 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.009 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8599999999999998e-07 2024-09-21T08:26:30.010 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899589+0000", 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1299999999999999e-06 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899592+0000", 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9190000000000001e-06 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.011 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899595+0000", 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9639999999999999e-06 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899654+0000", 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8958e-05 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899658+0000", 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2669999999999998e-06 2024-09-21T08:26:30.012 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511140+0000", 2024-09-21T08:26:30.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61148218399999998 2024-09-21T08:26:30.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.013 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.013 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:26:30.013 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511235+0000", 2024-09-21T08:26:30.013 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4927999999999996e-05 2024-09-21T08:26:30.013 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516004+0000", 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047691640000000002 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516060+0000", 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5726999999999997e-05 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.014 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516115+0000", 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5661999999999997e-05 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1187 7.0 7:0979c720:::smithi13779221-16:head [copy-from ver 34] snapc 26=[26,25] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:51.549803+0000", 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.387437425999998, 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012728847, 2024-09-21T08:26:30.015 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1187 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.016 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549803+0000", 2024-09-21T08:26:30.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549803+0000", 2024-09-21T08:26:30.017 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.017 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.017 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.017 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.017 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549805+0000", 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2019999999999998e-06 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549811+0000", 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3229999999999997e-06 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549822+0000", 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1093e-05 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.018 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549827+0000", 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3669999999999996e-06 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549875+0000", 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7865999999999999e-05 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.019 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549935+0000", 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0117000000000001e-05 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.554466+0000", 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045311630000000004 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.558093+0000", 2024-09-21T08:26:30.020 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036264240000000001 2024-09-21T08:26:30.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.562399+0000", 2024-09-21T08:26:30.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043062860000000003 2024-09-21T08:26:30.021 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.021 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.021 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.021 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.562467+0000", 2024-09-21T08:26:30.021 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7776999999999998e-05 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.562502+0000", 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5117999999999998e-05 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.562532+0000", 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0111e-05 2024-09-21T08:26:30.022 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.023 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.023 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.023 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.023 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.023 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1188 7.0 7:0979c720:::smithi13779221-16:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e158)", 2024-09-21T08:26:30.023 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:51.549839+0000", 2024-09-21T08:26:30.023 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.387401388999997, 2024-09-21T08:26:30.023 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075194659999999998, 2024-09-21T08:26:30.023 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.023 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1188 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549839+0000", 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.024 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549839+0000", 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549840+0000", 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0299999999999999e-07 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549841+0000", 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7179999999999999e-06 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.025 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549846+0000", 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5709999999999999e-06 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549849+0000", 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9720000000000001e-06 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.550060+0000", 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021093300000000001 2024-09-21T08:26:30.026 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.550093+0000", 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2869e-05 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.557212+0000", 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071193189999999998 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.557280+0000", 2024-09-21T08:26:30.027 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8207999999999995e-05 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.557359+0000", 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8373000000000003e-05 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:1198 7.6 e158/94 7:611b1c4b:::smithi13779221-3:head v 158'69, mlcod=151'65)", 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:51.677052+0000", 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 38.260188436999996, 2024-09-21T08:26:30.028 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095895809999999998, 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.677052+0000", 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.677052+0000", 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.029 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.677054+0000", 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.018e-06 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.682128+0000", 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005074411 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.030 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.682143+0000", 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4148000000000001e-05 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.682149+0000", 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6710000000000002e-06 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.682213+0000", 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3485000000000006e-05 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.031 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.682236+0000", 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3184e-05 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.686605+0000", 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.004369436 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.686642+0000", 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6227999999999999e-05 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.032 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1214 7.0 7:00f4c0d4:::smithi13779221-18:head [copy-from ver 91] snapc 27=[25] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:54.540239+0000", 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.397001904, 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016093084000000001, 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:30.033 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1214 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540239+0000", 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540239+0000", 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.034 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540241+0000", 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0499999999999999e-06 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540249+0000", 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8049999999999996e-06 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540263+0000", 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3329999999999999e-05 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.035 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540268+0000", 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6470000000000003e-06 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540325+0000", 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6918e-05 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540389+0000", 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3990000000000002e-05 2024-09-21T08:26:30.036 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.547214+0000", 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068248240000000002 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.551118+0000", 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039037669999999998 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556219+0000", 2024-09-21T08:26:30.037 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051015089999999997 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556258+0000", 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8606999999999998e-05 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556298+0000", 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0096999999999999e-05 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.038 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556332+0000", 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3540000000000001e-05 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1215 7.0 7:00f4c0d4:::smithi13779221-18:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e161)", 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:54.540284+0000", 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.396956869999997, 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010181865, 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.039 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1215 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540284+0000", 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.040 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540284+0000", 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540284+0000", 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7899999999999997e-07 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540288+0000", 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1810000000000001e-06 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.041 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540297+0000", 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7419999999999992e-06 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540303+0000", 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8529999999999997e-06 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540519+0000", 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00021589800000000001 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.042 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540561+0000", 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1619999999999998e-05 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.550315+0000", 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0097543409999999997 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.550387+0000", 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1420000000000004e-05 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.043 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.550465+0000", 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8930999999999994e-05 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1366 7.0 7:0979c720:::smithi13779221-16:head [write 729470~543821,stat] snapc 2c=[2c,2b] ondisk+write+known_if_redirected+supports_pool_eio e170)", 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:03.517254+0000", 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 26.419986713, 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077259099999999999, 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.044 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1366 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.517254+0000", 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.045 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.517254+0000", 2024-09-21T08:26:30.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.046 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.046 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.517276+0000", 2024-09-21T08:26:30.046 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2684000000000002e-05 2024-09-21T08:26:30.046 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.046 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.517832+0000", 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055601299999999995 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.517850+0000", 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7686999999999999e-05 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.517859+0000", 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2350000000000005e-06 2024-09-21T08:26:30.047 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.517956+0000", 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6627999999999997e-05 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.517965+0000", 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3039999999999994e-06 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.519904+0000", 2024-09-21T08:26:30.048 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00193834 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.520022+0000", 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118478 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.520245+0000", 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000222988 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:30.049 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.522069+0000", 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018236070000000001 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.523214+0000", 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001144805 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.524866+0000", 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016521179999999999 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.050 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.524928+0000", 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1964000000000005e-05 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:03.524980+0000", 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2058999999999997e-05 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1430 7.b 7:d6c5a045:::smithi13779221-46:head [write 463926~462166,stat] snapc 2e=[2e] ondisk+write+known_if_redirected+supports_pool_eio e175)", 2024-09-21T08:26:30.051 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:08.540947+0000", 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.396293558, 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.007538889, 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1430 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.052 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.540947+0000", 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.540947+0000", 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.540949+0000", 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1430000000000001e-06 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.053 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.544821+0000", 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038718110000000002 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.544837+0000", 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5671e-05 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.544844+0000", 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1790000000000002e-06 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.054 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.544914+0000", 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0133000000000001e-05 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.545027+0000", 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112744 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.545167+0000", 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140001 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.055 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.546462+0000", 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001295563 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.548251+0000", 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017885279999999999 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.548405+0000", 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000154346 2024-09-21T08:26:30.056 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.548448+0000", 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3401e-05 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:08.548486+0000", 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7369000000000001e-05 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.057 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:1571 7.18 e182/181 7:1fd29771:::smithi13779221-22:head v 182'166, mlcod=0'0)", 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:16.510087+0000", 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 13.427153653, 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012367031000000001, 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.510087+0000", 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.058 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.510087+0000", 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.510088+0000", 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2600000000000001e-07 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.517229+0000", 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071412979999999999 2024-09-21T08:26:30.059 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.517245+0000", 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.624e-05 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.517254+0000", 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2989999999999997e-06 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.517313+0000", 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9883000000000002e-05 2024-09-21T08:26:30.060 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.517328+0000", 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.491e-05 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.522422+0000", 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050932089999999996 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.522454+0000", 2024-09-21T08:26:30.061 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2265999999999999e-05 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1633 7.b 7:d22ba293:::smithi13779221-29:head [write 517715~559301,stat] snapc 36=[36,35,34,32] ondisk+write+known_if_redirected+supports_pool_eio e188)", 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:21.520998+0000", 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.4162425770000002, 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082127299999999997, 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:30.062 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1633 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.520998+0000", 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.520998+0000", 2024-09-21T08:26:30.063 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.521002+0000", 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2629999999999997e-06 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.521698+0000", 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00069593599999999997 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.521717+0000", 2024-09-21T08:26:30.064 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8437e-05 2024-09-21T08:26:30.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.065 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:30.065 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.521744+0000", 2024-09-21T08:26:30.065 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7016999999999999e-05 2024-09-21T08:26:30.065 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.065 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.521815+0000", 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1260999999999994e-05 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.521820+0000", 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0830000000000003e-06 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.525199+0000", 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003379174 2024-09-21T08:26:30.066 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.525393+0000", 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019426299999999999 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.525507+0000", 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011325200000000001 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.526762+0000", 2024-09-21T08:26:30.067 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012553029999999999 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.528204+0000", 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014421049999999999 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.529139+0000", 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093466000000000003 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:30.068 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.529177+0000", 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7928e-05 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:21.529211+0000", 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4048000000000003e-05 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:30.069 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:26:30.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:30.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:30.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:30.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:30.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:30.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:30.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:30.316 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:30.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:30.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:30.338 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:30.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:30.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:30.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:30.461 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:30.550 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:26:30.557 INFO:teuthology.orchestra.run.smithi137.stderr:set 7.f pg_upmap_items mapping to [5->7,3->4,1->2] 2024-09-21T08:26:30.558 INFO:tasks.rados.rados.0.smithi137.stdout:640: done (2 left) 2024-09-21T08:26:30.558 INFO:tasks.rados.rados.0.smithi137.stdout:641: done (1 left) 2024-09-21T08:26:30.558 INFO:tasks.rados.rados.0.smithi137.stdout:643: done (0 left) 2024-09-21T08:26:30.558 INFO:tasks.rados.rados.0.smithi137.stdout:644: read oid 10 snap 55 2024-09-21T08:26:30.559 INFO:tasks.rados.rados.0.smithi137.stdout:644: expect deleted 2024-09-21T08:26:30.559 INFO:tasks.rados.rados.0.smithi137.stdout:645: write (excl) oid 49 current snap is 56 2024-09-21T08:26:30.559 INFO:tasks.rados.rados.0.smithi137.stdout:645: seq_num 261 ranges {786627=795991,2173212=213415} 2024-09-21T08:26:30.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:30.567 INFO:tasks.rados.rados.0.smithi137.stdout:645: writing smithi13779221-49 from 786627 to 1582618 tid 1 2024-09-21T08:26:30.572 INFO:tasks.rados.rados.0.smithi137.stdout:645: writing smithi13779221-49 from 2173212 to 2386627 tid 2 2024-09-21T08:26:30.572 INFO:tasks.rados.rados.0.smithi137.stdout:646: delete oid 32 current snap is 56 2024-09-21T08:26:30.576 INFO:tasks.rados.rados.0.smithi137.stdout:645: finishing write tid 1 to smithi13779221-49 2024-09-21T08:26:30.577 INFO:tasks.rados.rados.0.smithi137.stdout:645: finishing write tid 2 to smithi13779221-49 2024-09-21T08:26:30.578 INFO:tasks.rados.rados.0.smithi137.stdout:645: finishing write tid 3 to smithi13779221-49 2024-09-21T08:26:30.578 INFO:tasks.rados.rados.0.smithi137.stdout:645: finishing write tid 4 to smithi13779221-49 2024-09-21T08:26:30.578 INFO:tasks.rados.rados.0.smithi137.stdout:645: oid 49 updating version 0 to 266 2024-09-21T08:26:30.578 INFO:tasks.rados.rados.0.smithi137.stdout:645: oid 49 version 266 is already newer than 265 2024-09-21T08:26:30.578 INFO:tasks.rados.rados.0.smithi137.stdout:645: oid 49 version 266 is already newer than 264 2024-09-21T08:26:30.578 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 266 (ObjNum 261 snap 56 seq_num 261) dirty exists 2024-09-21T08:26:30.578 INFO:tasks.rados.rados.0.smithi137.stdout:645: left oid 49 (ObjNum 261 snap 56 seq_num 261) 2024-09-21T08:26:30.581 INFO:tasks.rados.rados.0.smithi137.stdout:644: done (2 left) 2024-09-21T08:26:30.581 INFO:tasks.rados.rados.0.smithi137.stdout:645: done (1 left) 2024-09-21T08:26:30.581 INFO:tasks.rados.rados.0.smithi137.stdout:646: done (0 left) 2024-09-21T08:26:30.581 INFO:tasks.rados.rados.0.smithi137.stdout:647: write oid 7 current snap is 56 2024-09-21T08:26:30.581 INFO:tasks.rados.rados.0.smithi137.stdout:647: seq_num 263 ranges {420241=546489,1373823=544414} 2024-09-21T08:26:30.587 INFO:tasks.rados.rados.0.smithi137.stdout:647: writing smithi13779221-7 from 420241 to 966730 tid 1 2024-09-21T08:26:30.590 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:30.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:30.591 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:30.594 INFO:tasks.rados.rados.0.smithi137.stdout:647: writing smithi13779221-7 from 1373823 to 1918237 tid 2 2024-09-21T08:26:30.594 INFO:tasks.rados.rados.0.smithi137.stdout:648: rollback oid 25 current snap is 56 2024-09-21T08:26:30.595 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 25 to 55 2024-09-21T08:26:30.595 INFO:tasks.rados.rados.0.smithi137.stdout:647: finishing write tid 1 to smithi13779221-7 2024-09-21T08:26:30.595 INFO:tasks.rados.rados.0.smithi137.stdout:649: read oid 38 snap -1 2024-09-21T08:26:30.595 INFO:tasks.rados.rados.0.smithi137.stdout:649: expect (ObjNum 202 snap 40 seq_num 202) 2024-09-21T08:26:30.595 INFO:tasks.rados.rados.0.smithi137.stdout:650: copy_from oid 21 from oid 2 current snap is 56 2024-09-21T08:26:30.595 INFO:tasks.rados.rados.0.smithi137.stdout:651: copy_from oid 39 from oid 3 current snap is 56 2024-09-21T08:26:30.595 INFO:tasks.rados.rados.0.smithi137.stdout:652: snap_create 2024-09-21T08:26:30.596 INFO:tasks.rados.rados.0.smithi137.stdout:648: finishing rollback tid 1 to smithi13779221-25 2024-09-21T08:26:30.596 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 0 (ObjNum 129 snap 0 seq_num 3760687520) dirty dne 2024-09-21T08:26:30.602 INFO:tasks.rados.rados.0.smithi137.stdout:651: finishing copy_from racing read to smithi13779221-39 2024-09-21T08:26:30.608 INFO:tasks.rados.rados.0.smithi137.stdout:649: expect (ObjNum 202 snap 40 seq_num 202) 2024-09-21T08:26:30.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:30.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:30.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:30.714 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:30.768 INFO:tasks.rados.rados.0.smithi137.stdout:647: finishing write tid 2 to smithi13779221-7 2024-09-21T08:26:30.808 INFO:tasks.rados.rados.0.smithi137.stdout:647: finishing write tid 3 to smithi13779221-7 2024-09-21T08:26:30.809 INFO:tasks.rados.rados.0.smithi137.stdout:647: finishing write tid 4 to smithi13779221-7 2024-09-21T08:26:30.809 INFO:tasks.rados.rados.0.smithi137.stdout:647: oid 7 updating version 0 to 289 2024-09-21T08:26:30.809 INFO:tasks.rados.rados.0.smithi137.stdout:647: oid 7 version 289 is already newer than 288 2024-09-21T08:26:30.809 INFO:tasks.rados.rados.0.smithi137.stdout:647: oid 7 updating version 289 to 290 2024-09-21T08:26:30.809 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 290 (ObjNum 263 snap 56 seq_num 263) dirty exists 2024-09-21T08:26:30.809 INFO:tasks.rados.rados.0.smithi137.stdout:647: left oid 7 (ObjNum 263 snap 56 seq_num 263) 2024-09-21T08:26:30.809 INFO:tasks.rados.rados.0.smithi137.stdout:650: finishing copy_from racing read to smithi13779221-21 2024-09-21T08:26:30.809 INFO:tasks.rados.rados.0.smithi137.stdout:651: finishing copy_from to smithi13779221-39 2024-09-21T08:26:30.810 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 143 (ObjNum 220 snap 46 seq_num 220) dirty exists 2024-09-21T08:26:30.810 INFO:tasks.rados.rados.0.smithi137.stdout:650: finishing copy_from to smithi13779221-21 2024-09-21T08:26:30.810 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 99 (ObjNum 200 snap 40 seq_num 200) dirty exists 2024-09-21T08:26:30.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T08:26:30.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:30.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: 7.17 scrub starts 2024-09-21T08:26:30.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: 7.17 scrub ok 2024-09-21T08:26:30.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: 7.14 scrub starts 2024-09-21T08:26:30.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: 7.14 scrub ok 2024-09-21T08:26:30.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: 7.13 scrub starts 2024-09-21T08:26:30.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: 7.13 scrub ok 2024-09-21T08:26:30.833 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: osdmap e196: 8 total, 8 up, 4 in 2024-09-21T08:26:30.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/196264018' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.f", "id": [5, 7, 3, 4, 1, 2]}]: dispatch 2024-09-21T08:26:30.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: from='client.45041 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.f", "id": [5, 7, 3, 4, 1, 2]}]: dispatch 2024-09-21T08:26:30.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:30.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:30 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1875549361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:30.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:30.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:30.837 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:30.845 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:26:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:26:30.845 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: 7.17 scrub starts 2024-09-21T08:26:30.845 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: 7.17 scrub ok 2024-09-21T08:26:30.845 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: 7.14 scrub starts 2024-09-21T08:26:30.845 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: 7.14 scrub ok 2024-09-21T08:26:30.845 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: 7.13 scrub starts 2024-09-21T08:26:30.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: 7.13 scrub ok 2024-09-21T08:26:30.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: osdmap e196: 8 total, 8 up, 4 in 2024-09-21T08:26:30.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/196264018' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.f", "id": [5, 7, 3, 4, 1, 2]}]: dispatch 2024-09-21T08:26:30.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: from='client.45041 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.f", "id": [5, 7, 3, 4, 1, 2]}]: dispatch 2024-09-21T08:26:30.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:30.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1875549361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:30.846 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: 7.17 scrub starts 2024-09-21T08:26:30.846 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: 7.17 scrub ok 2024-09-21T08:26:30.846 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: 7.14 scrub starts 2024-09-21T08:26:30.846 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: 7.14 scrub ok 2024-09-21T08:26:30.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: 7.13 scrub starts 2024-09-21T08:26:30.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: 7.13 scrub ok 2024-09-21T08:26:30.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: osdmap e196: 8 total, 8 up, 4 in 2024-09-21T08:26:30.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/196264018' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.f", "id": [5, 7, 3, 4, 1, 2]}]: dispatch 2024-09-21T08:26:30.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: from='client.45041 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "7.f", "id": [5, 7, 3, 4, 1, 2]}]: dispatch 2024-09-21T08:26:30.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:30.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:30 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1875549361' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:30.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:30.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:30.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:30.961 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:31.059 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:31.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:31.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:31.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:31.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:31.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:31.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:31.208 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:31.208 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:31.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:31.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:31.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:31.332 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:31.405 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:26:31.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:31.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:31.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:31.455 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:31.512 INFO:tasks.rados.rados.0.smithi137.stdout:647: done (5 left) 2024-09-21T08:26:31.512 INFO:tasks.rados.rados.0.smithi137.stdout:648: done (4 left) 2024-09-21T08:26:31.512 INFO:tasks.rados.rados.0.smithi137.stdout:649: done (3 left) 2024-09-21T08:26:31.512 INFO:tasks.rados.rados.0.smithi137.stdout:650: done (2 left) 2024-09-21T08:26:31.512 INFO:tasks.rados.rados.0.smithi137.stdout:651: done (1 left) 2024-09-21T08:26:31.512 INFO:tasks.rados.rados.0.smithi137.stdout:652: done (0 left) 2024-09-21T08:26:31.512 INFO:tasks.rados.rados.0.smithi137.stdout:653: copy_from oid 1 from oid 42 current snap is 57 2024-09-21T08:26:31.513 INFO:tasks.rados.rados.0.smithi137.stdout:654: read oid 44 snap -1 2024-09-21T08:26:31.513 INFO:tasks.rados.rados.0.smithi137.stdout:654: expect (ObjNum 98 snap 19 seq_num 98) 2024-09-21T08:26:31.513 INFO:tasks.rados.rados.0.smithi137.stdout:655: snap_create 2024-09-21T08:26:31.518 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:26:31.518 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:26:31.518 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:26:31.518 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:26:31.522 INFO:tasks.rados.rados.0.smithi137.stdout:654: expect (ObjNum 98 snap 19 seq_num 98) 2024-09-21T08:26:31.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:31.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:31.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:31.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:31.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:31.683 INFO:tasks.rados.rados.0.smithi137.stdout:653: finishing copy_from to smithi13779221-1 2024-09-21T08:26:31.683 INFO:tasks.rados.rados.0.smithi137.stdout:653: got expected ENOENT (src dne) 2024-09-21T08:26:31.683 INFO:tasks.rados.rados.0.smithi137.stdout:653: finishing copy_from racing read to smithi13779221-1 2024-09-21T08:26:31.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:31.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:31.701 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: 4.10 scrub starts 2024-09-21T08:26:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: 4.10 scrub ok 2024-09-21T08:26:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: 7.15 scrub starts 2024-09-21T08:26:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: 7.15 scrub ok 2024-09-21T08:26:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: pgmap v200: 158 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 139 active+clean; 129 MiB data, 457 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: 7.2 scrub starts 2024-09-21T08:26:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: 7.2 scrub ok 2024-09-21T08:26:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: 5.1d scrub starts 2024-09-21T08:26:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: 5.1d scrub ok 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: from='client.45041 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.f", "id": [5, 7, 3, 4, 1, 2]}]': finished 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1875549361' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: osdmap e197: 8 total, 8 up, 4 in 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[29634]: osdmap e198: 8 total, 8 up, 4 in 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: 4.10 scrub starts 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: 4.10 scrub ok 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: 7.15 scrub starts 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: 7.15 scrub ok 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: pgmap v200: 158 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 139 active+clean; 129 MiB data, 457 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: 7.2 scrub starts 2024-09-21T08:26:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: 7.2 scrub ok 2024-09-21T08:26:31.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: 5.1d scrub starts 2024-09-21T08:26:31.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: 5.1d scrub ok 2024-09-21T08:26:31.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:26:31.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:26:31.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: from='client.45041 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.f", "id": [5, 7, 3, 4, 1, 2]}]': finished 2024-09-21T08:26:31.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1875549361' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:31.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: osdmap e197: 8 total, 8 up, 4 in 2024-09-21T08:26:31.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:31 smithi137 ceph-mon[41044]: osdmap e198: 8 total, 8 up, 4 in 2024-09-21T08:26:31.801 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:31.813 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: 4.10 scrub starts 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: 4.10 scrub ok 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: 7.15 scrub starts 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: 7.15 scrub ok 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: pgmap v200: 158 pgs: 5 active+clean+snaptrim, 14 active+clean+snaptrim_wait, 139 active+clean; 129 MiB data, 457 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: 7.2 scrub starts 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: 7.2 scrub ok 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: 5.1d scrub starts 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: 5.1d scrub ok 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: from='client.45041 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "7.f", "id": [5, 7, 3, 4, 1, 2]}]': finished 2024-09-21T08:26:31.814 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1875549361' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: osdmap e197: 8 total, 8 up, 4 in 2024-09-21T08:26:31.815 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:31 smithi165 ceph-mon[32893]: osdmap e198: 8 total, 8 up, 4 in 2024-09-21T08:26:31.823 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:31.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:31.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:31.925 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:31.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:31.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:31.948 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:31.993 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:32.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:32.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:32.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:32.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:32.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T08:26:32.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:32.204 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:32.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:32.205 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:32.306 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:32.328 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:32.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:32.329 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:32.390 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:32.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:32.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:32.453 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:32.453 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:32.497 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:32.498 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:32.511 INFO:tasks.rados.rados.0.smithi137.stdout:653: done (2 left) 2024-09-21T08:26:32.511 INFO:tasks.rados.rados.0.smithi137.stdout:654: done (1 left) 2024-09-21T08:26:32.511 INFO:tasks.rados.rados.0.smithi137.stdout:655: done (0 left) 2024-09-21T08:26:32.512 INFO:tasks.rados.rados.0.smithi137.stdout:656: snap_create 2024-09-21T08:26:32.516 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:32.516 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:32.531 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:32.532 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:32.547 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:32.547 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:32.554 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:32.565 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:32.565 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:32.576 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:32.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:32.577 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:32.581 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:32.582 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:32.599 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:32.600 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:32.615 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:32.616 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:32.677 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:32.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:32.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:32.700 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:32 smithi165 ceph-mon[32893]: 7.5 scrub starts 2024-09-21T08:26:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:32 smithi165 ceph-mon[32893]: 7.5 scrub ok 2024-09-21T08:26:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:32 smithi165 ceph-mon[32893]: 7.3 deep-scrub starts 2024-09-21T08:26:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:32 smithi165 ceph-mon[32893]: 7.3 deep-scrub ok 2024-09-21T08:26:32.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:32 smithi165 ceph-mon[32893]: 7.9 scrub starts 2024-09-21T08:26:32.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:32 smithi165 ceph-mon[32893]: 7.9 scrub ok 2024-09-21T08:26:32.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:32 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3148065946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:32.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:32 smithi165 ceph-mon[32893]: osdmap e199: 8 total, 8 up, 4 in 2024-09-21T08:26:32.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[29634]: 7.5 scrub starts 2024-09-21T08:26:32.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[29634]: 7.5 scrub ok 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[29634]: 7.3 deep-scrub starts 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[29634]: 7.3 deep-scrub ok 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[29634]: 7.9 scrub starts 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[29634]: 7.9 scrub ok 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3148065946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[29634]: osdmap e199: 8 total, 8 up, 4 in 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[41044]: 7.5 scrub starts 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[41044]: 7.5 scrub ok 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[41044]: 7.3 deep-scrub starts 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[41044]: 7.3 deep-scrub ok 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[41044]: 7.9 scrub starts 2024-09-21T08:26:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[41044]: 7.9 scrub ok 2024-09-21T08:26:32.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3148065946' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:32.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:32 smithi137 ceph-mon[41044]: osdmap e199: 8 total, 8 up, 4 in 2024-09-21T08:26:32.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:32.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:32.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:32.824 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:32.918 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:32.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:32.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:32.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:32.948 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:33.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:33.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:33.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:33.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:33.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:33.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:26:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:33.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:33.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:33.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:33.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:33.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:33.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:33.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:33.445 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:33.445 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:33.453 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:33.513 INFO:tasks.rados.rados.0.smithi137.stdout:656: done (0 left) 2024-09-21T08:26:33.513 INFO:tasks.rados.rados.0.smithi137.stdout:657: read oid 27 snap -1 2024-09-21T08:26:33.513 INFO:tasks.rados.rados.0.smithi137.stdout:657: expect (ObjNum 193 snap 39 seq_num 193) 2024-09-21T08:26:33.513 INFO:tasks.rados.rados.0.smithi137.stdout:658: write oid 49 current snap is 59 2024-09-21T08:26:33.513 INFO:tasks.rados.rados.0.smithi137.stdout:658: seq_num 264 ranges {437048=621738,1832305=404425} 2024-09-21T08:26:33.518 INFO:tasks.rados.rados.0.smithi137.stdout:658: writing smithi13779221-49 from 437048 to 1058786 tid 1 2024-09-21T08:26:33.526 INFO:tasks.rados.rados.0.smithi137.stdout:658: writing smithi13779221-49 from 1832305 to 2236730 tid 2 2024-09-21T08:26:33.526 INFO:tasks.rados.rados.0.smithi137.stdout:659: delete oid 10 current snap is 59 2024-09-21T08:26:33.527 INFO:tasks.rados.rados.0.smithi137.stdout:659: done (2 left) 2024-09-21T08:26:33.527 INFO:tasks.rados.rados.0.smithi137.stdout:660: rollback oid 15 current snap is 59 2024-09-21T08:26:33.527 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 15 to 56 2024-09-21T08:26:33.527 INFO:tasks.rados.rados.0.smithi137.stdout:661: snap_remove snap 58 2024-09-21T08:26:33.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:33.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:33.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:33.570 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:33.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T08:26:33.670 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:33.692 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:33.692 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:33.692 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:33.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:33.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:33.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:33.816 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:33.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:33 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:33.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:33 smithi137 ceph-mon[29634]: pgmap v203: 158 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 147 active+clean; 143 MiB data, 489 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:33.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:33 smithi137 ceph-mon[29634]: 7.1 scrub starts 2024-09-21T08:26:33.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:33 smithi137 ceph-mon[29634]: 7.1 scrub ok 2024-09-21T08:26:33.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:33 smithi137 ceph-mon[29634]: osdmap e200: 8 total, 8 up, 4 in 2024-09-21T08:26:33.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:33 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:33.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:33 smithi137 ceph-mon[41044]: pgmap v203: 158 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 147 active+clean; 143 MiB data, 489 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:33.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:33 smithi137 ceph-mon[41044]: 7.1 scrub starts 2024-09-21T08:26:33.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:33 smithi137 ceph-mon[41044]: 7.1 scrub ok 2024-09-21T08:26:33.833 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:33 smithi137 ceph-mon[41044]: osdmap e200: 8 total, 8 up, 4 in 2024-09-21T08:26:33.869 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:33.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:33.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:33.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:33.939 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:33.990 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:33 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:33.990 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:33 smithi165 ceph-mon[32893]: pgmap v203: 158 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 147 active+clean; 143 MiB data, 489 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:33.990 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:33 smithi165 ceph-mon[32893]: 7.1 scrub starts 2024-09-21T08:26:33.990 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:33 smithi165 ceph-mon[32893]: 7.1 scrub ok 2024-09-21T08:26:33.990 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:33 smithi165 ceph-mon[32893]: osdmap e200: 8 total, 8 up, 4 in 2024-09-21T08:26:34.040 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:34.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:34.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:34.062 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:34.163 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:34.185 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:34.185 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:34.185 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:34.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:34.294 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:34.294 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:34.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:34.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:34.308 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:34.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:34.430 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:34.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:34.431 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:34.511 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:26:34.516 INFO:tasks.rados.rados.0.smithi137.stdout:661: done (3 left) 2024-09-21T08:26:34.516 INFO:tasks.rados.rados.0.smithi137.stdout:662: read oid 33 snap -1 2024-09-21T08:26:34.516 INFO:tasks.rados.rados.0.smithi137.stdout:662: expect (ObjNum 228 snap 47 seq_num 228) 2024-09-21T08:26:34.516 INFO:tasks.rados.rados.0.smithi137.stdout:663: snap_remove snap 57 2024-09-21T08:26:34.531 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:34.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:34.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:34.554 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:34.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:34.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:34.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:34.678 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: 3.19 deep-scrub starts 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: 3.19 deep-scrub ok 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: 3.1a scrub starts 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: 3.1a scrub ok 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: 7.16 scrub starts 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: 7.16 scrub ok 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1042839754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1042839754' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:26:34.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[29634]: osdmap e201: 8 total, 8 up, 4 in 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: 3.19 deep-scrub starts 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: 3.19 deep-scrub ok 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: 3.1a scrub starts 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: 3.1a scrub ok 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: 7.16 scrub starts 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: 7.16 scrub ok 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1042839754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1042839754' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:26:34.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:34 smithi137 ceph-mon[41044]: osdmap e201: 8 total, 8 up, 4 in 2024-09-21T08:26:34.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:34.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:34.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:34.801 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:34.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:34.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:34.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:34.925 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: 3.19 deep-scrub starts 2024-09-21T08:26:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: 3.19 deep-scrub ok 2024-09-21T08:26:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: 3.1a scrub starts 2024-09-21T08:26:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: 3.1a scrub ok 2024-09-21T08:26:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: 7.16 scrub starts 2024-09-21T08:26:34.948 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: 7.16 scrub ok 2024-09-21T08:26:34.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:26:34.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:26:34.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1042839754' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:34.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:34.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1042839754' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:26:34.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:34 smithi165 ceph-mon[32893]: osdmap e201: 8 total, 8 up, 4 in 2024-09-21T08:26:34.975 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T08:26:35.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:35.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:35.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:35.048 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:35.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:35.171 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:35.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:35.172 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:35.217 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:35.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:35.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:35.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:35.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:35.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:35.417 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:35.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:35.418 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:35.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:26:35.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:35.524 INFO:tasks.rados.rados.0.smithi137.stdout:663: done (4 left) 2024-09-21T08:26:35.524 INFO:tasks.rados.rados.0.smithi137.stdout:664: read oid 40 snap -1 2024-09-21T08:26:35.524 INFO:tasks.rados.rados.0.smithi137.stdout:664: expect (ObjNum 211 snap 43 seq_num 211) 2024-09-21T08:26:35.524 INFO:tasks.rados.rados.0.smithi137.stdout:665: copy_from oid 39 from oid 47 current snap is 59 2024-09-21T08:26:35.524 INFO:tasks.rados.rados.0.smithi137.stdout:666: copy_from oid 5 from oid 29 current snap is 59 2024-09-21T08:26:35.524 INFO:tasks.rados.rados.0.smithi137.stdout:667: copy_from oid 4 from oid 1 current snap is 59 2024-09-21T08:26:35.524 INFO:tasks.rados.rados.0.smithi137.stdout:668: write (excl) oid 34 current snap is 59 2024-09-21T08:26:35.524 INFO:tasks.rados.rados.0.smithi137.stdout:668: seq_num 266 ranges {470662=772236,1832916=408094} 2024-09-21T08:26:35.530 INFO:tasks.rados.rados.0.smithi137.stdout:668: writing smithi13779221-34 from 470662 to 1242898 tid 1 2024-09-21T08:26:35.538 INFO:tasks.rados.rados.0.smithi137.stdout:668: writing smithi13779221-34 from 1832916 to 2241010 tid 2 2024-09-21T08:26:35.539 INFO:tasks.rados.rados.0.smithi137.stdout:657: expect (ObjNum 193 snap 39 seq_num 193) 2024-09-21T08:26:35.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:35.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:35.554 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:35.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:35 smithi165 ceph-mon[32893]: 4.c scrub starts 2024-09-21T08:26:35.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:35 smithi165 ceph-mon[32893]: 4.c scrub ok 2024-09-21T08:26:35.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:35 smithi165 ceph-mon[32893]: pgmap v206: 158 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 147 active+clean; 143 MiB data, 489 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:35.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:35 smithi165 ceph-mon[32893]: 7.4 deep-scrub starts 2024-09-21T08:26:35.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:35 smithi165 ceph-mon[32893]: 7.4 deep-scrub ok 2024-09-21T08:26:35.610 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:35 smithi165 ceph-mon[32893]: osdmap e202: 8 total, 8 up, 4 in 2024-09-21T08:26:35.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:35.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:35.677 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:35.677 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:35.706 INFO:tasks.rados.rados.0.smithi137.stdout:657: done (8 left) 2024-09-21T08:26:35.706 INFO:tasks.rados.rados.0.smithi137.stdout:669: snap_create 2024-09-21T08:26:35.707 INFO:tasks.rados.rados.0.smithi137.stdout:658: finishing write tid 1 to smithi13779221-49 2024-09-21T08:26:35.707 INFO:tasks.rados.rados.0.smithi137.stdout:660: finishing rollback tid 1 to smithi13779221-15 2024-09-21T08:26:35.707 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 0 (ObjNum 129 snap 0 seq_num 3760760768) dirty dne 2024-09-21T08:26:35.707 INFO:tasks.rados.rados.0.smithi137.stdout:658: finishing write tid 2 to smithi13779221-49 2024-09-21T08:26:35.707 INFO:tasks.rados.rados.0.smithi137.stdout:658: finishing write tid 3 to smithi13779221-49 2024-09-21T08:26:35.707 INFO:tasks.rados.rados.0.smithi137.stdout:658: finishing write tid 4 to smithi13779221-49 2024-09-21T08:26:35.707 INFO:tasks.rados.rados.0.smithi137.stdout:658: oid 49 updating version 0 to 270 2024-09-21T08:26:35.707 INFO:tasks.rados.rados.0.smithi137.stdout:658: oid 49 version 270 is already newer than 269 2024-09-21T08:26:35.707 INFO:tasks.rados.rados.0.smithi137.stdout:658: oid 49 version 270 is already newer than 268 2024-09-21T08:26:35.707 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 270 (ObjNum 264 snap 59 seq_num 264) dirty exists 2024-09-21T08:26:35.708 INFO:tasks.rados.rados.0.smithi137.stdout:658: left oid 49 (ObjNum 264 snap 59 seq_num 264) 2024-09-21T08:26:35.708 INFO:tasks.rados.rados.0.smithi137.stdout:662: expect (ObjNum 228 snap 47 seq_num 228) 2024-09-21T08:26:35.717 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:35.717 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:35.723 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:35.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[29634]: 4.c scrub starts 2024-09-21T08:26:35.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[29634]: 4.c scrub ok 2024-09-21T08:26:35.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[29634]: pgmap v206: 158 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 147 active+clean; 143 MiB data, 489 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:35.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[29634]: 7.4 deep-scrub starts 2024-09-21T08:26:35.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[29634]: 7.4 deep-scrub ok 2024-09-21T08:26:35.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[29634]: osdmap e202: 8 total, 8 up, 4 in 2024-09-21T08:26:35.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[41044]: 4.c scrub starts 2024-09-21T08:26:35.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[41044]: 4.c scrub ok 2024-09-21T08:26:35.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[41044]: pgmap v206: 158 pgs: 5 active+clean+snaptrim, 6 active+clean+snaptrim_wait, 147 active+clean; 143 MiB data, 489 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:35.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[41044]: 7.4 deep-scrub starts 2024-09-21T08:26:35.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[41044]: 7.4 deep-scrub ok 2024-09-21T08:26:35.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:35 smithi137 ceph-mon[41044]: osdmap e202: 8 total, 8 up, 4 in 2024-09-21T08:26:35.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:35.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:35.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:35.800 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:35.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:35.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:35.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:35.923 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:35.969 INFO:tasks.rados.rados.0.smithi137.stdout:666: finishing copy_from racing read to smithi13779221-5 2024-09-21T08:26:35.969 INFO:tasks.rados.rados.0.smithi137.stdout:666: finishing copy_from to smithi13779221-5 2024-09-21T08:26:35.969 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 273 (ObjNum 250 snap 53 seq_num 250) dirty exists 2024-09-21T08:26:35.969 INFO:tasks.rados.rados.0.smithi137.stdout:664: expect (ObjNum 211 snap 43 seq_num 211) 2024-09-21T08:26:36.024 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:36.045 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:36.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:36.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:36.122 INFO:tasks.rados.rados.0.smithi137.stdout:667: finishing copy_from racing read to smithi13779221-4 2024-09-21T08:26:36.122 INFO:tasks.rados.rados.0.smithi137.stdout:668: finishing write tid 1 to smithi13779221-34 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:668: finishing write tid 2 to smithi13779221-34 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:668: finishing write tid 3 to smithi13779221-34 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:668: finishing write tid 4 to smithi13779221-34 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:668: oid 34 updating version 0 to 276 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:668: oid 34 updating version 276 to 277 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:668: oid 34 version 277 is already newer than 275 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 277 (ObjNum 266 snap 59 seq_num 266) dirty exists 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:668: left oid 34 (ObjNum 266 snap 59 seq_num 266) 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:665: finishing copy_from racing read to smithi13779221-39 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:667: finishing copy_from to smithi13779221-4 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 245 (ObjNum 231 snap 47 seq_num 231) dirty exists 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:665: finishing copy_from to smithi13779221-39 2024-09-21T08:26:36.123 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:26:36.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:36.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:36.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:36.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:36.190 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:36.190 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:36.206 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:36.206 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:36.222 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:36.222 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:36.237 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:36.238 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:36.254 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:36.254 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:36.269 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:36.270 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:36.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:36.286 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:36.286 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:36.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:36.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:36.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:36.302 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:36.302 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:36.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T08:26:36.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:36.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:36.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:36.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:36.517 INFO:tasks.rados.rados.0.smithi137.stdout:658: done (8 left) 2024-09-21T08:26:36.517 INFO:tasks.rados.rados.0.smithi137.stdout:660: done (7 left) 2024-09-21T08:26:36.517 INFO:tasks.rados.rados.0.smithi137.stdout:662: done (6 left) 2024-09-21T08:26:36.517 INFO:tasks.rados.rados.0.smithi137.stdout:664: done (5 left) 2024-09-21T08:26:36.518 INFO:tasks.rados.rados.0.smithi137.stdout:665: done (4 left) 2024-09-21T08:26:36.518 INFO:tasks.rados.rados.0.smithi137.stdout:666: done (3 left) 2024-09-21T08:26:36.518 INFO:tasks.rados.rados.0.smithi137.stdout:667: done (2 left) 2024-09-21T08:26:36.518 INFO:tasks.rados.rados.0.smithi137.stdout:668: done (1 left) 2024-09-21T08:26:36.518 INFO:tasks.rados.rados.0.smithi137.stdout:669: done (0 left) 2024-09-21T08:26:36.518 INFO:tasks.rados.rados.0.smithi137.stdout:670: write (excl) oid 24 current snap is 60 2024-09-21T08:26:36.518 INFO:tasks.rados.rados.0.smithi137.stdout:670: seq_num 267 ranges {487469=563838,1607751=551752} 2024-09-21T08:26:36.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:36.522 INFO:tasks.rados.rados.0.smithi137.stdout:670: writing smithi13779221-24 from 487469 to 1051307 tid 1 2024-09-21T08:26:36.529 INFO:tasks.rados.rados.0.smithi137.stdout:670: writing smithi13779221-24 from 1607751 to 2159503 tid 2 2024-09-21T08:26:36.529 INFO:tasks.rados.rados.0.smithi137.stdout:671: snap_remove snap 56 2024-09-21T08:26:36.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:36.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:36.546 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:36.639 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:36.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:36.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:36.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:36.669 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:36.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:36 smithi165 ceph-mon[32893]: 3.1d scrub starts 2024-09-21T08:26:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:36 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/343393191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:36.761 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:36 smithi165 ceph-mon[32893]: osdmap e203: 8 total, 8 up, 4 in 2024-09-21T08:26:36.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:36.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:36.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:36.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:36.887 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 3, 6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:26:36.887 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:26:36.887 INFO:tasks.thrashosds.thrasher:Reweighting osd 7 to 0.980429076411602 2024-09-21T08:26:36.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd reweight 7 0.980429076411602 2024-09-21T08:26:36.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:36.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:36 smithi137 ceph-mon[29634]: 3.1d scrub starts 2024-09-21T08:26:36.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:36 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/343393191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:36.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:36 smithi137 ceph-mon[29634]: osdmap e203: 8 total, 8 up, 4 in 2024-09-21T08:26:36.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:36 smithi137 ceph-mon[41044]: 3.1d scrub starts 2024-09-21T08:26:36.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:36 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/343393191' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:36.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:36 smithi137 ceph-mon[41044]: osdmap e203: 8 total, 8 up, 4 in 2024-09-21T08:26:36.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:36.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:36.919 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:37.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:37.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:37.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:37.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:37.138 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:37.142 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:26:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:26:37.142 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:26:37.143 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:26:37.143 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:26:37.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:37.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:37.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:37.166 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:37.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:26:37.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:37.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:37.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:37.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:37.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:37.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:37.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:37.411 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:37.512 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:37.513 INFO:tasks.rados.rados.0.smithi137.stdout:671: done (1 left) 2024-09-21T08:26:37.513 INFO:tasks.rados.rados.0.smithi137.stdout:672: write oid 29 current snap is 60 2024-09-21T08:26:37.513 INFO:tasks.rados.rados.0.smithi137.stdout:670: finishing write tid 1 to smithi13779221-24 2024-09-21T08:26:37.513 INFO:tasks.rados.rados.0.smithi137.stdout:672: seq_num 268 ranges {504276=639087,1666233=695410} 2024-09-21T08:26:37.518 INFO:tasks.rados.rados.0.smithi137.stdout:672: writing smithi13779221-29 from 504276 to 1143363 tid 1 2024-09-21T08:26:37.526 INFO:tasks.rados.rados.0.smithi137.stdout:672: writing smithi13779221-29 from 1666233 to 2361643 tid 2 2024-09-21T08:26:37.526 INFO:tasks.rados.rados.0.smithi137.stdout:673: delete oid 16 current snap is 60 2024-09-21T08:26:37.526 INFO:tasks.rados.rados.0.smithi137.stdout:670: finishing write tid 2 to smithi13779221-24 2024-09-21T08:26:37.526 INFO:tasks.rados.rados.0.smithi137.stdout:670: finishing write tid 3 to smithi13779221-24 2024-09-21T08:26:37.526 INFO:tasks.rados.rados.0.smithi137.stdout:670: finishing write tid 4 to smithi13779221-24 2024-09-21T08:26:37.527 INFO:tasks.rados.rados.0.smithi137.stdout:670: oid 24 updating version 0 to 226 2024-09-21T08:26:37.527 INFO:tasks.rados.rados.0.smithi137.stdout:670: oid 24 updating version 226 to 228 2024-09-21T08:26:37.527 INFO:tasks.rados.rados.0.smithi137.stdout:670: oid 24 version 228 is already newer than 227 2024-09-21T08:26:37.527 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 228 (ObjNum 267 snap 60 seq_num 267) dirty exists 2024-09-21T08:26:37.527 INFO:tasks.rados.rados.0.smithi137.stdout:670: left oid 24 (ObjNum 267 snap 60 seq_num 267) 2024-09-21T08:26:37.529 INFO:tasks.rados.rados.0.smithi137.stdout:672: finishing write tid 1 to smithi13779221-29 2024-09-21T08:26:37.533 INFO:tasks.rados.rados.0.smithi137.stdout:672: finishing write tid 2 to smithi13779221-29 2024-09-21T08:26:37.533 INFO:tasks.rados.rados.0.smithi137.stdout:672: finishing write tid 3 to smithi13779221-29 2024-09-21T08:26:37.534 INFO:tasks.rados.rados.0.smithi137.stdout:672: finishing write tid 4 to smithi13779221-29 2024-09-21T08:26:37.534 INFO:tasks.rados.rados.0.smithi137.stdout:672: oid 29 updating version 0 to 296 2024-09-21T08:26:37.534 INFO:tasks.rados.rados.0.smithi137.stdout:672: oid 29 updating version 296 to 297 2024-09-21T08:26:37.534 INFO:tasks.rados.rados.0.smithi137.stdout:672: oid 29 updating version 297 to 298 2024-09-21T08:26:37.534 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 298 (ObjNum 268 snap 60 seq_num 268) dirty exists 2024-09-21T08:26:37.535 INFO:tasks.rados.rados.0.smithi137.stdout:672: left oid 29 (ObjNum 268 snap 60 seq_num 268) 2024-09-21T08:26:37.535 INFO:tasks.rados.rados.0.smithi137.stdout:670: done (2 left) 2024-09-21T08:26:37.535 INFO:tasks.rados.rados.0.smithi137.stdout:672: done (1 left) 2024-09-21T08:26:37.535 INFO:tasks.rados.rados.0.smithi137.stdout:673: done (0 left) 2024-09-21T08:26:37.535 INFO:tasks.rados.rados.0.smithi137.stdout:674: rollback oid 6 current snap is 60 2024-09-21T08:26:37.535 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 6 to 55 2024-09-21T08:26:37.535 INFO:tasks.rados.rados.0.smithi137.stdout:675: write oid 35 current snap is 60 2024-09-21T08:26:37.535 INFO:tasks.rados.rados.0.smithi137.stdout:675: seq_num 270 ranges {537890=789585,1783197=699079} 2024-09-21T08:26:37.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:37.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:37.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:37.541 INFO:tasks.rados.rados.0.smithi137.stdout:675: writing smithi13779221-35 from 537890 to 1327475 tid 1 2024-09-21T08:26:37.548 INFO:tasks.rados.rados.0.smithi137.stdout:675: writing smithi13779221-35 from 1783197 to 2482276 tid 2 2024-09-21T08:26:37.548 INFO:tasks.rados.rados.0.smithi137.stdout:676: snap_create 2024-09-21T08:26:37.548 INFO:tasks.rados.rados.0.smithi137.stdout:674: finishing rollback tid 0 to smithi13779221-6 2024-09-21T08:26:37.548 INFO:tasks.rados.rados.0.smithi137.stdout:674: finishing rollback tid 1 to smithi13779221-6 2024-09-21T08:26:37.548 INFO:tasks.rados.rados.0.smithi137.stdout:674: finishing rollback tid 2 to smithi13779221-6 2024-09-21T08:26:37.548 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 48 (ObjNum 191 snap 39 seq_num 191) dirty exists 2024-09-21T08:26:37.549 INFO:tasks.rados.rados.0.smithi137.stdout:675: finishing write tid 1 to smithi13779221-35 2024-09-21T08:26:37.550 INFO:tasks.rados.rados.0.smithi137.stdout:675: finishing write tid 2 to smithi13779221-35 2024-09-21T08:26:37.551 INFO:tasks.rados.rados.0.smithi137.stdout:675: finishing write tid 3 to smithi13779221-35 2024-09-21T08:26:37.551 INFO:tasks.rados.rados.0.smithi137.stdout:675: finishing write tid 4 to smithi13779221-35 2024-09-21T08:26:37.551 INFO:tasks.rados.rados.0.smithi137.stdout:675: oid 35 updating version 0 to 230 2024-09-21T08:26:37.551 INFO:tasks.rados.rados.0.smithi137.stdout:675: oid 35 version 230 is already newer than 229 2024-09-21T08:26:37.551 INFO:tasks.rados.rados.0.smithi137.stdout:675: oid 35 updating version 230 to 231 2024-09-21T08:26:37.551 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 231 (ObjNum 270 snap 60 seq_num 270) dirty exists 2024-09-21T08:26:37.551 INFO:tasks.rados.rados.0.smithi137.stdout:675: left oid 35 (ObjNum 270 snap 60 seq_num 270) 2024-09-21T08:26:37.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:37.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:37.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:37.669 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:37.755 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:37 smithi165 ceph-mon[32893]: pgmap v209: 158 pgs: 158 active+clean; 140 MiB data, 473 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:37.755 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:37 smithi165 ceph-mon[32893]: osdmap e204: 8 total, 8 up, 4 in 2024-09-21T08:26:37.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:37.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T08:26:37.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:37.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:37.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:37.791 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:37 smithi137 ceph-mon[29634]: pgmap v209: 158 pgs: 158 active+clean; 140 MiB data, 473 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:37.792 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:37 smithi137 ceph-mon[29634]: osdmap e204: 8 total, 8 up, 4 in 2024-09-21T08:26:37.792 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:37 smithi137 ceph-mon[41044]: pgmap v209: 158 pgs: 158 active+clean; 140 MiB data, 473 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:37.792 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:37 smithi137 ceph-mon[41044]: osdmap e204: 8 total, 8 up, 4 in 2024-09-21T08:26:37.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:37.909 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:37.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:37.910 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:38.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:38.028 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:26:38.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:38.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:38.032 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:38.133 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:38.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:38.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:38.156 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:38.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:38.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:38.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:38.278 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:38.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:38.400 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:38.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:38.401 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:38.484 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1086 5.9 5:93e5b521:::notify.7:head [watch reconnect cookie 94837804044672 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159598+0000", 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 120.322541081, 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40422385100000002, 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:38.485 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1086 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159598+0000", 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159598+0000", 2024-09-21T08:26:38.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159601+0000", 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2270000000000002e-06 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159611+0000", 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9590000000000007e-06 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159634+0000", 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2931000000000001e-05 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159651+0000", 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6959000000000001e-05 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105437 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159772+0000", 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5319e-05 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559521+0000", 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39974934899999998 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559838+0000", 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031693299999999998 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.561288+0000", 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014493150000000001 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.561373+0000", 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4902999999999997e-05 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.563822+0000", 2024-09-21T08:26:38.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024495189999999998 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:277 7.5 7:a6c204bd:::smithi13779221-42:head [copy-from ver 6] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.588389+0000", 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 111.893750704, 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013364199, 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.491 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 277 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588389+0000", 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588389+0000", 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588391+0000", 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2759999999999999e-06 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588399+0000", 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5620000000000004e-06 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588413+0000", 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3451e-05 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588420+0000", 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6769999999999998e-06 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588486+0000", 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6549000000000003e-05 2024-09-21T08:26:38.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.588587+0000", 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000100789 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.592068+0000", 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034810430000000001 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.594819+0000", 2024-09-21T08:26:38.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027511670000000001 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.598876+0000", 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040573500000000004 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.601685+0000", 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028083639999999998 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.601720+0000", 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.54e-05 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.601753+0000", 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2570999999999997e-05 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.497 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:343 7.5 7:aa5ad64c:::smithi13779221-49:head [read 0~2806736,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567068+0000", 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 109.915071477, 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010394037, 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 343 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567068+0000", 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567068+0000", 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567069+0000", 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7899999999999998e-07 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567073+0000", 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5569999999999997e-06 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567079+0000", 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0900000000000001e-06 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567084+0000", 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2739999999999998e-06 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.574117+0000", 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070326829999999996 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.574229+0000", 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111501 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.577462+0000", 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032332530000000002 2024-09-21T08:26:38.501 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:353 7.c e96/94 7:361ee62b:::smithi13779221-40:head v 96'14, mlcod=93'11)", 2024-09-21T08:26:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.908212+0000", 2024-09-21T08:26:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 108.573926919, 2024-09-21T08:26:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.008046826, 2024-09-21T08:26:38.502 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.908212+0000", 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.908212+0000", 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.503 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.908213+0000", 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4700000000000002e-07 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.909706+0000", 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001493316 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.909725+0000", 2024-09-21T08:26:38.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8861000000000001e-05 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.909735+0000", 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0287e-05 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.909808+0000", 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2986999999999998e-05 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.909833+0000", 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5256e-05 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916211+0000", 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063771970000000002 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916259+0000", 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8575000000000002e-05 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.506 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:592 7.5 7:aaa5c288:::smithi13779221-20:head [copy-from ver 40] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:10.738788+0000", 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.743351535000002, 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015504372000000001, 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.507 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 592 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738788+0000", 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738788+0000", 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738790+0000", 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7349999999999999e-06 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738795+0000", 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.271e-06 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738807+0000", 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1919000000000001e-05 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738829+0000", 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2436999999999999e-05 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738881+0000", 2024-09-21T08:26:38.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2021000000000002e-05 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738943+0000", 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1517000000000001e-05 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.744334+0000", 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053909250000000004 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:38.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.747951+0000", 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036174990000000001 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.752148+0000", 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041967590000000004 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.754233+0000", 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020854340000000002 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.512 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.754266+0000", 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2697000000000002e-05 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.754292+0000", 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6157999999999999e-05 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.513 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:593 7.5 7:aaa5c288:::smithi13779221-20:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:26:38.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:38.514 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:26:38.515 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:10.738842+0000", 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 87.743297097999999, 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0083558569999999995, 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 593 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738842+0000", 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738842+0000", 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738843+0000", 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3099999999999998e-07 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738845+0000", 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.03e-06 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738849+0000", 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9790000000000004e-06 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.738852+0000", 2024-09-21T08:26:38.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.061e-06 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.739056+0000", 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000203895 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.739087+0000", 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1112000000000003e-05 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.747035+0000", 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0079478599999999993 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.747112+0000", 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7483000000000003e-05 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:10.747198+0000", 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5906e-05 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.520 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:974 7.5 7:a9dd6c7d:::smithi13779221-47:head [copy-from ver 121] snapc 20=[20,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e144)", 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:37.754220+0000", 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 60.727919104999998, 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026134228999999998, 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.521 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 974 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754220+0000", 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754220+0000", 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.522 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754222+0000", 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2819999999999999e-06 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754232+0000", 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9620000000000005e-06 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754248+0000", 2024-09-21T08:26:38.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5074e-05 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754256+0000", 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4710000000000007e-06 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754326+0000", 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0407999999999994e-05 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754336+0000", 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7010000000000008e-06 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.756699+0000", 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002362655 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.756796+0000", 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7063000000000002e-05 2024-09-21T08:26:38.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.771644+0000", 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014848309000000001 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.775122+0000", 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003477787 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.780106+0000", 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049839280000000003 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.780246+0000", 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013988499999999999 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.780304+0000", 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8455999999999997e-05 2024-09-21T08:26:38.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.780354+0000", 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0247999999999997e-05 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:975 7.5 7:a9dd6c7d:::smithi13779221-47:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e144)", 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:37.754276+0000", 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 60.727862868999999, 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020233128, 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.528 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 975 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754276+0000", 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754276+0000", 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754277+0000", 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3000000000000001e-07 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754279+0000", 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1210000000000001e-06 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754284+0000", 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2959999999999998e-06 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754288+0000", 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.439e-06 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754386+0000", 2024-09-21T08:26:38.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7777999999999998e-05 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.754389+0000", 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8e-06 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.756916+0000", 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002526169 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.756943+0000", 2024-09-21T08:26:38.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7365e-05 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.774354+0000", 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017411045999999999 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.774427+0000", 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2831999999999997e-05 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:37.774510+0000", 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2751999999999996e-05 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:1051 7.8 e147/94 7:1fd29771:::smithi13779221-22:head v 147'111, mlcod=147'107)", 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:41.124824+0000", 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 57.357315376999999, 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0091017519999999994, 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.124824+0000", 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.124824+0000", 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.124825+0000", 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4169999999999999e-06 2024-09-21T08:26:38.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.129584+0000", 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047582309999999999 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.129597+0000", 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3233e-05 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.129604+0000", 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9090000000000003e-06 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.129653+0000", 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9456000000000002e-05 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.129668+0000", 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4401e-05 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.133892+0000", 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042240419999999999 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:41.133926+0000", 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4063000000000002e-05 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.538 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1179 7.5 7:a9dd6c7d:::smithi13779221-47:head [write 692488~638288] snapc 26=[26,25] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:51.528932+0000", 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.953207751000001, 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0085503049999999994, 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1179 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.528932+0000", 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.528932+0000", 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.528935+0000", 2024-09-21T08:26:38.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5700000000000001e-06 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.529878+0000", 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000942632 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.529895+0000", 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7689e-05 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.541 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.529907+0000", 2024-09-21T08:26:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1487e-05 2024-09-21T08:26:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.530024+0000", 2024-09-21T08:26:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117207 2024-09-21T08:26:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.530096+0000", 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2162000000000005e-05 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.530218+0000", 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122065 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.531697+0000", 2024-09-21T08:26:38.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014788399999999999 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.537383+0000", 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056853090000000004 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.537417+0000", 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4870000000000003e-05 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.537450+0000", 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2903e-05 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.537482+0000", 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1571e-05 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.545 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:1198 7.6 e158/94 7:611b1c4b:::smithi13779221-3:head v 158'69, mlcod=151'65)", 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:51.677070+0000", 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 46.805069240999998, 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093777240000000005, 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.677070+0000", 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.677070+0000", 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.677071+0000", 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2799999999999995e-07 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.681701+0000", 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0046298340000000002 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.681716+0000", 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5733e-05 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.681725+0000", 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1880000000000002e-06 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.681782+0000", 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7800999999999998e-05 2024-09-21T08:26:38.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.681800+0000", 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7226000000000001e-05 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.686397+0000", 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045969390000000004 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.686448+0000", 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1174999999999997e-05 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1392 7.c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 18] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:05.561950+0000", 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.920188901000003, 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014185672999999999, 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.550 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1392 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.561950+0000", 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.561950+0000", 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.561953+0000", 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8559999999999998e-06 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.561965+0000", 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1443999999999999e-05 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.561991+0000", 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6693e-05 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.562005+0000", 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3793e-05 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.562134+0000", 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000128978 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.562288+0000", 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015371300000000001 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.565631+0000", 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003343057 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.575609+0000", 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.009978097 2024-09-21T08:26:38.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.576034+0000", 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000424571 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.576071+0000", 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7679000000000003e-05 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.576107+0000", 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5534000000000002e-05 2024-09-21T08:26:38.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:05.576136+0000", 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9258e-05 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:1571 7.18 e182/181 7:1fd29771:::smithi13779221-22:head v 182'166, mlcod=0'0)", 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:16.510099+0000", 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.972040140000001, 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012084246, 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.556 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.510099+0000", 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.510099+0000", 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.510100+0000", 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4900000000000004e-07 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.516226+0000", 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061262360000000002 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.516245+0000", 2024-09-21T08:26:38.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8501000000000001e-05 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.516260+0000", 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4837e-05 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.516326+0000", 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6027999999999999e-05 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.516345+0000", 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9610999999999999e-05 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.522130+0000", 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057850790000000003 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.522183+0000", 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3004999999999998e-05 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1588 7.11 7:88bf4ad2:::smithi13779221-6:head [copy-from ver 0] snapc 32=[32] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:16.907002+0000", 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.575137522999999, 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.031641867, 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1588 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.561 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907002+0000", 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907002+0000", 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907004+0000", 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4430000000000002e-06 2024-09-21T08:26:38.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907020+0000", 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6277e-05 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907035+0000", 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4790000000000001e-05 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907045+0000", 2024-09-21T08:26:38.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8460000000000003e-06 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907108+0000", 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2638000000000006e-05 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907115+0000", 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0099999999999998e-06 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.910406+0000", 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032914730000000001 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.910417+0000", 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0322e-05 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.921994+0000", 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.01157721 2024-09-21T08:26:38.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:26:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.925853+0000", 2024-09-21T08:26:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038596580000000002 2024-09-21T08:26:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.937700+0000", 2024-09-21T08:26:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011846382000000001 2024-09-21T08:26:38.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.937753+0000", 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3501000000000002e-05 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.938644+0000", 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089031699999999995 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1589 7.11 7:88bf4ad2:::smithi13779221-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:16.907152+0000", 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 21.574987460999999, 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.031681281, 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1589 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907152+0000", 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907152+0000", 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907153+0000", 2024-09-21T08:26:38.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0759999999999999e-06 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907157+0000", 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8820000000000003e-06 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907164+0000", 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1740000000000003e-06 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907170+0000", 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7649999999999999e-06 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907282+0000", 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011179300000000001 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.907287+0000", 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5090000000000001e-06 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.910440+0000", 2024-09-21T08:26:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003152565 2024-09-21T08:26:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:16.910447+0000", 2024-09-21T08:26:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3080000000000002e-06 2024-09-21T08:26:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.926315+0000", 2024-09-21T08:26:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0158684629999999 2024-09-21T08:26:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:26:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.926374+0000", 2024-09-21T08:26:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8276000000000002e-05 2024-09-21T08:26:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.937889+0000", 2024-09-21T08:26:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011515689000000001 2024-09-21T08:26:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.937914+0000", 2024-09-21T08:26:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4854e-05 2024-09-21T08:26:38.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.938689+0000", 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00077465499999999996 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.938747+0000", 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8285000000000002e-05 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.938833+0000", 2024-09-21T08:26:38.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5987000000000002e-05 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1601 7.15 7:a8d20cef:::smithi13779221-34:head [copy-from ver 85] snapc 33=[33,32] ondisk+write+known_if_redirected+supports_pool_eio e184)", 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:17.913175+0000", 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.568964276999999, 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0368445209999999, 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.577 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1601 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913175+0000", 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913175+0000", 2024-09-21T08:26:38.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913177+0000", 2024-09-21T08:26:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2089999999999999e-06 2024-09-21T08:26:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913185+0000", 2024-09-21T08:26:38.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0509999999999997e-06 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913196+0000", 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0959e-05 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913206+0000", 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6569999999999992e-06 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.914037+0000", 2024-09-21T08:26:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00083100100000000005 2024-09-21T08:26:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.914051+0000", 2024-09-21T08:26:38.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4221e-05 2024-09-21T08:26:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.937608+0000", 2024-09-21T08:26:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0235564850000001 2024-09-21T08:26:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.937751+0000", 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014308499999999999 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.940751+0000", 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029998120000000001 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.945316+0000", 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045651939999999998 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:38.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.946136+0000", 2024-09-21T08:26:38.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00082026199999999999 2024-09-21T08:26:38.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.949891+0000", 2024-09-21T08:26:38.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037549950000000001 2024-09-21T08:26:38.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.949933+0000", 2024-09-21T08:26:38.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2178999999999997e-05 2024-09-21T08:26:38.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.950020+0000", 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6410999999999996e-05 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1602 7.15 7:a8d20cef:::smithi13779221-34:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e184)", 2024-09-21T08:26:38.586 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:17.913222+0000", 2024-09-21T08:26:38.587 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 20.568916859000002, 2024-09-21T08:26:38.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.031828242, 2024-09-21T08:26:38.587 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.587 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.587 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:26:38.587 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.587 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.587 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.587 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1602 2024-09-21T08:26:38.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.588 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913222+0000", 2024-09-21T08:26:38.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913222+0000", 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913223+0000", 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.16e-07 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913227+0000", 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.292e-06 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913234+0000", 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8779999999999998e-06 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.913240+0000", 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9139999999999996e-06 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.914085+0000", 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00084439200000000002 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:17.914093+0000", 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4619999999999996e-06 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.938099+0000", 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0240062379999999 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.938156+0000", 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6709000000000002e-05 2024-09-21T08:26:38.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.944784+0000", 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066274649999999999 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.944875+0000", 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1260000000000004e-05 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:18.945051+0000", 2024-09-21T08:26:38.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000175816 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1729 7.15 7:aa5ad64c:::smithi13779221-49:head [write 786627~795991,stat] snapc 39=[39] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:30.567866+0000", 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.9142731399999997, 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.008432864, 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.593 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1729 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.567866+0000", 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.567866+0000", 2024-09-21T08:26:38.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.567868+0000", 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8050000000000001e-06 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.568703+0000", 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00083546800000000002 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.568715+0000", 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1933000000000001e-05 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.568722+0000", 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.024e-06 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.568771+0000", 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8686000000000001e-05 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.596 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:26:38.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.568776+0000", 2024-09-21T08:26:38.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2109999999999997e-06 2024-09-21T08:26:38.597 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.597 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.597 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.597 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.572320+0000", 2024-09-21T08:26:38.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003543334 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.572377+0000", 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7544000000000003e-05 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.572493+0000", 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115902 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:26:38.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.575386+0000", 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028925650000000002 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.575944+0000", 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055857900000000002 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.576193+0000", 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000249077 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.599 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.576252+0000", 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9095000000000003e-05 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.576299+0000", 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6641000000000003e-05 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.600 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:1744 7.a e197/154 7:53537bd3:::smithi13779221-39:head v 197'56, mlcod=193'53)", 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:30.598298+0000", 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.8838412380000005, 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012985231999999999, 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.598298+0000", 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.598298+0000", 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.598300+0000", 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.328e-06 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605256+0000", 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069555729999999996 2024-09-21T08:26:38.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605277+0000", 2024-09-21T08:26:38.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0684999999999999e-05 2024-09-21T08:26:38.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605379+0000", 2024-09-21T08:26:38.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010247900000000001 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605449+0000", 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9382999999999996e-05 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605470+0000", 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1673999999999999e-05 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.611205+0000", 2024-09-21T08:26:38.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057352549999999999 2024-09-21T08:26:38.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.611283+0000", 2024-09-21T08:26:38.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7855000000000006e-05 2024-09-21T08:26:38.605 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:1742 7.d e197/154 7:b31d99ce:::smithi13779221-21:head v 197'58, mlcod=195'55)", 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:30.605430+0000", 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 7.8767092569999999, 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011197215, 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:26:38.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:26:38.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605430+0000", 2024-09-21T08:26:38.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:26:38.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605430+0000", 2024-09-21T08:26:38.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:26:38.609 INFO:tasks.rados.rados.0.smithi137.stdout:674: done (2 left) 2024-09-21T08:26:38.609 INFO:tasks.rados.rados.0.smithi137.stdout:675: done (1 left) 2024-09-21T08:26:38.610 INFO:tasks.rados.rados.0.smithi137.stdout:676: done (0 left) 2024-09-21T08:26:38.610 INFO:tasks.rados.rados.0.smithi137.stdout:677: read oid 1 snap -1 2024-09-21T08:26:38.610 INFO:tasks.rados.rados.0.smithi137.stdout:677: expect (ObjNum 231 snap 47 seq_num 231) 2024-09-21T08:26:38.610 INFO:tasks.rados.rados.0.smithi137.stdout:678: snap_create 2024-09-21T08:26:38.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605433+0000", 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6479999999999999e-06 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.612561+0000", 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071285919999999996 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.612575+0000", 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4028e-05 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.612582+0000", 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8619999999999997e-06 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.612637+0000", 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.499e-05 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.612658+0000", 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1115999999999999e-05 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.616592+0000", 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039338949999999998 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.616627+0000", 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5083999999999998e-05 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:26:38.613 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.7 to 0.980429 (fafd) 2024-09-21T08:26:38.614 INFO:tasks.rados.rados.0.smithi137.stdout:677: expect (ObjNum 231 snap 47 seq_num 231) 2024-09-21T08:26:38.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:38.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:38.630 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:38.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:38.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:38.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:38.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:38 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/447421590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.980429076411602}]: dispatch 2024-09-21T08:26:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:38 smithi165 ceph-mon[32893]: from='client.45071 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.980429076411602}]: dispatch 2024-09-21T08:26:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:38 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3081923855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:38 smithi165 ceph-mon[32893]: from='client.45071 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.980429076411602}]': finished 2024-09-21T08:26:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:38 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3081923855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:38.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:38 smithi165 ceph-mon[32893]: osdmap e205: 8 total, 8 up, 4 in 2024-09-21T08:26:38.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/447421590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.980429076411602}]: dispatch 2024-09-21T08:26:38.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[29634]: from='client.45071 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.980429076411602}]: dispatch 2024-09-21T08:26:38.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3081923855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:38.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[29634]: from='client.45071 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.980429076411602}]': finished 2024-09-21T08:26:38.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3081923855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:38.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[29634]: osdmap e205: 8 total, 8 up, 4 in 2024-09-21T08:26:38.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/447421590' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.980429076411602}]: dispatch 2024-09-21T08:26:38.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[41044]: from='client.45071 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 0.980429076411602}]: dispatch 2024-09-21T08:26:38.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3081923855' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:38.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[41044]: from='client.45071 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 0.980429076411602}]': finished 2024-09-21T08:26:38.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3081923855' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:38.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:38 smithi137 ceph-mon[41044]: osdmap e205: 8 total, 8 up, 4 in 2024-09-21T08:26:38.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:38.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:38.878 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:38.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:38.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:39.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:39.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:39.001 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:39.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:26:39.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:39.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:39.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:39.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:39.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:39.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:39.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:39.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:39.369 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:39.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:39.369 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:39.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T08:26:39.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:39.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:39.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:39.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:39.518 INFO:tasks.rados.rados.0.smithi137.stdout:677: done (1 left) 2024-09-21T08:26:39.518 INFO:tasks.rados.rados.0.smithi137.stdout:678: done (0 left) 2024-09-21T08:26:39.518 INFO:tasks.rados.rados.0.smithi137.stdout:679: snap_create 2024-09-21T08:26:39.594 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:39.615 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:39.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:39.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:39.717 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:39.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:39.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:39.739 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:39 smithi165 ceph-mon[32893]: pgmap v212: 158 pgs: 158 active+clean; 140 MiB data, 473 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:39 smithi165 ceph-mon[32893]: osdmap e206: 8 total, 8 up, 4 in 2024-09-21T08:26:39.837 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:39 smithi137 ceph-mon[29634]: pgmap v212: 158 pgs: 158 active+clean; 140 MiB data, 473 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:39.837 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:39 smithi137 ceph-mon[29634]: osdmap e206: 8 total, 8 up, 4 in 2024-09-21T08:26:39.837 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:39 smithi137 ceph-mon[41044]: pgmap v212: 158 pgs: 158 active+clean; 140 MiB data, 473 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:39.837 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:39 smithi137 ceph-mon[41044]: osdmap e206: 8 total, 8 up, 4 in 2024-09-21T08:26:39.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:39.862 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:39.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:39.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:39.863 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:39.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:39.985 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:39.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:39.985 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:40.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:40.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:40.108 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:40.108 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:40.209 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:40.231 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:40.231 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:40.232 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:40.330 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:40.330 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:40.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:40.348 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:40.349 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:40.354 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:40.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:40.355 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:40.374 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:40.375 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:40.387 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:40.396 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:40.396 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:40.421 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:40.421 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:40.438 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:40.439 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:40.454 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:40.455 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:40.456 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:40.470 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:40.471 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:40.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:40.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:40.478 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:40.518 INFO:tasks.rados.rados.0.smithi137.stdout:679: done (0 left) 2024-09-21T08:26:40.519 INFO:tasks.rados.rados.0.smithi137.stdout:680: snap_create 2024-09-21T08:26:40.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:40.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:40.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:40.601 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:40.630 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:40 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3201229012' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:40.630 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:40 smithi137 ceph-mon[29634]: osdmap e207: 8 total, 8 up, 4 in 2024-09-21T08:26:40.630 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:40 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3201229012' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:40.630 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:40 smithi137 ceph-mon[41044]: osdmap e207: 8 total, 8 up, 4 in 2024-09-21T08:26:40.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:40.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:40.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:40.726 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:40.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:40.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:40.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:40.849 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:40.936 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:40 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:26:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:26:40.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:40.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:40.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:40.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:40 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3201229012' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:40 smithi165 ceph-mon[32893]: osdmap e207: 8 total, 8 up, 4 in 2024-09-21T08:26:41.042 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:41.042 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:41.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:41.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:41.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:41.102 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:41.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:41.227 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:41.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:41.228 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:41.250 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:26:41.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:41.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:41.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:41.354 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:41.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:41.478 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:41.479 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:41.479 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:41.518 INFO:tasks.rados.rados.0.smithi137.stdout:680: done (0 left) 2024-09-21T08:26:41.518 INFO:tasks.rados.rados.0.smithi137.stdout:681: snap_remove snap 61 2024-09-21T08:26:41.580 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:41.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:41.604 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:41.604 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:41.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:41.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:41.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:41.731 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:41.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:41.855 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:41.856 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:41.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:41.957 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:41.968 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:41 smithi137 ceph-mon[29634]: pgmap v215: 158 pgs: 158 active+clean; 140 MiB data, 473 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:41.968 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:41 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:41.969 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:41 smithi137 ceph-mon[29634]: osdmap e208: 8 total, 8 up, 4 in 2024-09-21T08:26:41.969 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:41 smithi137 ceph-mon[41044]: pgmap v215: 158 pgs: 158 active+clean; 140 MiB data, 473 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:41.969 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:41 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:41.969 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:41 smithi137 ceph-mon[41044]: osdmap e208: 8 total, 8 up, 4 in 2024-09-21T08:26:41.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:41.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:41.984 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:41.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T08:26:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:41 smithi165 ceph-mon[32893]: pgmap v215: 158 pgs: 158 active+clean; 140 MiB data, 473 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:42.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:41 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:42.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:41 smithi165 ceph-mon[32893]: osdmap e208: 8 total, 8 up, 4 in 2024-09-21T08:26:42.014 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:26:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:26:42.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:42.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:42.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:42.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:42.131 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:42.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:42.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:42.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:42.241 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:42.343 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:42.367 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:42.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:42.368 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:42.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:42.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:42.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:42.500 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:42.520 INFO:tasks.rados.rados.0.smithi137.stdout:681: done (0 left) 2024-09-21T08:26:42.520 INFO:tasks.rados.rados.0.smithi137.stdout:682: rollback oid 33 current snap is 64 2024-09-21T08:26:42.521 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 33 to 63 2024-09-21T08:26:42.521 INFO:tasks.rados.rados.0.smithi137.stdout:683: copy_from oid 7 from oid 16 current snap is 64 2024-09-21T08:26:42.521 INFO:tasks.rados.rados.0.smithi137.stdout:684: write oid 38 current snap is 64 2024-09-21T08:26:42.521 INFO:tasks.rados.rados.0.smithi137.stdout:684: seq_num 271 ranges {554697=464834,1725326=559090} 2024-09-21T08:26:42.526 INFO:tasks.rados.rados.0.smithi137.stdout:684: writing smithi13779221-38 from 554697 to 1019531 tid 1 2024-09-21T08:26:42.534 INFO:tasks.rados.rados.0.smithi137.stdout:684: writing smithi13779221-38 from 1725326 to 2284416 tid 2 2024-09-21T08:26:42.534 INFO:tasks.rados.rados.0.smithi137.stdout:685: copy_from oid 41 from oid 15 current snap is 64 2024-09-21T08:26:42.534 INFO:tasks.rados.rados.0.smithi137.stdout:684: finishing write tid 1 to smithi13779221-38 2024-09-21T08:26:42.534 INFO:tasks.rados.rados.0.smithi137.stdout:682: finishing rollback tid 0 to smithi13779221-33 2024-09-21T08:26:42.535 INFO:tasks.rados.rados.0.smithi137.stdout:682: finishing rollback tid 1 to smithi13779221-33 2024-09-21T08:26:42.535 INFO:tasks.rados.rados.0.smithi137.stdout:683: finishing copy_from to smithi13779221-7 2024-09-21T08:26:42.535 INFO:tasks.rados.rados.0.smithi137.stdout:683: got expected ENOENT (src dne) 2024-09-21T08:26:42.535 INFO:tasks.rados.rados.0.smithi137.stdout:683: finishing copy_from racing read to smithi13779221-7 2024-09-21T08:26:42.535 INFO:tasks.rados.rados.0.smithi137.stdout:682: finishing rollback tid 2 to smithi13779221-33 2024-09-21T08:26:42.535 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 54 (ObjNum 228 snap 47 seq_num 228) dirty exists 2024-09-21T08:26:42.535 INFO:tasks.rados.rados.0.smithi137.stdout:682: done (3 left) 2024-09-21T08:26:42.535 INFO:tasks.rados.rados.0.smithi137.stdout:683: done (2 left) 2024-09-21T08:26:42.535 INFO:tasks.rados.rados.0.smithi137.stdout:686: write (excl) oid 2 current snap is 64 2024-09-21T08:26:42.535 INFO:tasks.rados.rados.0.smithi137.stdout:686: seq_num 272 ranges {571504=540083,1783808=702748} 2024-09-21T08:26:42.540 INFO:tasks.rados.rados.0.smithi137.stdout:686: writing smithi13779221-2 from 571504 to 1111587 tid 1 2024-09-21T08:26:42.548 INFO:tasks.rados.rados.0.smithi137.stdout:686: writing smithi13779221-2 from 1783808 to 2486556 tid 2 2024-09-21T08:26:42.548 INFO:tasks.rados.rados.0.smithi137.stdout:687: read oid 32 snap 63 2024-09-21T08:26:42.548 INFO:tasks.rados.rados.0.smithi137.stdout:687: expect deleted 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:688: rollback oid 26 current snap is 64 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 26 to 60 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:685: finishing copy_from to smithi13779221-41 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:685: got expected ENOENT (src dne) 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:685: finishing copy_from racing read to smithi13779221-41 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:684: finishing write tid 2 to smithi13779221-38 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:684: finishing write tid 3 to smithi13779221-38 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:684: finishing write tid 4 to smithi13779221-38 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:684: oid 38 updating version 0 to 301 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:684: oid 38 updating version 301 to 302 2024-09-21T08:26:42.549 INFO:tasks.rados.rados.0.smithi137.stdout:684: oid 38 version 302 is already newer than 300 2024-09-21T08:26:42.550 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 302 (ObjNum 271 snap 64 seq_num 271) dirty exists 2024-09-21T08:26:42.550 INFO:tasks.rados.rados.0.smithi137.stdout:684: left oid 38 (ObjNum 271 snap 64 seq_num 271) 2024-09-21T08:26:42.550 INFO:tasks.rados.rados.0.smithi137.stdout:686: finishing write tid 1 to smithi13779221-2 2024-09-21T08:26:42.550 INFO:tasks.rados.rados.0.smithi137.stdout:684: done (4 left) 2024-09-21T08:26:42.550 INFO:tasks.rados.rados.0.smithi137.stdout:685: done (3 left) 2024-09-21T08:26:42.550 INFO:tasks.rados.rados.0.smithi137.stdout:689: delete oid 29 current snap is 64 2024-09-21T08:26:42.551 INFO:tasks.rados.rados.0.smithi137.stdout:687: done (3 left) 2024-09-21T08:26:42.551 INFO:tasks.rados.rados.0.smithi137.stdout:689: done (2 left) 2024-09-21T08:26:42.551 INFO:tasks.rados.rados.0.smithi137.stdout:690: read oid 31 snap -1 2024-09-21T08:26:42.551 INFO:tasks.rados.rados.0.smithi137.stdout:690: expect (ObjNum 203 snap 40 seq_num 203) 2024-09-21T08:26:42.551 INFO:tasks.rados.rados.0.smithi137.stdout:691: write oid 10 current snap is 64 2024-09-21T08:26:42.551 INFO:tasks.rados.rados.0.smithi137.stdout:691: seq_num 274 ranges {605118=406934,1617125=706417} 2024-09-21T08:26:42.556 INFO:tasks.rados.rados.0.smithi137.stdout:691: writing smithi13779221-10 from 605118 to 1012052 tid 1 2024-09-21T08:26:42.565 INFO:tasks.rados.rados.0.smithi137.stdout:691: writing smithi13779221-10 from 1617125 to 2323542 tid 2 2024-09-21T08:26:42.565 INFO:tasks.rados.rados.0.smithi137.stdout:692: read oid 44 snap -1 2024-09-21T08:26:42.565 INFO:tasks.rados.rados.0.smithi137.stdout:692: expect (ObjNum 98 snap 19 seq_num 98) 2024-09-21T08:26:42.565 INFO:tasks.rados.rados.0.smithi137.stdout:693: read oid 4 snap -1 2024-09-21T08:26:42.565 INFO:tasks.rados.rados.0.smithi137.stdout:693: expect (ObjNum 231 snap 47 seq_num 231) 2024-09-21T08:26:42.565 INFO:tasks.rados.rados.0.smithi137.stdout:694: delete oid 3 current snap is 64 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:688: finishing rollback tid 0 to smithi13779221-26 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:688: finishing rollback tid 1 to smithi13779221-26 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:688: finishing rollback tid 2 to smithi13779221-26 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 262 (ObjNum 200 snap 40 seq_num 200) dirty exists 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:686: finishing write tid 2 to smithi13779221-2 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:686: finishing write tid 3 to smithi13779221-2 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:686: finishing write tid 4 to smithi13779221-2 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:686: oid 2 updating version 0 to 107 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:686: oid 2 version 107 is already newer than 105 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:686: oid 2 version 107 is already newer than 106 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 107 (ObjNum 272 snap 64 seq_num 272) dirty exists 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:686: left oid 2 (ObjNum 272 snap 64 seq_num 272) 2024-09-21T08:26:42.566 INFO:tasks.rados.rados.0.smithi137.stdout:690: expect (ObjNum 203 snap 40 seq_num 203) 2024-09-21T08:26:42.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:42.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:42.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:42.624 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:42.643 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:42.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:42.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:42.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:42.748 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:42.748 INFO:tasks.rados.rados.0.smithi137.stdout:691: finishing write tid 1 to smithi13779221-10 2024-09-21T08:26:42.748 INFO:tasks.rados.rados.0.smithi137.stdout:691: finishing write tid 2 to smithi13779221-10 2024-09-21T08:26:42.748 INFO:tasks.rados.rados.0.smithi137.stdout:686: done (6 left) 2024-09-21T08:26:42.748 INFO:tasks.rados.rados.0.smithi137.stdout:688: done (5 left) 2024-09-21T08:26:42.749 INFO:tasks.rados.rados.0.smithi137.stdout:690: done (4 left) 2024-09-21T08:26:42.749 INFO:tasks.rados.rados.0.smithi137.stdout:694: done (3 left) 2024-09-21T08:26:42.749 INFO:tasks.rados.rados.0.smithi137.stdout:695: snap_remove snap 63 2024-09-21T08:26:42.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:42.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:42.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:42.875 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:42.976 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:43.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:43.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:43.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:43.069 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:43.069 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:43.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:43.123 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:43.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:43.124 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:43.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:43.245 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:43.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:43.246 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:43.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:43.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:43.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:43.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:43.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:43.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:43.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:43.501 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:43.592 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:26:43.595 INFO:tasks.rados.rados.0.smithi137.stdout:695: done (3 left) 2024-09-21T08:26:43.595 INFO:tasks.rados.rados.0.smithi137.stdout:696: write (excl) oid 29 current snap is 64 2024-09-21T08:26:43.595 INFO:tasks.rados.rados.0.smithi137.stdout:691: finishing write tid 3 to smithi13779221-10 2024-09-21T08:26:43.595 INFO:tasks.rados.rados.0.smithi137.stdout:696: seq_num 276 ranges {638732=557432,1617736=710086} 2024-09-21T08:26:43.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:43.602 INFO:tasks.rados.rados.0.smithi137.stdout:696: writing smithi13779221-29 from 638732 to 1196164 tid 1 2024-09-21T08:26:43.609 INFO:tasks.rados.rados.0.smithi137.stdout:696: writing smithi13779221-29 from 1617736 to 2327822 tid 2 2024-09-21T08:26:43.610 INFO:tasks.rados.rados.0.smithi137.stdout:691: finishing write tid 4 to smithi13779221-10 2024-09-21T08:26:43.610 INFO:tasks.rados.rados.0.smithi137.stdout:691: oid 10 updating version 0 to 203 2024-09-21T08:26:43.610 INFO:tasks.rados.rados.0.smithi137.stdout:691: oid 10 updating version 203 to 204 2024-09-21T08:26:43.611 INFO:tasks.rados.rados.0.smithi137.stdout:691: oid 10 version 204 is already newer than 202 2024-09-21T08:26:43.611 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 204 (ObjNum 274 snap 64 seq_num 274) dirty exists 2024-09-21T08:26:43.611 INFO:tasks.rados.rados.0.smithi137.stdout:691: left oid 10 (ObjNum 274 snap 64 seq_num 274) 2024-09-21T08:26:43.611 INFO:tasks.rados.rados.0.smithi137.stdout:692: expect (ObjNum 98 snap 19 seq_num 98) 2024-09-21T08:26:43.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:26:43.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:43.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:43.631 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:43.697 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:43 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:43.697 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:43 smithi137 ceph-mon[29634]: pgmap v218: 158 pgs: 158 active+clean; 140 MiB data, 503 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 962 KiB/s wr, 7 op/s 2024-09-21T08:26:43.697 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:43 smithi137 ceph-mon[29634]: osdmap e209: 8 total, 8 up, 4 in 2024-09-21T08:26:43.697 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:43 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3304409019' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:43.697 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:43 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:43.697 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:43 smithi137 ceph-mon[41044]: pgmap v218: 158 pgs: 158 active+clean; 140 MiB data, 503 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 962 KiB/s wr, 7 op/s 2024-09-21T08:26:43.697 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:43 smithi137 ceph-mon[41044]: osdmap e209: 8 total, 8 up, 4 in 2024-09-21T08:26:43.697 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:43 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3304409019' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:43.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T08:26:43.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:43.778 INFO:tasks.rados.rados.0.smithi137.stdout:693: expect (ObjNum 231 snap 47 seq_num 231) 2024-09-21T08:26:43.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:43.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:43.789 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:43.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:43.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:43.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:43.914 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:43.957 INFO:tasks.rados.rados.0.smithi137.stdout:696: finishing write tid 1 to smithi13779221-29 2024-09-21T08:26:43.957 INFO:tasks.rados.rados.0.smithi137.stdout:696: finishing write tid 2 to smithi13779221-29 2024-09-21T08:26:43.957 INFO:tasks.rados.rados.0.smithi137.stdout:696: finishing write tid 3 to smithi13779221-29 2024-09-21T08:26:43.957 INFO:tasks.rados.rados.0.smithi137.stdout:696: finishing write tid 4 to smithi13779221-29 2024-09-21T08:26:43.957 INFO:tasks.rados.rados.0.smithi137.stdout:696: oid 29 updating version 0 to 310 2024-09-21T08:26:43.957 INFO:tasks.rados.rados.0.smithi137.stdout:696: oid 29 version 310 is already newer than 309 2024-09-21T08:26:43.957 INFO:tasks.rados.rados.0.smithi137.stdout:696: oid 29 updating version 310 to 311 2024-09-21T08:26:43.957 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 311 (ObjNum 276 snap 64 seq_num 276) dirty exists 2024-09-21T08:26:43.958 INFO:tasks.rados.rados.0.smithi137.stdout:696: left oid 29 (ObjNum 276 snap 64 seq_num 276) 2024-09-21T08:26:43.958 INFO:tasks.rados.rados.0.smithi137.stdout:691: done (3 left) 2024-09-21T08:26:43.958 INFO:tasks.rados.rados.0.smithi137.stdout:692: done (2 left) 2024-09-21T08:26:43.958 INFO:tasks.rados.rados.0.smithi137.stdout:693: done (1 left) 2024-09-21T08:26:43.958 INFO:tasks.rados.rados.0.smithi137.stdout:696: done (0 left) 2024-09-21T08:26:43.958 INFO:tasks.rados.rados.0.smithi137.stdout:697: delete oid 33 current snap is 64 2024-09-21T08:26:43.958 INFO:tasks.rados.rados.0.smithi137.stdout:697: done (0 left) 2024-09-21T08:26:43.959 INFO:tasks.rados.rados.0.smithi137.stdout:698: read oid 18 snap -1 2024-09-21T08:26:43.959 INFO:tasks.rados.rados.0.smithi137.stdout:698: expect deleted 2024-09-21T08:26:43.959 INFO:tasks.rados.rados.0.smithi137.stdout:699: write (excl) oid 7 current snap is 64 2024-09-21T08:26:43.959 INFO:tasks.rados.rados.0.smithi137.stdout:699: seq_num 278 ranges {672346=707930,2134700=537646} 2024-09-21T08:26:43.966 INFO:tasks.rados.rados.0.smithi137.stdout:699: writing smithi13779221-7 from 672346 to 1380276 tid 1 2024-09-21T08:26:43.967 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:43.975 INFO:tasks.rados.rados.0.smithi137.stdout:699: writing smithi13779221-7 from 2134700 to 2672346 tid 2 2024-09-21T08:26:43.975 INFO:tasks.rados.rados.0.smithi137.stdout:700: snap_remove snap 60 2024-09-21T08:26:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:43 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:43 smithi165 ceph-mon[32893]: pgmap v218: 158 pgs: 158 active+clean; 140 MiB data, 503 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 962 KiB/s wr, 7 op/s 2024-09-21T08:26:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:43 smithi165 ceph-mon[32893]: osdmap e209: 8 total, 8 up, 4 in 2024-09-21T08:26:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:43 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3304409019' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:26:44.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:44.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:44.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:44.037 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:44.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:44.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:44.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:44.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:44.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:44.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:44.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:44.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:44.389 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:44.415 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:44.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:44.416 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:44.423 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:44.516 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:44.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:44.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:44.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:44.601 INFO:tasks.rados.rados.0.smithi137.stdout:700: done (2 left) 2024-09-21T08:26:44.601 INFO:tasks.rados.rados.0.smithi137.stdout:701: write (excl) oid 45 current snap is 64 2024-09-21T08:26:44.601 INFO:tasks.rados.rados.0.smithi137.stdout:701: seq_num 279 ranges {689153=783179,2076829=457413} 2024-09-21T08:26:44.609 INFO:tasks.rados.rados.0.smithi137.stdout:701: writing smithi13779221-45 from 689153 to 1472332 tid 1 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:701: writing smithi13779221-45 from 2076829 to 2534242 tid 2 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:699: finishing write tid 1 to smithi13779221-7 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:699: finishing write tid 2 to smithi13779221-7 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:699: finishing write tid 3 to smithi13779221-7 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:699: finishing write tid 4 to smithi13779221-7 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:699: oid 7 updating version 0 to 314 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:699: oid 7 updating version 314 to 315 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:699: oid 7 version 315 is already newer than 313 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 315 (ObjNum 278 snap 64 seq_num 278) dirty exists 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:699: left oid 7 (ObjNum 278 snap 64 seq_num 278) 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:698: done (2 left) 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:699: done (1 left) 2024-09-21T08:26:44.616 INFO:tasks.rados.rados.0.smithi137.stdout:702: read oid 16 snap 55 2024-09-21T08:26:44.617 INFO:tasks.rados.rados.0.smithi137.stdout:702: expect (ObjNum 210 snap 43 seq_num 210) 2024-09-21T08:26:44.617 INFO:tasks.rados.rados.0.smithi137.stdout:703: write (excl) oid 1 current snap is 64 2024-09-21T08:26:44.617 INFO:tasks.rados.rados.0.smithi137.stdout:703: seq_num 280 ranges {705960=458428,1735311=717424} 2024-09-21T08:26:44.620 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:44 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3304409019' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:26:44.620 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:44 smithi137 ceph-mon[29634]: osdmap e210: 8 total, 8 up, 4 in 2024-09-21T08:26:44.620 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:44 smithi137 ceph-mon[29634]: 7.7 scrub starts 2024-09-21T08:26:44.620 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:44 smithi137 ceph-mon[29634]: 7.7 scrub ok 2024-09-21T08:26:44.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:44 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3304409019' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:26:44.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:44 smithi137 ceph-mon[41044]: osdmap e210: 8 total, 8 up, 4 in 2024-09-21T08:26:44.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:44 smithi137 ceph-mon[41044]: 7.7 scrub starts 2024-09-21T08:26:44.621 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:44 smithi137 ceph-mon[41044]: 7.7 scrub ok 2024-09-21T08:26:44.622 INFO:tasks.rados.rados.0.smithi137.stdout:703: writing smithi13779221-1 from 705960 to 1164388 tid 1 2024-09-21T08:26:44.631 INFO:tasks.rados.rados.0.smithi137.stdout:703: writing smithi13779221-1 from 1735311 to 2452735 tid 2 2024-09-21T08:26:44.632 INFO:tasks.rados.rados.0.smithi137.stdout:701: finishing write tid 1 to smithi13779221-45 2024-09-21T08:26:44.632 INFO:tasks.rados.rados.0.smithi137.stdout:701: finishing write tid 2 to smithi13779221-45 2024-09-21T08:26:44.632 INFO:tasks.rados.rados.0.smithi137.stdout:701: finishing write tid 3 to smithi13779221-45 2024-09-21T08:26:44.632 INFO:tasks.rados.rados.0.smithi137.stdout:702: expect (ObjNum 210 snap 43 seq_num 210) 2024-09-21T08:26:44.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:44.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:44.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:44.667 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:44.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:44.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:44.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:44.791 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:44.868 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 3, 6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:26:44.868 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:26:44.868 INFO:tasks.thrashosds.thrasher:Adding osd 3 2024-09-21T08:26:44.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd in 3 2024-09-21T08:26:44.884 INFO:tasks.rados.rados.0.smithi137.stdout:701: finishing write tid 4 to smithi13779221-45 2024-09-21T08:26:44.884 INFO:tasks.rados.rados.0.smithi137.stdout:701: oid 45 updating version 0 to 249 2024-09-21T08:26:44.884 INFO:tasks.rados.rados.0.smithi137.stdout:701: oid 45 updating version 249 to 250 2024-09-21T08:26:44.884 INFO:tasks.rados.rados.0.smithi137.stdout:701: oid 45 updating version 250 to 251 2024-09-21T08:26:44.884 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 251 (ObjNum 279 snap 64 seq_num 279) dirty exists 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:701: left oid 45 (ObjNum 279 snap 64 seq_num 279) 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:701: done (2 left) 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:702: done (1 left) 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:704: write (excl) oid 38 current snap is 64 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:703: finishing write tid 1 to smithi13779221-1 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:703: finishing write tid 2 to smithi13779221-1 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:703: finishing write tid 3 to smithi13779221-1 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:703: finishing write tid 4 to smithi13779221-1 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:703: oid 1 updating version 0 to 247 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:703: oid 1 updating version 247 to 248 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:703: oid 1 version 248 is already newer than 246 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 248 (ObjNum 280 snap 64 seq_num 280) dirty exists 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:703: left oid 1 (ObjNum 280 snap 64 seq_num 280) 2024-09-21T08:26:44.885 INFO:tasks.rados.rados.0.smithi137.stdout:704: seq_num 281 ranges {722767=533677,1793793=461082} 2024-09-21T08:26:44.891 INFO:tasks.rados.rados.0.smithi137.stdout:704: writing smithi13779221-38 from 722767 to 1256444 tid 1 2024-09-21T08:26:44.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:44.897 INFO:tasks.rados.rados.0.smithi137.stdout:704: writing smithi13779221-38 from 1793793 to 2254875 tid 2 2024-09-21T08:26:44.898 INFO:tasks.rados.rados.0.smithi137.stdout:704: finishing write tid 1 to smithi13779221-38 2024-09-21T08:26:44.898 INFO:tasks.rados.rados.0.smithi137.stdout:703: done (1 left) 2024-09-21T08:26:44.898 INFO:tasks.rados.rados.0.smithi137.stdout:705: snap_create 2024-09-21T08:26:44.901 INFO:tasks.rados.rados.0.smithi137.stdout:704: finishing write tid 2 to smithi13779221-38 2024-09-21T08:26:44.901 INFO:tasks.rados.rados.0.smithi137.stdout:704: finishing write tid 3 to smithi13779221-38 2024-09-21T08:26:44.902 INFO:tasks.rados.rados.0.smithi137.stdout:704: finishing write tid 4 to smithi13779221-38 2024-09-21T08:26:44.902 INFO:tasks.rados.rados.0.smithi137.stdout:704: oid 38 updating version 0 to 318 2024-09-21T08:26:44.902 INFO:tasks.rados.rados.0.smithi137.stdout:704: oid 38 version 318 is already newer than 317 2024-09-21T08:26:44.902 INFO:tasks.rados.rados.0.smithi137.stdout:704: oid 38 version 318 is already newer than 316 2024-09-21T08:26:44.902 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 318 (ObjNum 281 snap 64 seq_num 281) dirty exists 2024-09-21T08:26:44.902 INFO:tasks.rados.rados.0.smithi137.stdout:704: left oid 38 (ObjNum 281 snap 64 seq_num 281) 2024-09-21T08:26:44.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:44.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:44.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:44 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3304409019' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:26:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:44 smithi165 ceph-mon[32893]: osdmap e210: 8 total, 8 up, 4 in 2024-09-21T08:26:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:44 smithi165 ceph-mon[32893]: 7.7 scrub starts 2024-09-21T08:26:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:44 smithi165 ceph-mon[32893]: 7.7 scrub ok 2024-09-21T08:26:45.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:45.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:45.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:45.043 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:45.102 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:45.103 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:45.119 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:45.120 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:45.136 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:45.136 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:45.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:45.153 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:45.153 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:45.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:45.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:45.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:45.170 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:45.170 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:45.186 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:45.187 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:45.203 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:45.203 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:45.220 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:45.220 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:45.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:45.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:45.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:45.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:45.333 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:26:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:26:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:26:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:26:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:336 7.9 7:9d4d36a0:::smithi13779221-27:head [copy-from ver 7] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:26:45.333 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 116.761202082, 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050439811000000001, 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 336 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569727+0000", 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4899999999999999e-06 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569733+0000", 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9120000000000003e-06 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569741+0000", 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4999999999999999e-06 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569749+0000", 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2099999999999996e-06 2024-09-21T08:26:45.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.583356+0000", 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013607731 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584178+0000", 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082153700000000001 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.610330+0000", 2024-09-21T08:26:45.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026152526999999998 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.617047+0000", 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067167470000000003 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.619683+0000", 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026361039999999998 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:45.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620040+0000", 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035616499999999998 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620096+0000", 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6922000000000002e-05 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620165+0000", 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8966000000000003e-05 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.339 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:337 7.9 7:9d4d36a0:::smithi13779221-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 116.75326225000001, 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.038882593999999999, 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.340 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 337 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577668+0000", 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2960000000000001e-06 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577674+0000", 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5479999999999998e-06 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577687+0000", 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2829e-05 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577698+0000", 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0832999999999999e-05 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584334+0000", 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066355900000000002 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584379+0000", 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5575999999999999e-05 2024-09-21T08:26:45.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616393+0000", 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032013592 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616468+0000", 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5150999999999999e-05 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616548+0000", 2024-09-21T08:26:45.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0179000000000006e-05 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:348 7.9 7:92b92e8a:::smithi13779221-35:head [copy-from ver 33] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:49.567369+0000", 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 115.763558909, 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014025052, 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.345 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 348 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567369+0000", 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567369+0000", 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567371+0000", 2024-09-21T08:26:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4030000000000001e-06 2024-09-21T08:26:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567378+0000", 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9280000000000004e-06 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567390+0000", 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2027e-05 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567399+0000", 2024-09-21T08:26:45.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7930000000000003e-06 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567455+0000", 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5983000000000003e-05 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567462+0000", 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9430000000000003e-06 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.569717+0000", 2024-09-21T08:26:45.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022547579999999999 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.569810+0000", 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3554999999999995e-05 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.575654+0000", 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058440419999999998 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.580118+0000", 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044635400000000002 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.581293+0000", 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011749709999999999 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.581328+0000", 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5018000000000003e-05 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.581366+0000", 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7889999999999998e-05 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.581394+0000", 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8201e-05 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.352 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:349 7.9 7:92b92e8a:::smithi13779221-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:49.567414+0000", 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 115.763513394, 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011978051, 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 349 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567414+0000", 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567414+0000", 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567415+0000", 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6700000000000002e-07 2024-09-21T08:26:45.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567418+0000", 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1219999999999999e-06 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567425+0000", 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4659999999999996e-06 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.355 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567431+0000", 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2489999999999998e-06 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567536+0000", 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104713 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.567553+0000", 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7795000000000002e-05 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.570154+0000", 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026001900000000001 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.570197+0000", 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3000999999999997e-05 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.579201+0000", 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090041860000000008 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.579285+0000", 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3881999999999994e-05 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:49.579392+0000", 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010758 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:569 7.e e117/80 7:78d05d33:::smithi13779221-41:head v 117'23, mlcod=100'19)", 2024-09-21T08:26:45.358 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.565684+0000", 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 94.765243697000002, 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013671371999999999, 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565684+0000", 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565684+0000", 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565685+0000", 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9899999999999999e-07 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567942+0000", 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022574980000000001 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567966+0000", 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3577000000000001e-05 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567978+0000", 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2068e-05 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.568075+0000", 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7194000000000005e-05 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.568100+0000", 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4593999999999999e-05 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579300+0000", 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011200217 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579355+0000", 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5325000000000003e-05 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.362 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:601 7.9 7:9d4d36a0:::smithi13779221-27:head [write 1651964~401685] snapc 15=[15,14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e118)", 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:11.503430+0000", 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 93.827497265999995, 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01254068, 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 601 2024-09-21T08:26:45.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503430+0000", 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503430+0000", 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503433+0000", 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3369999999999998e-06 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503559+0000", 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012637600000000001 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503581+0000", 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1925000000000001e-05 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503589+0000", 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3189999999999995e-06 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503647+0000", 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7720999999999998e-05 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503760+0000", 2024-09-21T08:26:45.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112897 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.503933+0000", 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017294200000000001 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.508305+0000", 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0043725439999999999 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.514072+0000", 2024-09-21T08:26:45.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057668290000000002 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.515855+0000", 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017826249999999999 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.515910+0000", 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4879000000000001e-05 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:11.515971+0000", 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1285999999999996e-05 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:833 7.6 7:611b1c4b:::smithi13779221-3:head [copy-from ver 0] snapc 1b=[1b,1a] ondisk+write+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.705983+0000", 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.624944425999999, 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013624030000000001, 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.369 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 833 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705983+0000", 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705983+0000", 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705987+0000", 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8840000000000004e-06 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705997+0000", 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0370999999999999e-05 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706017+0000", 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9979e-05 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706028+0000", 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0128e-05 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706105+0000", 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7667e-05 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:45.372 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706114+0000", 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4540000000000007e-06 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.713988+0000", 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078742559999999996 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.714091+0000", 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010333200000000001 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.719607+0000", 2024-09-21T08:26:45.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005515959 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:834 7.6 7:611b1c4b:::smithi13779221-3:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.624876094000001, 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014179074, 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.374 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 834 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.23e-07 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706056+0000", 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.365e-06 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706064+0000", 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3399999999999998e-06 2024-09-21T08:26:45.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706070+0000", 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0719999999999996e-06 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706167+0000", 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6844999999999999e-05 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706174+0000", 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9630000000000001e-06 2024-09-21T08:26:45.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.714277+0000", 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081029469999999992 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.714328+0000", 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1270999999999998e-05 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.720039+0000", 2024-09-21T08:26:45.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057105020000000001 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.720127+0000", 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7916000000000003e-05 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.720231+0000", 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010403000000000001 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1198 7.6 7:611b1c4b:::smithi13779221-3:head [copy-from ver 136] snapc 26=[26,25] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:51.673794+0000", 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 53.657133694000002, 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013117772999999999, 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1198 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.673794+0000", 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.673794+0000", 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.673794+0000", 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3099999999999998e-07 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.673799+0000", 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3819999999999997e-06 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.673802+0000", 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3359999999999999e-06 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.673808+0000", 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2770000000000002e-06 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.673845+0000", 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6251999999999998e-05 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.673917+0000", 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2512e-05 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.676994+0000", 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0030763549999999998 2024-09-21T08:26:45.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.680687+0000", 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036936809999999999 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.686691+0000", 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0060040680000000004 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.686818+0000", 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012647099999999999 2024-09-21T08:26:45.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.686868+0000", 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0095000000000001e-05 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:51.686912+0000", 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3912999999999998e-05 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1228 7.1 7:88bf4ad2:::smithi13779221-6:head [write 410137~481384] snapc 28=[] ondisk+write+known_if_redirected+supports_pool_eio e164)", 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:57.534479+0000", 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.796448869000002, 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014085208, 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1228 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.534479+0000", 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.534479+0000", 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.534482+0000", 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5379999999999998e-06 2024-09-21T08:26:45.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.534748+0000", 2024-09-21T08:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026530100000000001 2024-09-21T08:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.534765+0000", 2024-09-21T08:26:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7507999999999999e-05 2024-09-21T08:26:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.389 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.534774+0000", 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1579999999999994e-06 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.534855+0000", 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0311999999999999e-05 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.534864+0000", 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1290000000000002e-06 2024-09-21T08:26:45.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.541803+0000", 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0069395189999999999 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.541883+0000", 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9352999999999997e-05 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.542015+0000", 2024-09-21T08:26:45.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013195599999999999 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.543799+0000", 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017848320000000001 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.544624+0000", 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082489399999999995 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.548466+0000", 2024-09-21T08:26:45.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00384214 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.548519+0000", 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2405999999999999e-05 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:57.548564+0000", 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5161999999999999e-05 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1378 7.9 7:92b92e8a:::smithi13779221-35:head [copy-from ver 66] snapc 2c=[] ondisk+write+known_if_redirected+supports_pool_eio e172)", 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:05.517752+0000", 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 39.813175557999998, 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018606989000000001, 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1378 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.394 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517752+0000", 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517752+0000", 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517755+0000", 2024-09-21T08:26:45.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7269999999999999e-06 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517764+0000", 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3000000000000007e-06 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517781+0000", 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7336000000000001e-05 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517794+0000", 2024-09-21T08:26:45.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2163e-05 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517866+0000", 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2105000000000002e-05 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517874+0000", 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1340000000000004e-06 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.520832+0000", 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029577370000000002 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.520935+0000", 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103515 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.529384+0000", 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0084484599999999997 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.534407+0000", 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005023507 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.535911+0000", 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0015034650000000001 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.536258+0000", 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034711099999999999 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.536315+0000", 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7506999999999997e-05 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.536359+0000", 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3921999999999997e-05 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1379 7.9 7:92b92e8a:::smithi13779221-35:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e172)", 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:05.517813+0000", 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 39.813114179000003, 2024-09-21T08:26:45.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014767521, 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1379 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517813+0000", 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517813+0000", 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517814+0000", 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2999999999999999e-07 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517820+0000", 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.677e-06 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517827+0000", 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2590000000000003e-06 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517834+0000", 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3069999999999999e-06 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517932+0000", 2024-09-21T08:26:45.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7885000000000002e-05 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.517938+0000", 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3099999999999997e-06 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.521071+0000", 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031335909999999998 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:45.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.521111+0000", 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9258000000000003e-05 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.532386+0000", 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011275432 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.532482+0000", 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6329000000000006e-05 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.405 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:05.532581+0000", 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8542999999999995e-05 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1594 7.16 7:6a4067f0:::smithi13779221-23:head [write 433680~466703] snapc 32=[32] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.326885+0000", 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 28.004043073999998, 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.62075412799999996, 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1594 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.326885+0000", 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.326885+0000", 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.326887+0000", 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9489999999999999e-06 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327091+0000", 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020473400000000001 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327103+0000", 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.153e-05 2024-09-21T08:26:45.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327108+0000", 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4489999999999998e-06 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327157+0000", 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8429e-05 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327164+0000", 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6899999999999992e-06 2024-09-21T08:26:45.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.925242+0000", 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59807755799999995 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928049+0000", 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028070149999999999 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.944633+0000", 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016583665000000001 2024-09-21T08:26:45.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.944714+0000", 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1816000000000004e-05 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.944826+0000", 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111823 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:45.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946814+0000", 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001987815 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946867+0000", 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3052e-05 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947560+0000", 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069266300000000005 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947595+0000", 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5394e-05 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947639+0000", 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3545999999999999e-05 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1595 7.16 7:6a4067f0:::smithi13779221-23:head [write 1577796~455884] snapc 32=[32] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.332973+0000", 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 27.997954676999999, 2024-09-21T08:26:45.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61484140200000004, 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1595 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.332973+0000", 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.332973+0000", 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.332975+0000", 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5680000000000001e-06 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333220+0000", 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024565600000000002 2024-09-21T08:26:45.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333230+0000", 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0223e-05 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333236+0000", 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5230000000000003e-06 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333274+0000", 2024-09-21T08:26:45.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8232999999999998e-05 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333280+0000", 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2040000000000002e-06 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928405+0000", 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59512471499999997 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928464+0000", 2024-09-21T08:26:45.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9123000000000003e-05 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.945609+0000", 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017145059000000001 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.945660+0000", 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0562999999999997e-05 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:26:45.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.945757+0000", 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6949999999999998e-05 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947123+0000", 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001366173 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947230+0000", 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010691700000000001 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947760+0000", 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053026199999999999 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947791+0000", 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0923000000000002e-05 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947814+0000", 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3309999999999999e-05 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1596 7.16 7:6a4067f0:::smithi13779221-23:head [setxattr _header (57) in=7b,truncate 2033680] snapc 32=[32] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.333246+0000", 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 27.9976816, 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61472346600000005, 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1596 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333246+0000", 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333246+0000", 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333247+0000", 2024-09-21T08:26:45.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1299999999999998e-07 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333251+0000", 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1699999999999999e-06 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333256+0000", 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3340000000000001e-06 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333262+0000", 2024-09-21T08:26:45.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2780000000000001e-06 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333310+0000", 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8572000000000002e-05 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333320+0000", 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8099999999999992e-06 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928508+0000", 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59518784499999999 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928553+0000", 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4545000000000001e-05 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946225+0000", 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017672832999999999 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946285+0000", 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9466999999999998e-05 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946398+0000", 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011340000000000001 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947240+0000", 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00084159800000000004 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947726+0000", 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048660799999999997 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947904+0000", 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017706899999999999 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947931+0000", 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7742e-05 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947970+0000", 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8281999999999997e-05 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1597 7.16 7:6a4067f0:::smithi13779221-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.333275+0000", 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 27.997652295999998, 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61483585500000004, 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.428 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1597 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333275+0000", 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333275+0000", 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333276+0000", 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2099999999999996e-07 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333278+0000", 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4229999999999999e-06 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333285+0000", 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0820000000000001e-06 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333291+0000", 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2089999999999996e-06 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333334+0000", 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3482999999999997e-05 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333338+0000", 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0169999999999998e-06 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928654+0000", 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59531576200000003 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928713+0000", 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8971999999999997e-05 2024-09-21T08:26:45.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946717+0000", 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018003894999999999 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946799+0000", 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2521000000000005e-05 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.948007+0000", 2024-09-21T08:26:45.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012079180000000001 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.948053+0000", 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5661000000000002e-05 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.948111+0000", 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8291000000000001e-05 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.434 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1742 7.d 7:b31d99ce:::smithi13779221-21:head [copy-from ver 99] snapc 39=[39] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:30.594911+0000", 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.736016355, 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021953376, 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1742 2024-09-21T08:26:45.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594911+0000", 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594911+0000", 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594913+0000", 2024-09-21T08:26:45.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2190000000000001e-06 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594919+0000", 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2049999999999999e-06 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594930+0000", 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1416e-05 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595001+0000", 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0678000000000003e-05 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595054+0000", 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2966000000000001e-05 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595167+0000", 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113167 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.605259+0000", 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010091829 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.610873+0000", 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056143 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616680+0000", 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058072159999999996 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616794+0000", 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113318 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616826+0000", 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2264999999999997e-05 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616865+0000", 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8797000000000001e-05 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.440 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1743 7.d 7:b31d99ce:::smithi13779221-21:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e197)", 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:30.595306+0000", 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 14.735621492, 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014710656000000001, 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.441 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1743 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595306+0000", 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595306+0000", 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595307+0000", 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1090000000000001e-06 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595349+0000", 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2190000000000001e-05 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595358+0000", 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0609999999999996e-06 2024-09-21T08:26:45.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595425+0000", 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7491999999999997e-05 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595519+0000", 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4378999999999999e-05 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595605+0000", 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5864000000000003e-05 2024-09-21T08:26:45.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.609862+0000", 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014257115000000001 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.609939+0000", 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6161999999999993e-05 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.610017+0000", 2024-09-21T08:26:45.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8283999999999998e-05 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1747 7.f 7:f1c8dcbe:::smithi13779221-1:head [copy-from ver 0] snapc 3a=[3a,39] ondisk+write+known_if_redirected+supports_pool_eio e198)", 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:31.512068+0000", 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.818859173, 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013572206999999999, 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.446 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1747 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512068+0000", 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512068+0000", 2024-09-21T08:26:45.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512072+0000", 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8430000000000003e-06 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512086+0000", 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4105e-05 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512102+0000", 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5610000000000001e-05 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512114+0000", 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2054e-05 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512179+0000", 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4548000000000006e-05 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512184+0000", 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9010000000000002e-06 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.520463+0000", 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082790009999999994 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.520602+0000", 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013953699999999999 2024-09-21T08:26:45.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.525641+0000", 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005038608 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1748 7.f 7:f1c8dcbe:::smithi13779221-1:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e198)", 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:31.512139+0000", 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 13.818789158, 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01381858, 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:26:45.451 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1748 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512139+0000", 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512139+0000", 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512139+0000", 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0400000000000005e-07 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512143+0000", 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.163e-06 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512151+0000", 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0669999999999992e-06 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512157+0000", 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3110000000000002e-06 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512242+0000", 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5154000000000005e-05 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.512249+0000", 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4960000000000001e-06 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.520843+0000", 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0085938999999999998 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.520899+0000", 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6613000000000002e-05 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.525733+0000", 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048334329999999998 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.525836+0000", 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103613 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:31.525957+0000", 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120926 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.456 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:45.457 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:45.457 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:26:45.457 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:45.478 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:45.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:45.479 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:45.580 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:45.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:45.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:45.605 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:45.608 INFO:tasks.rados.rados.0.smithi137.stdout:704: done (1 left) 2024-09-21T08:26:45.609 INFO:tasks.rados.rados.0.smithi137.stdout:705: done (0 left) 2024-09-21T08:26:45.609 INFO:tasks.rados.rados.0.smithi137.stdout:706: write oid 14 current snap is 65 2024-09-21T08:26:45.609 INFO:tasks.rados.rados.0.smithi137.stdout:706: seq_num 282 ranges {739574=725279,1968628=721093} 2024-09-21T08:26:45.617 INFO:tasks.rados.rados.0.smithi137.stdout:706: writing smithi13779221-14 from 739574 to 1464853 tid 1 2024-09-21T08:26:45.625 INFO:tasks.rados.rados.0.smithi137.stdout:706: writing smithi13779221-14 from 1968628 to 2689721 tid 2 2024-09-21T08:26:45.625 INFO:tasks.rados.rados.0.smithi137.stdout:707: read oid 32 snap -1 2024-09-21T08:26:45.625 INFO:tasks.rados.rados.0.smithi137.stdout:707: expect deleted 2024-09-21T08:26:45.625 INFO:tasks.rados.rados.0.smithi137.stdout:708: read oid 39 snap -1 2024-09-21T08:26:45.625 INFO:tasks.rados.rados.0.smithi137.stdout:708: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:26:45.625 INFO:tasks.rados.rados.0.smithi137.stdout:709: delete oid 26 current snap is 65 2024-09-21T08:26:45.628 INFO:tasks.rados.rados.0.smithi137.stdout:707: done (3 left) 2024-09-21T08:26:45.629 INFO:tasks.rados.rados.0.smithi137.stdout:709: done (2 left) 2024-09-21T08:26:45.629 INFO:tasks.rados.rados.0.smithi137.stdout:710: read oid 8 snap 64 2024-09-21T08:26:45.629 INFO:tasks.rados.rados.0.smithi137.stdout:710: expect (ObjNum 254 snap 55 seq_num 254) 2024-09-21T08:26:45.629 INFO:tasks.rados.rados.0.smithi137.stdout:711: read oid 21 snap -1 2024-09-21T08:26:45.629 INFO:tasks.rados.rados.0.smithi137.stdout:711: expect (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:26:45.629 INFO:tasks.rados.rados.0.smithi137.stdout:712: snap_create 2024-09-21T08:26:45.630 INFO:tasks.rados.rados.0.smithi137.stdout:706: finishing write tid 1 to smithi13779221-14 2024-09-21T08:26:45.630 INFO:tasks.rados.rados.0.smithi137.stdout:706: finishing write tid 2 to smithi13779221-14 2024-09-21T08:26:45.630 INFO:tasks.rados.rados.0.smithi137.stdout:706: finishing write tid 3 to smithi13779221-14 2024-09-21T08:26:45.630 INFO:tasks.rados.rados.0.smithi137.stdout:706: finishing write tid 4 to smithi13779221-14 2024-09-21T08:26:45.630 INFO:tasks.rados.rados.0.smithi137.stdout:706: oid 14 updating version 0 to 50 2024-09-21T08:26:45.630 INFO:tasks.rados.rados.0.smithi137.stdout:706: oid 14 updating version 50 to 51 2024-09-21T08:26:45.630 INFO:tasks.rados.rados.0.smithi137.stdout:706: oid 14 version 51 is already newer than 49 2024-09-21T08:26:45.630 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 51 (ObjNum 282 snap 65 seq_num 282) dirty exists 2024-09-21T08:26:45.630 INFO:tasks.rados.rados.0.smithi137.stdout:706: left oid 14 (ObjNum 282 snap 65 seq_num 282) 2024-09-21T08:26:45.632 INFO:tasks.rados.rados.0.smithi137.stdout:708: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:26:45.706 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:45.729 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:45.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:45.730 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:45.831 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:45.845 INFO:tasks.rados.rados.0.smithi137.stdout:710: expect (ObjNum 254 snap 55 seq_num 254) 2024-09-21T08:26:45.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:45.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:45.852 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:45.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:45.966 INFO:tasks.rados.rados.0.smithi137.stdout:711: expect (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:26:45.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:45.978 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:45.979 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: 7.c scrub starts 2024-09-21T08:26:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: 7.c scrub ok 2024-09-21T08:26:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: pgmap v221: 158 pgs: 158 active+clean; 140 MiB data, 503 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 962 KiB/s wr, 7 op/s 2024-09-21T08:26:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: 7.a scrub starts 2024-09-21T08:26:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: 7.a scrub ok 2024-09-21T08:26:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: osdmap e211: 8 total, 8 up, 4 in 2024-09-21T08:26:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: 5.e scrub starts 2024-09-21T08:26:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: 5.e scrub ok 2024-09-21T08:26:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4131277697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: 5.5 scrub starts 2024-09-21T08:26:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:45 smithi165 ceph-mon[32893]: 5.5 scrub ok 2024-09-21T08:26:46.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: 7.c scrub starts 2024-09-21T08:26:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: 7.c scrub ok 2024-09-21T08:26:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: pgmap v221: 158 pgs: 158 active+clean; 140 MiB data, 503 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 962 KiB/s wr, 7 op/s 2024-09-21T08:26:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: 7.a scrub starts 2024-09-21T08:26:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: 7.a scrub ok 2024-09-21T08:26:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: osdmap e211: 8 total, 8 up, 4 in 2024-09-21T08:26:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: 5.e scrub starts 2024-09-21T08:26:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: 5.e scrub ok 2024-09-21T08:26:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:46.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4131277697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:46.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: 5.5 scrub starts 2024-09-21T08:26:46.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[29634]: 5.5 scrub ok 2024-09-21T08:26:46.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: 7.c scrub starts 2024-09-21T08:26:46.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: 7.c scrub ok 2024-09-21T08:26:46.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: pgmap v221: 158 pgs: 158 active+clean; 140 MiB data, 503 MiB used, 357 GiB / 358 GiB avail; 2.8 MiB/s rd, 962 KiB/s wr, 7 op/s 2024-09-21T08:26:46.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: 7.a scrub starts 2024-09-21T08:26:46.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: 7.a scrub ok 2024-09-21T08:26:46.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: osdmap e211: 8 total, 8 up, 4 in 2024-09-21T08:26:46.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: 5.e scrub starts 2024-09-21T08:26:46.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: 5.e scrub ok 2024-09-21T08:26:46.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:26:46.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4131277697' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:46.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: 5.5 scrub starts 2024-09-21T08:26:46.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:45 smithi137 ceph-mon[41044]: 5.5 scrub ok 2024-09-21T08:26:46.080 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:46.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:46.101 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:46.101 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:46.202 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:46.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:46.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:46.224 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:46.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:46.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:46.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:46.347 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:46.396 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:26:46.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:46.474 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:46.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:46.475 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:46.517 INFO:tasks.rados.rados.0.smithi137.stdout:706: done (4 left) 2024-09-21T08:26:46.517 INFO:tasks.rados.rados.0.smithi137.stdout:708: done (3 left) 2024-09-21T08:26:46.517 INFO:tasks.rados.rados.0.smithi137.stdout:710: done (2 left) 2024-09-21T08:26:46.517 INFO:tasks.rados.rados.0.smithi137.stdout:711: done (1 left) 2024-09-21T08:26:46.518 INFO:tasks.rados.rados.0.smithi137.stdout:712: done (0 left) 2024-09-21T08:26:46.518 INFO:tasks.rados.rados.0.smithi137.stdout:713: read oid 18 snap -1 2024-09-21T08:26:46.518 INFO:tasks.rados.rados.0.smithi137.stdout:713: expect deleted 2024-09-21T08:26:46.518 INFO:tasks.rados.rados.0.smithi137.stdout:714: write oid 16 current snap is 66 2024-09-21T08:26:46.518 INFO:tasks.rados.rados.0.smithi137.stdout:714: seq_num 284 ranges {773188=475777,1969239=724762} 2024-09-21T08:26:46.525 INFO:tasks.rados.rados.0.smithi137.stdout:714: writing smithi13779221-16 from 773188 to 1248965 tid 1 2024-09-21T08:26:46.535 INFO:tasks.rados.rados.0.smithi137.stdout:714: writing smithi13779221-16 from 1969239 to 2694001 tid 2 2024-09-21T08:26:46.535 INFO:tasks.rados.rados.0.smithi137.stdout:715: write oid 23 current snap is 66 2024-09-21T08:26:46.535 INFO:tasks.rados.rados.0.smithi137.stdout:715: seq_num 285 ranges {789995=551026,2027721=468420} 2024-09-21T08:26:46.542 INFO:tasks.rados.rados.0.smithi137.stdout:715: writing smithi13779221-23 from 789995 to 1341021 tid 1 2024-09-21T08:26:46.550 INFO:tasks.rados.rados.0.smithi137.stdout:715: writing smithi13779221-23 from 2027721 to 2496141 tid 2 2024-09-21T08:26:46.550 INFO:tasks.rados.rados.0.smithi137.stdout:716: delete oid 19 current snap is 66 2024-09-21T08:26:46.550 INFO:tasks.rados.rados.0.smithi137.stdout:714: finishing write tid 1 to smithi13779221-16 2024-09-21T08:26:46.550 INFO:tasks.rados.rados.0.smithi137.stdout:714: finishing write tid 2 to smithi13779221-16 2024-09-21T08:26:46.550 INFO:tasks.rados.rados.0.smithi137.stdout:714: finishing write tid 3 to smithi13779221-16 2024-09-21T08:26:46.550 INFO:tasks.rados.rados.0.smithi137.stdout:714: finishing write tid 4 to smithi13779221-16 2024-09-21T08:26:46.550 INFO:tasks.rados.rados.0.smithi137.stdout:714: oid 16 updating version 0 to 105 2024-09-21T08:26:46.550 INFO:tasks.rados.rados.0.smithi137.stdout:714: oid 16 updating version 105 to 106 2024-09-21T08:26:46.550 INFO:tasks.rados.rados.0.smithi137.stdout:714: oid 16 updating version 106 to 107 2024-09-21T08:26:46.551 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 107 (ObjNum 284 snap 66 seq_num 284) dirty exists 2024-09-21T08:26:46.551 INFO:tasks.rados.rados.0.smithi137.stdout:714: left oid 16 (ObjNum 284 snap 66 seq_num 284) 2024-09-21T08:26:46.551 INFO:tasks.rados.rados.0.smithi137.stdout:715: finishing write tid 1 to smithi13779221-23 2024-09-21T08:26:46.552 INFO:tasks.rados.rados.0.smithi137.stdout:715: finishing write tid 2 to smithi13779221-23 2024-09-21T08:26:46.552 INFO:tasks.rados.rados.0.smithi137.stdout:715: finishing write tid 3 to smithi13779221-23 2024-09-21T08:26:46.552 INFO:tasks.rados.rados.0.smithi137.stdout:713: done (3 left) 2024-09-21T08:26:46.552 INFO:tasks.rados.rados.0.smithi137.stdout:714: done (2 left) 2024-09-21T08:26:46.552 INFO:tasks.rados.rados.0.smithi137.stdout:716: done (1 left) 2024-09-21T08:26:46.552 INFO:tasks.rados.rados.0.smithi137.stdout:717: read oid 18 snap -1 2024-09-21T08:26:46.552 INFO:tasks.rados.rados.0.smithi137.stdout:717: expect deleted 2024-09-21T08:26:46.552 INFO:tasks.rados.rados.0.smithi137.stdout:718: snap_remove snap 64 2024-09-21T08:26:46.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:46.597 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:46.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:46.598 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:46.699 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:46.723 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:46.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:46.724 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:46.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:46.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:46.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:46.848 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:46.904 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:46 smithi137 ceph-mon[29634]: 4.1c scrub starts 2024-09-21T08:26:46.904 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:46 smithi137 ceph-mon[29634]: 4.1c scrub ok 2024-09-21T08:26:46.904 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:46 smithi137 ceph-mon[29634]: osdmap e212: 8 total, 8 up, 4 in 2024-09-21T08:26:46.904 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:46 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:46.905 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:46 smithi137 ceph-mon[29634]: osdmap e213: 8 total, 8 up, 4 in 2024-09-21T08:26:46.905 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:46 smithi137 ceph-mon[41044]: 4.1c scrub starts 2024-09-21T08:26:46.905 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:46 smithi137 ceph-mon[41044]: 4.1c scrub ok 2024-09-21T08:26:46.905 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:46 smithi137 ceph-mon[41044]: osdmap e212: 8 total, 8 up, 4 in 2024-09-21T08:26:46.905 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:46 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:46.905 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:46 smithi137 ceph-mon[41044]: osdmap e213: 8 total, 8 up, 4 in 2024-09-21T08:26:46.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T08:26:46.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:46.989 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:46.991 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:46.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:46.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:47.000 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:46 smithi165 ceph-mon[32893]: 4.1c scrub starts 2024-09-21T08:26:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:46 smithi165 ceph-mon[32893]: 4.1c scrub ok 2024-09-21T08:26:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:46 smithi165 ceph-mon[32893]: osdmap e212: 8 total, 8 up, 4 in 2024-09-21T08:26:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:46 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:46 smithi165 ceph-mon[32893]: osdmap e213: 8 total, 8 up, 4 in 2024-09-21T08:26:47.101 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:47.122 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:47.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:47.123 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:47.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:47.250 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:47.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:47.251 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:47.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:47.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:47.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:47.379 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:47.480 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:47.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:47.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:47.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:47.517 INFO:tasks.rados.rados.0.smithi137.stdout:718: done (2 left) 2024-09-21T08:26:47.517 INFO:tasks.rados.rados.0.smithi137.stdout:719: snap_remove snap 65 2024-09-21T08:26:47.607 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:47.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:47.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:47.632 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:47.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[29634]: 6.6 scrub starts 2024-09-21T08:26:47.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[29634]: 6.6 scrub ok 2024-09-21T08:26:47.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[29634]: pgmap v224: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 154 MiB data, 534 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T08:26:47.661 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[29634]: 6.1 scrub starts 2024-09-21T08:26:47.662 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[29634]: 6.1 scrub ok 2024-09-21T08:26:47.662 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[29634]: osdmap e214: 8 total, 8 up, 4 in 2024-09-21T08:26:47.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[41044]: 6.6 scrub starts 2024-09-21T08:26:47.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[41044]: 6.6 scrub ok 2024-09-21T08:26:47.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[41044]: pgmap v224: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 154 MiB data, 534 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T08:26:47.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[41044]: 6.1 scrub starts 2024-09-21T08:26:47.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[41044]: 6.1 scrub ok 2024-09-21T08:26:47.662 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:47 smithi137 ceph-mon[41044]: osdmap e214: 8 total, 8 up, 4 in 2024-09-21T08:26:47.733 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:47.759 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:47.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:47.760 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:47.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:47.883 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:47.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:47.884 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:47.958 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:47.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:48.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:48.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:48.012 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:47 smithi165 ceph-mon[32893]: 6.6 scrub starts 2024-09-21T08:26:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:47 smithi165 ceph-mon[32893]: 6.6 scrub ok 2024-09-21T08:26:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:47 smithi165 ceph-mon[32893]: pgmap v224: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 154 MiB data, 534 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T08:26:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:47 smithi165 ceph-mon[32893]: 6.1 scrub starts 2024-09-21T08:26:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:47 smithi165 ceph-mon[32893]: 6.1 scrub ok 2024-09-21T08:26:48.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:47 smithi165 ceph-mon[32893]: osdmap e214: 8 total, 8 up, 4 in 2024-09-21T08:26:48.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:48.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:48.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:48.139 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:48.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:48.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:26:48.261 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:48.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:48.262 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:48.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:48.380 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29611]: 2024-09-21T08:26:48.097+0000 7f1058bb7640 -1 mon.a@0(leader).osd e214 definitely_dead 0 2024-09-21T08:26:48.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:48.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:48.391 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:48.492 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:48.515 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:26:48.515 INFO:teuthology.orchestra.run.smithi137.stderr:marked in osd.3. 2024-09-21T08:26:48.518 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:48.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:48.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:48.524 INFO:tasks.rados.rados.0.smithi137.stdout:719: done (2 left) 2024-09-21T08:26:48.524 INFO:tasks.rados.rados.0.smithi137.stdout:720: rollback oid 39 current snap is 66 2024-09-21T08:26:48.524 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 39 to 55 2024-09-21T08:26:48.524 INFO:tasks.rados.rados.0.smithi137.stdout:721: write oid 1 current snap is 66 2024-09-21T08:26:48.524 INFO:tasks.rados.rados.0.smithi137.stdout:721: seq_num 287 ranges {423609=701524,1628332=472089,2648987=174622} 2024-09-21T08:26:48.533 INFO:tasks.rados.rados.0.smithi137.stdout:721: writing smithi13779221-1 from 423609 to 1125133 tid 1 2024-09-21T08:26:48.543 INFO:tasks.rados.rados.0.smithi137.stdout:721: writing smithi13779221-1 from 1628332 to 2100421 tid 2 2024-09-21T08:26:48.548 INFO:tasks.rados.rados.0.smithi137.stdout:721: writing smithi13779221-1 from 2648987 to 2823609 tid 3 2024-09-21T08:26:48.549 INFO:tasks.rados.rados.0.smithi137.stdout:715: finishing write tid 4 to smithi13779221-23 2024-09-21T08:26:48.549 INFO:tasks.rados.rados.0.smithi137.stdout:715: oid 23 updating version 0 to 168 2024-09-21T08:26:48.549 INFO:tasks.rados.rados.0.smithi137.stdout:715: oid 23 updating version 168 to 170 2024-09-21T08:26:48.549 INFO:tasks.rados.rados.0.smithi137.stdout:715: oid 23 version 170 is already newer than 169 2024-09-21T08:26:48.549 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 170 (ObjNum 285 snap 66 seq_num 285) dirty exists 2024-09-21T08:26:48.549 INFO:tasks.rados.rados.0.smithi137.stdout:715: left oid 23 (ObjNum 285 snap 66 seq_num 285) 2024-09-21T08:26:48.549 INFO:tasks.rados.rados.0.smithi137.stdout:715: done (3 left) 2024-09-21T08:26:48.549 INFO:tasks.rados.rados.0.smithi137.stdout:717: done (2 left) 2024-09-21T08:26:48.550 INFO:tasks.rados.rados.0.smithi137.stdout:722: write (excl) oid 20 current snap is 66 2024-09-21T08:26:48.550 INFO:tasks.rados.rados.0.smithi137.stdout:722: seq_num 288 ranges {440416=776773,1686814=732100} 2024-09-21T08:26:48.557 INFO:tasks.rados.rados.0.smithi137.stdout:722: writing smithi13779221-20 from 440416 to 1217189 tid 1 2024-09-21T08:26:48.565 INFO:tasks.rados.rados.0.smithi137.stdout:722: writing smithi13779221-20 from 1686814 to 2418914 tid 2 2024-09-21T08:26:48.566 INFO:tasks.rados.rados.0.smithi137.stdout:723: delete oid 45 current snap is 66 2024-09-21T08:26:48.573 INFO:tasks.rados.rados.0.smithi137.stdout:722: finishing write tid 1 to smithi13779221-20 2024-09-21T08:26:48.574 INFO:tasks.rados.rados.0.smithi137.stdout:722: finishing write tid 2 to smithi13779221-20 2024-09-21T08:26:48.574 INFO:tasks.rados.rados.0.smithi137.stdout:722: finishing write tid 3 to smithi13779221-20 2024-09-21T08:26:48.574 INFO:tasks.rados.rados.0.smithi137.stdout:722: finishing write tid 4 to smithi13779221-20 2024-09-21T08:26:48.574 INFO:tasks.rados.rados.0.smithi137.stdout:722: oid 20 updating version 0 to 282 2024-09-21T08:26:48.574 INFO:tasks.rados.rados.0.smithi137.stdout:722: oid 20 version 282 is already newer than 281 2024-09-21T08:26:48.574 INFO:tasks.rados.rados.0.smithi137.stdout:722: oid 20 updating version 282 to 283 2024-09-21T08:26:48.574 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 283 (ObjNum 288 snap 66 seq_num 288) dirty exists 2024-09-21T08:26:48.574 INFO:tasks.rados.rados.0.smithi137.stdout:722: left oid 20 (ObjNum 288 snap 66 seq_num 288) 2024-09-21T08:26:48.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:48.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:48.643 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:48.643 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:48.647 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: 4.f scrub starts 2024-09-21T08:26:48.647 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: 4.f scrub ok 2024-09-21T08:26:48.647 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: 3.1d scrub starts 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: 3.1d scrub ok 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2769137850' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3106949187' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: 6.16 scrub starts 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: 6.16 scrub ok 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2769137850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3106949187' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[29634]: osdmap e215: 8 total, 8 up, 5 in 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: 4.f scrub starts 2024-09-21T08:26:48.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: 4.f scrub ok 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: 3.1d scrub starts 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: 3.1d scrub ok 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2769137850' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3106949187' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: 6.16 scrub starts 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: 6.16 scrub ok 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2769137850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3106949187' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-21T08:26:48.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:48 smithi137 ceph-mon[41044]: osdmap e215: 8 total, 8 up, 5 in 2024-09-21T08:26:48.650 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:26:48.650 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:26:48.650 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:26:48.650 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:26:48.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:48.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:48.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:48.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:48.867 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:48.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:48.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:48.893 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:48.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: 4.f scrub starts 2024-09-21T08:26:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: 4.f scrub ok 2024-09-21T08:26:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: 3.1d scrub starts 2024-09-21T08:26:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: 3.1d scrub ok 2024-09-21T08:26:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2769137850' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:26:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3106949187' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["3"]}]: dispatch 2024-09-21T08:26:49.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: 6.16 scrub starts 2024-09-21T08:26:49.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: 6.16 scrub ok 2024-09-21T08:26:49.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:26:49.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:26:49.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2769137850' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:26:49.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3106949187' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["3"]}]': finished 2024-09-21T08:26:49.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:48 smithi165 ceph-mon[32893]: osdmap e215: 8 total, 8 up, 5 in 2024-09-21T08:26:49.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:49.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:49.016 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:49.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:49.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:49.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:49.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:49.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:49.261 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:49.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:49.262 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:49.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T08:26:49.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:49.384 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:49.385 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:49.385 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:49.486 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:49.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:49.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:49.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:49.587 INFO:tasks.rados.rados.0.smithi137.stdout:720: finishing rollback tid 0 to smithi13779221-39 2024-09-21T08:26:49.588 INFO:tasks.rados.rados.0.smithi137.stdout:720: finishing rollback tid 1 to smithi13779221-39 2024-09-21T08:26:49.588 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 219 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-21T08:26:49.589 INFO:tasks.rados.rados.0.smithi137.stdout:721: finishing write tid 1 to smithi13779221-1 2024-09-21T08:26:49.591 INFO:tasks.rados.rados.0.smithi137.stdout:721: finishing write tid 2 to smithi13779221-1 2024-09-21T08:26:49.591 INFO:tasks.rados.rados.0.smithi137.stdout:721: finishing write tid 3 to smithi13779221-1 2024-09-21T08:26:49.591 INFO:tasks.rados.rados.0.smithi137.stdout:721: finishing write tid 4 to smithi13779221-1 2024-09-21T08:26:49.592 INFO:tasks.rados.rados.0.smithi137.stdout:721: finishing write tid 5 to smithi13779221-1 2024-09-21T08:26:49.592 INFO:tasks.rados.rados.0.smithi137.stdout:721: oid 1 updating version 0 to 251 2024-09-21T08:26:49.592 INFO:tasks.rados.rados.0.smithi137.stdout:721: oid 1 version 251 is already newer than 250 2024-09-21T08:26:49.592 INFO:tasks.rados.rados.0.smithi137.stdout:721: oid 1 updating version 251 to 252 2024-09-21T08:26:49.592 INFO:tasks.rados.rados.0.smithi137.stdout:721: oid 1 version 252 is already newer than 249 2024-09-21T08:26:49.592 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 252 (ObjNum 287 snap 66 seq_num 287) dirty exists 2024-09-21T08:26:49.592 INFO:tasks.rados.rados.0.smithi137.stdout:721: left oid 1 (ObjNum 287 snap 66 seq_num 287) 2024-09-21T08:26:49.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:49.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:49.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:49.646 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:49.747 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:49.768 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:49.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:49.769 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:49.870 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:49.892 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:49.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:49.893 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:49.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:49 smithi165 ceph-mon[32893]: 6.12 scrub starts 2024-09-21T08:26:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:49 smithi165 ceph-mon[32893]: 6.12 scrub ok 2024-09-21T08:26:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:49 smithi165 ceph-mon[32893]: pgmap v227: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 154 MiB data, 534 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T08:26:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:49 smithi165 ceph-mon[32893]: osdmap e216: 8 total, 8 up, 5 in 2024-09-21T08:26:50.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:50.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:50.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:50.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:49 smithi137 ceph-mon[29634]: 6.12 scrub starts 2024-09-21T08:26:50.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:49 smithi137 ceph-mon[29634]: 6.12 scrub ok 2024-09-21T08:26:50.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:49 smithi137 ceph-mon[29634]: pgmap v227: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 154 MiB data, 534 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T08:26:50.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:49 smithi137 ceph-mon[29634]: osdmap e216: 8 total, 8 up, 5 in 2024-09-21T08:26:50.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:49 smithi137 ceph-mon[41044]: 6.12 scrub starts 2024-09-21T08:26:50.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:49 smithi137 ceph-mon[41044]: 6.12 scrub ok 2024-09-21T08:26:50.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:49 smithi137 ceph-mon[41044]: pgmap v227: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 154 MiB data, 534 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 0 B/s wr, 20 op/s 2024-09-21T08:26:50.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:49 smithi137 ceph-mon[41044]: osdmap e216: 8 total, 8 up, 5 in 2024-09-21T08:26:50.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:50.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:50.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:50.157 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:50.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:50.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:50.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:50.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:50.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:50.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:50.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:50.402 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:50.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:50.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:50.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:50.531 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:50.552 INFO:tasks.thrashosds.thrasher:Added osd 3 2024-09-21T08:26:50.553 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:50.571 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:50.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:50.637 INFO:tasks.rados.rados.0.smithi137.stdout:720: done (3 left) 2024-09-21T08:26:50.637 INFO:tasks.rados.rados.0.smithi137.stdout:721: done (2 left) 2024-09-21T08:26:50.637 INFO:tasks.rados.rados.0.smithi137.stdout:722: done (1 left) 2024-09-21T08:26:50.637 INFO:tasks.rados.rados.0.smithi137.stdout:723: done (0 left) 2024-09-21T08:26:50.638 INFO:tasks.rados.rados.0.smithi137.stdout:724: write (excl) oid 42 current snap is 66 2024-09-21T08:26:50.638 INFO:tasks.rados.rados.0.smithi137.stdout:724: seq_num 290 ranges {474030=643624,1520131=735769} 2024-09-21T08:26:50.644 INFO:tasks.rados.rados.0.smithi137.stdout:724: writing smithi13779221-42 from 474030 to 1117654 tid 1 2024-09-21T08:26:50.651 INFO:tasks.rados.rados.0.smithi137.stdout:724: writing smithi13779221-42 from 1520131 to 2255900 tid 2 2024-09-21T08:26:50.651 INFO:tasks.rados.rados.0.smithi137.stdout:725: read oid 4 snap 55 2024-09-21T08:26:50.651 INFO:tasks.rados.rados.0.smithi137.stdout:725: expect (ObjNum 244 snap 52 seq_num 244) 2024-09-21T08:26:50.651 INFO:tasks.rados.rados.0.smithi137.stdout:726: read oid 35 snap -1 2024-09-21T08:26:50.651 INFO:tasks.rados.rados.0.smithi137.stdout:726: expect (ObjNum 270 snap 60 seq_num 270) 2024-09-21T08:26:50.651 INFO:tasks.rados.rados.0.smithi137.stdout:727: snap_create 2024-09-21T08:26:50.651 INFO:tasks.rados.rados.0.smithi137.stdout:724: finishing write tid 1 to smithi13779221-42 2024-09-21T08:26:50.655 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:50.655 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:50.656 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:50.656 INFO:tasks.rados.rados.0.smithi137.stdout:724: finishing write tid 2 to smithi13779221-42 2024-09-21T08:26:50.657 INFO:tasks.rados.rados.0.smithi137.stdout:725: expect (ObjNum 244 snap 52 seq_num 244) 2024-09-21T08:26:50.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:50.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:50 smithi137 ceph-mon[29634]: 7.14 scrub starts 2024-09-21T08:26:50.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:50 smithi137 ceph-mon[29634]: 7.14 scrub ok 2024-09-21T08:26:50.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:50 smithi137 ceph-mon[29634]: osdmap e217: 8 total, 8 up, 5 in 2024-09-21T08:26:50.768 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:26:50 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:26:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:26:50.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:50 smithi137 ceph-mon[41044]: 7.14 scrub starts 2024-09-21T08:26:50.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:50 smithi137 ceph-mon[41044]: 7.14 scrub ok 2024-09-21T08:26:50.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:50 smithi137 ceph-mon[41044]: osdmap e217: 8 total, 8 up, 5 in 2024-09-21T08:26:50.772 INFO:tasks.rados.rados.0.smithi137.stdout:726: expect (ObjNum 270 snap 60 seq_num 270) 2024-09-21T08:26:50.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:50.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:50.778 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:50.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:50.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:50.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:50.901 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:50.929 INFO:tasks.rados.rados.0.smithi137.stdout:724: finishing write tid 3 to smithi13779221-42 2024-09-21T08:26:50.929 INFO:tasks.rados.rados.0.smithi137.stdout:724: finishing write tid 4 to smithi13779221-42 2024-09-21T08:26:50.930 INFO:tasks.rados.rados.0.smithi137.stdout:724: oid 42 updating version 0 to 248 2024-09-21T08:26:50.930 INFO:tasks.rados.rados.0.smithi137.stdout:724: oid 42 version 248 is already newer than 246 2024-09-21T08:26:50.930 INFO:tasks.rados.rados.0.smithi137.stdout:724: oid 42 version 248 is already newer than 247 2024-09-21T08:26:50.930 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 248 (ObjNum 290 snap 66 seq_num 290) dirty exists 2024-09-21T08:26:50.930 INFO:tasks.rados.rados.0.smithi137.stdout:724: left oid 42 (ObjNum 290 snap 66 seq_num 290) 2024-09-21T08:26:51.002 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:50 smithi165 ceph-mon[32893]: 7.14 scrub starts 2024-09-21T08:26:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:50 smithi165 ceph-mon[32893]: 7.14 scrub ok 2024-09-21T08:26:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:50 smithi165 ceph-mon[32893]: osdmap e217: 8 total, 8 up, 5 in 2024-09-21T08:26:51.023 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:51.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:51.024 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:51.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:51.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:51.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:51.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:51.247 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:51.268 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:51.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:51.269 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:51.319 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:51.319 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:51.335 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:51.335 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:51.351 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:51.351 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:51.367 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:51.367 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:51.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:51.383 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:51.383 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:51.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:51.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:51.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:51.400 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:51.400 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:51.416 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:51.416 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:51.433 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:51.434 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:51.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:51.518 INFO:tasks.rados.rados.0.smithi137.stdout:724: done (3 left) 2024-09-21T08:26:51.518 INFO:tasks.rados.rados.0.smithi137.stdout:725: done (2 left) 2024-09-21T08:26:51.518 INFO:tasks.rados.rados.0.smithi137.stdout:726: done (1 left) 2024-09-21T08:26:51.518 INFO:tasks.rados.rados.0.smithi137.stdout:727: done (0 left) 2024-09-21T08:26:51.518 INFO:tasks.rados.rados.0.smithi137.stdout:728: write (excl) oid 30 current snap is 67 2024-09-21T08:26:51.518 INFO:tasks.rados.rados.0.smithi137.stdout:728: seq_num 291 ranges {490837=718873,1862260=479427,2875032=15805} 2024-09-21T08:26:51.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:51.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:51.520 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:51.525 INFO:tasks.rados.rados.0.smithi137.stdout:728: writing smithi13779221-30 from 490837 to 1209710 tid 1 2024-09-21T08:26:51.534 INFO:tasks.rados.rados.0.smithi137.stdout:728: writing smithi13779221-30 from 1862260 to 2341687 tid 2 2024-09-21T08:26:51.536 INFO:tasks.rados.rados.0.smithi137.stdout:728: writing smithi13779221-30 from 2875032 to 2890837 tid 3 2024-09-21T08:26:51.537 INFO:tasks.rados.rados.0.smithi137.stdout:729: write (excl) oid 3 current snap is 67 2024-09-21T08:26:51.537 INFO:tasks.rados.rados.0.smithi137.stdout:728: finishing write tid 1 to smithi13779221-30 2024-09-21T08:26:51.537 INFO:tasks.rados.rados.0.smithi137.stdout:729: seq_num 292 ranges {507644=794122,1920742=623085} 2024-09-21T08:26:51.541 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:51.541 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:51.544 INFO:tasks.rados.rados.0.smithi137.stdout:729: writing smithi13779221-3 from 507644 to 1301766 tid 1 2024-09-21T08:26:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:729: writing smithi13779221-3 from 1920742 to 2543827 tid 2 2024-09-21T08:26:51.552 INFO:tasks.rados.rados.0.smithi137.stdout:730: delete oid 10 current snap is 67 2024-09-21T08:26:51.552 INFO:tasks.rados.rados.0.smithi137.stdout:728: finishing write tid 2 to smithi13779221-30 2024-09-21T08:26:51.552 INFO:tasks.rados.rados.0.smithi137.stdout:728: finishing write tid 3 to smithi13779221-30 2024-09-21T08:26:51.552 INFO:tasks.rados.rados.0.smithi137.stdout:728: finishing write tid 4 to smithi13779221-30 2024-09-21T08:26:51.552 INFO:tasks.rados.rados.0.smithi137.stdout:728: finishing write tid 5 to smithi13779221-30 2024-09-21T08:26:51.552 INFO:tasks.rados.rados.0.smithi137.stdout:728: oid 30 updating version 0 to 329 2024-09-21T08:26:51.552 INFO:tasks.rados.rados.0.smithi137.stdout:728: oid 30 version 329 is already newer than 327 2024-09-21T08:26:51.552 INFO:tasks.rados.rados.0.smithi137.stdout:728: oid 30 version 329 is already newer than 328 2024-09-21T08:26:51.553 INFO:tasks.rados.rados.0.smithi137.stdout:728: oid 30 version 329 is already newer than 326 2024-09-21T08:26:51.553 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 329 (ObjNum 291 snap 67 seq_num 291) dirty exists 2024-09-21T08:26:51.553 INFO:tasks.rados.rados.0.smithi137.stdout:728: left oid 30 (ObjNum 291 snap 67 seq_num 291) 2024-09-21T08:26:51.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:51.643 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:51.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:51.644 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:51.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:51.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:51.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:51.768 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:51.869 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:51.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:51.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:51.892 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:51.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:51 smithi165 ceph-mon[32893]: 3.15 scrub starts 2024-09-21T08:26:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:51 smithi165 ceph-mon[32893]: 3.15 scrub ok 2024-09-21T08:26:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:51 smithi165 ceph-mon[32893]: pgmap v230: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 154 MiB data, 534 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:51 smithi165 ceph-mon[32893]: 7.3 scrub starts 2024-09-21T08:26:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:51 smithi165 ceph-mon[32893]: 7.3 scrub ok 2024-09-21T08:26:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:51 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3797047615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:51 smithi165 ceph-mon[32893]: osdmap e218: 8 total, 8 up, 5 in 2024-09-21T08:26:52.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:26:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:26:52.016 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:52.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:52.017 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[29634]: 3.15 scrub starts 2024-09-21T08:26:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[29634]: 3.15 scrub ok 2024-09-21T08:26:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[29634]: pgmap v230: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 154 MiB data, 534 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[29634]: 7.3 scrub starts 2024-09-21T08:26:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[29634]: 7.3 scrub ok 2024-09-21T08:26:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3797047615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:52.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[29634]: osdmap e218: 8 total, 8 up, 5 in 2024-09-21T08:26:52.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[41044]: 3.15 scrub starts 2024-09-21T08:26:52.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[41044]: 3.15 scrub ok 2024-09-21T08:26:52.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[41044]: pgmap v230: 158 pgs: 5 active+clean+snaptrim, 153 active+clean; 154 MiB data, 534 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:26:52.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[41044]: 7.3 scrub starts 2024-09-21T08:26:52.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[41044]: 7.3 scrub ok 2024-09-21T08:26:52.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3797047615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:52.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:51 smithi137 ceph-mon[41044]: osdmap e218: 8 total, 8 up, 5 in 2024-09-21T08:26:52.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:26:52.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:52.139 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:52.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:52.140 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:52.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:52.262 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:52.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:52.263 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:52.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:52.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:52.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:52.389 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:52.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:52.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:52.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:52.514 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:52.537 INFO:tasks.rados.rados.0.smithi137.stdout:728: done (2 left) 2024-09-21T08:26:52.537 INFO:tasks.rados.rados.0.smithi137.stdout:730: done (1 left) 2024-09-21T08:26:52.537 INFO:tasks.rados.rados.0.smithi137.stdout:731: read oid 17 snap -1 2024-09-21T08:26:52.537 INFO:tasks.rados.rados.0.smithi137.stdout:731: expect (ObjNum 192 snap 39 seq_num 192) 2024-09-21T08:26:52.538 INFO:tasks.rados.rados.0.smithi137.stdout:732: copy_from oid 50 from oid 42 current snap is 67 2024-09-21T08:26:52.538 INFO:tasks.rados.rados.0.smithi137.stdout:733: rollback oid 24 current snap is 67 2024-09-21T08:26:52.538 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 24 to 55 2024-09-21T08:26:52.538 INFO:tasks.rados.rados.0.smithi137.stdout:734: read oid 27 snap -1 2024-09-21T08:26:52.538 INFO:tasks.rados.rados.0.smithi137.stdout:734: expect (ObjNum 193 snap 39 seq_num 193) 2024-09-21T08:26:52.538 INFO:tasks.rados.rados.0.smithi137.stdout:735: snap_create 2024-09-21T08:26:52.538 INFO:tasks.rados.rados.0.smithi137.stdout:729: finishing write tid 1 to smithi13779221-3 2024-09-21T08:26:52.540 INFO:tasks.rados.rados.0.smithi137.stdout:729: finishing write tid 2 to smithi13779221-3 2024-09-21T08:26:52.540 INFO:tasks.rados.rados.0.smithi137.stdout:729: finishing write tid 3 to smithi13779221-3 2024-09-21T08:26:52.540 INFO:tasks.rados.rados.0.smithi137.stdout:729: finishing write tid 4 to smithi13779221-3 2024-09-21T08:26:52.540 INFO:tasks.rados.rados.0.smithi137.stdout:729: oid 3 updating version 0 to 167 2024-09-21T08:26:52.540 INFO:tasks.rados.rados.0.smithi137.stdout:729: oid 3 version 167 is already newer than 166 2024-09-21T08:26:52.540 INFO:tasks.rados.rados.0.smithi137.stdout:729: oid 3 updating version 167 to 168 2024-09-21T08:26:52.540 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 168 (ObjNum 292 snap 67 seq_num 292) dirty exists 2024-09-21T08:26:52.540 INFO:tasks.rados.rados.0.smithi137.stdout:729: left oid 3 (ObjNum 292 snap 67 seq_num 292) 2024-09-21T08:26:52.543 INFO:tasks.rados.rados.0.smithi137.stdout:731: expect (ObjNum 192 snap 39 seq_num 192) 2024-09-21T08:26:52.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:52.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:52.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:52.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:52.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:52.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:52.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:52.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:52.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:52 smithi137 ceph-mon[41044]: 6.e scrub starts 2024-09-21T08:26:52.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:52 smithi137 ceph-mon[41044]: 6.e scrub ok 2024-09-21T08:26:52.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:52 smithi137 ceph-mon[41044]: osdmap e219: 8 total, 8 up, 5 in 2024-09-21T08:26:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:52 smithi137 ceph-mon[29634]: 6.e scrub starts 2024-09-21T08:26:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:52 smithi137 ceph-mon[29634]: 6.e scrub ok 2024-09-21T08:26:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:52 smithi137 ceph-mon[29634]: osdmap e219: 8 total, 8 up, 5 in 2024-09-21T08:26:52.769 INFO:tasks.rados.rados.0.smithi137.stdout:733: finishing rollback tid 0 to smithi13779221-24 2024-09-21T08:26:52.769 INFO:tasks.rados.rados.0.smithi137.stdout:733: finishing rollback tid 1 to smithi13779221-24 2024-09-21T08:26:52.769 INFO:tasks.rados.rados.0.smithi137.stdout:733: finishing rollback tid 2 to smithi13779221-24 2024-09-21T08:26:52.769 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 234 (ObjNum 192 snap 39 seq_num 192) dirty exists 2024-09-21T08:26:52.769 INFO:tasks.rados.rados.0.smithi137.stdout:734: expect (ObjNum 193 snap 39 seq_num 193) 2024-09-21T08:26:52.817 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T08:26:52.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:52.883 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:52.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:52.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:52.887 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:52.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:52 smithi165 ceph-mon[32893]: 6.e scrub starts 2024-09-21T08:26:52.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:52 smithi165 ceph-mon[32893]: 6.e scrub ok 2024-09-21T08:26:52.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:52 smithi165 ceph-mon[32893]: osdmap e219: 8 total, 8 up, 5 in 2024-09-21T08:26:52.935 INFO:tasks.rados.rados.0.smithi137.stdout:732: finishing copy_from racing read to smithi13779221-50 2024-09-21T08:26:52.935 INFO:tasks.rados.rados.0.smithi137.stdout:732: finishing copy_from to smithi13779221-50 2024-09-21T08:26:52.935 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 248 (ObjNum 290 snap 66 seq_num 290) dirty exists 2024-09-21T08:26:52.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:53.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:53.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:53.011 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:53.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:53.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:53.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:53.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:53.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:53.258 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:53.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:53.259 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:53.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:53.381 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:53.382 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:53.464 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:53.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:53.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:53.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:53.506 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:53.531 INFO:tasks.rados.rados.0.smithi137.stdout:729: done (5 left) 2024-09-21T08:26:53.531 INFO:tasks.rados.rados.0.smithi137.stdout:731: done (4 left) 2024-09-21T08:26:53.531 INFO:tasks.rados.rados.0.smithi137.stdout:732: done (3 left) 2024-09-21T08:26:53.531 INFO:tasks.rados.rados.0.smithi137.stdout:733: done (2 left) 2024-09-21T08:26:53.531 INFO:tasks.rados.rados.0.smithi137.stdout:734: done (1 left) 2024-09-21T08:26:53.531 INFO:tasks.rados.rados.0.smithi137.stdout:735: done (0 left) 2024-09-21T08:26:53.531 INFO:tasks.rados.rados.0.smithi137.stdout:736: copy_from oid 6 from oid 24 current snap is 68 2024-09-21T08:26:53.531 INFO:tasks.rados.rados.0.smithi137.stdout:737: snap_create 2024-09-21T08:26:53.549 INFO:tasks.rados.rados.0.smithi137.stdout:736: finishing copy_from racing read to smithi13779221-6 2024-09-21T08:26:53.555 INFO:tasks.rados.rados.0.smithi137.stdout:736: finishing copy_from to smithi13779221-6 2024-09-21T08:26:53.555 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 234 (ObjNum 192 snap 39 seq_num 192) dirty exists 2024-09-21T08:26:53.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:53.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:53.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:53.630 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:53.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:53.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:53.753 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:53.853 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:53.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:53.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: 3.e scrub starts 2024-09-21T08:26:53.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: 3.e scrub ok 2024-09-21T08:26:53.862 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: pgmap v233: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+snaptrim_wait, 2 active+remapped, 2 active+undersized+remapped, 2 active+clean+snaptrim, 149 active+clean; 162 MiB data, 585 MiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s; 1/804 objects degraded (0.124%); 1/804 objects misplaced (0.124%); 14 MiB/s, 45 objects/s recovering 2024-09-21T08:26:53.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: 3.0 scrub starts 2024-09-21T08:26:53.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: 3.0 scrub ok 2024-09-21T08:26:53.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: Health check failed: Degraded data redundancy: 1/804 objects degraded (0.124%), 1 pg degraded (PG_DEGRADED) 2024-09-21T08:26:53.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: 3.2 scrub starts 2024-09-21T08:26:53.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: 3.2 scrub ok 2024-09-21T08:26:53.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: 7.6 scrub starts 2024-09-21T08:26:53.863 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: 7.6 scrub ok 2024-09-21T08:26:53.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: osdmap e220: 8 total, 8 up, 5 in 2024-09-21T08:26:53.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/649714715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:53.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[41044]: from='client.45119 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:53.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:53.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: 3.e scrub starts 2024-09-21T08:26:53.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: 3.e scrub ok 2024-09-21T08:26:53.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: pgmap v233: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+snaptrim_wait, 2 active+remapped, 2 active+undersized+remapped, 2 active+clean+snaptrim, 149 active+clean; 162 MiB data, 585 MiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s; 1/804 objects degraded (0.124%); 1/804 objects misplaced (0.124%); 14 MiB/s, 45 objects/s recovering 2024-09-21T08:26:53.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: 3.0 scrub starts 2024-09-21T08:26:53.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: 3.0 scrub ok 2024-09-21T08:26:53.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: Health check failed: Degraded data redundancy: 1/804 objects degraded (0.124%), 1 pg degraded (PG_DEGRADED) 2024-09-21T08:26:53.865 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: 3.2 scrub starts 2024-09-21T08:26:53.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: 3.2 scrub ok 2024-09-21T08:26:53.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: 7.6 scrub starts 2024-09-21T08:26:53.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: 7.6 scrub ok 2024-09-21T08:26:53.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: osdmap e220: 8 total, 8 up, 5 in 2024-09-21T08:26:53.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/649714715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:53.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:53 smithi137 ceph-mon[29634]: from='client.45119 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:53.878 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:53.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:53.970 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:26:53.970 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:26:53.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:54.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:54.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:54.001 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:26:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: 3.e scrub starts 2024-09-21T08:26:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: 3.e scrub ok 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: pgmap v233: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+snaptrim_wait, 2 active+remapped, 2 active+undersized+remapped, 2 active+clean+snaptrim, 149 active+clean; 162 MiB data, 585 MiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s; 1/804 objects degraded (0.124%); 1/804 objects misplaced (0.124%); 14 MiB/s, 45 objects/s recovering 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: 3.0 scrub starts 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: 3.0 scrub ok 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: Health check failed: Degraded data redundancy: 1/804 objects degraded (0.124%), 1 pg degraded (PG_DEGRADED) 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: 3.2 scrub starts 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: 3.2 scrub ok 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: 7.6 scrub starts 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: 7.6 scrub ok 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: osdmap e220: 8 total, 8 up, 5 in 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/649714715' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:53 smithi165 ceph-mon[32893]: from='client.45119 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:26:54.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:54.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:54.125 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:54.125 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:54.139 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:26:54.226 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:54.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:54.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:54.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:54.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:54.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:54.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:54.371 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:54.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:54.493 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:54.494 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:54.515 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:54.532 INFO:tasks.rados.rados.0.smithi137.stdout:736: done (1 left) 2024-09-21T08:26:54.533 INFO:tasks.rados.rados.0.smithi137.stdout:737: done (0 left) 2024-09-21T08:26:54.533 INFO:tasks.rados.rados.0.smithi137.stdout:738: write (excl) oid 13 current snap is 69 2024-09-21T08:26:54.533 INFO:tasks.rados.rados.0.smithi137.stdout:738: seq_num 294 ranges {541258=544620,1637706=626754,2840948=100310} 2024-09-21T08:26:54.533 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:26:54.539 INFO:tasks.rados.rados.0.smithi137.stdout:738: writing smithi13779221-13 from 541258 to 1085878 tid 1 2024-09-21T08:26:54.546 INFO:tasks.rados.rados.0.smithi137.stdout:738: writing smithi13779221-13 from 1637706 to 2264460 tid 2 2024-09-21T08:26:54.551 INFO:tasks.rados.rados.0.smithi137.stdout:738: writing smithi13779221-13 from 2840948 to 2941258 tid 3 2024-09-21T08:26:54.551 INFO:tasks.rados.rados.0.smithi137.stdout:739: rollback oid 12 current snap is 69 2024-09-21T08:26:54.551 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 12 to 62 2024-09-21T08:26:54.551 INFO:tasks.rados.rados.0.smithi137.stdout:738: finishing write tid 1 to smithi13779221-13 2024-09-21T08:26:54.551 INFO:tasks.rados.rados.0.smithi137.stdout:740: delete oid 49 current snap is 69 2024-09-21T08:26:54.551 INFO:tasks.rados.rados.0.smithi137.stdout:738: finishing write tid 2 to smithi13779221-13 2024-09-21T08:26:54.553 INFO:tasks.rados.rados.0.smithi137.stdout:738: finishing write tid 3 to smithi13779221-13 2024-09-21T08:26:54.553 INFO:tasks.rados.rados.0.smithi137.stdout:738: finishing write tid 4 to smithi13779221-13 2024-09-21T08:26:54.553 INFO:tasks.rados.rados.0.smithi137.stdout:739: finishing rollback tid 0 to smithi13779221-12 2024-09-21T08:26:54.553 INFO:tasks.rados.rados.0.smithi137.stdout:738: finishing write tid 5 to smithi13779221-13 2024-09-21T08:26:54.553 INFO:tasks.rados.rados.0.smithi137.stdout:738: oid 13 updating version 0 to 333 2024-09-21T08:26:54.553 INFO:tasks.rados.rados.0.smithi137.stdout:738: oid 13 version 333 is already newer than 331 2024-09-21T08:26:54.553 INFO:tasks.rados.rados.0.smithi137.stdout:738: oid 13 version 333 is already newer than 332 2024-09-21T08:26:54.553 INFO:tasks.rados.rados.0.smithi137.stdout:738: oid 13 updating version 333 to 334 2024-09-21T08:26:54.553 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 334 (ObjNum 294 snap 69 seq_num 294) dirty exists 2024-09-21T08:26:54.553 INFO:tasks.rados.rados.0.smithi137.stdout:738: left oid 13 (ObjNum 294 snap 69 seq_num 294) 2024-09-21T08:26:54.554 INFO:tasks.rados.rados.0.smithi137.stdout:739: finishing rollback tid 1 to smithi13779221-12 2024-09-21T08:26:54.554 INFO:tasks.rados.rados.0.smithi137.stdout:739: finishing rollback tid 2 to smithi13779221-12 2024-09-21T08:26:54.554 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 276 (ObjNum 201 snap 40 seq_num 201) dirty exists 2024-09-21T08:26:54.555 INFO:tasks.rados.rados.0.smithi137.stdout:738: done (2 left) 2024-09-21T08:26:54.555 INFO:tasks.rados.rados.0.smithi137.stdout:739: done (1 left) 2024-09-21T08:26:54.555 INFO:tasks.rados.rados.0.smithi137.stdout:740: done (0 left) 2024-09-21T08:26:54.555 INFO:tasks.rados.rados.0.smithi137.stdout:741: delete oid 14 current snap is 69 2024-09-21T08:26:54.557 INFO:tasks.rados.rados.0.smithi137.stdout:741: done (0 left) 2024-09-21T08:26:54.558 INFO:tasks.rados.rados.0.smithi137.stdout:742: read oid 5 snap -1 2024-09-21T08:26:54.558 INFO:tasks.rados.rados.0.smithi137.stdout:742: expect (ObjNum 250 snap 53 seq_num 250) 2024-09-21T08:26:54.558 INFO:tasks.rados.rados.0.smithi137.stdout:743: read oid 48 snap 66 2024-09-21T08:26:54.558 INFO:tasks.rados.rados.0.smithi137.stdout:743: expect (ObjNum 259 snap 55 seq_num 259) 2024-09-21T08:26:54.558 INFO:tasks.rados.rados.0.smithi137.stdout:744: write oid 41 current snap is 69 2024-09-21T08:26:54.558 INFO:tasks.rados.rados.0.smithi137.stdout:744: seq_num 297 ranges {591679=486720,1813152=490434} 2024-09-21T08:26:54.565 INFO:tasks.rados.rados.0.smithi137.stdout:744: writing smithi13779221-41 from 591679 to 1078399 tid 1 2024-09-21T08:26:54.572 INFO:tasks.rados.rados.0.smithi137.stdout:744: writing smithi13779221-41 from 1813152 to 2303586 tid 2 2024-09-21T08:26:54.572 INFO:tasks.rados.rados.0.smithi137.stdout:745: snap_remove snap 55 2024-09-21T08:26:54.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:54.617 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:54.618 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:54.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:54.740 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:54.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:54.741 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:54.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:54.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:54.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:54.864 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:54.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:26:54.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:26:54.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[29634]: 5.16 deep-scrub starts 2024-09-21T08:26:54.951 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[29634]: 5.16 deep-scrub ok 2024-09-21T08:26:54.951 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[29634]: 4.0 scrub starts 2024-09-21T08:26:54.951 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[29634]: 4.0 scrub ok 2024-09-21T08:26:54.951 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:54.951 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[29634]: from='client.45119 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:26:54.951 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[29634]: osdmap e221: 8 total, 8 up, 5 in 2024-09-21T08:26:54.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:26:54.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:26:54.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[41044]: 5.16 deep-scrub starts 2024-09-21T08:26:54.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[41044]: 5.16 deep-scrub ok 2024-09-21T08:26:54.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[41044]: 4.0 scrub starts 2024-09-21T08:26:54.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[41044]: 4.0 scrub ok 2024-09-21T08:26:54.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:54.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[41044]: from='client.45119 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:26:54.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:54 smithi137 ceph-mon[41044]: osdmap e221: 8 total, 8 up, 5 in 2024-09-21T08:26:54.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:54.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T08:26:54.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:54.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:54.990 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:54 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:26:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:54 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:26:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:54 smithi165 ceph-mon[32893]: 5.16 deep-scrub starts 2024-09-21T08:26:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:54 smithi165 ceph-mon[32893]: 5.16 deep-scrub ok 2024-09-21T08:26:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:54 smithi165 ceph-mon[32893]: 4.0 scrub starts 2024-09-21T08:26:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:54 smithi165 ceph-mon[32893]: 4.0 scrub ok 2024-09-21T08:26:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:54 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:26:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:54 smithi165 ceph-mon[32893]: from='client.45119 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:26:55.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:54 smithi165 ceph-mon[32893]: osdmap e221: 8 total, 8 up, 5 in 2024-09-21T08:26:55.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:55.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:55.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:55.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:55.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:55.236 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:55.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:55.237 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:55.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:55.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:55.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:55.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:55.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:55.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:55.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:55.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:55.531 INFO:tasks.rados.rados.0.smithi137.stdout:745: done (3 left) 2024-09-21T08:26:55.531 INFO:tasks.rados.rados.0.smithi137.stdout:746: snap_create 2024-09-21T08:26:55.531 INFO:tasks.rados.rados.0.smithi137.stdout:742: expect (ObjNum 250 snap 53 seq_num 250) 2024-09-21T08:26:55.552 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [4, 6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:26:55.553 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:26:55.553 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T08:26:55.553 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T08:26:55.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:55.617 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:55.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:55.618 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:55.640 INFO:tasks.rados.rados.0.smithi137.stdout:743: expect (ObjNum 259 snap 55 seq_num 259) 2024-09-21T08:26:55.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:55.742 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:55.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:55.743 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:55.766 INFO:tasks.rados.rados.0.smithi137.stdout:744: finishing write tid 1 to smithi13779221-41 2024-09-21T08:26:55.767 INFO:tasks.rados.rados.0.smithi137.stdout:744: finishing write tid 2 to smithi13779221-41 2024-09-21T08:26:55.767 INFO:tasks.rados.rados.0.smithi137.stdout:744: finishing write tid 3 to smithi13779221-41 2024-09-21T08:26:55.767 INFO:tasks.rados.rados.0.smithi137.stdout:744: finishing write tid 4 to smithi13779221-41 2024-09-21T08:26:55.767 INFO:tasks.rados.rados.0.smithi137.stdout:744: oid 41 updating version 0 to 265 2024-09-21T08:26:55.767 INFO:tasks.rados.rados.0.smithi137.stdout:744: oid 41 updating version 265 to 266 2024-09-21T08:26:55.767 INFO:tasks.rados.rados.0.smithi137.stdout:744: oid 41 version 266 is already newer than 264 2024-09-21T08:26:55.767 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 266 (ObjNum 297 snap 69 seq_num 297) dirty exists 2024-09-21T08:26:55.767 INFO:tasks.rados.rados.0.smithi137.stdout:744: left oid 41 (ObjNum 297 snap 69 seq_num 297) 2024-09-21T08:26:55.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:55.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:55.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:55.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:55.867 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:55 smithi137 ceph-mon[29634]: pgmap v236: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+snaptrim_wait, 2 active+remapped, 2 active+undersized+remapped, 2 active+clean+snaptrim, 149 active+clean; 162 MiB data, 585 MiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s; 1/804 objects degraded (0.124%); 1/804 objects misplaced (0.124%); 14 MiB/s, 45 objects/s recovering 2024-09-21T08:26:55.867 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:55 smithi137 ceph-mon[29634]: osdmap e222: 8 total, 8 up, 5 in 2024-09-21T08:26:55.867 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:55 smithi137 ceph-mon[41044]: pgmap v236: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+snaptrim_wait, 2 active+remapped, 2 active+undersized+remapped, 2 active+clean+snaptrim, 149 active+clean; 162 MiB data, 585 MiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s; 1/804 objects degraded (0.124%); 1/804 objects misplaced (0.124%); 14 MiB/s, 45 objects/s recovering 2024-09-21T08:26:55.867 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:55 smithi137 ceph-mon[41044]: osdmap e222: 8 total, 8 up, 5 in 2024-09-21T08:26:55.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:55.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:55.989 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:55.989 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:55 smithi165 ceph-mon[32893]: pgmap v236: 158 pgs: 1 active+recovering+undersized+degraded+remapped, 2 active+clean+snaptrim_wait, 2 active+remapped, 2 active+undersized+remapped, 2 active+clean+snaptrim, 149 active+clean; 162 MiB data, 585 MiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 1.8 MiB/s wr, 13 op/s; 1/804 objects degraded (0.124%); 1/804 objects misplaced (0.124%); 14 MiB/s, 45 objects/s recovering 2024-09-21T08:26:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:55 smithi165 ceph-mon[32893]: osdmap e222: 8 total, 8 up, 5 in 2024-09-21T08:26:56.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:56.091 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:26:56.091 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:26:56.108 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:26:56.108 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:26:56.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:56.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:56.112 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:56.126 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:26:56.126 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:26:56.131 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:56.131 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:56.146 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:26:56.146 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:26:56.164 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:26:56.164 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:26:56.181 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:26:56.181 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:26:56.197 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:26:56.198 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:26:56.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:56.214 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:26:56.214 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:26:56.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:56.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:56.236 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:56.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:56.358 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:56.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:56.359 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:56.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:56.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:56.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:56.483 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:56.519 INFO:tasks.rados.rados.0.smithi137.stdout:742: done (3 left) 2024-09-21T08:26:56.519 INFO:tasks.rados.rados.0.smithi137.stdout:743: done (2 left) 2024-09-21T08:26:56.519 INFO:tasks.rados.rados.0.smithi137.stdout:744: done (1 left) 2024-09-21T08:26:56.519 INFO:tasks.rados.rados.0.smithi137.stdout:746: done (0 left) 2024-09-21T08:26:56.519 INFO:tasks.rados.rados.0.smithi137.stdout:747: read oid 10 snap 68 2024-09-21T08:26:56.519 INFO:tasks.rados.rados.0.smithi137.stdout:747: expect deleted 2024-09-21T08:26:56.520 INFO:tasks.rados.rados.0.smithi137.stdout:748: write oid 26 current snap is 70 2024-09-21T08:26:56.520 INFO:tasks.rados.rados.0.smithi137.stdout:748: seq_num 298 ranges {608486=561969,1871634=634092,2950640=57846} 2024-09-21T08:26:56.525 INFO:tasks.rados.rados.0.smithi137.stdout:748: writing smithi13779221-26 from 608486 to 1170455 tid 1 2024-09-21T08:26:56.533 INFO:tasks.rados.rados.0.smithi137.stdout:748: writing smithi13779221-26 from 1871634 to 2505726 tid 2 2024-09-21T08:26:56.536 INFO:tasks.rados.rados.0.smithi137.stdout:748: writing smithi13779221-26 from 2950640 to 3008486 tid 3 2024-09-21T08:26:56.536 INFO:tasks.rados.rados.0.smithi137.stdout:749: rollback oid 48 current snap is 70 2024-09-21T08:26:56.536 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 48 to 66 2024-09-21T08:26:56.536 INFO:tasks.rados.rados.0.smithi137.stdout:748: finishing write tid 1 to smithi13779221-26 2024-09-21T08:26:56.536 INFO:tasks.rados.rados.0.smithi137.stdout:748: finishing write tid 2 to smithi13779221-26 2024-09-21T08:26:56.536 INFO:tasks.rados.rados.0.smithi137.stdout:747: done (2 left) 2024-09-21T08:26:56.536 INFO:tasks.rados.rados.0.smithi137.stdout:750: rollback oid 42 current snap is 70 2024-09-21T08:26:56.536 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 42 to 66 2024-09-21T08:26:56.536 INFO:tasks.rados.rados.0.smithi137.stdout:751: delete oid 12 current snap is 70 2024-09-21T08:26:56.537 INFO:tasks.rados.rados.0.smithi137.stdout:748: finishing write tid 3 to smithi13779221-26 2024-09-21T08:26:56.537 INFO:tasks.rados.rados.0.smithi137.stdout:748: finishing write tid 4 to smithi13779221-26 2024-09-21T08:26:56.538 INFO:tasks.rados.rados.0.smithi137.stdout:748: finishing write tid 5 to smithi13779221-26 2024-09-21T08:26:56.538 INFO:tasks.rados.rados.0.smithi137.stdout:748: oid 26 updating version 0 to 270 2024-09-21T08:26:56.538 INFO:tasks.rados.rados.0.smithi137.stdout:748: oid 26 version 270 is already newer than 269 2024-09-21T08:26:56.538 INFO:tasks.rados.rados.0.smithi137.stdout:748: oid 26 version 270 is already newer than 268 2024-09-21T08:26:56.538 INFO:tasks.rados.rados.0.smithi137.stdout:748: oid 26 version 270 is already newer than 267 2024-09-21T08:26:56.538 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 270 (ObjNum 298 snap 70 seq_num 298) dirty exists 2024-09-21T08:26:56.538 INFO:tasks.rados.rados.0.smithi137.stdout:748: left oid 26 (ObjNum 298 snap 70 seq_num 298) 2024-09-21T08:26:56.539 INFO:tasks.rados.rados.0.smithi137.stdout:749: finishing rollback tid 0 to smithi13779221-48 2024-09-21T08:26:56.539 INFO:tasks.rados.rados.0.smithi137.stdout:748: done (3 left) 2024-09-21T08:26:56.539 INFO:tasks.rados.rados.0.smithi137.stdout:751: done (2 left) 2024-09-21T08:26:56.539 INFO:tasks.rados.rados.0.smithi137.stdout:752: rollback oid 32 current snap is 70 2024-09-21T08:26:56.539 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 32 to 62 2024-09-21T08:26:56.539 INFO:tasks.rados.rados.0.smithi137.stdout:753: delete oid 8 current snap is 70 2024-09-21T08:26:56.540 INFO:tasks.rados.rados.0.smithi137.stdout:749: finishing rollback tid 1 to smithi13779221-48 2024-09-21T08:26:56.540 INFO:tasks.rados.rados.0.smithi137.stdout:749: finishing rollback tid 2 to smithi13779221-48 2024-09-21T08:26:56.540 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 210 (ObjNum 259 snap 55 seq_num 259) dirty exists 2024-09-21T08:26:56.540 INFO:tasks.rados.rados.0.smithi137.stdout:752: finishing rollback tid 1 to smithi13779221-32 2024-09-21T08:26:56.541 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 190 (ObjNum 97 snap 0 seq_num 1627259487) dirty dne 2024-09-21T08:26:56.541 INFO:tasks.rados.rados.0.smithi137.stdout:749: done (3 left) 2024-09-21T08:26:56.541 INFO:tasks.rados.rados.0.smithi137.stdout:752: done (2 left) 2024-09-21T08:26:56.541 INFO:tasks.rados.rados.0.smithi137.stdout:753: done (1 left) 2024-09-21T08:26:56.541 INFO:tasks.rados.rados.0.smithi137.stdout:754: write oid 10 current snap is 70 2024-09-21T08:26:56.541 INFO:tasks.rados.rados.0.smithi137.stdout:754: seq_num 301 ranges {658907=787716,1930727=497772,3032909=25998} 2024-09-21T08:26:56.548 INFO:tasks.rados.rados.0.smithi137.stdout:754: writing smithi13779221-10 from 658907 to 1446623 tid 1 2024-09-21T08:26:56.554 INFO:tasks.rados.rados.0.smithi137.stdout:754: writing smithi13779221-10 from 1930727 to 2428499 tid 2 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:754: writing smithi13779221-10 from 3032909 to 3058907 tid 3 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:755: read oid 45 snap -1 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:755: expect deleted 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:756: write (excl) oid 2 current snap is 70 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:750: finishing rollback tid 0 to smithi13779221-42 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:750: finishing rollback tid 1 to smithi13779221-42 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:750: finishing rollback tid 2 to smithi13779221-42 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 252 (ObjNum 290 snap 66 seq_num 290) dirty exists 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:754: finishing write tid 1 to smithi13779221-10 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:754: finishing write tid 2 to smithi13779221-10 2024-09-21T08:26:56.558 INFO:tasks.rados.rados.0.smithi137.stdout:756: seq_num 302 ranges {675714=462965,1589209=641430,2943979=131735} 2024-09-21T08:26:56.564 INFO:tasks.rados.rados.0.smithi137.stdout:756: writing smithi13779221-2 from 675714 to 1138679 tid 1 2024-09-21T08:26:56.570 INFO:tasks.rados.rados.0.smithi137.stdout:756: writing smithi13779221-2 from 1589209 to 2230639 tid 2 2024-09-21T08:26:56.574 INFO:tasks.rados.rados.0.smithi137.stdout:756: writing smithi13779221-2 from 2943979 to 3075714 tid 3 2024-09-21T08:26:56.574 INFO:tasks.rados.rados.0.smithi137.stdout:750: done (3 left) 2024-09-21T08:26:56.575 INFO:tasks.rados.rados.0.smithi137.stdout:757: snap_remove snap 62 2024-09-21T08:26:56.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:56.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:56.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:56.607 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:56.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:56.730 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:56.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:56.731 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:56.831 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:56.840 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:26:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:26:56.840 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:26:56.841 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:26:56.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:56.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:56.854 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:56.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:56.965 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:56 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/519341566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:56.965 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:56 smithi137 ceph-mon[29634]: osdmap e223: 8 total, 8 up, 5 in 2024-09-21T08:26:56.966 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:56 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/519341566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:56.966 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:56 smithi137 ceph-mon[41044]: osdmap e223: 8 total, 8 up, 5 in 2024-09-21T08:26:56.976 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:56.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:56.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:56 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/519341566' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:26:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:56 smithi165 ceph-mon[32893]: osdmap e223: 8 total, 8 up, 5 in 2024-09-21T08:26:57.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:57.099 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:57.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:57.100 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:57.201 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:57.222 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:57.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:57.223 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 223, 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "ec8566d6-77f1-11ef-baf6-efdc52797490", 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-21T08:18:21.794737+0000", 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-21T08:26:56.513250+0000", 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-21T08:23:19.473450+0000", 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-21T08:26:48.512819+0000", 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5801984, 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-21T08:26:57.231 INFO:teuthology.orchestra.run.smithi137.stdout: "noscrub", 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 7, 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-21T08:26:57.232 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:21:15.037508+0000", 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:57.233 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 1, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 1, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 1, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 1, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 1, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:57.234 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "122", 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:57.235 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:57.236 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-21T08:26:57.237 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 7.8899998664855957, 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 7.8899998664855957, 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 3, 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 3, 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:57.238 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:28.246186+0000", 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:57.239 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:57.240 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 68, 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:57.241 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:57.242 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:57.243 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.6500000953674316, 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.6500000953674316, 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.6699999570846558, 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.6699999570846558, 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:57.244 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:29.209879+0000", 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:57.245 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:57.246 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:57.247 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:57.247 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:57.247 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:57.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.247 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "67", 2024-09-21T08:26:57.247 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:57.248 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:57.249 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:57.250 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9800000190734863, 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9800000190734863, 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-21T08:26:57.251 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:30.851833+0000", 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:26:57.252 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "69", 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:57.253 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:57.254 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:57.255 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.4800000190734863, 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.4800000190734863, 2024-09-21T08:26:57.256 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.190000057220459, 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.190000057220459, 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:32.798630+0000", 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:57.257 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:26:57.258 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "71", 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:26:57.259 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:57.260 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:57.261 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9800000190734863, 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9800000190734863, 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:57.262 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:34.858938+0000", 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:57.263 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:57.264 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "73", 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:57.265 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:57.266 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-21T08:26:57.267 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 1.9800000190734863, 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 1.9800000190734863, 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.25, 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.25, 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:57.268 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_0", 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:24:33.441067+0000", 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:26:57.269 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 26, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 26, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 26, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 26, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 26, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:26:57.270 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "223", 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "181", 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 71, 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 223, 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:26:57.271 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:26:57.272 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:26:57.273 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.0499999523162842, 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.0499999523162842, 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.62999999523162842, 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.9199999570846558, 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.9199999570846558, 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 1, 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 1, 2024-09-21T08:26:57.274 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 1 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977", 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-21T08:26:57.275 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 218, 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.277 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/3938795530", 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/3938795530", 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/3938795530", 2024-09-21T08:26:57.278 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/3938795530", 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d", 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:57.279 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 102, 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:57.280 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.281 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/2743268361", 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/2743268361", 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/2743268361", 2024-09-21T08:26:57.282 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/2743268361", 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d", 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:57.283 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 218, 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1500397557", 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1500397557", 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1500397557", 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1500397557", 2024-09-21T08:26:57.286 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "73f21606-38be-4578-a86b-bb417217c0e0", 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:57.287 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 25, 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 218, 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/617155021", 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/617155021", 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/617155021", 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/617155021", 2024-09-21T08:26:57.290 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae", 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 31, 2024-09-21T08:26:57.291 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 80, 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6800", 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6801", 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6803", 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6802", 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6800/70360958", 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6801/70360958", 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6803/70360958", 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6802/70360958", 2024-09-21T08:26:57.294 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd", 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:57.295 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 41, 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 218, 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6804", 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:57.296 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.297 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.297 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6805", 2024-09-21T08:26:57.297 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:26:57.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.297 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6807", 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.298 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6806", 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6804/1835779380", 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6805/1835779380", 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6807/1835779380", 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6806/1835779380", 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b", 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 49, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 94, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:57.300 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6808", 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6809", 2024-09-21T08:26:57.301 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6811", 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6810", 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6808/3820354208", 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6809/3820354208", 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6811/3820354208", 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6810/3820354208", 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:57.303 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f", 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0.9804229736328125, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 59, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 215, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:26:57.304 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6812", 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6813", 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:26:57.305 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6815", 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.306 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6814", 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6812/1933524273", 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6813/1933524273", 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6815/1933524273", 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6814/1933524273", 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.307 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:25:46.489431+0000", 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-21T08:26:57.308 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:26:00.210653+0000", 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:26:57.309 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:26:02.707590+0000", 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:26:53.651451+0000", 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-21T08:26:57.310 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:26:18.932348+0000", 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:26:57.311 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:26:19.772946+0000", 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:26:33.799562+0000", 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-21T08:26:57.312 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:26:57.313 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:26:57.313 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:26:57.313 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:26:57.313 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:26:57.313 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:25:57.391802+0000", 2024-09-21T08:26:57.313 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:26:57.313 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.313 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:57.313 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [], 2024-09-21T08:26:57.313 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "2.1", 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 0, 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-21T08:26:57.314 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1478868457": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1738156348": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2643910317": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/705906958": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3936900881": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/223603733": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1718301516": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2580000250": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3166961425": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/700535732": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1900990534": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1825289826": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3010106803": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3442439903": "2024-09-22T08:23:59.682278+0000" 2024-09-21T08:26:57.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [ 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:26:57.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 57, 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 66, 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 2 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [], 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [ 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.317 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 62, 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 2 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 65, 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-21T08:26:57.318 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-21T08:26:57.319 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-21T08:26:57.319 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-21T08:26:57.319 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-21T08:26:57.319 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-21T08:26:57.319 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-21T08:26:57.319 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-21T08:26:57.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:26:57.319 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:26:57.323 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:57.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:57.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:57.346 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:57.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:57.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:57.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:57.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:57.526 INFO:tasks.rados.rados.0.smithi137.stdout:754: finishing write tid 3 to smithi13779221-10 2024-09-21T08:26:57.526 INFO:tasks.rados.rados.0.smithi137.stdout:754: finishing write tid 4 to smithi13779221-10 2024-09-21T08:26:57.526 INFO:tasks.rados.rados.0.smithi137.stdout:754: finishing write tid 5 to smithi13779221-10 2024-09-21T08:26:57.526 INFO:tasks.rados.rados.0.smithi137.stdout:754: oid 10 updating version 0 to 214 2024-09-21T08:26:57.526 INFO:tasks.rados.rados.0.smithi137.stdout:754: oid 10 version 214 is already newer than 213 2024-09-21T08:26:57.526 INFO:tasks.rados.rados.0.smithi137.stdout:754: oid 10 version 214 is already newer than 211 2024-09-21T08:26:57.526 INFO:tasks.rados.rados.0.smithi137.stdout:754: oid 10 version 214 is already newer than 212 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 214 (ObjNum 301 snap 70 seq_num 301) dirty exists 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:754: left oid 10 (ObjNum 301 snap 70 seq_num 301) 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:756: finishing write tid 1 to smithi13779221-2 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:756: finishing write tid 2 to smithi13779221-2 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:756: finishing write tid 3 to smithi13779221-2 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:756: finishing write tid 4 to smithi13779221-2 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:756: finishing write tid 5 to smithi13779221-2 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:756: oid 2 updating version 0 to 113 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:756: oid 2 updating version 113 to 115 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:756: oid 2 version 115 is already newer than 112 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:756: oid 2 version 115 is already newer than 114 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 115 (ObjNum 302 snap 70 seq_num 302) dirty exists 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:756: left oid 2 (ObjNum 302 snap 70 seq_num 302) 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:754: done (3 left) 2024-09-21T08:26:57.527 INFO:tasks.rados.rados.0.smithi137.stdout:755: done (2 left) 2024-09-21T08:26:57.528 INFO:tasks.rados.rados.0.smithi137.stdout:756: done (1 left) 2024-09-21T08:26:57.528 INFO:tasks.rados.rados.0.smithi137.stdout:757: done (0 left) 2024-09-21T08:26:57.528 INFO:tasks.rados.rados.0.smithi137.stdout:758: write oid 12 current snap is 70 2024-09-21T08:26:57.528 INFO:tasks.rados.rados.0.smithi137.stdout:758: seq_num 303 ranges {692521=538214,1931338=785088} 2024-09-21T08:26:57.532 INFO:tasks.rados.rados.0.smithi137.stdout:758: writing smithi13779221-12 from 692521 to 1230735 tid 1 2024-09-21T08:26:57.541 INFO:tasks.rados.rados.0.smithi137.stdout:758: writing smithi13779221-12 from 1931338 to 2716426 tid 2 2024-09-21T08:26:57.541 INFO:tasks.rados.rados.0.smithi137.stdout:758: finishing write tid 1 to smithi13779221-12 2024-09-21T08:26:57.541 INFO:tasks.rados.rados.0.smithi137.stdout:759: delete oid 8 current snap is 70 2024-09-21T08:26:57.545 INFO:tasks.rados.rados.0.smithi137.stdout:758: finishing write tid 2 to smithi13779221-12 2024-09-21T08:26:57.545 INFO:tasks.rados.rados.0.smithi137.stdout:758: finishing write tid 3 to smithi13779221-12 2024-09-21T08:26:57.545 INFO:tasks.rados.rados.0.smithi137.stdout:759: done (1 left) 2024-09-21T08:26:57.545 INFO:tasks.rados.rados.0.smithi137.stdout:760: write (excl) oid 11 current snap is 70 2024-09-21T08:26:57.545 INFO:tasks.rados.rados.0.smithi137.stdout:758: finishing write tid 4 to smithi13779221-12 2024-09-21T08:26:57.545 INFO:tasks.rados.rados.0.smithi137.stdout:758: oid 12 updating version 0 to 278 2024-09-21T08:26:57.545 INFO:tasks.rados.rados.0.smithi137.stdout:758: oid 12 updating version 278 to 280 2024-09-21T08:26:57.545 INFO:tasks.rados.rados.0.smithi137.stdout:758: oid 12 version 280 is already newer than 279 2024-09-21T08:26:57.545 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 280 (ObjNum 303 snap 70 seq_num 303) dirty exists 2024-09-21T08:26:57.545 INFO:tasks.rados.rados.0.smithi137.stdout:758: left oid 12 (ObjNum 303 snap 70 seq_num 303) 2024-09-21T08:26:57.546 INFO:tasks.rados.rados.0.smithi137.stdout:760: seq_num 305 ranges {726135=405065,1764655=788757,3026248=99887} 2024-09-21T08:26:57.551 INFO:tasks.rados.rados.0.smithi137.stdout:760: writing smithi13779221-11 from 726135 to 1131200 tid 1 2024-09-21T08:26:57.560 INFO:tasks.rados.rados.0.smithi137.stdout:760: writing smithi13779221-11 from 1764655 to 2553412 tid 2 2024-09-21T08:26:57.563 INFO:tasks.rados.rados.0.smithi137.stdout:760: writing smithi13779221-11 from 3026248 to 3126135 tid 3 2024-09-21T08:26:57.563 INFO:tasks.rados.rados.0.smithi137.stdout:758: done (1 left) 2024-09-21T08:26:57.563 INFO:tasks.rados.rados.0.smithi137.stdout:761: delete oid 39 current snap is 70 2024-09-21T08:26:57.563 INFO:tasks.rados.rados.0.smithi137.stdout:760: finishing write tid 1 to smithi13779221-11 2024-09-21T08:26:57.563 INFO:tasks.rados.rados.0.smithi137.stdout:760: finishing write tid 2 to smithi13779221-11 2024-09-21T08:26:57.565 INFO:tasks.rados.rados.0.smithi137.stdout:761: done (1 left) 2024-09-21T08:26:57.565 INFO:tasks.rados.rados.0.smithi137.stdout:762: read oid 7 snap -1 2024-09-21T08:26:57.565 INFO:tasks.rados.rados.0.smithi137.stdout:762: expect (ObjNum 278 snap 64 seq_num 278) 2024-09-21T08:26:57.565 INFO:tasks.rados.rados.0.smithi137.stdout:763: delete oid 27 current snap is 70 2024-09-21T08:26:57.565 INFO:tasks.rados.rados.0.smithi137.stdout:760: finishing write tid 3 to smithi13779221-11 2024-09-21T08:26:57.565 INFO:tasks.rados.rados.0.smithi137.stdout:760: finishing write tid 4 to smithi13779221-11 2024-09-21T08:26:57.566 INFO:tasks.rados.rados.0.smithi137.stdout:760: finishing write tid 5 to smithi13779221-11 2024-09-21T08:26:57.566 INFO:tasks.rados.rados.0.smithi137.stdout:760: oid 11 updating version 0 to 192 2024-09-21T08:26:57.566 INFO:tasks.rados.rados.0.smithi137.stdout:760: oid 11 version 192 is already newer than 191 2024-09-21T08:26:57.566 INFO:tasks.rados.rados.0.smithi137.stdout:760: oid 11 updating version 192 to 194 2024-09-21T08:26:57.566 INFO:tasks.rados.rados.0.smithi137.stdout:760: oid 11 version 194 is already newer than 193 2024-09-21T08:26:57.566 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 194 (ObjNum 305 snap 70 seq_num 305) dirty exists 2024-09-21T08:26:57.566 INFO:tasks.rados.rados.0.smithi137.stdout:760: left oid 11 (ObjNum 305 snap 70 seq_num 305) 2024-09-21T08:26:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:760: done (2 left) 2024-09-21T08:26:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:763: done (1 left) 2024-09-21T08:26:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:764: rollback oid 3 current snap is 70 2024-09-21T08:26:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 3 to 68 2024-09-21T08:26:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:765: read oid 38 snap -1 2024-09-21T08:26:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:765: expect (ObjNum 281 snap 64 seq_num 281) 2024-09-21T08:26:57.567 INFO:tasks.rados.rados.0.smithi137.stdout:766: delete oid 48 current snap is 70 2024-09-21T08:26:57.568 INFO:tasks.rados.rados.0.smithi137.stdout:764: finishing rollback tid 0 to smithi13779221-3 2024-09-21T08:26:57.568 INFO:tasks.rados.rados.0.smithi137.stdout:766: done (3 left) 2024-09-21T08:26:57.569 INFO:tasks.rados.rados.0.smithi137.stdout:767: copy_from oid 9 from oid 50 current snap is 70 2024-09-21T08:26:57.569 INFO:tasks.rados.rados.0.smithi137.stdout:768: rollback oid 26 current snap is 70 2024-09-21T08:26:57.569 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 26 to 59 2024-09-21T08:26:57.569 INFO:tasks.rados.rados.0.smithi137.stdout:769: write (excl) oid 43 current snap is 70 2024-09-21T08:26:57.569 INFO:tasks.rados.rados.0.smithi137.stdout:769: seq_num 309 ranges {793363=706061,2282230=796095} 2024-09-21T08:26:57.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:26:57.578 INFO:tasks.rados.rados.0.smithi137.stdout:769: writing smithi13779221-43 from 793363 to 1499424 tid 1 2024-09-21T08:26:57.589 INFO:tasks.rados.rados.0.smithi137.stdout:769: writing smithi13779221-43 from 2282230 to 3078325 tid 2 2024-09-21T08:26:57.589 INFO:tasks.rados.rados.0.smithi137.stdout:770: write (excl) oid 28 current snap is 70 2024-09-21T08:26:57.589 INFO:tasks.rados.rados.0.smithi137.stdout:770: seq_num 310 ranges {410170=781310,1940712=656106,3163363=46807} 2024-09-21T08:26:57.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:57.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:57.593 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:26:57.595 INFO:tasks.rados.rados.0.smithi137.stdout:770: writing smithi13779221-28 from 410170 to 1191480 tid 1 2024-09-21T08:26:57.603 INFO:tasks.rados.rados.0.smithi137.stdout:770: writing smithi13779221-28 from 1940712 to 2596818 tid 2 2024-09-21T08:26:57.607 INFO:tasks.rados.rados.0.smithi137.stdout:770: writing smithi13779221-28 from 3163363 to 3210170 tid 3 2024-09-21T08:26:57.607 INFO:tasks.rados.rados.0.smithi137.stdout:771: copy_from oid 20 from oid 10 current snap is 70 2024-09-21T08:26:57.607 INFO:tasks.rados.rados.0.smithi137.stdout:764: finishing rollback tid 1 to smithi13779221-3 2024-09-21T08:26:57.607 INFO:tasks.rados.rados.0.smithi137.stdout:764: finishing rollback tid 2 to smithi13779221-3 2024-09-21T08:26:57.607 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 171 (ObjNum 292 snap 67 seq_num 292) dirty exists 2024-09-21T08:26:57.607 INFO:tasks.rados.rados.0.smithi137.stdout:768: finishing rollback tid 0 to smithi13779221-26 2024-09-21T08:26:57.607 INFO:tasks.rados.rados.0.smithi137.stdout:768: finishing rollback tid 1 to smithi13779221-26 2024-09-21T08:26:57.607 INFO:tasks.rados.rados.0.smithi137.stdout:768: finishing rollback tid 2 to smithi13779221-26 2024-09-21T08:26:57.607 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 273 (ObjNum 200 snap 40 seq_num 200) dirty exists 2024-09-21T08:26:57.607 INFO:tasks.rados.rados.0.smithi137.stdout:762: expect (ObjNum 278 snap 64 seq_num 278) 2024-09-21T08:26:57.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:26:57.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:57.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:57.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:57.716 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:57.718 INFO:tasks.rados.rados.0.smithi137.stdout:765: expect (ObjNum 281 snap 64 seq_num 281) 2024-09-21T08:26:57.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:57.839 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:57.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:57.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:57.923 INFO:tasks.rados.rados.0.smithi137.stdout:767: finishing copy_from racing read to smithi13779221-9 2024-09-21T08:26:57.923 INFO:tasks.rados.rados.0.smithi137.stdout:762: done (7 left) 2024-09-21T08:26:57.923 INFO:tasks.rados.rados.0.smithi137.stdout:764: done (6 left) 2024-09-21T08:26:57.923 INFO:tasks.rados.rados.0.smithi137.stdout:765: done (5 left) 2024-09-21T08:26:57.923 INFO:tasks.rados.rados.0.smithi137.stdout:768: done (4 left) 2024-09-21T08:26:57.923 INFO:tasks.rados.rados.0.smithi137.stdout:772: rollback oid 13 current snap is 70 2024-09-21T08:26:57.923 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 13 to 67 2024-09-21T08:26:57.923 INFO:tasks.rados.rados.0.smithi137.stdout:767: finishing copy_from to smithi13779221-9 2024-09-21T08:26:57.923 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 9 v 248 (ObjNum 290 snap 66 seq_num 290) dirty exists 2024-09-21T08:26:57.923 INFO:tasks.rados.rados.0.smithi137.stdout:769: finishing write tid 1 to smithi13779221-43 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:769: finishing write tid 2 to smithi13779221-43 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:769: finishing write tid 3 to smithi13779221-43 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:769: finishing write tid 4 to smithi13779221-43 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:769: oid 43 updating version 0 to 276 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:769: oid 43 version 276 is already newer than 274 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:769: oid 43 version 276 is already newer than 275 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 276 (ObjNum 309 snap 70 seq_num 309) dirty exists 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:769: left oid 43 (ObjNum 309 snap 70 seq_num 309) 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:770: finishing write tid 1 to smithi13779221-28 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:770: finishing write tid 2 to smithi13779221-28 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:770: finishing write tid 3 to smithi13779221-28 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:770: finishing write tid 4 to smithi13779221-28 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:770: finishing write tid 5 to smithi13779221-28 2024-09-21T08:26:57.924 INFO:tasks.rados.rados.0.smithi137.stdout:770: oid 28 updating version 0 to 122 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:770: oid 28 updating version 122 to 123 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:770: oid 28 version 123 is already newer than 121 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:770: oid 28 updating version 123 to 124 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 124 (ObjNum 310 snap 70 seq_num 310) dirty exists 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:770: left oid 28 (ObjNum 310 snap 70 seq_num 310) 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:771: finishing copy_from racing read to smithi13779221-20 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:771: finishing copy_from to smithi13779221-20 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 214 (ObjNum 301 snap 70 seq_num 301) dirty exists 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:767: done (4 left) 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:769: done (3 left) 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:770: done (2 left) 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:771: done (1 left) 2024-09-21T08:26:57.925 INFO:tasks.rados.rados.0.smithi137.stdout:773: delete oid 27 current snap is 70 2024-09-21T08:26:57.926 INFO:tasks.rados.rados.0.smithi137.stdout:773: done (1 left) 2024-09-21T08:26:57.926 INFO:tasks.rados.rados.0.smithi137.stdout:774: write oid 6 current snap is 70 2024-09-21T08:26:57.926 INFO:tasks.rados.rados.0.smithi137.stdout:774: seq_num 312 ranges {443784=648161,1657676=659775,2818209=425575} 2024-09-21T08:26:57.930 INFO:tasks.rados.rados.0.smithi137.stdout:774: writing smithi13779221-6 from 443784 to 1091945 tid 1 2024-09-21T08:26:57.936 INFO:tasks.rados.rados.0.smithi137.stdout:774: writing smithi13779221-6 from 1657676 to 2317451 tid 2 2024-09-21T08:26:57.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:57.942 INFO:tasks.rados.rados.0.smithi137.stdout:774: writing smithi13779221-6 from 2818209 to 3243784 tid 3 2024-09-21T08:26:57.942 INFO:tasks.rados.rados.0.smithi137.stdout:775: read oid 47 snap -1 2024-09-21T08:26:57.942 INFO:tasks.rados.rados.0.smithi137.stdout:775: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:26:57.942 INFO:tasks.rados.rados.0.smithi137.stdout:776: read oid 19 snap 68 2024-09-21T08:26:57.942 INFO:tasks.rados.rados.0.smithi137.stdout:776: expect deleted 2024-09-21T08:26:57.942 INFO:tasks.rados.rados.0.smithi137.stdout:777: read oid 27 snap -1 2024-09-21T08:26:57.943 INFO:tasks.rados.rados.0.smithi137.stdout:777: expect deleted 2024-09-21T08:26:57.943 INFO:tasks.rados.rados.0.smithi137.stdout:778: read oid 37 snap -1 2024-09-21T08:26:57.943 INFO:tasks.rados.rados.0.smithi137.stdout:778: expect (ObjNum 243 snap 50 seq_num 243) 2024-09-21T08:26:57.943 INFO:tasks.rados.rados.0.smithi137.stdout:779: snap_create 2024-09-21T08:26:57.943 INFO:tasks.rados.rados.0.smithi137.stdout:772: finishing rollback tid 0 to smithi13779221-13 2024-09-21T08:26:57.943 INFO:tasks.rados.rados.0.smithi137.stdout:772: finishing rollback tid 1 to smithi13779221-13 2024-09-21T08:26:57.943 INFO:tasks.rados.rados.0.smithi137.stdout:772: finishing rollback tid 2 to smithi13779221-13 2024-09-21T08:26:57.943 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 346 (ObjNum 255 snap 55 seq_num 255) dirty exists 2024-09-21T08:26:57.943 INFO:tasks.rados.rados.0.smithi137.stdout:774: finishing write tid 1 to smithi13779221-6 2024-09-21T08:26:57.943 INFO:tasks.rados.rados.0.smithi137.stdout:774: finishing write tid 2 to smithi13779221-6 2024-09-21T08:26:57.955 INFO:tasks.rados.rados.0.smithi137.stdout:778: expect (ObjNum 243 snap 50 seq_num 243) 2024-09-21T08:26:57.963 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:57.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:57.964 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:57 smithi165 ceph-mon[32893]: pgmap v239: 158 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 145 active+clean; 167 MiB data, 614 MiB used, 446 GiB / 447 GiB avail; 2.2 MiB/s rd, 2 op/s 2024-09-21T08:26:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:57 smithi165 ceph-mon[32893]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/804 objects degraded (0.124%), 1 pg degraded) 2024-09-21T08:26:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:57 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2576062013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:26:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:57 smithi165 ceph-mon[32893]: osdmap e224: 8 total, 8 up, 5 in 2024-09-21T08:26:58.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:57 smithi137 ceph-mon[29634]: pgmap v239: 158 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 145 active+clean; 167 MiB data, 614 MiB used, 446 GiB / 447 GiB avail; 2.2 MiB/s rd, 2 op/s 2024-09-21T08:26:58.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:57 smithi137 ceph-mon[29634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/804 objects degraded (0.124%), 1 pg degraded) 2024-09-21T08:26:58.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:57 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2576062013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:26:58.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:57 smithi137 ceph-mon[29634]: osdmap e224: 8 total, 8 up, 5 in 2024-09-21T08:26:58.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:57 smithi137 ceph-mon[41044]: pgmap v239: 158 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 145 active+clean; 167 MiB data, 614 MiB used, 446 GiB / 447 GiB avail; 2.2 MiB/s rd, 2 op/s 2024-09-21T08:26:58.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:57 smithi137 ceph-mon[41044]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1/804 objects degraded (0.124%), 1 pg degraded) 2024-09-21T08:26:58.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:57 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2576062013' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:26:58.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:57 smithi137 ceph-mon[41044]: osdmap e224: 8 total, 8 up, 5 in 2024-09-21T08:26:58.041 INFO:tasks.rados.rados.0.smithi137.stdout:775: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:26:58.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:58.087 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:58.087 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:58.087 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:58.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:58.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:58.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:58.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:58.255 INFO:tasks.rados.rados.0.smithi137.stdout:774: finishing write tid 3 to smithi13779221-6 2024-09-21T08:26:58.255 INFO:tasks.rados.rados.0.smithi137.stdout:774: finishing write tid 4 to smithi13779221-6 2024-09-21T08:26:58.255 INFO:tasks.rados.rados.0.smithi137.stdout:774: finishing write tid 5 to smithi13779221-6 2024-09-21T08:26:58.256 INFO:tasks.rados.rados.0.smithi137.stdout:774: oid 6 updating version 0 to 238 2024-09-21T08:26:58.256 INFO:tasks.rados.rados.0.smithi137.stdout:774: oid 6 version 238 is already newer than 237 2024-09-21T08:26:58.256 INFO:tasks.rados.rados.0.smithi137.stdout:774: oid 6 updating version 238 to 239 2024-09-21T08:26:58.256 INFO:tasks.rados.rados.0.smithi137.stdout:774: oid 6 version 239 is already newer than 236 2024-09-21T08:26:58.256 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 239 (ObjNum 312 snap 70 seq_num 312) dirty exists 2024-09-21T08:26:58.256 INFO:tasks.rados.rados.0.smithi137.stdout:774: left oid 6 (ObjNum 312 snap 70 seq_num 312) 2024-09-21T08:26:58.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:26:58.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:58.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:58.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:26:58.437 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:26:58.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:58.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:58.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:26:58.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:58.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T08:26:58.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:58.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:58.585 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:58.651 INFO:tasks.rados.rados.0.smithi137.stdout:772: done (6 left) 2024-09-21T08:26:58.651 INFO:tasks.rados.rados.0.smithi137.stdout:774: done (5 left) 2024-09-21T08:26:58.651 INFO:tasks.rados.rados.0.smithi137.stdout:775: done (4 left) 2024-09-21T08:26:58.651 INFO:tasks.rados.rados.0.smithi137.stdout:776: done (3 left) 2024-09-21T08:26:58.651 INFO:tasks.rados.rados.0.smithi137.stdout:777: done (2 left) 2024-09-21T08:26:58.651 INFO:tasks.rados.rados.0.smithi137.stdout:778: done (1 left) 2024-09-21T08:26:58.651 INFO:tasks.rados.rados.0.smithi137.stdout:779: done (0 left) 2024-09-21T08:26:58.651 INFO:tasks.rados.rados.0.smithi137.stdout:780: delete oid 5 current snap is 71 2024-09-21T08:26:58.655 INFO:tasks.rados.rados.0.smithi137.stdout:780: done (0 left) 2024-09-21T08:26:58.655 INFO:tasks.rados.rados.0.smithi137.stdout:781: rollback oid 43 current snap is 71 2024-09-21T08:26:58.655 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 43 to 69 2024-09-21T08:26:58.656 INFO:tasks.rados.rados.0.smithi137.stdout:782: write (excl) oid 28 current snap is 71 2024-09-21T08:26:58.656 INFO:tasks.rados.rados.0.smithi137.stdout:782: seq_num 314 ranges {477398=798659,1774640=663444,2873055=404343} 2024-09-21T08:26:58.662 INFO:tasks.rados.rados.0.smithi137.stdout:782: writing smithi13779221-28 from 477398 to 1276057 tid 1 2024-09-21T08:26:58.669 INFO:tasks.rados.rados.0.smithi137.stdout:782: writing smithi13779221-28 from 1774640 to 2438084 tid 2 2024-09-21T08:26:58.674 INFO:tasks.rados.rados.0.smithi137.stdout:782: writing smithi13779221-28 from 2873055 to 3277398 tid 3 2024-09-21T08:26:58.674 INFO:tasks.rados.rados.0.smithi137.stdout:783: snap_create 2024-09-21T08:26:58.674 INFO:tasks.rados.rados.0.smithi137.stdout:781: finishing rollback tid 0 to smithi13779221-43 2024-09-21T08:26:58.674 INFO:tasks.rados.rados.0.smithi137.stdout:781: finishing rollback tid 1 to smithi13779221-43 2024-09-21T08:26:58.674 INFO:tasks.rados.rados.0.smithi137.stdout:781: finishing rollback tid 2 to smithi13779221-43 2024-09-21T08:26:58.674 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 279 (ObjNum 156 snap 31 seq_num 156) dirty exists 2024-09-21T08:26:58.675 INFO:tasks.rados.rados.0.smithi137.stdout:782: finishing write tid 1 to smithi13779221-28 2024-09-21T08:26:58.675 INFO:tasks.rados.rados.0.smithi137.stdout:782: finishing write tid 2 to smithi13779221-28 2024-09-21T08:26:58.676 INFO:tasks.rados.rados.0.smithi137.stdout:782: finishing write tid 3 to smithi13779221-28 2024-09-21T08:26:58.676 INFO:tasks.rados.rados.0.smithi137.stdout:782: finishing write tid 4 to smithi13779221-28 2024-09-21T08:26:58.676 INFO:tasks.rados.rados.0.smithi137.stdout:782: finishing write tid 5 to smithi13779221-28 2024-09-21T08:26:58.677 INFO:tasks.rados.rados.0.smithi137.stdout:782: oid 28 updating version 0 to 126 2024-09-21T08:26:58.677 INFO:tasks.rados.rados.0.smithi137.stdout:782: oid 28 updating version 126 to 128 2024-09-21T08:26:58.677 INFO:tasks.rados.rados.0.smithi137.stdout:782: oid 28 updating version 128 to 129 2024-09-21T08:26:58.677 INFO:tasks.rados.rados.0.smithi137.stdout:782: oid 28 version 129 is already newer than 127 2024-09-21T08:26:58.677 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 129 (ObjNum 314 snap 71 seq_num 314) dirty exists 2024-09-21T08:26:58.677 INFO:tasks.rados.rados.0.smithi137.stdout:782: left oid 28 (ObjNum 314 snap 71 seq_num 314) 2024-09-21T08:26:58.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:58.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:58.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:58.708 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:58.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:58.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:58.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:58.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:58.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:58.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:58.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:58.957 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:59.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:26:59.079 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:59.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:59.080 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:26:59.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:26:59.202 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:59.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:59.204 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:26:59.243 INFO:tasks.thrashosds.thrasher:j is {'epoch': 223, 'fsid': 'ec8566d6-77f1-11ef-baf6-efdc52797490', 'created': '2024-09-21T08:18:21.794737+0000', 'modified': '2024-09-21T08:26:56.513250+0000', 'last_up_change': '2024-09-21T08:23:19.473450+0000', 'last_in_change': '2024-09-21T08:26:48.512819+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T08:21:15.037508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '122', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 7.889999866485596, 'score_stable': 7.889999866485596, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T08:23:28.246186+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.6500000953674316, 'score_stable': 2.6500000953674316, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.6699999570846558, 'raw_score_stable': 1.6699999570846558, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T08:23:29.209879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9800000190734863, 'score_stable': 1.9800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T08:23:30.851833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.4800000190734863, 'score_stable': 3.4800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 2.190000057220459, 'raw_score_stable': 2.190000057220459, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T08:23:32.798630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9800000190734863, 'score_stable': 1.9800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T08:23:34.858938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.9800000190734863, 'score_stable': 1.9800000190734863, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.25, 'raw_score_stable': 1.25, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-21T08:24:33.441067+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 26, 'pg_placement_num': 26, 'pg_placement_num_target': 26, 'pg_num_target': 26, 'pg_num_pending': 26, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '223', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '181', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 71, 'snap_epoch': 223, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.049999952316284, 'score_stable': 3.049999952316284, 'optimal_score': 0.6299999952316284, 'raw_score_acting': 1.9199999570846558, 'raw_score_stable': 1.9199999570846558, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}], 'osds': [{'osd': 0, 'uuid': '1b007d1d-6c6b-4a68-9e68-2c5fd7d26977', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 3938795530}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 3938795530}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 3938795530}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 3938795530}]}, 'public_addr': '172.21.15.137:6801/3938795530', 'cluster_addr': '172.21.15.137:6802/3938795530', 'heartbeat_back_addr': '172.21.15.137:6804/3938795530', 'heartbeat_front_addr': '172.21.15.137:6803/3938795530', 'state': ['exists', 'up']}, {'osd': 1, 'uuid': '8651068e-694f-4bd4-9917-5c55670aca5d', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 102, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 2743268361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 2743268361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 2743268361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 2743268361}]}, 'public_addr': '172.21.15.137:6805/2743268361', 'cluster_addr': '172.21.15.137:6806/2743268361', 'heartbeat_back_addr': '172.21.15.137:6808/2743268361', 'heartbeat_front_addr': '172.21.15.137:6807/2743268361', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2a22f0f7-c610-4995-98a7-d4785efb1e5d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1500397557}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1500397557}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1500397557}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1500397557}]}, 'public_addr': '172.21.15.137:6809/1500397557', 'cluster_addr': '172.21.15.137:6810/1500397557', 'heartbeat_back_addr': '172.21.15.137:6812/1500397557', 'heartbeat_front_addr': '172.21.15.137:6811/1500397557', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '73f21606-38be-4578-a86b-bb417217c0e0', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 617155021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 617155021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 617155021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 617155021}]}, 'public_addr': '172.21.15.137:6813/617155021', 'cluster_addr': '172.21.15.137:6814/617155021', 'heartbeat_back_addr': '172.21.15.137:6816/617155021', 'heartbeat_front_addr': '172.21.15.137:6815/617155021', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e5f6762b-65f5-4dc7-8e54-9087149ae6ae', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 70360958}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 70360958}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 70360958}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 70360958}]}, 'public_addr': '172.21.15.165:6800/70360958', 'cluster_addr': '172.21.15.165:6801/70360958', 'heartbeat_back_addr': '172.21.15.165:6803/70360958', 'heartbeat_front_addr': '172.21.15.165:6802/70360958', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f7f8c456-b77f-4435-a860-202e64dda8dd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 218, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 1835779380}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 1835779380}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 1835779380}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 1835779380}]}, 'public_addr': '172.21.15.165:6804/1835779380', 'cluster_addr': '172.21.15.165:6805/1835779380', 'heartbeat_back_addr': '172.21.15.165:6807/1835779380', 'heartbeat_front_addr': '172.21.15.165:6806/1835779380', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b8b3df16-73fa-41a5-a93f-117c599d488b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 3820354208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 3820354208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 3820354208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 3820354208}]}, 'public_addr': '172.21.15.165:6808/3820354208', 'cluster_addr': '172.21.15.165:6809/3820354208', 'heartbeat_back_addr': '172.21.15.165:6811/3820354208', 'heartbeat_front_addr': '172.21.15.165:6810/3820354208', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ddf234e4-1494-40bd-a31b-333b01717c3f', 'up': 1, 'in': 1, 'weight': 0.9804229736328125, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 215, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 1933524273}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 1933524273}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 1933524273}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 1933524273}]}, 'public_addr': '172.21.15.165:6812/1933524273', 'cluster_addr': '172.21.15.165:6813/1933524273', 'heartbeat_back_addr': '172.21.15.165:6815/1933524273', 'heartbeat_front_addr': '172.21.15.165:6814/1933524273', 'state': ['exists', 'up']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:25:46.489431+0000', 'dead_epoch': 0}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:26:00.210653+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:26:02.707590+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:26:53.651451+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:26:18.932348+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:26:19.772946+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:26:33.799562+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:25:57.391802+0000', 'dead_epoch': 0}], 'pg_upmap': [], 'pg_upmap_items': [{'pgid': '2.1', 'mappings': [{'from': 0, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:0/1478868457': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/1738156348': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/2643910317': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/705906958': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/3936900881': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/223603733': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/1718301516': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:6800/2580000250': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/3166961425': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/700535732': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:6800/1900990534': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:6800/1825289826': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/3010106803': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/3442439903': '2024-09-22T08:23:59.682278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 57, 'length': 1}, {'begin': 66, 'length': 2}]}], 'new_removed_snaps': [], 'new_purged_snaps': [{'pool': 7, 'snaps': [{'begin': 62, 'length': 2}, {'begin': 65, 'length': 1}]}], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T08:26:59.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:26:59.246 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:59.273 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:26:59.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:59.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:59.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:59.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:59.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:59.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:59.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:59.450 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:59.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:26:59.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:59.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:59.572 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:26:59.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:26:59.653 INFO:tasks.rados.rados.0.smithi137.stdout:781: done (2 left) 2024-09-21T08:26:59.653 INFO:tasks.rados.rados.0.smithi137.stdout:782: done (1 left) 2024-09-21T08:26:59.653 INFO:tasks.rados.rados.0.smithi137.stdout:783: done (0 left) 2024-09-21T08:26:59.653 INFO:tasks.rados.rados.0.smithi137.stdout:784: read oid 47 snap 67 2024-09-21T08:26:59.653 INFO:tasks.rados.rados.0.smithi137.stdout:784: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:26:59.654 INFO:tasks.rados.rados.0.smithi137.stdout:785: write (excl) oid 12 current snap is 72 2024-09-21T08:26:59.654 INFO:tasks.rados.rados.0.smithi137.stdout:785: seq_num 315 ranges {494205=473908,1433122=407102,2384125=462684} 2024-09-21T08:26:59.658 INFO:tasks.rados.rados.0.smithi137.stdout:785: writing smithi13779221-12 from 494205 to 968113 tid 1 2024-09-21T08:26:59.665 INFO:tasks.rados.rados.0.smithi137.stdout:785: writing smithi13779221-12 from 1433122 to 1840224 tid 2 2024-09-21T08:26:59.672 INFO:tasks.rados.rados.0.smithi137.stdout:785: writing smithi13779221-12 from 2384125 to 2846809 tid 3 2024-09-21T08:26:59.672 INFO:tasks.rados.rados.0.smithi137.stdout:786: read oid 29 snap -1 2024-09-21T08:26:59.672 INFO:tasks.rados.rados.0.smithi137.stdout:786: expect (ObjNum 276 snap 64 seq_num 276) 2024-09-21T08:26:59.672 INFO:tasks.rados.rados.0.smithi137.stdout:787: read oid 39 snap 69 2024-09-21T08:26:59.672 INFO:tasks.rados.rados.0.smithi137.stdout:787: expect deleted 2024-09-21T08:26:59.672 INFO:tasks.rados.rados.0.smithi137.stdout:788: snap_remove snap 68 2024-09-21T08:26:59.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:26:59.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:26:59.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:59.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:26:59.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:59 smithi137 ceph-mon[29634]: pgmap v242: 158 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 145 active+clean; 167 MiB data, 614 MiB used, 446 GiB / 447 GiB avail; 2.2 MiB/s rd, 2 op/s 2024-09-21T08:26:59.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:59 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:26:59.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:59 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:26:59.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:26:59 smithi137 ceph-mon[29634]: osdmap e225: 8 total, 8 up, 5 in 2024-09-21T08:26:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:59 smithi137 ceph-mon[41044]: pgmap v242: 158 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 145 active+clean; 167 MiB data, 614 MiB used, 446 GiB / 447 GiB avail; 2.2 MiB/s rd, 2 op/s 2024-09-21T08:26:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:59 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:26:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:59 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:26:59.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:26:59 smithi137 ceph-mon[41044]: osdmap e225: 8 total, 8 up, 5 in 2024-09-21T08:26:59.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:59.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:59.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:59.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:26:59.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:26:59.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:26:59.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:26:59.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:59 smithi165 ceph-mon[32893]: pgmap v242: 158 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 145 active+clean; 167 MiB data, 614 MiB used, 446 GiB / 447 GiB avail; 2.2 MiB/s rd, 2 op/s 2024-09-21T08:27:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:59 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:27:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:59 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:27:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:26:59 smithi165 ceph-mon[32893]: osdmap e225: 8 total, 8 up, 5 in 2024-09-21T08:27:00.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:00.072 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:00.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:00.073 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:00.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:00.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:00.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:00.196 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:00.232 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:00.232 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:27:00.232 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:27:00.233 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:27:00.233 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:27:00.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:00.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:00.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:00.320 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:00.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:00.436 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:00.436 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:00.444 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:00.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:00.446 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:00.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:00.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:00.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:00.569 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:00.659 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:27:00.667 INFO:tasks.rados.rados.0.smithi137.stdout:788: done (4 left) 2024-09-21T08:27:00.668 INFO:tasks.rados.rados.0.smithi137.stdout:789: read oid 7 snap 71 2024-09-21T08:27:00.668 INFO:tasks.rados.rados.0.smithi137.stdout:789: expect (ObjNum 278 snap 64 seq_num 278) 2024-09-21T08:27:00.668 INFO:tasks.rados.rados.0.smithi137.stdout:790: read oid 18 snap -1 2024-09-21T08:27:00.668 INFO:tasks.rados.rados.0.smithi137.stdout:790: expect deleted 2024-09-21T08:27:00.668 INFO:tasks.rados.rados.0.smithi137.stdout:791: delete oid 27 current snap is 72 2024-09-21T08:27:00.668 INFO:tasks.rados.rados.0.smithi137.stdout:785: finishing write tid 1 to smithi13779221-12 2024-09-21T08:27:00.668 INFO:tasks.rados.rados.0.smithi137.stdout:785: finishing write tid 2 to smithi13779221-12 2024-09-21T08:27:00.668 INFO:tasks.rados.rados.0.smithi137.stdout:784: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:27:00.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:00.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:00.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:00.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:00.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:00.823 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:00.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:00.824 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: osdmap e226: 8 total, 8 up, 5 in 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]: dispatch 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.c", "id": [7, 5]}]: dispatch 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]: dispatch 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.c", "id": [7, 5]}]: dispatch 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:00.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2279012522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: osdmap e226: 8 total, 8 up, 5 in 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]: dispatch 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.c", "id": [7, 5]}]: dispatch 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]: dispatch 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.c", "id": [7, 5]}]: dispatch 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:00.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2279012522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:00.862 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:27:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:27:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:27:00.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T08:27:00.888 INFO:tasks.rados.rados.0.smithi137.stdout:785: finishing write tid 3 to smithi13779221-12 2024-09-21T08:27:00.888 INFO:tasks.rados.rados.0.smithi137.stdout:785: finishing write tid 4 to smithi13779221-12 2024-09-21T08:27:00.888 INFO:tasks.rados.rados.0.smithi137.stdout:784: done (6 left) 2024-09-21T08:27:00.888 INFO:tasks.rados.rados.0.smithi137.stdout:787: done (5 left) 2024-09-21T08:27:00.888 INFO:tasks.rados.rados.0.smithi137.stdout:791: done (4 left) 2024-09-21T08:27:00.888 INFO:tasks.rados.rados.0.smithi137.stdout:792: snap_create 2024-09-21T08:27:00.888 INFO:tasks.rados.rados.0.smithi137.stdout:785: finishing write tid 5 to smithi13779221-12 2024-09-21T08:27:00.888 INFO:tasks.rados.rados.0.smithi137.stdout:785: oid 12 updating version 0 to 282 2024-09-21T08:27:00.888 INFO:tasks.rados.rados.0.smithi137.stdout:785: oid 12 updating version 282 to 285 2024-09-21T08:27:00.889 INFO:tasks.rados.rados.0.smithi137.stdout:785: oid 12 version 285 is already newer than 284 2024-09-21T08:27:00.889 INFO:tasks.rados.rados.0.smithi137.stdout:785: oid 12 version 285 is already newer than 283 2024-09-21T08:27:00.889 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 285 (ObjNum 315 snap 72 seq_num 315) dirty exists 2024-09-21T08:27:00.889 INFO:tasks.rados.rados.0.smithi137.stdout:785: left oid 12 (ObjNum 315 snap 72 seq_num 315) 2024-09-21T08:27:00.889 INFO:tasks.rados.rados.0.smithi137.stdout:786: expect (ObjNum 276 snap 64 seq_num 276) 2024-09-21T08:27:00.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:00.952 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:00.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:00.953 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:00.980 INFO:tasks.rados.rados.0.smithi137.stdout:789: expect (ObjNum 278 snap 64 seq_num 278) 2024-09-21T08:27:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: osdmap e226: 8 total, 8 up, 5 in 2024-09-21T08:27:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-09-21T08:27:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]: dispatch 2024-09-21T08:27:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]: dispatch 2024-09-21T08:27:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-09-21T08:27:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.c", "id": [7, 5]}]: dispatch 2024-09-21T08:27:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]: dispatch 2024-09-21T08:27:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]: dispatch 2024-09-21T08:27:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.c", "id": [7, 5]}]: dispatch 2024-09-21T08:27:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:00 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2279012522' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:01.054 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:01.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:01.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:01.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:01.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:01.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:01.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:01.203 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:01.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:01.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:01.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:01.327 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:01.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:01.450 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:01.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:01.451 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:01.523 INFO:tasks.rados.rados.0.smithi137.stdout:785: done (4 left) 2024-09-21T08:27:01.523 INFO:tasks.rados.rados.0.smithi137.stdout:786: done (3 left) 2024-09-21T08:27:01.523 INFO:tasks.rados.rados.0.smithi137.stdout:789: done (2 left) 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:790: done (1 left) 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:792: done (0 left) 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:793: read oid 19 snap -1 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:793: expect deleted 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:794: read oid 42 snap -1 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:794: expect (ObjNum 290 snap 66 seq_num 290) 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:795: read oid 26 snap 72 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:795: expect (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:796: rollback oid 3 current snap is 73 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 3 to 59 2024-09-21T08:27:01.524 INFO:tasks.rados.rados.0.smithi137.stdout:797: copy_from oid 33 from oid 43 current snap is 73 2024-09-21T08:27:01.525 INFO:tasks.rados.rados.0.smithi137.stdout:798: copy_from oid 36 from oid 38 current snap is 73 2024-09-21T08:27:01.526 INFO:tasks.rados.rados.0.smithi137.stdout:799: write oid 17 current snap is 73 2024-09-21T08:27:01.526 INFO:tasks.rados.rados.0.smithi137.stdout:799: seq_num 317 ranges {527819=624406,1833733=410771,2722618=601581} 2024-09-21T08:27:01.530 INFO:tasks.rados.rados.0.smithi137.stdout:799: writing smithi13779221-17 from 527819 to 1152225 tid 1 2024-09-21T08:27:01.536 INFO:tasks.rados.rados.0.smithi137.stdout:799: writing smithi13779221-17 from 1833733 to 2244504 tid 2 2024-09-21T08:27:01.544 INFO:tasks.rados.rados.0.smithi137.stdout:799: writing smithi13779221-17 from 2722618 to 3324199 tid 3 2024-09-21T08:27:01.545 INFO:tasks.rados.rados.0.smithi137.stdout:800: snap_remove snap 71 2024-09-21T08:27:01.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:01.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:01.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:01.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:01.676 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:01.698 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:01.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:01.699 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:01.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:01.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:01.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:01.823 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:01.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:01.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:01.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:01.947 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:02.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:27:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:27:02.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:01 smithi165 ceph-mon[32893]: pgmap v245: 158 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 145 active+clean; 167 MiB data, 614 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:27:02.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:01 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:02.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:01 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-09-21T08:27:02.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:01 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]': finished 2024-09-21T08:27:02.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:01 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2024-09-21T08:27:02.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:01 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.c", "id": [7, 5]}]': finished 2024-09-21T08:27:02.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:01 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2279012522' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:02.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:01 smithi165 ceph-mon[32893]: osdmap e227: 8 total, 8 up, 5 in 2024-09-21T08:27:02.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:01 smithi165 ceph-mon[32893]: osdmap e228: 8 total, 8 up, 5 in 2024-09-21T08:27:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[29634]: pgmap v245: 158 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 145 active+clean; 167 MiB data, 614 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:27:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-09-21T08:27:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]': finished 2024-09-21T08:27:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2024-09-21T08:27:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.c", "id": [7, 5]}]': finished 2024-09-21T08:27:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2279012522' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[29634]: osdmap e227: 8 total, 8 up, 5 in 2024-09-21T08:27:02.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[29634]: osdmap e228: 8 total, 8 up, 5 in 2024-09-21T08:27:02.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[41044]: pgmap v245: 158 pgs: 6 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 145 active+clean; 167 MiB data, 614 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:27:02.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:02.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 2]}]': finished 2024-09-21T08:27:02.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.8", "id": [3, 2]}]': finished 2024-09-21T08:27:02.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [0, 2]}]': finished 2024-09-21T08:27:02.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "7.c", "id": [7, 5]}]': finished 2024-09-21T08:27:02.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2279012522' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:02.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[41044]: osdmap e227: 8 total, 8 up, 5 in 2024-09-21T08:27:02.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:01 smithi137 ceph-mon[41044]: osdmap e228: 8 total, 8 up, 5 in 2024-09-21T08:27:02.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:02.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:02.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:02.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:02.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:02.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:02.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:02.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:02.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:02.317 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:02.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:02.318 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:02.417 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:27:02.419 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:27:02.419 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:02.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:02.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:02.446 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:02.505 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:02.505 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:02.522 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:02.523 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:02.526 INFO:tasks.rados.rados.0.smithi137.stdout:800: done (7 left) 2024-09-21T08:27:02.526 INFO:tasks.rados.rados.0.smithi137.stdout:801: copy_from oid 11 from oid 21 current snap is 73 2024-09-21T08:27:02.527 INFO:tasks.rados.rados.0.smithi137.stdout:796: finishing rollback tid 0 to smithi13779221-3 2024-09-21T08:27:02.527 INFO:tasks.rados.rados.0.smithi137.stdout:796: finishing rollback tid 1 to smithi13779221-3 2024-09-21T08:27:02.527 INFO:tasks.rados.rados.0.smithi137.stdout:796: finishing rollback tid 2 to smithi13779221-3 2024-09-21T08:27:02.527 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 174 (ObjNum 220 snap 46 seq_num 220) dirty exists 2024-09-21T08:27:02.527 INFO:tasks.rados.rados.0.smithi137.stdout:795: expect (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:27:02.546 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:02.546 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:02.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:02.567 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:02.568 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:02.574 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:02.574 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:02.596 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:02.596 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:02.623 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:02.623 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:02.651 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:02.651 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:02.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:02.674 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:02.675 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:02.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:02.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:02.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:02.714 INFO:tasks.rados.rados.0.smithi137.stdout:798: finishing copy_from racing read to smithi13779221-36 2024-09-21T08:27:02.714 INFO:tasks.rados.rados.0.smithi137.stdout:799: finishing write tid 1 to smithi13779221-17 2024-09-21T08:27:02.714 INFO:tasks.rados.rados.0.smithi137.stdout:799: finishing write tid 2 to smithi13779221-17 2024-09-21T08:27:02.714 INFO:tasks.rados.rados.0.smithi137.stdout:794: expect (ObjNum 290 snap 66 seq_num 290) 2024-09-21T08:27:02.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:02.806 INFO:tasks.rados.rados.0.smithi137.stdout:797: finishing copy_from racing read to smithi13779221-33 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:797: finishing copy_from to smithi13779221-33 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 279 (ObjNum 156 snap 31 seq_num 156) dirty exists 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:799: finishing write tid 3 to smithi13779221-17 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:793: done (7 left) 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:794: done (6 left) 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:795: done (5 left) 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:796: done (4 left) 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:797: done (3 left) 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:802: read oid 7 snap 69 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:802: expect (ObjNum 278 snap 64 seq_num 278) 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:803: snap_create 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:798: finishing copy_from to smithi13779221-36 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 318 (ObjNum 281 snap 64 seq_num 281) dirty exists 2024-09-21T08:27:02.807 INFO:tasks.rados.rados.0.smithi137.stdout:799: finishing write tid 4 to smithi13779221-17 2024-09-21T08:27:02.808 INFO:tasks.rados.rados.0.smithi137.stdout:799: finishing write tid 5 to smithi13779221-17 2024-09-21T08:27:02.808 INFO:tasks.rados.rados.0.smithi137.stdout:799: oid 17 updating version 0 to 237 2024-09-21T08:27:02.808 INFO:tasks.rados.rados.0.smithi137.stdout:799: oid 17 version 237 is already newer than 236 2024-09-21T08:27:02.808 INFO:tasks.rados.rados.0.smithi137.stdout:799: oid 17 version 237 is already newer than 235 2024-09-21T08:27:02.808 INFO:tasks.rados.rados.0.smithi137.stdout:799: oid 17 updating version 237 to 238 2024-09-21T08:27:02.808 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 238 (ObjNum 317 snap 73 seq_num 317) dirty exists 2024-09-21T08:27:02.808 INFO:tasks.rados.rados.0.smithi137.stdout:799: left oid 17 (ObjNum 317 snap 73 seq_num 317) 2024-09-21T08:27:02.808 INFO:tasks.rados.rados.0.smithi137.stdout:801: finishing copy_from racing read to smithi13779221-11 2024-09-21T08:27:02.808 INFO:tasks.rados.rados.0.smithi137.stdout:801: finishing copy_from to smithi13779221-11 2024-09-21T08:27:02.808 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 99 (ObjNum 200 snap 40 seq_num 200) dirty exists 2024-09-21T08:27:02.813 INFO:tasks.rados.rados.0.smithi137.stdout:802: expect (ObjNum 278 snap 64 seq_num 278) 2024-09-21T08:27:02.823 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:02.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:02.824 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:02.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:02.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:02.948 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:02.948 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:03.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:02 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:03.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:02 smithi165 ceph-mon[32893]: pgmap v248: 158 pgs: 4 peering, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 194 MiB data, 650 MiB used, 446 GiB / 447 GiB avail; 20 MiB/s rd, 4.8 MiB/s wr, 38 op/s 2024-09-21T08:27:03.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:02 smithi165 ceph-mon[32893]: from='client.15252 v1:172.21.15.137:0/2810845237' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:27:03.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:02 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1688475086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:03.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:02 smithi165 ceph-mon[32893]: osdmap e229: 8 total, 8 up, 5 in 2024-09-21T08:27:03.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:02 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:03.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:02 smithi137 ceph-mon[29634]: pgmap v248: 158 pgs: 4 peering, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 194 MiB data, 650 MiB used, 446 GiB / 447 GiB avail; 20 MiB/s rd, 4.8 MiB/s wr, 38 op/s 2024-09-21T08:27:03.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:02 smithi137 ceph-mon[29634]: from='client.15252 v1:172.21.15.137:0/2810845237' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:27:03.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:02 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1688475086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:03.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:02 smithi137 ceph-mon[29634]: osdmap e229: 8 total, 8 up, 5 in 2024-09-21T08:27:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:02 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:02 smithi137 ceph-mon[41044]: pgmap v248: 158 pgs: 4 peering, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 194 MiB data, 650 MiB used, 446 GiB / 447 GiB avail; 20 MiB/s rd, 4.8 MiB/s wr, 38 op/s 2024-09-21T08:27:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:02 smithi137 ceph-mon[41044]: from='client.15252 v1:172.21.15.137:0/2810845237' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:27:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:02 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1688475086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:02 smithi137 ceph-mon[41044]: osdmap e229: 8 total, 8 up, 5 in 2024-09-21T08:27:03.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:03.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:03.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:03.072 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:03.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:03.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:03.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:03.197 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:03.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:03.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:03.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:03.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:03.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:03.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:03.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:03.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:03.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:03.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:03.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:03.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:03.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:03.676 INFO:tasks.rados.rados.0.smithi137.stdout:798: done (4 left) 2024-09-21T08:27:03.676 INFO:tasks.rados.rados.0.smithi137.stdout:799: done (3 left) 2024-09-21T08:27:03.676 INFO:tasks.rados.rados.0.smithi137.stdout:801: done (2 left) 2024-09-21T08:27:03.676 INFO:tasks.rados.rados.0.smithi137.stdout:802: done (1 left) 2024-09-21T08:27:03.676 INFO:tasks.rados.rados.0.smithi137.stdout:803: done (0 left) 2024-09-21T08:27:03.676 INFO:tasks.rados.rados.0.smithi137.stdout:804: read oid 38 snap -1 2024-09-21T08:27:03.676 INFO:tasks.rados.rados.0.smithi137.stdout:804: expect (ObjNum 281 snap 64 seq_num 281) 2024-09-21T08:27:03.676 INFO:tasks.rados.rados.0.smithi137.stdout:805: snap_remove snap 72 2024-09-21T08:27:03.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:03.696 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:03.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:27:03.793 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":248,"stamp":"2024-09-21T08:27:02.072435+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":203727800,"num_objects":292,"num_object_clones":46,"num_object_copies":876,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":292,"num_whiteouts":11,"num_read":1887,"num_read_kb":787155,"num_write":1184,"num_write_kb":210643,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":554,"num_bytes_recovered":115403499,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4032,"ondisk_log_size":4032,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":666064,"kb_used_data":480976,"kb_used_omap":7,"kb_used_meta":184952,"kb_avail":468059696,"statfs":{"total":479975178240,"available":479293128704,"internally_reserved":0,"allocated":492519424,"data_stored":593805841,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7968,"internal_metadata":189391072},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28811928,"num_objects":18,"num_object_clones":17,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":87,"num_read_kb":83562,"num_write":67,"num_write_kb":19819,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.000806"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.674572+0000","last_change":"2024-09-21T08:25:38.378517+0000","last_active":"2024-09-21T08:27:00.674572+0000","last_peered":"2024-09-21T08:27:00.674572+0000","last_clean":"2024-09-21T08:27:00.674572+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:27:00.674572+0000","last_undegraded":"2024-09-21T08:27:00.674572+0000","last_fullsized":"2024-09-21T08:27:00.674572+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.245122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":43,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667094+0000","last_change":"2024-09-21T08:26:49.666610+0000","last_active":"2024-09-21T08:27:00.667094+0000","last_peered":"2024-09-21T08:27:00.667094+0000","last_clean":"2024-09-21T08:27:00.667094+0000","last_became_active":"2024-09-21T08:26:49.560883+0000","last_became_peered":"2024-09-21T08:26:49.560883+0000","last_unstale":"2024-09-21T08:27:00.667094+0000","last_undegraded":"2024-09-21T08:27:00.667094+0000","last_fullsized":"2024-09-21T08:27:00.667094+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:24:50.572709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.572709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":156,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666058+0000","last_change":"2024-09-21T08:25:48.912078+0000","last_active":"2024-09-21T08:27:00.666058+0000","last_peered":"2024-09-21T08:27:00.666058+0000","last_clean":"2024-09-21T08:27:00.666058+0000","last_became_active":"2024-09-21T08:25:48.911674+0000","last_became_peered":"2024-09-21T08:25:48.911674+0000","last_unstale":"2024-09-21T08:27:00.666058+0000","last_undegraded":"2024-09-21T08:27:00.666058+0000","last_fullsized":"2024-09-21T08:27:00.666058+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":303,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666586+0000","last_change":"2024-09-21T08:25:48.914509+0000","last_active":"2024-09-21T08:27:00.666586+0000","last_peered":"2024-09-21T08:27:00.666586+0000","last_clean":"2024-09-21T08:27:00.666586+0000","last_became_active":"2024-09-21T08:25:48.911853+0000","last_became_peered":"2024-09-21T08:25:48.911853+0000","last_unstale":"2024-09-21T08:27:00.666586+0000","last_undegraded":"2024-09-21T08:27:00.666586+0000","last_fullsized":"2024-09-21T08:27:00.666586+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667415+0000","last_change":"2024-09-21T08:26:49.560554+0000","last_active":"2024-09-21T08:27:00.667415+0000","last_peered":"2024-09-21T08:27:00.667415+0000","last_clean":"2024-09-21T08:27:00.667415+0000","last_became_active":"2024-09-21T08:26:49.560166+0000","last_became_peered":"2024-09-21T08:26:49.560166+0000","last_unstale":"2024-09-21T08:27:00.667415+0000","last_undegraded":"2024-09-21T08:27:00.667415+0000","last_fullsized":"2024-09-21T08:27:00.667415+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668981+0000","last_change":"2024-09-21T08:26:49.558121+0000","last_active":"2024-09-21T08:27:00.668981+0000","last_peered":"2024-09-21T08:27:00.668981+0000","last_clean":"2024-09-21T08:27:00.668981+0000","last_became_active":"2024-09-21T08:26:49.557060+0000","last_became_peered":"2024-09-21T08:26:49.557060+0000","last_unstale":"2024-09-21T08:27:00.668981+0000","last_undegraded":"2024-09-21T08:27:00.668981+0000","last_fullsized":"2024-09-21T08:27:00.668981+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:19.906906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:19.906906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999449+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":49,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667104+0000","last_change":"2024-09-21T08:26:49.675965+0000","last_active":"2024-09-21T08:27:00.667104+0000","last_peered":"2024-09-21T08:27:00.667104+0000","last_clean":"2024-09-21T08:27:00.667104+0000","last_became_active":"2024-09-21T08:26:49.563017+0000","last_became_peered":"2024-09-21T08:26:49.563017+0000","last_unstale":"2024-09-21T08:27:00.667104+0000","last_undegraded":"2024-09-21T08:27:00.667104+0000","last_fullsized":"2024-09-21T08:27:00.667104+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:25:20.941990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:20.941990+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,0],"acting":[3,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":161,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662161+0000","last_change":"2024-09-21T08:25:48.912528+0000","last_active":"2024-09-21T08:27:00.662161+0000","last_peered":"2024-09-21T08:27:00.662161+0000","last_clean":"2024-09-21T08:27:00.662161+0000","last_became_active":"2024-09-21T08:25:48.911583+0000","last_became_peered":"2024-09-21T08:25:48.911583+0000","last_unstale":"2024-09-21T08:27:00.662161+0000","last_undegraded":"2024-09-21T08:27:00.662161+0000","last_fullsized":"2024-09-21T08:27:00.662161+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.640831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.640831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"227'179","reported_seq":987,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.679612+0000","last_change":"2024-09-21T08:27:00.679850+0000","last_active":"2024-09-21T08:27:00.679612+0000","last_peered":"2024-09-21T08:27:00.679612+0000","last_clean":"2024-09-21T08:27:00.679612+0000","last_became_active":"2024-09-21T08:26:20.935479+0000","last_became_peered":"2024-09-21T08:26:20.935479+0000","last_unstale":"2024-09-21T08:27:00.679612+0000","last_undegraded":"2024-09-21T08:27:00.679612+0000","last_fullsized":"2024-09-21T08:27:00.679612+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'167","last_scrub_stamp":"2024-09-21T08:26:26.899804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:26.899804+0000","objects_scrubbed":1,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:37.458820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.012644614,"stat_sum":{"num_bytes":3243751,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":18,"num_read_kb":25352,"num_write":2,"num_write_kb":3168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"},{"start":"46","length":"1"}]},{"pgid":"3.1d","version":"0'0","reported_seq":369,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667310+0000","last_change":"2024-09-21T08:26:49.559305+0000","last_active":"2024-09-21T08:27:00.667310+0000","last_peered":"2024-09-21T08:27:00.667310+0000","last_clean":"2024-09-21T08:27:00.667310+0000","last_became_active":"2024-09-21T08:26:49.558311+0000","last_became_peered":"2024-09-21T08:26:49.558311+0000","last_unstale":"2024-09-21T08:27:00.667310+0000","last_undegraded":"2024-09-21T08:27:00.667310+0000","last_fullsized":"2024-09-21T08:27:00.667310+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.780545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.780545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:01.000834+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":43,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.672615+0000","last_change":"2024-09-21T08:26:49.631668+0000","last_active":"2024-09-21T08:27:00.672615+0000","last_peered":"2024-09-21T08:27:00.672615+0000","last_clean":"2024-09-21T08:27:00.672615+0000","last_became_active":"2024-09-21T08:26:49.558180+0000","last_became_peered":"2024-09-21T08:26:49.558180+0000","last_unstale":"2024-09-21T08:27:00.672615+0000","last_undegraded":"2024-09-21T08:27:00.672615+0000","last_fullsized":"2024-09-21T08:27:00.672615+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":334,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667118+0000","last_change":"2024-09-21T08:25:22.581281+0000","last_active":"2024-09-21T08:27:00.667118+0000","last_peered":"2024-09-21T08:27:00.667118+0000","last_clean":"2024-09-21T08:27:00.667118+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:27:00.667118+0000","last_undegraded":"2024-09-21T08:27:00.667118+0000","last_fullsized":"2024-09-21T08:27:00.667118+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":327,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669902+0000","last_change":"2024-09-21T08:25:48.885985+0000","last_active":"2024-09-21T08:27:00.669902+0000","last_peered":"2024-09-21T08:27:00.669902+0000","last_clean":"2024-09-21T08:27:00.669902+0000","last_became_active":"2024-09-21T08:25:48.885136+0000","last_became_peered":"2024-09-21T08:25:48.885136+0000","last_unstale":"2024-09-21T08:27:00.669902+0000","last_undegraded":"2024-09-21T08:27:00.669902+0000","last_fullsized":"2024-09-21T08:27:00.669902+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"190'170","reported_seq":933,"reported_epoch":227,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:00.667828+0000","last_change":"2024-09-21T08:26:57.534466+0000","last_active":"2024-09-21T08:27:00.667828+0000","last_peered":"2024-09-21T08:27:00.667828+0000","last_clean":"2024-09-21T08:27:00.667828+0000","last_became_active":"2024-09-21T08:26:20.935965+0000","last_became_peered":"2024-09-21T08:26:20.935965+0000","last_unstale":"2024-09-21T08:27:00.667828+0000","last_undegraded":"2024-09-21T08:27:00.667828+0000","last_fullsized":"2024-09-21T08:27:00.667828+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"190'170","last_scrub_stamp":"2024-09-21T08:26:28.392516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.392516+0000","objects_scrubbed":2,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:38.609353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001462156,"stat_sum":{"num_bytes":7109361,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":9,"num_read_kb":16664,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"42","length":"2"}]},{"pgid":"3.1c","version":"0'0","reported_seq":360,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666832+0000","last_change":"2024-09-21T08:25:48.912437+0000","last_active":"2024-09-21T08:27:00.666832+0000","last_peered":"2024-09-21T08:27:00.666832+0000","last_clean":"2024-09-21T08:27:00.666832+0000","last_became_active":"2024-09-21T08:25:48.912043+0000","last_became_peered":"2024-09-21T08:25:48.912043+0000","last_unstale":"2024-09-21T08:27:00.666832+0000","last_undegraded":"2024-09-21T08:27:00.666832+0000","last_fullsized":"2024-09-21T08:27:00.666832+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":39,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667325+0000","last_change":"2024-09-21T08:26:49.581307+0000","last_active":"2024-09-21T08:27:00.667325+0000","last_peered":"2024-09-21T08:27:00.667325+0000","last_clean":"2024-09-21T08:27:00.667325+0000","last_became_active":"2024-09-21T08:26:49.556200+0000","last_became_peered":"2024-09-21T08:26:49.556200+0000","last_unstale":"2024-09-21T08:27:00.667325+0000","last_undegraded":"2024-09-21T08:27:00.667325+0000","last_fullsized":"2024-09-21T08:27:00.667325+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:24:32.569846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:32.569846+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":349,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669180+0000","last_change":"2024-09-21T08:26:49.557008+0000","last_active":"2024-09-21T08:27:00.669180+0000","last_peered":"2024-09-21T08:27:00.669180+0000","last_clean":"2024-09-21T08:27:00.669180+0000","last_became_active":"2024-09-21T08:26:49.556672+0000","last_became_peered":"2024-09-21T08:26:49.556672+0000","last_unstale":"2024-09-21T08:27:00.669180+0000","last_undegraded":"2024-09-21T08:27:00.669180+0000","last_fullsized":"2024-09-21T08:27:00.669180+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_clean_scrub_stamp":"2024-09-21T08:26:09.782307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:19.256521+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":353,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666270+0000","last_change":"2024-09-21T08:26:49.620432+0000","last_active":"2024-09-21T08:27:00.666270+0000","last_peered":"2024-09-21T08:27:00.666270+0000","last_clean":"2024-09-21T08:27:00.666270+0000","last_became_active":"2024-09-21T08:26:49.554541+0000","last_became_peered":"2024-09-21T08:26:49.554541+0000","last_unstale":"2024-09-21T08:27:00.666270+0000","last_undegraded":"2024-09-21T08:27:00.666270+0000","last_fullsized":"2024-09-21T08:27:00.666270+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":305,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667315+0000","last_change":"2024-09-21T08:26:49.555907+0000","last_active":"2024-09-21T08:27:00.667315+0000","last_peered":"2024-09-21T08:27:00.667315+0000","last_clean":"2024-09-21T08:27:00.667315+0000","last_became_active":"2024-09-21T08:26:49.555557+0000","last_became_peered":"2024-09-21T08:26:49.555557+0000","last_unstale":"2024-09-21T08:27:00.667315+0000","last_undegraded":"2024-09-21T08:27:00.667315+0000","last_fullsized":"2024-09-21T08:27:00.667315+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.855448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.855448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":2,"reported_epoch":227,"state":"peering","last_fresh":"2024-09-21T08:26:59.661332+0000","last_change":"2024-09-21T08:27:00.714167+0000","last_active":"2024-09-21T08:27:00.714167+0000","last_peered":"2024-09-21T08:26:59.661332+0000","last_clean":"2024-09-21T08:26:59.661332+0000","last_became_active":"2024-09-21T08:26:49.568926+0000","last_became_peered":"2024-09-21T08:26:49.568926+0000","last_unstale":"2024-09-21T08:26:59.661332+0000","last_undegraded":"2024-09-21T08:26:59.661332+0000","last_fullsized":"2024-09-21T08:26:59.661332+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":436,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662970+0000","last_change":"2024-09-21T08:26:49.588793+0000","last_active":"2024-09-21T08:27:00.662970+0000","last_peered":"2024-09-21T08:27:00.662970+0000","last_clean":"2024-09-21T08:27:00.662970+0000","last_became_active":"2024-09-21T08:26:49.554314+0000","last_became_peered":"2024-09-21T08:26:49.554314+0000","last_unstale":"2024-09-21T08:27:00.662970+0000","last_undegraded":"2024-09-21T08:27:00.662970+0000","last_fullsized":"2024-09-21T08:27:00.662970+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:26:45.263994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.263994+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:00.313491+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":207,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670407+0000","last_change":"2024-09-21T08:26:49.566278+0000","last_active":"2024-09-21T08:27:00.670407+0000","last_peered":"2024-09-21T08:27:00.670407+0000","last_clean":"2024-09-21T08:27:00.670407+0000","last_became_active":"2024-09-21T08:26:49.565939+0000","last_became_peered":"2024-09-21T08:26:49.565939+0000","last_unstale":"2024-09-21T08:27:00.670407+0000","last_undegraded":"2024-09-21T08:27:00.670407+0000","last_fullsized":"2024-09-21T08:27:00.670407+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:30.310628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.310628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:39.760178+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669812+0000","last_change":"2024-09-21T08:26:49.558948+0000","last_active":"2024-09-21T08:27:00.669812+0000","last_peered":"2024-09-21T08:27:00.669812+0000","last_clean":"2024-09-21T08:27:00.669812+0000","last_became_active":"2024-09-21T08:26:49.557020+0000","last_became_peered":"2024-09-21T08:26:49.557020+0000","last_unstale":"2024-09-21T08:27:00.669812+0000","last_undegraded":"2024-09-21T08:27:00.669812+0000","last_fullsized":"2024-09-21T08:27:00.669812+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":166,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662338+0000","last_change":"2024-09-21T08:26:33.237763+0000","last_active":"2024-09-21T08:27:00.662338+0000","last_peered":"2024-09-21T08:27:00.662338+0000","last_clean":"2024-09-21T08:27:00.662338+0000","last_became_active":"2024-09-21T08:25:48.899796+0000","last_became_peered":"2024-09-21T08:25:48.899796+0000","last_unstale":"2024-09-21T08:27:00.662338+0000","last_undegraded":"2024-09-21T08:27:00.662338+0000","last_fullsized":"2024-09-21T08:27:00.662338+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:33.237691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.237691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:57.612618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":45,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669775+0000","last_change":"2024-09-21T08:26:49.636840+0000","last_active":"2024-09-21T08:27:00.669775+0000","last_peered":"2024-09-21T08:27:00.669775+0000","last_clean":"2024-09-21T08:27:00.669775+0000","last_became_active":"2024-09-21T08:26:49.558955+0000","last_became_peered":"2024-09-21T08:26:49.558955+0000","last_unstale":"2024-09-21T08:27:00.669775+0000","last_undegraded":"2024-09-21T08:27:00.669775+0000","last_fullsized":"2024-09-21T08:27:00.669775+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:21.424414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.424414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670610+0000","last_change":"2024-09-21T08:26:49.563435+0000","last_active":"2024-09-21T08:27:00.670610+0000","last_peered":"2024-09-21T08:27:00.670610+0000","last_clean":"2024-09-21T08:27:00.670610+0000","last_became_active":"2024-09-21T08:26:49.560143+0000","last_became_peered":"2024-09-21T08:26:49.560143+0000","last_unstale":"2024-09-21T08:27:00.670610+0000","last_undegraded":"2024-09-21T08:27:00.670610+0000","last_fullsized":"2024-09-21T08:27:00.670610+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.797084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":320,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669535+0000","last_change":"2024-09-21T08:23:36.584003+0000","last_active":"2024-09-21T08:27:00.669535+0000","last_peered":"2024-09-21T08:27:00.669535+0000","last_clean":"2024-09-21T08:27:00.669535+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:27:00.669535+0000","last_undegraded":"2024-09-21T08:27:00.669535+0000","last_fullsized":"2024-09-21T08:27:00.669535+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"198'2","reported_seq":37,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667584+0000","last_change":"2024-09-21T08:26:49.659949+0000","last_active":"2024-09-21T08:27:00.667584+0000","last_peered":"2024-09-21T08:27:00.667584+0000","last_clean":"2024-09-21T08:27:00.667584+0000","last_became_active":"2024-09-21T08:26:49.560808+0000","last_became_peered":"2024-09-21T08:26:49.560808+0000","last_unstale":"2024-09-21T08:27:00.667584+0000","last_undegraded":"2024-09-21T08:27:00.667584+0000","last_fullsized":"2024-09-21T08:27:00.667584+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'2","last_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_deep_scrub":"198'2","last_deep_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_clean_scrub_stamp":"2024-09-21T08:26:32.819313+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:39.504304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":398,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.673880+0000","last_change":"2024-09-21T08:26:49.666681+0000","last_active":"2024-09-21T08:27:00.673880+0000","last_peered":"2024-09-21T08:27:00.673880+0000","last_clean":"2024-09-21T08:27:00.673880+0000","last_became_active":"2024-09-21T08:26:49.569195+0000","last_became_peered":"2024-09-21T08:26:49.569195+0000","last_unstale":"2024-09-21T08:27:00.673880+0000","last_undegraded":"2024-09-21T08:27:00.673880+0000","last_fullsized":"2024-09-21T08:27:00.673880+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"217'13","reported_seq":421,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.671257+0000","last_change":"2024-09-21T08:26:49.622656+0000","last_active":"2024-09-21T08:27:00.671257+0000","last_peered":"2024-09-21T08:27:00.671257+0000","last_clean":"2024-09-21T08:27:00.671257+0000","last_became_active":"2024-09-21T08:26:49.557194+0000","last_became_peered":"2024-09-21T08:26:49.557194+0000","last_unstale":"2024-09-21T08:27:00.671257+0000","last_undegraded":"2024-09-21T08:27:00.671257+0000","last_fullsized":"2024-09-21T08:27:00.671257+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.594212+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"225'129","reported_seq":774,"reported_epoch":226,"state":"peering","last_fresh":"2024-09-21T08:26:59.652318+0000","last_change":"2024-09-21T08:27:00.667776+0000","last_active":"2024-09-21T08:27:00.667776+0000","last_peered":"2024-09-21T08:26:59.652318+0000","last_clean":"2024-09-21T08:26:59.652318+0000","last_became_active":"2024-09-21T08:26:52.534542+0000","last_became_peered":"2024-09-21T08:26:52.534542+0000","last_unstale":"2024-09-21T08:26:59.652318+0000","last_undegraded":"2024-09-21T08:26:59.652318+0000","last_fullsized":"2024-09-21T08:26:59.652318+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'119","last_scrub_stamp":"2024-09-21T08:26:43.765848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:43.765848+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.86185442199999995,"stat_sum":{"num_bytes":10443223,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":105,"num_read_kb":107231,"num_write":67,"num_write_kb":33235,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":19311715,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,5,7],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"}]},{"pgid":"4.f","version":"73'15","reported_seq":49,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669535+0000","last_change":"2024-09-21T08:26:49.591809+0000","last_active":"2024-09-21T08:27:00.669535+0000","last_peered":"2024-09-21T08:27:00.669535+0000","last_clean":"2024-09-21T08:27:00.669535+0000","last_became_active":"2024-09-21T08:26:49.553271+0000","last_became_peered":"2024-09-21T08:26:49.553271+0000","last_unstale":"2024-09-21T08:27:00.669535+0000","last_undegraded":"2024-09-21T08:27:00.669535+0000","last_fullsized":"2024-09-21T08:27:00.669535+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:26:46.777115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.777115+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:04.093322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":2,"reported_epoch":227,"state":"peering","last_fresh":"2024-09-21T08:26:59.650426+0000","last_change":"2024-09-21T08:27:00.714096+0000","last_active":"2024-09-21T08:27:00.714096+0000","last_peered":"2024-09-21T08:26:59.650426+0000","last_clean":"2024-09-21T08:26:59.650426+0000","last_became_active":"2024-09-21T08:26:49.552339+0000","last_became_peered":"2024-09-21T08:26:49.552339+0000","last_unstale":"2024-09-21T08:26:59.650426+0000","last_undegraded":"2024-09-21T08:26:59.650426+0000","last_fullsized":"2024-09-21T08:26:59.650426+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:45.434055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:45.434055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.e","version":"85'9","reported_seq":372,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667643+0000","last_change":"2024-09-21T08:26:44.741364+0000","last_active":"2024-09-21T08:27:00.667643+0000","last_peered":"2024-09-21T08:27:00.667643+0000","last_clean":"2024-09-21T08:27:00.667643+0000","last_became_active":"2024-09-21T08:24:38.548789+0000","last_became_peered":"2024-09-21T08:24:38.548789+0000","last_unstale":"2024-09-21T08:27:00.667643+0000","last_undegraded":"2024-09-21T08:27:00.667643+0000","last_fullsized":"2024-09-21T08:27:00.667643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:26:44.741275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.741275+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:57.665735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":336,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667440+0000","last_change":"2024-09-21T08:26:08.864503+0000","last_active":"2024-09-21T08:27:00.667440+0000","last_peered":"2024-09-21T08:27:00.667440+0000","last_clean":"2024-09-21T08:27:00.667440+0000","last_became_active":"2024-09-21T08:25:48.914453+0000","last_became_peered":"2024-09-21T08:25:48.914453+0000","last_unstale":"2024-09-21T08:27:00.667440+0000","last_undegraded":"2024-09-21T08:27:00.667440+0000","last_fullsized":"2024-09-21T08:27:00.667440+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:08.864399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:08.864399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:29.676635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"179'19","reported_seq":66,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667726+0000","last_change":"2024-09-21T08:27:00.668033+0000","last_active":"2024-09-21T08:27:00.667726+0000","last_peered":"2024-09-21T08:27:00.667726+0000","last_clean":"2024-09-21T08:27:00.667726+0000","last_became_active":"2024-09-21T08:26:49.556057+0000","last_became_peered":"2024-09-21T08:26:49.556057+0000","last_unstale":"2024-09-21T08:27:00.667726+0000","last_undegraded":"2024-09-21T08:27:00.667726+0000","last_fullsized":"2024-09-21T08:27:00.667726+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:26:51.043840+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.043840+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:20.721199+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064820600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"},{"start":"46","length":"1"}]},{"pgid":"4.0","version":"73'18","reported_seq":59,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669252+0000","last_change":"2024-09-21T08:26:54.003250+0000","last_active":"2024-09-21T08:27:00.669252+0000","last_peered":"2024-09-21T08:27:00.669252+0000","last_clean":"2024-09-21T08:27:00.669252+0000","last_became_active":"2024-09-21T08:26:49.560206+0000","last_became_peered":"2024-09-21T08:26:49.560206+0000","last_unstale":"2024-09-21T08:27:00.669252+0000","last_undegraded":"2024-09-21T08:27:00.669252+0000","last_fullsized":"2024-09-21T08:27:00.669252+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-09-21T08:26:54.003173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:54.003173+0000","objects_scrubbed":8,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:01.184353+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669626+0000","last_change":"2024-09-21T08:26:49.572789+0000","last_active":"2024-09-21T08:27:00.669626+0000","last_peered":"2024-09-21T08:27:00.669626+0000","last_clean":"2024-09-21T08:27:00.669626+0000","last_became_active":"2024-09-21T08:26:49.572501+0000","last_became_peered":"2024-09-21T08:26:49.572501+0000","last_unstale":"2024-09-21T08:27:00.669626+0000","last_undegraded":"2024-09-21T08:27:00.669626+0000","last_fullsized":"2024-09-21T08:27:00.669626+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999403+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666943+0000","last_change":"2024-09-21T08:26:49.565902+0000","last_active":"2024-09-21T08:27:00.666943+0000","last_peered":"2024-09-21T08:27:00.666943+0000","last_clean":"2024-09-21T08:27:00.666943+0000","last_became_active":"2024-09-21T08:26:49.564128+0000","last_became_peered":"2024-09-21T08:26:49.564128+0000","last_unstale":"2024-09-21T08:27:00.666943+0000","last_undegraded":"2024-09-21T08:27:00.666943+0000","last_fullsized":"2024-09-21T08:27:00.666943+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.672515+0000","last_change":"2024-09-21T08:26:49.556170+0000","last_active":"2024-09-21T08:27:00.672515+0000","last_peered":"2024-09-21T08:27:00.672515+0000","last_clean":"2024-09-21T08:27:00.672515+0000","last_became_active":"2024-09-21T08:26:49.555833+0000","last_became_peered":"2024-09-21T08:26:49.555833+0000","last_unstale":"2024-09-21T08:27:00.672515+0000","last_undegraded":"2024-09-21T08:27:00.672515+0000","last_fullsized":"2024-09-21T08:27:00.672515+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999187+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.2","version":"227'219","reported_seq":1005,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:00.671451+0000","last_change":"2024-09-21T08:27:00.671451+0000","last_active":"2024-09-21T08:27:00.671451+0000","last_peered":"2024-09-21T08:27:00.671451+0000","last_clean":"2024-09-21T08:27:00.671451+0000","last_became_active":"2024-09-21T08:26:52.533818+0000","last_became_peered":"2024-09-21T08:26:52.533818+0000","last_unstale":"2024-09-21T08:27:00.671451+0000","last_undegraded":"2024-09-21T08:27:00.671451+0000","last_fullsized":"2024-09-21T08:27:00.671451+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'198","last_scrub_stamp":"2024-09-21T08:26:30.265653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.265653+0000","objects_scrubbed":2,"log_size":219,"log_dups_size":0,"ondisk_log_size":219,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:38.881218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.86268748900000003,"stat_sum":{"num_bytes":8017038,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":10,"num_read_kb":9187,"num_write":13,"num_write_kb":2371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":3460603,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"}]},{"pgid":"4.1","version":"73'14","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666001+0000","last_change":"2024-09-21T08:24:56.561414+0000","last_active":"2024-09-21T08:27:00.666001+0000","last_peered":"2024-09-21T08:27:00.666001+0000","last_clean":"2024-09-21T08:27:00.666001+0000","last_became_active":"2024-09-21T08:24:56.509169+0000","last_became_peered":"2024-09-21T08:24:56.509169+0000","last_unstale":"2024-09-21T08:27:00.666001+0000","last_undegraded":"2024-09-21T08:27:00.666001+0000","last_fullsized":"2024-09-21T08:27:00.666001+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":362,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.673605+0000","last_change":"2024-09-21T08:25:48.919957+0000","last_active":"2024-09-21T08:27:00.673605+0000","last_peered":"2024-09-21T08:27:00.673605+0000","last_clean":"2024-09-21T08:27:00.673605+0000","last_became_active":"2024-09-21T08:25:48.893688+0000","last_became_peered":"2024-09-21T08:25:48.893688+0000","last_unstale":"2024-09-21T08:27:00.673605+0000","last_undegraded":"2024-09-21T08:27:00.673605+0000","last_fullsized":"2024-09-21T08:27:00.673605+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.245063+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"217'12","reported_seq":40,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667501+0000","last_change":"2024-09-21T08:26:49.662341+0000","last_active":"2024-09-21T08:27:00.667501+0000","last_peered":"2024-09-21T08:27:00.667501+0000","last_clean":"2024-09-21T08:27:00.667501+0000","last_became_active":"2024-09-21T08:26:49.560772+0000","last_became_peered":"2024-09-21T08:26:49.560772+0000","last_unstale":"2024-09-21T08:27:00.667501+0000","last_undegraded":"2024-09-21T08:27:00.667501+0000","last_fullsized":"2024-09-21T08:27:00.667501+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":339,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669276+0000","last_change":"2024-09-21T08:25:48.900309+0000","last_active":"2024-09-21T08:27:00.669276+0000","last_peered":"2024-09-21T08:27:00.669276+0000","last_clean":"2024-09-21T08:27:00.669276+0000","last_became_active":"2024-09-21T08:25:48.899971+0000","last_became_peered":"2024-09-21T08:25:48.899971+0000","last_unstale":"2024-09-21T08:27:00.669276+0000","last_undegraded":"2024-09-21T08:27:00.669276+0000","last_fullsized":"2024-09-21T08:27:00.669276+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:21.561826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.561826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802353+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"179'43","reported_seq":62,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668119+0000","last_change":"2024-09-21T08:27:00.670153+0000","last_active":"2024-09-21T08:27:00.668119+0000","last_peered":"2024-09-21T08:27:00.668119+0000","last_clean":"2024-09-21T08:27:00.668119+0000","last_became_active":"2024-09-21T08:26:49.566765+0000","last_became_peered":"2024-09-21T08:26:49.566765+0000","last_unstale":"2024-09-21T08:27:00.668119+0000","last_undegraded":"2024-09-21T08:27:00.668119+0000","last_fullsized":"2024-09-21T08:27:00.668119+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:26:32.285475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:32.285475+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:55.744704+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0028820959999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"},{"start":"46","length":"1"}]},{"pgid":"4.2","version":"73'10","reported_seq":210,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666206+0000","last_change":"2024-09-21T08:26:05.809673+0000","last_active":"2024-09-21T08:27:00.666206+0000","last_peered":"2024-09-21T08:27:00.666206+0000","last_clean":"2024-09-21T08:27:00.666206+0000","last_became_active":"2024-09-21T08:25:48.894952+0000","last_became_peered":"2024-09-21T08:25:48.894952+0000","last_unstale":"2024-09-21T08:27:00.666206+0000","last_undegraded":"2024-09-21T08:27:00.666206+0000","last_fullsized":"2024-09-21T08:27:00.666206+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:26:05.809597+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.809597+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:07.999081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":370,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666824+0000","last_change":"2024-09-21T08:26:49.649582+0000","last_active":"2024-09-21T08:27:00.666824+0000","last_peered":"2024-09-21T08:27:00.666824+0000","last_clean":"2024-09-21T08:27:00.666824+0000","last_became_active":"2024-09-21T08:26:49.557390+0000","last_became_peered":"2024-09-21T08:26:49.557390+0000","last_unstale":"2024-09-21T08:27:00.666824+0000","last_undegraded":"2024-09-21T08:27:00.666824+0000","last_fullsized":"2024-09-21T08:27:00.666824+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"217'12","reported_seq":415,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669067+0000","last_change":"2024-09-21T08:26:49.660192+0000","last_active":"2024-09-21T08:27:00.669067+0000","last_peered":"2024-09-21T08:27:00.669067+0000","last_clean":"2024-09-21T08:27:00.669067+0000","last_became_active":"2024-09-21T08:26:49.570273+0000","last_became_peered":"2024-09-21T08:26:49.570273+0000","last_unstale":"2024-09-21T08:27:00.669067+0000","last_undegraded":"2024-09-21T08:27:00.669067+0000","last_fullsized":"2024-09-21T08:27:00.669067+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'11","last_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_deep_scrub":"176'11","last_deep_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_clean_scrub_stamp":"2024-09-21T08:26:10.440547+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:18.534349+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":333,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669690+0000","last_change":"2024-09-21T08:26:49.567144+0000","last_active":"2024-09-21T08:27:00.669690+0000","last_peered":"2024-09-21T08:27:00.669690+0000","last_clean":"2024-09-21T08:27:00.669690+0000","last_became_active":"2024-09-21T08:26:49.566834+0000","last_became_peered":"2024-09-21T08:26:49.566834+0000","last_unstale":"2024-09-21T08:27:00.669690+0000","last_undegraded":"2024-09-21T08:27:00.669690+0000","last_fullsized":"2024-09-21T08:27:00.669690+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.245027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,2],"acting":[0,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"227'119","reported_seq":928,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:00.667748+0000","last_change":"2024-09-21T08:27:00.667748+0000","last_active":"2024-09-21T08:27:00.667748+0000","last_peered":"2024-09-21T08:27:00.667748+0000","last_clean":"2024-09-21T08:27:00.667748+0000","last_became_active":"2024-09-21T08:26:52.532334+0000","last_became_peered":"2024-09-21T08:26:52.532334+0000","last_unstale":"2024-09-21T08:27:00.667748+0000","last_undegraded":"2024-09-21T08:27:00.667748+0000","last_fullsized":"2024-09-21T08:27:00.667748+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'100","last_scrub_stamp":"2024-09-21T08:26:26.838731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:26.838731+0000","objects_scrubbed":1,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:32.467738+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.88713140899999998,"stat_sum":{"num_bytes":6212836,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":13,"num_read_kb":14915,"num_write":9,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":3062097,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"}]},{"pgid":"4.3","version":"73'19","reported_seq":365,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668874+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:27:00.668874+0000","last_peered":"2024-09-21T08:27:00.668874+0000","last_clean":"2024-09-21T08:27:00.668874+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:27:00.668874+0000","last_undegraded":"2024-09-21T08:27:00.668874+0000","last_fullsized":"2024-09-21T08:27:00.668874+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244880+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":157,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662779+0000","last_change":"2024-09-21T08:25:48.906191+0000","last_active":"2024-09-21T08:27:00.662779+0000","last_peered":"2024-09-21T08:27:00.662779+0000","last_clean":"2024-09-21T08:27:00.662779+0000","last_became_active":"2024-09-21T08:25:48.905818+0000","last_became_peered":"2024-09-21T08:25:48.905818+0000","last_unstale":"2024-09-21T08:27:00.662779+0000","last_undegraded":"2024-09-21T08:27:00.662779+0000","last_fullsized":"2024-09-21T08:27:00.662779+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.529672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.529672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":268,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.676090+0000","last_change":"2024-09-21T08:25:39.339196+0000","last_active":"2024-09-21T08:27:00.676090+0000","last_peered":"2024-09-21T08:27:00.676090+0000","last_clean":"2024-09-21T08:27:00.676090+0000","last_became_active":"2024-09-21T08:24:56.499628+0000","last_became_peered":"2024-09-21T08:24:56.499628+0000","last_unstale":"2024-09-21T08:27:00.676090+0000","last_undegraded":"2024-09-21T08:27:00.676090+0000","last_fullsized":"2024-09-21T08:27:00.676090+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.339126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.339126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:03.207166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":233,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662285+0000","last_change":"2024-09-21T08:26:46.264057+0000","last_active":"2024-09-21T08:27:00.662285+0000","last_peered":"2024-09-21T08:27:00.662285+0000","last_clean":"2024-09-21T08:27:00.662285+0000","last_became_active":"2024-09-21T08:25:48.885120+0000","last_became_peered":"2024-09-21T08:25:48.885120+0000","last_unstale":"2024-09-21T08:27:00.662285+0000","last_undegraded":"2024-09-21T08:27:00.662285+0000","last_fullsized":"2024-09-21T08:27:00.662285+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:46.263984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.263984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:50.138045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"224'83","reported_seq":865,"reported_epoch":227,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:00.675200+0000","last_change":"2024-09-21T08:26:55.530907+0000","last_active":"2024-09-21T08:27:00.675200+0000","last_peered":"2024-09-21T08:27:00.675200+0000","last_clean":"2024-09-21T08:27:00.675200+0000","last_became_active":"2024-09-21T08:26:49.567761+0000","last_became_peered":"2024-09-21T08:26:49.567761+0000","last_unstale":"2024-09-21T08:27:00.675200+0000","last_undegraded":"2024-09-21T08:27:00.675200+0000","last_fullsized":"2024-09-21T08:27:00.675200+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":83,"log_dups_size":0,"ondisk_log_size":83,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:58.340201+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.112811172,"stat_sum":{"num_bytes":7865246,"num_objects":4,"num_object_clones":2,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":6571537,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"2"}]},{"pgid":"4.4","version":"217'32","reported_seq":290,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.663260+0000","last_change":"2024-09-21T08:26:49.595566+0000","last_active":"2024-09-21T08:27:00.663260+0000","last_peered":"2024-09-21T08:27:00.663260+0000","last_clean":"2024-09-21T08:27:00.663260+0000","last_became_active":"2024-09-21T08:26:49.554321+0000","last_became_peered":"2024-09-21T08:26:49.554321+0000","last_unstale":"2024-09-21T08:27:00.663260+0000","last_undegraded":"2024-09-21T08:27:00.663260+0000","last_fullsized":"2024-09-21T08:27:00.663260+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":716,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,7],"acting":[2,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":314,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669308+0000","last_change":"2024-09-21T08:26:49.570768+0000","last_active":"2024-09-21T08:27:00.669308+0000","last_peered":"2024-09-21T08:27:00.669308+0000","last_clean":"2024-09-21T08:27:00.669308+0000","last_became_active":"2024-09-21T08:26:49.569363+0000","last_became_peered":"2024-09-21T08:26:49.569363+0000","last_unstale":"2024-09-21T08:27:00.669308+0000","last_undegraded":"2024-09-21T08:27:00.669308+0000","last_fullsized":"2024-09-21T08:27:00.669308+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":382,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666001+0000","last_change":"2024-09-21T08:26:49.601390+0000","last_active":"2024-09-21T08:27:00.666001+0000","last_peered":"2024-09-21T08:27:00.666001+0000","last_clean":"2024-09-21T08:27:00.666001+0000","last_became_active":"2024-09-21T08:26:49.555217+0000","last_became_peered":"2024-09-21T08:26:49.555217+0000","last_unstale":"2024-09-21T08:27:00.666001+0000","last_undegraded":"2024-09-21T08:27:00.666001+0000","last_fullsized":"2024-09-21T08:27:00.666001+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:32.010627+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":57,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":357,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670267+0000","last_change":"2024-09-21T08:26:49.565404+0000","last_active":"2024-09-21T08:27:00.670267+0000","last_peered":"2024-09-21T08:27:00.670267+0000","last_clean":"2024-09-21T08:27:00.670267+0000","last_became_active":"2024-09-21T08:26:49.565080+0000","last_became_peered":"2024-09-21T08:26:49.565080+0000","last_unstale":"2024-09-21T08:27:00.670267+0000","last_undegraded":"2024-09-21T08:27:00.670267+0000","last_fullsized":"2024-09-21T08:27:00.670267+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:45.237804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.237804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:13.899797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669271+0000","last_change":"2024-09-21T08:26:49.568643+0000","last_active":"2024-09-21T08:27:00.669271+0000","last_peered":"2024-09-21T08:27:00.669271+0000","last_clean":"2024-09-21T08:27:00.669271+0000","last_became_active":"2024-09-21T08:26:49.568114+0000","last_became_peered":"2024-09-21T08:26:49.568114+0000","last_unstale":"2024-09-21T08:27:00.669271+0000","last_undegraded":"2024-09-21T08:27:00.669271+0000","last_fullsized":"2024-09-21T08:27:00.669271+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:45.761359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.761359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:54.452736+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.4","version":"223'61","reported_seq":844,"reported_epoch":227,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:00.672209+0000","last_change":"2024-09-21T08:26:59.537191+0000","last_active":"2024-09-21T08:27:00.672209+0000","last_peered":"2024-09-21T08:27:00.672209+0000","last_clean":"2024-09-21T08:27:00.672209+0000","last_became_active":"2024-09-21T08:26:49.565205+0000","last_became_peered":"2024-09-21T08:26:49.565205+0000","last_unstale":"2024-09-21T08:27:00.672209+0000","last_undegraded":"2024-09-21T08:27:00.672209+0000","last_fullsized":"2024-09-21T08:27:00.672209+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:26:34.278834+0000","objects_scrubbed":1,"log_size":61,"log_dups_size":0,"ondisk_log_size":61,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:59.792676+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.883862131,"stat_sum":{"num_bytes":3831996,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":9,"num_read_kb":17663,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":2750472,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"42","length":"2"}]},{"pgid":"4.7","version":"73'13","reported_seq":214,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670095+0000","last_change":"2024-09-21T08:25:48.937399+0000","last_active":"2024-09-21T08:27:00.670095+0000","last_peered":"2024-09-21T08:27:00.670095+0000","last_clean":"2024-09-21T08:27:00.670095+0000","last_became_active":"2024-09-21T08:25:48.908351+0000","last_became_peered":"2024-09-21T08:25:48.908351+0000","last_unstale":"2024-09-21T08:27:00.670095+0000","last_undegraded":"2024-09-21T08:27:00.670095+0000","last_fullsized":"2024-09-21T08:27:00.670095+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":245,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662122+0000","last_change":"2024-09-21T08:26:52.225812+0000","last_active":"2024-09-21T08:27:00.662122+0000","last_peered":"2024-09-21T08:27:00.662122+0000","last_clean":"2024-09-21T08:27:00.662122+0000","last_became_active":"2024-09-21T08:25:48.911245+0000","last_became_peered":"2024-09-21T08:25:48.911245+0000","last_unstale":"2024-09-21T08:27:00.662122+0000","last_undegraded":"2024-09-21T08:27:00.662122+0000","last_fullsized":"2024-09-21T08:27:00.662122+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:52.225741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:52.225741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:08.931292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":390,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662171+0000","last_change":"2024-09-21T08:26:49.575983+0000","last_active":"2024-09-21T08:27:00.662171+0000","last_peered":"2024-09-21T08:27:00.662171+0000","last_clean":"2024-09-21T08:27:00.662171+0000","last_became_active":"2024-09-21T08:26:49.554412+0000","last_became_peered":"2024-09-21T08:26:49.554412+0000","last_unstale":"2024-09-21T08:27:00.662171+0000","last_undegraded":"2024-09-21T08:27:00.662171+0000","last_fullsized":"2024-09-21T08:27:00.662171+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:50.675673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.675673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":333,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.664193+0000","last_change":"2024-09-21T08:25:42.021037+0000","last_active":"2024-09-21T08:27:00.664193+0000","last_peered":"2024-09-21T08:27:00.664193+0000","last_clean":"2024-09-21T08:27:00.664193+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:27:00.664193+0000","last_undegraded":"2024-09-21T08:27:00.664193+0000","last_fullsized":"2024-09-21T08:27:00.664193+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:42.020928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.020928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:04.279526+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":337,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670927+0000","last_change":"2024-09-21T08:26:49.560337+0000","last_active":"2024-09-21T08:27:00.670927+0000","last_peered":"2024-09-21T08:27:00.670927+0000","last_clean":"2024-09-21T08:27:00.670927+0000","last_became_active":"2024-09-21T08:26:49.559016+0000","last_became_peered":"2024-09-21T08:26:49.559016+0000","last_unstale":"2024-09-21T08:27:00.670927+0000","last_undegraded":"2024-09-21T08:27:00.670927+0000","last_fullsized":"2024-09-21T08:27:00.670927+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802152+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"227'254","reported_seq":997,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.682662+0000","last_change":"2024-09-21T08:27:00.682843+0000","last_active":"2024-09-21T08:27:00.682662+0000","last_peered":"2024-09-21T08:27:00.682662+0000","last_clean":"2024-09-21T08:27:00.682662+0000","last_became_active":"2024-09-21T08:26:15.915988+0000","last_became_peered":"2024-09-21T08:26:15.915988+0000","last_unstale":"2024-09-21T08:27:00.682662+0000","last_undegraded":"2024-09-21T08:27:00.682662+0000","last_fullsized":"2024-09-21T08:27:00.682662+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'245","last_scrub_stamp":"2024-09-21T08:26:30.864040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.864040+0000","objects_scrubbed":0,"log_size":254,"log_dups_size":0,"ondisk_log_size":254,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:53.332383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011575968000000001,"stat_sum":{"num_bytes":2874030,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":6,"num_write_kb":1348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"},{"start":"46","length":"1"}]},{"pgid":"4.6","version":"73'12","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.673786+0000","last_change":"2024-09-21T08:26:07.384721+0000","last_active":"2024-09-21T08:27:00.673786+0000","last_peered":"2024-09-21T08:27:00.673786+0000","last_clean":"2024-09-21T08:27:00.673786+0000","last_became_active":"2024-09-21T08:25:48.888592+0000","last_became_peered":"2024-09-21T08:25:48.888592+0000","last_unstale":"2024-09-21T08:27:00.673786+0000","last_undegraded":"2024-09-21T08:27:00.673786+0000","last_fullsized":"2024-09-21T08:27:00.673786+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:26:07.384640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:07.384640+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:17.751790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.673115+0000","last_change":"2024-09-21T08:26:49.567141+0000","last_active":"2024-09-21T08:27:00.673115+0000","last_peered":"2024-09-21T08:27:00.673115+0000","last_clean":"2024-09-21T08:27:00.673115+0000","last_became_active":"2024-09-21T08:26:49.566816+0000","last_became_peered":"2024-09-21T08:26:49.566816+0000","last_unstale":"2024-09-21T08:27:00.673115+0000","last_undegraded":"2024-09-21T08:27:00.673115+0000","last_fullsized":"2024-09-21T08:27:00.673115+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":574,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670134+0000","last_change":"2024-09-21T08:26:49.597638+0000","last_active":"2024-09-21T08:27:00.670134+0000","last_peered":"2024-09-21T08:27:00.670134+0000","last_clean":"2024-09-21T08:27:00.670134+0000","last_became_active":"2024-09-21T08:26:49.556434+0000","last_became_peered":"2024-09-21T08:26:49.556434+0000","last_unstale":"2024-09-21T08:27:00.670134+0000","last_undegraded":"2024-09-21T08:27:00.670134+0000","last_fullsized":"2024-09-21T08:27:00.670134+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":202,"num_read_kb":197,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":340,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670073+0000","last_change":"2024-09-21T08:25:48.909001+0000","last_active":"2024-09-21T08:27:00.670073+0000","last_peered":"2024-09-21T08:27:00.670073+0000","last_clean":"2024-09-21T08:27:00.670073+0000","last_became_active":"2024-09-21T08:25:48.908492+0000","last_became_peered":"2024-09-21T08:25:48.908492+0000","last_unstale":"2024-09-21T08:27:00.670073+0000","last_undegraded":"2024-09-21T08:27:00.670073+0000","last_fullsized":"2024-09-21T08:27:00.670073+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.812789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.812789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":237,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666847+0000","last_change":"2024-09-21T08:26:49.556316+0000","last_active":"2024-09-21T08:27:00.666847+0000","last_peered":"2024-09-21T08:27:00.666847+0000","last_clean":"2024-09-21T08:27:00.666847+0000","last_became_active":"2024-09-21T08:26:49.555991+0000","last_became_peered":"2024-09-21T08:26:49.555991+0000","last_unstale":"2024-09-21T08:27:00.666847+0000","last_undegraded":"2024-09-21T08:27:00.666847+0000","last_fullsized":"2024-09-21T08:27:00.666847+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,2],"acting":[5,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"224'109","reported_seq":886,"reported_epoch":227,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:00.674173+0000","last_change":"2024-09-21T08:26:55.531365+0000","last_active":"2024-09-21T08:27:00.674173+0000","last_peered":"2024-09-21T08:27:00.674173+0000","last_clean":"2024-09-21T08:27:00.674173+0000","last_became_active":"2024-09-21T08:26:52.533639+0000","last_became_peered":"2024-09-21T08:26:52.533639+0000","last_unstale":"2024-09-21T08:27:00.674173+0000","last_undegraded":"2024-09-21T08:27:00.674173+0000","last_fullsized":"2024-09-21T08:27:00.674173+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'105","last_scrub_stamp":"2024-09-21T08:26:53.224145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.224145+0000","objects_scrubbed":2,"log_size":109,"log_dups_size":0,"ondisk_log_size":109,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:06.777425+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0024093959999999998,"stat_sum":{"num_bytes":6605184,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"2"}]},{"pgid":"4.5","version":"217'19","reported_seq":370,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670723+0000","last_change":"2024-09-21T08:26:49.639667+0000","last_active":"2024-09-21T08:27:00.670723+0000","last_peered":"2024-09-21T08:27:00.670723+0000","last_clean":"2024-09-21T08:27:00.670723+0000","last_became_active":"2024-09-21T08:26:49.559515+0000","last_became_peered":"2024-09-21T08:26:49.559515+0000","last_unstale":"2024-09-21T08:27:00.670723+0000","last_undegraded":"2024-09-21T08:27:00.670723+0000","last_fullsized":"2024-09-21T08:27:00.670723+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-09-21T08:24:48.583137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.583137+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":2,"reported_epoch":227,"state":"peering","last_fresh":"2024-09-21T08:26:59.650898+0000","last_change":"2024-09-21T08:27:00.714152+0000","last_active":"2024-09-21T08:27:00.714152+0000","last_peered":"2024-09-21T08:26:59.650898+0000","last_clean":"2024-09-21T08:26:59.650898+0000","last_became_active":"2024-09-21T08:26:49.552205+0000","last_became_peered":"2024-09-21T08:26:49.552205+0000","last_unstale":"2024-09-21T08:26:59.650898+0000","last_undegraded":"2024-09-21T08:26:59.650898+0000","last_fullsized":"2024-09-21T08:26:59.650898+0000","mapping_epoch":227,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.045353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.045353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":625,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667855+0000","last_change":"2024-09-21T08:25:51.506619+0000","last_active":"2024-09-21T08:27:00.667855+0000","last_peered":"2024-09-21T08:27:00.667855+0000","last_clean":"2024-09-21T08:27:00.667855+0000","last_became_active":"2024-09-21T08:25:51.506298+0000","last_became_peered":"2024-09-21T08:25:51.506298+0000","last_unstale":"2024-09-21T08:27:00.667855+0000","last_undegraded":"2024-09-21T08:27:00.667855+0000","last_fullsized":"2024-09-21T08:27:00.667855+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:04.766419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":331,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669755+0000","last_change":"2024-09-21T08:25:41.700634+0000","last_active":"2024-09-21T08:27:00.669755+0000","last_peered":"2024-09-21T08:27:00.669755+0000","last_clean":"2024-09-21T08:27:00.669755+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:27:00.669755+0000","last_undegraded":"2024-09-21T08:27:00.669755+0000","last_fullsized":"2024-09-21T08:27:00.669755+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:04.766398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":354,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670192+0000","last_change":"2024-09-21T08:26:49.557820+0000","last_active":"2024-09-21T08:27:00.670192+0000","last_peered":"2024-09-21T08:27:00.670192+0000","last_clean":"2024-09-21T08:27:00.670192+0000","last_became_active":"2024-09-21T08:26:49.557131+0000","last_became_peered":"2024-09-21T08:26:49.557131+0000","last_unstale":"2024-09-21T08:27:00.670192+0000","last_undegraded":"2024-09-21T08:27:00.670192+0000","last_fullsized":"2024-09-21T08:27:00.670192+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"225'66","reported_seq":742,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:00.670631+0000","last_change":"2024-09-21T08:26:59.537209+0000","last_active":"2024-09-21T08:27:00.670631+0000","last_peered":"2024-09-21T08:27:00.670631+0000","last_clean":"2024-09-21T08:27:00.670631+0000","last_became_active":"2024-09-21T08:26:49.566892+0000","last_became_peered":"2024-09-21T08:26:49.566892+0000","last_unstale":"2024-09-21T08:27:00.670631+0000","last_undegraded":"2024-09-21T08:27:00.670631+0000","last_fullsized":"2024-09-21T08:27:00.670631+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.124092722,"stat_sum":{"num_bytes":5731220,"num_objects":3,"num_object_clones":1,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":49,"num_read_kb":44197,"num_write":43,"num_write_kb":18389,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":12585743,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"2"}]},{"pgid":"4.e","version":"73'11","reported_seq":45,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668786+0000","last_change":"2024-09-21T08:26:49.585939+0000","last_active":"2024-09-21T08:27:00.668786+0000","last_peered":"2024-09-21T08:27:00.668786+0000","last_clean":"2024-09-21T08:27:00.668786+0000","last_became_active":"2024-09-21T08:26:49.553464+0000","last_became_peered":"2024-09-21T08:26:49.553464+0000","last_unstale":"2024-09-21T08:27:00.668786+0000","last_undegraded":"2024-09-21T08:27:00.668786+0000","last_fullsized":"2024-09-21T08:27:00.668786+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":311,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662231+0000","last_change":"2024-09-21T08:25:48.907133+0000","last_active":"2024-09-21T08:27:00.662231+0000","last_peered":"2024-09-21T08:27:00.662231+0000","last_clean":"2024-09-21T08:27:00.662231+0000","last_became_active":"2024-09-21T08:25:48.906832+0000","last_became_peered":"2024-09-21T08:25:48.906832+0000","last_unstale":"2024-09-21T08:27:00.662231+0000","last_undegraded":"2024-09-21T08:27:00.662231+0000","last_fullsized":"2024-09-21T08:27:00.662231+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.756905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.756905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":350,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670271+0000","last_change":"2024-09-21T08:26:49.555599+0000","last_active":"2024-09-21T08:27:00.670271+0000","last_peered":"2024-09-21T08:27:00.670271+0000","last_clean":"2024-09-21T08:27:00.670271+0000","last_became_active":"2024-09-21T08:26:49.555268+0000","last_became_peered":"2024-09-21T08:26:49.555268+0000","last_unstale":"2024-09-21T08:27:00.670271+0000","last_undegraded":"2024-09-21T08:27:00.670271+0000","last_fullsized":"2024-09-21T08:27:00.670271+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670521+0000","last_change":"2024-09-21T08:26:49.552971+0000","last_active":"2024-09-21T08:27:00.670521+0000","last_peered":"2024-09-21T08:27:00.670521+0000","last_clean":"2024-09-21T08:27:00.670521+0000","last_became_active":"2024-09-21T08:26:49.552216+0000","last_became_peered":"2024-09-21T08:26:49.552216+0000","last_unstale":"2024-09-21T08:27:00.670521+0000","last_undegraded":"2024-09-21T08:27:00.670521+0000","last_fullsized":"2024-09-21T08:27:00.670521+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.e","version":"227'154","reported_seq":873,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:00.667275+0000","last_change":"2024-09-21T08:27:00.664315+0000","last_active":"2024-09-21T08:27:00.667275+0000","last_peered":"2024-09-21T08:27:00.667275+0000","last_clean":"2024-09-21T08:27:00.667275+0000","last_became_active":"2024-09-21T08:24:34.508974+0000","last_became_peered":"2024-09-21T08:24:34.508974+0000","last_unstale":"2024-09-21T08:27:00.667275+0000","last_undegraded":"2024-09-21T08:27:00.667275+0000","last_fullsized":"2024-09-21T08:27:00.667275+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.89471831199999996,"stat_sum":{"num_bytes":16883987,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":70,"num_read_kb":64284,"num_write":77,"num_write_kb":28152,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"}]},{"pgid":"4.d","version":"73'17","reported_seq":418,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.663070+0000","last_change":"2024-09-21T08:25:48.934395+0000","last_active":"2024-09-21T08:27:00.663070+0000","last_peered":"2024-09-21T08:27:00.663070+0000","last_clean":"2024-09-21T08:27:00.663070+0000","last_became_active":"2024-09-21T08:25:48.904254+0000","last_became_peered":"2024-09-21T08:25:48.904254+0000","last_unstale":"2024-09-21T08:27:00.663070+0000","last_undegraded":"2024-09-21T08:27:00.663070+0000","last_fullsized":"2024-09-21T08:27:00.663070+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-09-21T08:25:42.978787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.978787+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670512+0000","last_change":"2024-09-21T08:26:49.567990+0000","last_active":"2024-09-21T08:27:00.670512+0000","last_peered":"2024-09-21T08:27:00.670512+0000","last_clean":"2024-09-21T08:27:00.670512+0000","last_became_active":"2024-09-21T08:26:49.567678+0000","last_became_peered":"2024-09-21T08:26:49.567678+0000","last_unstale":"2024-09-21T08:27:00.670512+0000","last_undegraded":"2024-09-21T08:27:00.670512+0000","last_fullsized":"2024-09-21T08:27:00.670512+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,7],"acting":[3,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":235,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.672317+0000","last_change":"2024-09-21T08:25:48.889493+0000","last_active":"2024-09-21T08:27:00.672317+0000","last_peered":"2024-09-21T08:27:00.672317+0000","last_clean":"2024-09-21T08:27:00.672317+0000","last_became_active":"2024-09-21T08:25:48.889206+0000","last_became_peered":"2024-09-21T08:25:48.889206+0000","last_unstale":"2024-09-21T08:27:00.672317+0000","last_undegraded":"2024-09-21T08:27:00.672317+0000","last_fullsized":"2024-09-21T08:27:00.672317+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244953+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":359,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.663380+0000","last_change":"2024-09-21T08:26:49.538837+0000","last_active":"2024-09-21T08:27:00.663380+0000","last_peered":"2024-09-21T08:27:00.663380+0000","last_clean":"2024-09-21T08:27:00.663380+0000","last_became_active":"2024-09-21T08:26:49.538545+0000","last_became_peered":"2024-09-21T08:26:49.538545+0000","last_unstale":"2024-09-21T08:27:00.663380+0000","last_undegraded":"2024-09-21T08:27:00.663380+0000","last_fullsized":"2024-09-21T08:27:00.663380+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.766815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,5],"acting":[2,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"216'28","reported_seq":766,"reported_epoch":227,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:00.675009+0000","last_change":"2024-09-21T08:26:55.530692+0000","last_active":"2024-09-21T08:27:00.675009+0000","last_peered":"2024-09-21T08:27:00.675009+0000","last_clean":"2024-09-21T08:27:00.675009+0000","last_became_active":"2024-09-21T08:26:49.567589+0000","last_became_peered":"2024-09-21T08:26:49.567589+0000","last_unstale":"2024-09-21T08:27:00.675009+0000","last_undegraded":"2024-09-21T08:27:00.675009+0000","last_fullsized":"2024-09-21T08:27:00.675009+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.11268840099999999,"stat_sum":{"num_bytes":5622221,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":33,"num_read_kb":32227,"num_write":20,"num_write_kb":11169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":7538304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"2"}]},{"pgid":"4.c","version":"73'10","reported_seq":45,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670695+0000","last_change":"2024-09-21T08:26:49.668296+0000","last_active":"2024-09-21T08:27:00.670695+0000","last_peered":"2024-09-21T08:27:00.670695+0000","last_clean":"2024-09-21T08:27:00.670695+0000","last_became_active":"2024-09-21T08:26:49.560923+0000","last_became_peered":"2024-09-21T08:26:49.560923+0000","last_unstale":"2024-09-21T08:27:00.670695+0000","last_undegraded":"2024-09-21T08:27:00.670695+0000","last_fullsized":"2024-09-21T08:27:00.670695+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:26:33.823950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.823950+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:44.257340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669287+0000","last_change":"2024-09-21T08:26:49.560312+0000","last_active":"2024-09-21T08:27:00.669287+0000","last_peered":"2024-09-21T08:27:00.669287+0000","last_clean":"2024-09-21T08:27:00.669287+0000","last_became_active":"2024-09-21T08:26:49.558587+0000","last_became_peered":"2024-09-21T08:26:49.558587+0000","last_unstale":"2024-09-21T08:27:00.669287+0000","last_undegraded":"2024-09-21T08:27:00.669287+0000","last_fullsized":"2024-09-21T08:27:00.669287+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":384,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.663015+0000","last_change":"2024-09-21T08:24:51.689976+0000","last_active":"2024-09-21T08:27:00.663015+0000","last_peered":"2024-09-21T08:27:00.663015+0000","last_clean":"2024-09-21T08:27:00.663015+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:27:00.663015+0000","last_undegraded":"2024-09-21T08:27:00.663015+0000","last_fullsized":"2024-09-21T08:27:00.663015+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:24:51.689901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.689901+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":167,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.663999+0000","last_change":"2024-09-21T08:26:51.222803+0000","last_active":"2024-09-21T08:27:00.663999+0000","last_peered":"2024-09-21T08:27:00.663999+0000","last_clean":"2024-09-21T08:27:00.663999+0000","last_became_active":"2024-09-21T08:25:48.903536+0000","last_became_peered":"2024-09-21T08:25:48.903536+0000","last_unstale":"2024-09-21T08:27:00.663999+0000","last_undegraded":"2024-09-21T08:27:00.663999+0000","last_fullsized":"2024-09-21T08:27:00.663999+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:51.222707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.222707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:05.138581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"227'204","reported_seq":1017,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:00.662980+0000","last_change":"2024-09-21T08:27:00.662980+0000","last_active":"2024-09-21T08:27:00.662980+0000","last_peered":"2024-09-21T08:27:00.662980+0000","last_clean":"2024-09-21T08:27:00.662980+0000","last_became_active":"2024-09-21T08:26:52.532950+0000","last_became_peered":"2024-09-21T08:26:52.532950+0000","last_unstale":"2024-09-21T08:27:00.662980+0000","last_undegraded":"2024-09-21T08:27:00.662980+0000","last_fullsized":"2024-09-21T08:27:00.662980+0000","mapping_epoch":218,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":219,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'178","last_scrub_stamp":"2024-09-21T08:26:28.217842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.217842+0000","objects_scrubbed":4,"log_size":204,"log_dups_size":0,"ondisk_log_size":204,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:39.002226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.89613807199999995,"stat_sum":{"num_bytes":10134654,"num_objects":5,"num_object_clones":2,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":30,"num_read_kb":35626,"num_write":7,"num_write_kb":5004,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":10628267,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,3],"acting":[2,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"}]},{"pgid":"4.b","version":"73'9","reported_seq":417,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668972+0000","last_change":"2024-09-21T08:26:49.657957+0000","last_active":"2024-09-21T08:27:00.668972+0000","last_peered":"2024-09-21T08:27:00.668972+0000","last_clean":"2024-09-21T08:27:00.668972+0000","last_became_active":"2024-09-21T08:26:49.569804+0000","last_became_peered":"2024-09-21T08:26:49.569804+0000","last_unstale":"2024-09-21T08:27:00.668972+0000","last_undegraded":"2024-09-21T08:27:00.668972+0000","last_fullsized":"2024-09-21T08:27:00.668972+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:25:22.369811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.369811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.245093+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,5],"acting":[0,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":366,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666952+0000","last_change":"2024-09-21T08:26:49.554711+0000","last_active":"2024-09-21T08:27:00.666952+0000","last_peered":"2024-09-21T08:27:00.666952+0000","last_clean":"2024-09-21T08:27:00.666952+0000","last_became_active":"2024-09-21T08:26:49.554344+0000","last_became_peered":"2024-09-21T08:26:49.554344+0000","last_unstale":"2024-09-21T08:27:00.666952+0000","last_undegraded":"2024-09-21T08:27:00.666952+0000","last_fullsized":"2024-09-21T08:27:00.666952+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.665444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.665444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,0],"acting":[5,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":357,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662554+0000","last_change":"2024-09-21T08:26:49.555105+0000","last_active":"2024-09-21T08:27:00.662554+0000","last_peered":"2024-09-21T08:27:00.662554+0000","last_clean":"2024-09-21T08:27:00.662554+0000","last_became_active":"2024-09-21T08:26:49.554491+0000","last_became_peered":"2024-09-21T08:26:49.554491+0000","last_unstale":"2024-09-21T08:27:00.662554+0000","last_undegraded":"2024-09-21T08:27:00.662554+0000","last_fullsized":"2024-09-21T08:27:00.662554+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":321,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.672481+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:27:00.672481+0000","last_peered":"2024-09-21T08:27:00.672481+0000","last_clean":"2024-09-21T08:27:00.672481+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:27:00.672481+0000","last_undegraded":"2024-09-21T08:27:00.672481+0000","last_fullsized":"2024-09-21T08:27:00.672481+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"222'187","reported_seq":1003,"reported_epoch":227,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:00.672761+0000","last_change":"2024-09-21T08:26:57.534763+0000","last_active":"2024-09-21T08:27:00.672761+0000","last_peered":"2024-09-21T08:27:00.672761+0000","last_clean":"2024-09-21T08:27:00.672761+0000","last_became_active":"2024-09-21T08:26:15.917755+0000","last_became_peered":"2024-09-21T08:26:15.917755+0000","last_unstale":"2024-09-21T08:27:00.672761+0000","last_undegraded":"2024-09-21T08:27:00.672761+0000","last_fullsized":"2024-09-21T08:27:00.672761+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:56.698484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.017328706,"stat_sum":{"num_bytes":17182185,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":36,"num_read_kb":49770,"num_write":10,"num_write_kb":2545,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"42","length":"2"}]},{"pgid":"4.a","version":"73'19","reported_seq":225,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667890+0000","last_change":"2024-09-21T08:25:48.964103+0000","last_active":"2024-09-21T08:27:00.667890+0000","last_peered":"2024-09-21T08:27:00.667890+0000","last_clean":"2024-09-21T08:27:00.667890+0000","last_became_active":"2024-09-21T08:25:48.895647+0000","last_became_peered":"2024-09-21T08:25:48.895647+0000","last_unstale":"2024-09-21T08:27:00.667890+0000","last_undegraded":"2024-09-21T08:27:00.667890+0000","last_fullsized":"2024-09-21T08:27:00.667890+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670490+0000","last_change":"2024-09-21T08:26:49.560321+0000","last_active":"2024-09-21T08:27:00.670490+0000","last_peered":"2024-09-21T08:27:00.670490+0000","last_clean":"2024-09-21T08:27:00.670490+0000","last_became_active":"2024-09-21T08:26:49.558284+0000","last_became_peered":"2024-09-21T08:26:49.558284+0000","last_unstale":"2024-09-21T08:27:00.670490+0000","last_undegraded":"2024-09-21T08:27:00.670490+0000","last_fullsized":"2024-09-21T08:27:00.670490+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.740674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.740674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802249+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,3],"acting":[7,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":325,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662533+0000","last_change":"2024-09-21T08:23:34.564923+0000","last_active":"2024-09-21T08:27:00.662533+0000","last_peered":"2024-09-21T08:27:00.662533+0000","last_clean":"2024-09-21T08:27:00.662533+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:27:00.662533+0000","last_undegraded":"2024-09-21T08:27:00.662533+0000","last_fullsized":"2024-09-21T08:27:00.662533+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":345,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667057+0000","last_change":"2024-09-21T08:26:49.538933+0000","last_active":"2024-09-21T08:27:00.667057+0000","last_peered":"2024-09-21T08:27:00.667057+0000","last_clean":"2024-09-21T08:27:00.667057+0000","last_became_active":"2024-09-21T08:26:49.538560+0000","last_became_peered":"2024-09-21T08:26:49.538560+0000","last_unstale":"2024-09-21T08:27:00.667057+0000","last_undegraded":"2024-09-21T08:27:00.667057+0000","last_fullsized":"2024-09-21T08:27:00.667057+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802308+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"224'66","reported_seq":81,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.673491+0000","last_change":"2024-09-21T08:27:00.676165+0000","last_active":"2024-09-21T08:27:00.673491+0000","last_peered":"2024-09-21T08:27:00.673491+0000","last_clean":"2024-09-21T08:27:00.673491+0000","last_became_active":"2024-09-21T08:26:49.556205+0000","last_became_peered":"2024-09-21T08:26:49.556205+0000","last_unstale":"2024-09-21T08:27:00.673491+0000","last_undegraded":"2024-09-21T08:27:00.673491+0000","last_fullsized":"2024-09-21T08:27:00.673491+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'60","last_scrub_stamp":"2024-09-21T08:26:44.214809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.214809+0000","objects_scrubbed":1,"log_size":66,"log_dups_size":0,"ondisk_log_size":66,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:50.695034+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0041121389999999999,"stat_sum":{"num_bytes":3092488,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":35,"num_read_kb":32896,"num_write":30,"num_write_kb":11916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":5644096,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"},{"start":"46","length":"1"}]},{"pgid":"4.9","version":"73'12","reported_seq":45,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667679+0000","last_change":"2024-09-21T08:26:49.643079+0000","last_active":"2024-09-21T08:27:00.667679+0000","last_peered":"2024-09-21T08:27:00.667679+0000","last_clean":"2024-09-21T08:27:00.667679+0000","last_became_active":"2024-09-21T08:26:49.560628+0000","last_became_peered":"2024-09-21T08:26:49.560628+0000","last_unstale":"2024-09-21T08:27:00.667679+0000","last_undegraded":"2024-09-21T08:27:00.667679+0000","last_fullsized":"2024-09-21T08:27:00.667679+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:41.463820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.463820+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.699556+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,7],"acting":[3,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670005+0000","last_change":"2024-09-21T08:26:51.769124+0000","last_active":"2024-09-21T08:27:00.670005+0000","last_peered":"2024-09-21T08:27:00.670005+0000","last_clean":"2024-09-21T08:27:00.670005+0000","last_became_active":"2024-09-21T08:25:48.896222+0000","last_became_peered":"2024-09-21T08:25:48.896222+0000","last_unstale":"2024-09-21T08:27:00.670005+0000","last_undegraded":"2024-09-21T08:27:00.670005+0000","last_fullsized":"2024-09-21T08:27:00.670005+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:51.769044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.769044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:06.214839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":339,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662308+0000","last_change":"2024-09-21T08:25:48.913293+0000","last_active":"2024-09-21T08:27:00.662308+0000","last_peered":"2024-09-21T08:27:00.662308+0000","last_clean":"2024-09-21T08:27:00.662308+0000","last_became_active":"2024-09-21T08:25:48.912993+0000","last_became_peered":"2024-09-21T08:25:48.912993+0000","last_unstale":"2024-09-21T08:27:00.662308+0000","last_undegraded":"2024-09-21T08:27:00.662308+0000","last_fullsized":"2024-09-21T08:27:00.662308+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:44.914394+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:44.914394+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:07.844366+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667663+0000","last_change":"2024-09-21T08:26:49.553467+0000","last_active":"2024-09-21T08:27:00.667663+0000","last_peered":"2024-09-21T08:27:00.667663+0000","last_clean":"2024-09-21T08:27:00.667663+0000","last_became_active":"2024-09-21T08:26:49.552319+0000","last_became_peered":"2024-09-21T08:26:49.552319+0000","last_unstale":"2024-09-21T08:27:00.667663+0000","last_undegraded":"2024-09-21T08:27:00.667663+0000","last_fullsized":"2024-09-21T08:27:00.667663+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999495+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.b","version":"225'350","reported_seq":1231,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:00.672734+0000","last_change":"2024-09-21T08:26:59.544775+0000","last_active":"2024-09-21T08:27:00.672734+0000","last_peered":"2024-09-21T08:27:00.672734+0000","last_clean":"2024-09-21T08:27:00.672734+0000","last_became_active":"2024-09-21T08:25:51.507525+0000","last_became_peered":"2024-09-21T08:25:51.507525+0000","last_unstale":"2024-09-21T08:27:00.672734+0000","last_undegraded":"2024-09-21T08:27:00.672734+0000","last_fullsized":"2024-09-21T08:27:00.672734+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.88627527399999995,"stat_sum":{"num_bytes":32431761,"num_objects":15,"num_object_clones":7,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":295,"num_read_kb":286333,"num_write":217,"num_write_kb":60789,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31871318,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"39","length":"1"},{"start":"42","length":"2"}]},{"pgid":"4.8","version":"73'15","reported_seq":429,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.665976+0000","last_change":"2024-09-21T08:26:49.594814+0000","last_active":"2024-09-21T08:27:00.665976+0000","last_peered":"2024-09-21T08:27:00.665976+0000","last_clean":"2024-09-21T08:27:00.665976+0000","last_became_active":"2024-09-21T08:26:49.555130+0000","last_became_peered":"2024-09-21T08:26:49.555130+0000","last_unstale":"2024-09-21T08:27:00.665976+0000","last_undegraded":"2024-09-21T08:27:00.665976+0000","last_fullsized":"2024-09-21T08:27:00.665976+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":375,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.671335+0000","last_change":"2024-09-21T08:25:48.975855+0000","last_active":"2024-09-21T08:27:00.671335+0000","last_peered":"2024-09-21T08:27:00.671335+0000","last_clean":"2024-09-21T08:27:00.671335+0000","last_became_active":"2024-09-21T08:25:48.897974+0000","last_became_peered":"2024-09-21T08:25:48.897974+0000","last_unstale":"2024-09-21T08:27:00.671335+0000","last_undegraded":"2024-09-21T08:27:00.671335+0000","last_fullsized":"2024-09-21T08:27:00.671335+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802218+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"217'12","reported_seq":423,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668330+0000","last_change":"2024-09-21T08:26:49.575766+0000","last_active":"2024-09-21T08:27:00.668330+0000","last_peered":"2024-09-21T08:27:00.668330+0000","last_clean":"2024-09-21T08:27:00.668330+0000","last_became_active":"2024-09-21T08:26:49.538615+0000","last_became_peered":"2024-09-21T08:26:49.538615+0000","last_unstale":"2024-09-21T08:27:00.668330+0000","last_undegraded":"2024-09-21T08:27:00.668330+0000","last_fullsized":"2024-09-21T08:27:00.668330+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'10","last_scrub_stamp":"2024-09-21T08:25:42.708458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.708458+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":328,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666520+0000","last_change":"2024-09-21T08:24:56.500567+0000","last_active":"2024-09-21T08:27:00.666520+0000","last_peered":"2024-09-21T08:27:00.666520+0000","last_clean":"2024-09-21T08:27:00.666520+0000","last_became_active":"2024-09-21T08:24:56.500287+0000","last_became_peered":"2024-09-21T08:24:56.500287+0000","last_unstale":"2024-09-21T08:27:00.666520+0000","last_undegraded":"2024-09-21T08:27:00.666520+0000","last_fullsized":"2024-09-21T08:27:00.666520+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"180'50","reported_seq":66,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.671141+0000","last_change":"2024-09-21T08:27:00.671414+0000","last_active":"2024-09-21T08:27:00.671141+0000","last_peered":"2024-09-21T08:27:00.671141+0000","last_clean":"2024-09-21T08:27:00.671141+0000","last_became_active":"2024-09-21T08:26:49.555707+0000","last_became_peered":"2024-09-21T08:26:49.555707+0000","last_unstale":"2024-09-21T08:27:00.671141+0000","last_undegraded":"2024-09-21T08:27:00.671141+0000","last_fullsized":"2024-09-21T08:27:00.671141+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:26:50.014780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:50.014780+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:58.621409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00044696100000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"},{"start":"46","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":265,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669607+0000","last_change":"2024-09-21T08:26:49.567839+0000","last_active":"2024-09-21T08:27:00.669607+0000","last_peered":"2024-09-21T08:27:00.669607+0000","last_clean":"2024-09-21T08:27:00.669607+0000","last_became_active":"2024-09-21T08:26:49.567535+0000","last_became_peered":"2024-09-21T08:26:49.567535+0000","last_unstale":"2024-09-21T08:27:00.669607+0000","last_undegraded":"2024-09-21T08:27:00.669607+0000","last_fullsized":"2024-09-21T08:27:00.669607+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":41,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669716+0000","last_change":"2024-09-21T08:26:49.627154+0000","last_active":"2024-09-21T08:27:00.669716+0000","last_peered":"2024-09-21T08:27:00.669716+0000","last_clean":"2024-09-21T08:27:00.669716+0000","last_became_active":"2024-09-21T08:26:49.557128+0000","last_became_peered":"2024-09-21T08:26:49.557128+0000","last_unstale":"2024-09-21T08:27:00.669716+0000","last_undegraded":"2024-09-21T08:27:00.669716+0000","last_fullsized":"2024-09-21T08:27:00.669716+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999172+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":358,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667130+0000","last_change":"2024-09-21T08:26:53.795433+0000","last_active":"2024-09-21T08:27:00.667130+0000","last_peered":"2024-09-21T08:27:00.667130+0000","last_clean":"2024-09-21T08:27:00.667130+0000","last_became_active":"2024-09-21T08:26:49.554703+0000","last_became_peered":"2024-09-21T08:26:49.554703+0000","last_unstale":"2024-09-21T08:27:00.667130+0000","last_undegraded":"2024-09-21T08:27:00.667130+0000","last_fullsized":"2024-09-21T08:27:00.667130+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.795325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:02.067016+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":331,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.671061+0000","last_change":"2024-09-21T08:24:56.506870+0000","last_active":"2024-09-21T08:27:00.671061+0000","last_peered":"2024-09-21T08:27:00.671061+0000","last_clean":"2024-09-21T08:27:00.671061+0000","last_became_active":"2024-09-21T08:24:56.502937+0000","last_became_peered":"2024-09-21T08:24:56.502937+0000","last_unstale":"2024-09-21T08:27:00.671061+0000","last_undegraded":"2024-09-21T08:27:00.671061+0000","last_fullsized":"2024-09-21T08:27:00.671061+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802166+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"227'299","reported_seq":1101,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.682270+0000","last_change":"2024-09-21T08:27:00.682471+0000","last_active":"2024-09-21T08:27:00.682270+0000","last_peered":"2024-09-21T08:27:00.682270+0000","last_clean":"2024-09-21T08:27:00.682270+0000","last_became_active":"2024-09-21T08:26:20.938955+0000","last_became_peered":"2024-09-21T08:26:20.938955+0000","last_unstale":"2024-09-21T08:27:00.682270+0000","last_undegraded":"2024-09-21T08:27:00.682270+0000","last_fullsized":"2024-09-21T08:27:00.682270+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'262","last_scrub_stamp":"2024-09-21T08:26:29.875830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.875830+0000","objects_scrubbed":3,"log_size":299,"log_dups_size":0,"ondisk_log_size":299,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:43.183568+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.012096099000000001,"stat_sum":{"num_bytes":16450395,"num_objects":7,"num_object_clones":3,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":26,"num_read_kb":26013,"num_write":18,"num_write_kb":7606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"},{"start":"46","length":"1"}]},{"pgid":"4.16","version":"73'9","reported_seq":377,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.673698+0000","last_change":"2024-09-21T08:26:49.600026+0000","last_active":"2024-09-21T08:27:00.673698+0000","last_peered":"2024-09-21T08:27:00.673698+0000","last_clean":"2024-09-21T08:27:00.673698+0000","last_became_active":"2024-09-21T08:26:49.564379+0000","last_became_peered":"2024-09-21T08:26:49.564379+0000","last_unstale":"2024-09-21T08:27:00.673698+0000","last_undegraded":"2024-09-21T08:27:00.673698+0000","last_fullsized":"2024-09-21T08:27:00.673698+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244862+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,7],"acting":[0,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":357,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668052+0000","last_change":"2024-09-21T08:26:49.545470+0000","last_active":"2024-09-21T08:27:00.668052+0000","last_peered":"2024-09-21T08:27:00.668052+0000","last_clean":"2024-09-21T08:27:00.668052+0000","last_became_active":"2024-09-21T08:26:49.545227+0000","last_became_peered":"2024-09-21T08:26:49.545227+0000","last_unstale":"2024-09-21T08:27:00.668052+0000","last_undegraded":"2024-09-21T08:27:00.668052+0000","last_fullsized":"2024-09-21T08:27:00.668052+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:46.719545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:46.719545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:05.075220+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669253+0000","last_change":"2024-09-21T08:26:49.558756+0000","last_active":"2024-09-21T08:27:00.669253+0000","last_peered":"2024-09-21T08:27:00.669253+0000","last_clean":"2024-09-21T08:27:00.669253+0000","last_became_active":"2024-09-21T08:26:49.558434+0000","last_became_peered":"2024-09-21T08:26:49.558434+0000","last_unstale":"2024-09-21T08:27:00.669253+0000","last_undegraded":"2024-09-21T08:27:00.669253+0000","last_fullsized":"2024-09-21T08:27:00.669253+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:05.814753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.814753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:17.784870+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,0],"acting":[3,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":335,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.664115+0000","last_change":"2024-09-21T08:24:53.746385+0000","last_active":"2024-09-21T08:27:00.664115+0000","last_peered":"2024-09-21T08:27:00.664115+0000","last_clean":"2024-09-21T08:27:00.664115+0000","last_became_active":"2024-09-21T08:24:38.539845+0000","last_became_peered":"2024-09-21T08:24:38.539845+0000","last_unstale":"2024-09-21T08:27:00.664115+0000","last_undegraded":"2024-09-21T08:27:00.664115+0000","last_fullsized":"2024-09-21T08:27:00.664115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.746295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.746295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"225'112","reported_seq":869,"reported_epoch":227,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:00.673455+0000","last_change":"2024-09-21T08:26:59.538358+0000","last_active":"2024-09-21T08:27:00.673455+0000","last_peered":"2024-09-21T08:27:00.673455+0000","last_clean":"2024-09-21T08:27:00.673455+0000","last_became_active":"2024-09-21T08:26:17.924805+0000","last_became_peered":"2024-09-21T08:26:17.924805+0000","last_unstale":"2024-09-21T08:27:00.673455+0000","last_undegraded":"2024-09-21T08:27:00.673455+0000","last_fullsized":"2024-09-21T08:27:00.673455+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'102","last_scrub_stamp":"2024-09-21T08:26:33.255256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.255256+0000","objects_scrubbed":1,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:58.563703+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.112203122,"stat_sum":{"num_bytes":3346980,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":997,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"42","length":"2"}]},{"pgid":"4.15","version":"73'9","reported_seq":394,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667062+0000","last_change":"2024-09-21T08:26:49.647879+0000","last_active":"2024-09-21T08:27:00.667062+0000","last_peered":"2024-09-21T08:27:00.667062+0000","last_clean":"2024-09-21T08:27:00.667062+0000","last_became_active":"2024-09-21T08:26:49.557951+0000","last_became_peered":"2024-09-21T08:26:49.557951+0000","last_unstale":"2024-09-21T08:27:00.667062+0000","last_undegraded":"2024-09-21T08:27:00.667062+0000","last_fullsized":"2024-09-21T08:27:00.667062+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.675877+0000","last_change":"2024-09-21T08:26:49.566193+0000","last_active":"2024-09-21T08:27:00.675877+0000","last_peered":"2024-09-21T08:27:00.675877+0000","last_clean":"2024-09-21T08:27:00.675877+0000","last_became_active":"2024-09-21T08:26:49.565055+0000","last_became_peered":"2024-09-21T08:26:49.565055+0000","last_unstale":"2024-09-21T08:27:00.675877+0000","last_undegraded":"2024-09-21T08:27:00.675877+0000","last_fullsized":"2024-09-21T08:27:00.675877+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"217'12","reported_seq":40,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670782+0000","last_change":"2024-09-21T08:26:49.668308+0000","last_active":"2024-09-21T08:27:00.670782+0000","last_peered":"2024-09-21T08:27:00.670782+0000","last_clean":"2024-09-21T08:27:00.670782+0000","last_became_active":"2024-09-21T08:26:49.560725+0000","last_became_peered":"2024-09-21T08:26:49.560725+0000","last_unstale":"2024-09-21T08:27:00.670782+0000","last_undegraded":"2024-09-21T08:27:00.670782+0000","last_fullsized":"2024-09-21T08:27:00.670782+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999419+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":305,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.663325+0000","last_change":"2024-09-21T08:25:39.989977+0000","last_active":"2024-09-21T08:27:00.663325+0000","last_peered":"2024-09-21T08:27:00.663325+0000","last_clean":"2024-09-21T08:27:00.663325+0000","last_became_active":"2024-09-21T08:24:38.542854+0000","last_became_peered":"2024-09-21T08:24:38.542854+0000","last_unstale":"2024-09-21T08:27:00.663325+0000","last_undegraded":"2024-09-21T08:27:00.663325+0000","last_fullsized":"2024-09-21T08:27:00.663325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.989838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.989838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"179'63","reported_seq":805,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.679959+0000","last_change":"2024-09-21T08:27:00.680304+0000","last_active":"2024-09-21T08:27:00.679959+0000","last_peered":"2024-09-21T08:27:00.679959+0000","last_clean":"2024-09-21T08:27:00.679959+0000","last_became_active":"2024-09-21T08:26:17.919628+0000","last_became_peered":"2024-09-21T08:26:17.919628+0000","last_unstale":"2024-09-21T08:27:00.679959+0000","last_undegraded":"2024-09-21T08:27:00.679959+0000","last_fullsized":"2024-09-21T08:27:00.679959+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'63","last_scrub_stamp":"2024-09-21T08:26:28.855711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.855711+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:32.670799+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011632251,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"},{"start":"46","length":"1"}]},{"pgid":"4.14","version":"104'15","reported_seq":45,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670430+0000","last_change":"2024-09-21T08:26:49.673980+0000","last_active":"2024-09-21T08:27:00.670430+0000","last_peered":"2024-09-21T08:27:00.670430+0000","last_clean":"2024-09-21T08:27:00.670430+0000","last_became_active":"2024-09-21T08:26:49.562964+0000","last_became_peered":"2024-09-21T08:26:49.562964+0000","last_unstale":"2024-09-21T08:27:00.670430+0000","last_undegraded":"2024-09-21T08:27:00.670430+0000","last_fullsized":"2024-09-21T08:27:00.670430+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999388+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,2],"acting":[3,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":365,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668304+0000","last_change":"2024-09-21T08:24:56.709528+0000","last_active":"2024-09-21T08:27:00.668304+0000","last_peered":"2024-09-21T08:27:00.668304+0000","last_clean":"2024-09-21T08:27:00.668304+0000","last_became_active":"2024-09-21T08:24:56.504665+0000","last_became_peered":"2024-09-21T08:24:56.504665+0000","last_unstale":"2024-09-21T08:27:00.668304+0000","last_undegraded":"2024-09-21T08:27:00.668304+0000","last_fullsized":"2024-09-21T08:27:00.668304+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:56.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.709437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802294+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"217'10","reported_seq":405,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667365+0000","last_change":"2024-09-21T08:26:49.652202+0000","last_active":"2024-09-21T08:27:00.667365+0000","last_peered":"2024-09-21T08:27:00.667365+0000","last_clean":"2024-09-21T08:27:00.667365+0000","last_became_active":"2024-09-21T08:26:49.558669+0000","last_became_peered":"2024-09-21T08:26:49.558669+0000","last_unstale":"2024-09-21T08:27:00.667365+0000","last_undegraded":"2024-09-21T08:27:00.667365+0000","last_fullsized":"2024-09-21T08:27:00.667365+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":341,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.675498+0000","last_change":"2024-09-21T08:26:49.569546+0000","last_active":"2024-09-21T08:27:00.675498+0000","last_peered":"2024-09-21T08:27:00.675498+0000","last_clean":"2024-09-21T08:27:00.675498+0000","last_became_active":"2024-09-21T08:26:49.569251+0000","last_became_peered":"2024-09-21T08:26:49.569251+0000","last_unstale":"2024-09-21T08:27:00.675498+0000","last_undegraded":"2024-09-21T08:27:00.675498+0000","last_fullsized":"2024-09-21T08:27:00.675498+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:48.251366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:48.251366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:53.604250+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,3],"acting":[0,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"222'101","reported_seq":72,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667446+0000","last_change":"2024-09-21T08:27:00.669974+0000","last_active":"2024-09-21T08:27:00.667446+0000","last_peered":"2024-09-21T08:27:00.667446+0000","last_clean":"2024-09-21T08:27:00.667446+0000","last_became_active":"2024-09-21T08:26:49.558443+0000","last_became_peered":"2024-09-21T08:26:49.558443+0000","last_unstale":"2024-09-21T08:27:00.667446+0000","last_undegraded":"2024-09-21T08:27:00.667446+0000","last_fullsized":"2024-09-21T08:27:00.667446+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'94","last_scrub_stamp":"2024-09-21T08:26:27.208234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.208234+0000","objects_scrubbed":1,"log_size":101,"log_dups_size":0,"ondisk_log_size":101,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:51.671091+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027074040000000001,"stat_sum":{"num_bytes":6302658,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":13071,"num_write":5,"num_write_kb":1173,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":4359892,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,7],"acting":[3,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"},{"start":"46","length":"1"}]},{"pgid":"4.13","version":"73'11","reported_seq":208,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666468+0000","last_change":"2024-09-21T08:25:48.979008+0000","last_active":"2024-09-21T08:27:00.666468+0000","last_peered":"2024-09-21T08:27:00.666468+0000","last_clean":"2024-09-21T08:27:00.666468+0000","last_became_active":"2024-09-21T08:25:48.914638+0000","last_became_peered":"2024-09-21T08:25:48.914638+0000","last_unstale":"2024-09-21T08:27:00.666468+0000","last_undegraded":"2024-09-21T08:27:00.666468+0000","last_fullsized":"2024-09-21T08:27:00.666468+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":301,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670030+0000","last_change":"2024-09-21T08:24:56.504037+0000","last_active":"2024-09-21T08:27:00.670030+0000","last_peered":"2024-09-21T08:27:00.670030+0000","last_clean":"2024-09-21T08:27:00.670030+0000","last_became_active":"2024-09-21T08:24:56.503232+0000","last_became_peered":"2024-09-21T08:24:56.503232+0000","last_unstale":"2024-09-21T08:27:00.670030+0000","last_undegraded":"2024-09-21T08:27:00.670030+0000","last_fullsized":"2024-09-21T08:27:00.670030+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802134+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":219,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666443+0000","last_change":"2024-09-21T08:26:49.554839+0000","last_active":"2024-09-21T08:27:00.666443+0000","last_peered":"2024-09-21T08:27:00.666443+0000","last_clean":"2024-09-21T08:27:00.666443+0000","last_became_active":"2024-09-21T08:26:49.554478+0000","last_became_peered":"2024-09-21T08:26:49.554478+0000","last_unstale":"2024-09-21T08:27:00.666443+0000","last_undegraded":"2024-09-21T08:27:00.666443+0000","last_fullsized":"2024-09-21T08:27:00.666443+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667021+0000","last_change":"2024-09-21T08:26:49.565673+0000","last_active":"2024-09-21T08:27:00.667021+0000","last_peered":"2024-09-21T08:27:00.667021+0000","last_clean":"2024-09-21T08:27:00.667021+0000","last_became_active":"2024-09-21T08:26:49.563741+0000","last_became_peered":"2024-09-21T08:26:49.563741+0000","last_unstale":"2024-09-21T08:27:00.667021+0000","last_undegraded":"2024-09-21T08:27:00.667021+0000","last_fullsized":"2024-09-21T08:27:00.667021+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.11","version":"227'68","reported_seq":681,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.679925+0000","last_change":"2024-09-21T08:27:00.680298+0000","last_active":"2024-09-21T08:27:00.679925+0000","last_peered":"2024-09-21T08:27:00.679925+0000","last_clean":"2024-09-21T08:27:00.679925+0000","last_became_active":"2024-09-21T08:26:49.556505+0000","last_became_peered":"2024-09-21T08:26:49.556505+0000","last_unstale":"2024-09-21T08:27:00.679925+0000","last_undegraded":"2024-09-21T08:27:00.679925+0000","last_fullsized":"2024-09-21T08:27:00.679925+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'44","last_scrub_stamp":"2024-09-21T08:26:27.882963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.882963+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:32.704260+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.012749523,"stat_sum":{"num_bytes":10910166,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":4,"num_read_kb":2,"num_write":13,"num_write_kb":6260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":4656509,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,2],"acting":[7,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"39","length":"1"},{"start":"40","length":"1"},{"start":"42","length":"2"},{"start":"46","length":"1"}]},{"pgid":"4.12","version":"73'9","reported_seq":201,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.662453+0000","last_change":"2024-09-21T08:25:48.953827+0000","last_active":"2024-09-21T08:27:00.662453+0000","last_peered":"2024-09-21T08:27:00.662453+0000","last_clean":"2024-09-21T08:27:00.662453+0000","last_became_active":"2024-09-21T08:25:48.910041+0000","last_became_peered":"2024-09-21T08:25:48.910041+0000","last_unstale":"2024-09-21T08:27:00.662453+0000","last_undegraded":"2024-09-21T08:27:00.662453+0000","last_fullsized":"2024-09-21T08:27:00.662453+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":363,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670533+0000","last_change":"2024-09-21T08:26:49.790968+0000","last_active":"2024-09-21T08:27:00.670533+0000","last_peered":"2024-09-21T08:27:00.670533+0000","last_clean":"2024-09-21T08:27:00.670533+0000","last_became_active":"2024-09-21T08:26:49.558025+0000","last_became_peered":"2024-09-21T08:26:49.558025+0000","last_unstale":"2024-09-21T08:27:00.670533+0000","last_undegraded":"2024-09-21T08:27:00.670533+0000","last_fullsized":"2024-09-21T08:27:00.670533+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:49.790900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:49.790900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:59.679681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,0],"acting":[7,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667524+0000","last_change":"2024-09-21T08:26:49.569459+0000","last_active":"2024-09-21T08:27:00.667524+0000","last_peered":"2024-09-21T08:27:00.667524+0000","last_clean":"2024-09-21T08:27:00.667524+0000","last_became_active":"2024-09-21T08:26:49.569171+0000","last_became_peered":"2024-09-21T08:26:49.569171+0000","last_unstale":"2024-09-21T08:27:00.667524+0000","last_undegraded":"2024-09-21T08:27:00.667524+0000","last_fullsized":"2024-09-21T08:27:00.667524+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":333,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670043+0000","last_change":"2024-09-21T08:26:49.566312+0000","last_active":"2024-09-21T08:27:00.670043+0000","last_peered":"2024-09-21T08:27:00.670043+0000","last_clean":"2024-09-21T08:27:00.670043+0000","last_became_active":"2024-09-21T08:26:49.566025+0000","last_became_peered":"2024-09-21T08:26:49.566025+0000","last_unstale":"2024-09-21T08:27:00.670043+0000","last_undegraded":"2024-09-21T08:27:00.670043+0000","last_fullsized":"2024-09-21T08:27:00.670043+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.244967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"226'237","reported_seq":1037,"reported_epoch":227,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:00.675768+0000","last_change":"2024-09-21T08:26:59.538323+0000","last_active":"2024-09-21T08:27:00.675768+0000","last_peered":"2024-09-21T08:27:00.675768+0000","last_clean":"2024-09-21T08:27:00.675768+0000","last_became_active":"2024-09-21T08:26:17.923264+0000","last_became_peered":"2024-09-21T08:26:17.923264+0000","last_unstale":"2024-09-21T08:27:00.675768+0000","last_undegraded":"2024-09-21T08:27:00.675768+0000","last_fullsized":"2024-09-21T08:27:00.675768+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'200","last_scrub_stamp":"2024-09-21T08:26:27.417028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.417028+0000","objects_scrubbed":3,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:50.148673+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.88586830999999999,"stat_sum":{"num_bytes":18839913,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":16,"num_read_kb":10761,"num_write":18,"num_write_kb":10371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"42","length":"2"}]},{"pgid":"4.11","version":"73'11","reported_seq":45,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666833+0000","last_change":"2024-09-21T08:26:49.674466+0000","last_active":"2024-09-21T08:27:00.666833+0000","last_peered":"2024-09-21T08:27:00.666833+0000","last_clean":"2024-09-21T08:27:00.666833+0000","last_became_active":"2024-09-21T08:26:49.566336+0000","last_became_peered":"2024-09-21T08:26:49.566336+0000","last_unstale":"2024-09-21T08:27:00.666833+0000","last_undegraded":"2024-09-21T08:27:00.666833+0000","last_fullsized":"2024-09-21T08:27:00.666833+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999341+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,5],"acting":[3,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":354,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666815+0000","last_change":"2024-09-21T08:26:49.556365+0000","last_active":"2024-09-21T08:27:00.666815+0000","last_peered":"2024-09-21T08:27:00.666815+0000","last_clean":"2024-09-21T08:27:00.666815+0000","last_became_active":"2024-09-21T08:26:49.556039+0000","last_became_peered":"2024-09-21T08:26:49.556039+0000","last_unstale":"2024-09-21T08:27:00.666815+0000","last_undegraded":"2024-09-21T08:27:00.666815+0000","last_fullsized":"2024-09-21T08:27:00.666815+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:12.313653+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":343,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668158+0000","last_change":"2024-09-21T08:24:58.688115+0000","last_active":"2024-09-21T08:27:00.668158+0000","last_peered":"2024-09-21T08:27:00.668158+0000","last_clean":"2024-09-21T08:27:00.668158+0000","last_became_active":"2024-09-21T08:24:56.503123+0000","last_became_peered":"2024-09-21T08:24:56.503123+0000","last_unstale":"2024-09-21T08:27:00.668158+0000","last_undegraded":"2024-09-21T08:27:00.668158+0000","last_fullsized":"2024-09-21T08:27:00.668158+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.688046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.688046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802323+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669357+0000","last_change":"2024-09-21T08:26:49.559001+0000","last_active":"2024-09-21T08:27:00.669357+0000","last_peered":"2024-09-21T08:27:00.669357+0000","last_clean":"2024-09-21T08:27:00.669357+0000","last_became_active":"2024-09-21T08:26:49.558373+0000","last_became_peered":"2024-09-21T08:26:49.558373+0000","last_unstale":"2024-09-21T08:27:00.669357+0000","last_undegraded":"2024-09-21T08:27:00.669357+0000","last_fullsized":"2024-09-21T08:27:00.669357+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999264+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"7.13","version":"197'23","reported_seq":750,"reported_epoch":227,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:00.668051+0000","last_change":"2024-09-21T08:26:57.535167+0000","last_active":"2024-09-21T08:27:00.668051+0000","last_peered":"2024-09-21T08:27:00.668051+0000","last_clean":"2024-09-21T08:27:00.668051+0000","last_became_active":"2024-09-21T08:26:17.925392+0000","last_became_peered":"2024-09-21T08:26:17.925392+0000","last_unstale":"2024-09-21T08:27:00.668051+0000","last_undegraded":"2024-09-21T08:27:00.668051+0000","last_fullsized":"2024-09-21T08:27:00.668051+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'22","last_scrub_stamp":"2024-09-21T08:26:29.353795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.353795+0000","objects_scrubbed":0,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:46.257699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020342910000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"39","length":"1"},{"start":"42","length":"2"}]},{"pgid":"4.10","version":"73'4","reported_seq":39,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668060+0000","last_change":"2024-09-21T08:26:49.676937+0000","last_active":"2024-09-21T08:27:00.668060+0000","last_peered":"2024-09-21T08:27:00.668060+0000","last_clean":"2024-09-21T08:27:00.668060+0000","last_became_active":"2024-09-21T08:26:49.567246+0000","last_became_peered":"2024-09-21T08:26:49.567246+0000","last_unstale":"2024-09-21T08:27:00.668060+0000","last_undegraded":"2024-09-21T08:27:00.668060+0000","last_fullsized":"2024-09-21T08:27:00.668060+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-09-21T08:26:29.821849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.821849+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:34.737752+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,2],"acting":[3,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":357,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.674489+0000","last_change":"2024-09-21T08:26:49.567262+0000","last_active":"2024-09-21T08:27:00.674489+0000","last_peered":"2024-09-21T08:27:00.674489+0000","last_clean":"2024-09-21T08:27:00.674489+0000","last_became_active":"2024-09-21T08:26:49.566755+0000","last_became_peered":"2024-09-21T08:26:49.566755+0000","last_unstale":"2024-09-21T08:27:00.674489+0000","last_undegraded":"2024-09-21T08:27:00.674489+0000","last_fullsized":"2024-09-21T08:27:00.674489+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.245013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,3],"acting":[0,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":308,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.668449+0000","last_change":"2024-09-21T08:26:49.561426+0000","last_active":"2024-09-21T08:27:00.668449+0000","last_peered":"2024-09-21T08:27:00.668449+0000","last_clean":"2024-09-21T08:27:00.668449+0000","last_became_active":"2024-09-21T08:26:49.559825+0000","last_became_peered":"2024-09-21T08:26:49.559825+0000","last_unstale":"2024-09-21T08:27:00.668449+0000","last_undegraded":"2024-09-21T08:27:00.668449+0000","last_fullsized":"2024-09-21T08:27:00.668449+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:00.802263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,3],"acting":[7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":351,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.669412+0000","last_change":"2024-09-21T08:26:49.557187+0000","last_active":"2024-09-21T08:27:00.669412+0000","last_peered":"2024-09-21T08:27:00.669412+0000","last_clean":"2024-09-21T08:27:00.669412+0000","last_became_active":"2024-09-21T08:26:49.556826+0000","last_became_peered":"2024-09-21T08:26:49.556826+0000","last_unstale":"2024-09-21T08:27:00.669412+0000","last_undegraded":"2024-09-21T08:27:00.669412+0000","last_fullsized":"2024-09-21T08:27:00.669412+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.736816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.736816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:59.699398+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,3],"acting":[7,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":195,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.667219+0000","last_change":"2024-09-21T08:26:49.558289+0000","last_active":"2024-09-21T08:27:00.667219+0000","last_peered":"2024-09-21T08:27:00.667219+0000","last_clean":"2024-09-21T08:27:00.667219+0000","last_became_active":"2024-09-21T08:26:49.556944+0000","last_became_peered":"2024-09-21T08:26:49.556944+0000","last_unstale":"2024-09-21T08:27:00.667219+0000","last_undegraded":"2024-09-21T08:27:00.667219+0000","last_fullsized":"2024-09-21T08:27:00.667219+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,3],"acting":[5,2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":33,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.666959+0000","last_change":"2024-09-21T08:26:49.561306+0000","last_active":"2024-09-21T08:27:00.666959+0000","last_peered":"2024-09-21T08:27:00.666959+0000","last_clean":"2024-09-21T08:27:00.666959+0000","last_became_active":"2024-09-21T08:26:49.558481+0000","last_became_peered":"2024-09-21T08:26:49.558481+0000","last_unstale":"2024-09-21T08:27:00.666959+0000","last_undegraded":"2024-09-21T08:27:00.666959+0000","last_fullsized":"2024-09-21T08:27:00.666959+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:02.999372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,2],"acting":[3,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":347,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.670210+0000","last_change":"2024-09-21T08:26:49.629621+0000","last_active":"2024-09-21T08:27:00.670210+0000","last_peered":"2024-09-21T08:27:00.670210+0000","last_clean":"2024-09-21T08:27:00.670210+0000","last_became_active":"2024-09-21T08:26:49.556108+0000","last_became_peered":"2024-09-21T08:26:49.556108+0000","last_unstale":"2024-09-21T08:27:00.670210+0000","last_undegraded":"2024-09-21T08:27:00.670210+0000","last_fullsized":"2024-09-21T08:27:00.670210+0000","mapping_epoch":215,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":216,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3],"acting":[5,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":333,"reported_epoch":227,"state":"active+clean","last_fresh":"2024-09-21T08:27:00.674018+0000","last_change":"2024-09-21T08:25:25.350265+0000","last_active":"2024-09-21T08:27:00.674018+0000","last_peered":"2024-09-21T08:27:00.674018+0000","last_clean":"2024-09-21T08:27:00.674018+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:27:00.674018+0000","last_undegraded":"2024-09-21T08:27:00.674018+0000","last_fullsized":"2024-09-21T08:27:00.674018+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.350197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.350197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:01.245108+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":203131293,"num_objects":93,"num_object_clones":46,"num_object_copies":279,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":93,"num_whiteouts":11,"num_read":774,"num_read_kb":786196,"num_write":569,"num_write_kb":208182,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":174,"num_bytes_recovered":112440553,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":551677952,"data_stored":666593102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3393,"ondisk_log_size":3393,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":39,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":88,"ondisk_log_size":88,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":675,"num_read_kb":432,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":326,"num_bytes_recovered":6524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":655360,"data_stored":17410,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":405,"ondisk_log_size":405,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":3708,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":7675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":202,"num_read_kb":197,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":835,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070511,"num_pgs":105,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":157112,"kb_used_data":117600,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93588040,"statfs":{"total":95995035648,"available":95834152960,"internally_reserved":0,"allocated":120422400,"data_stored":142011342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":40434122},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397557,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659192,"num_pgs":87,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":101412,"kb_used_data":66060,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93643740,"statfs":{"total":95995035648,"available":95891189760,"internally_reserved":0,"allocated":67645440,"data_stored":80086100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986238,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182466,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":105340,"kb_used_data":74144,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93639812,"statfs":{"total":95995035648,"available":95887167488,"internally_reserved":0,"allocated":75923456,"data_stored":86540633,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411400,"num_pgs":89,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":164684,"kb_used_data":125168,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93580468,"statfs":{"total":95995035648,"available":95826399232,"internally_reserved":0,"allocated":128172032,"data_stored":158862695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574924,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738449,"num_pgs":93,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":137516,"kb_used_data":98004,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93607636,"statfs":{"total":95995035648,"available":95854219264,"internally_reserved":0,"allocated":100356096,"data_stored":126305071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":389,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":2354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":57344,"data_stored":1458,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":2090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2602,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":97222656,"data_stored":124358036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45531136,"data_stored":53470237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":125071360,"data_stored":156916204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":73375744,"data_stored":85184354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":65110016,"data_stored":78730227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":117272576,"data_stored":140063629,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:27:03.799 INFO:tasks.thrashosds.thrasher:Setting 7.10 to [5, 3, 7] 2024-09-21T08:27:03.799 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '7.10', '5', '3', '7'] 2024-09-21T08:27:03.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pg-upmap 7.10 5 3 7 2024-09-21T08:27:03.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:03.803 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:03.823 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:03.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:03.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:03.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:03.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:03.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:03.948 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:04.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:04.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:04.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:04.072 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:04.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:04.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:04.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:04.200 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:04.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:04.322 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:04.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:04.323 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:04.333 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:04.333 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:04.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:04.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:04.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:04.447 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:04.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:04.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:04.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:04.572 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:04.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:04.674 INFO:tasks.rados.rados.0.smithi137.stdout:805: done (1 left) 2024-09-21T08:27:04.674 INFO:tasks.rados.rados.0.smithi137.stdout:806: read oid 24 snap -1 2024-09-21T08:27:04.678 INFO:tasks.rados.rados.0.smithi137.stdout:806: expect (ObjNum 192 snap 39 seq_num 192) 2024-09-21T08:27:04.678 INFO:tasks.rados.rados.0.smithi137.stdout:807: snap_create 2024-09-21T08:27:04.678 INFO:tasks.rados.rados.0.smithi137.stdout:804: expect (ObjNum 281 snap 64 seq_num 281) 2024-09-21T08:27:04.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:04.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:04.701 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:04.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:04 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:27:04.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:04 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:27:04.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:04 smithi165 ceph-mon[32893]: osdmap e230: 8 total, 8 up, 5 in 2024-09-21T08:27:04.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:04 smithi165 ceph-mon[32893]: pgmap v251: 158 pgs: 4 peering, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 194 MiB data, 650 MiB used, 446 GiB / 447 GiB avail; 20 MiB/s rd, 4.8 MiB/s wr, 38 op/s 2024-09-21T08:27:04.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:04.823 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:04.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:04.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:04.883 INFO:tasks.rados.rados.0.smithi137.stdout:806: expect (ObjNum 192 snap 39 seq_num 192) 2024-09-21T08:27:04.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:04.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:04 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:27:04.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:04 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:27:04.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:04 smithi137 ceph-mon[29634]: osdmap e230: 8 total, 8 up, 5 in 2024-09-21T08:27:04.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:04 smithi137 ceph-mon[29634]: pgmap v251: 158 pgs: 4 peering, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 194 MiB data, 650 MiB used, 446 GiB / 447 GiB avail; 20 MiB/s rd, 4.8 MiB/s wr, 38 op/s 2024-09-21T08:27:04.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:04 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:27:04.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:04 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:27:04.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:04 smithi137 ceph-mon[41044]: osdmap e230: 8 total, 8 up, 5 in 2024-09-21T08:27:04.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:04 smithi137 ceph-mon[41044]: pgmap v251: 158 pgs: 4 peering, 8 active+clean+snaptrim_wait, 7 active+clean+snaptrim, 139 active+clean; 194 MiB data, 650 MiB used, 446 GiB / 447 GiB avail; 20 MiB/s rd, 4.8 MiB/s wr, 38 op/s 2024-09-21T08:27:04.943 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:04.944 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:04.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:04.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:04.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:04.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T08:27:05.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:05.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:05.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:05.076 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:05.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:05.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:05.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:05.204 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:05.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:05.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:27:05.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:05.326 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:05.326 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:05.427 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:05.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:05.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:05.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:05.526 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:05.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:05.582 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:05.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:05.583 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:05.675 INFO:tasks.rados.rados.0.smithi137.stdout:804: done (2 left) 2024-09-21T08:27:05.675 INFO:tasks.rados.rados.0.smithi137.stdout:806: done (1 left) 2024-09-21T08:27:05.675 INFO:tasks.rados.rados.0.smithi137.stdout:807: done (0 left) 2024-09-21T08:27:05.675 INFO:tasks.rados.rados.0.smithi137.stdout:808: rollback oid 25 current snap is 75 2024-09-21T08:27:05.675 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 25 to 70 2024-09-21T08:27:05.675 INFO:tasks.rados.rados.0.smithi137.stdout:809: rollback oid 19 current snap is 75 2024-09-21T08:27:05.675 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 19 to 69 2024-09-21T08:27:05.676 INFO:tasks.rados.rados.0.smithi137.stdout:810: read oid 39 snap 66 2024-09-21T08:27:05.676 INFO:tasks.rados.rados.0.smithi137.stdout:810: expect deleted 2024-09-21T08:27:05.676 INFO:tasks.rados.rados.0.smithi137.stdout:811: rollback oid 37 current snap is 75 2024-09-21T08:27:05.676 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 37 to 59 2024-09-21T08:27:05.676 INFO:tasks.rados.rados.0.smithi137.stdout:812: read oid 16 snap 73 2024-09-21T08:27:05.676 INFO:tasks.rados.rados.0.smithi137.stdout:812: expect (ObjNum 284 snap 66 seq_num 284) 2024-09-21T08:27:05.676 INFO:tasks.rados.rados.0.smithi137.stdout:813: snap_remove snap 67 2024-09-21T08:27:05.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:05.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:05.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:05.706 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:05.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:05.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:05.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:05.829 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:05.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:05 smithi137 ceph-mon[29634]: osdmap e231: 8 total, 8 up, 5 in 2024-09-21T08:27:05.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:05 smithi137 ceph-mon[29634]: osdmap e232: 8 total, 8 up, 5 in 2024-09-21T08:27:05.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:05 smithi137 ceph-mon[41044]: osdmap e231: 8 total, 8 up, 5 in 2024-09-21T08:27:05.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:05 smithi137 ceph-mon[41044]: osdmap e232: 8 total, 8 up, 5 in 2024-09-21T08:27:05.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:05.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:05.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:05.951 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:05 smithi165 ceph-mon[32893]: osdmap e231: 8 total, 8 up, 5 in 2024-09-21T08:27:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:05 smithi165 ceph-mon[32893]: osdmap e232: 8 total, 8 up, 5 in 2024-09-21T08:27:06.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:06.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:06.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:06.074 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:06.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:06.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:06.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:06.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:06.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:06.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:06.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:06.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:06.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:06.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:06.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:06.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:06.445 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:06.448 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:06.448 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:06.518 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:27:06.518 INFO:teuthology.orchestra.run.smithi137.stderr:set 7.10 pg_upmap mapping to [5,3,7] 2024-09-21T08:27:06.524 INFO:tasks.rados.rados.0.smithi137.stdout:813: done (5 left) 2024-09-21T08:27:06.524 INFO:tasks.rados.rados.0.smithi137.stdout:814: write (excl) oid 10 current snap is 75 2024-09-21T08:27:06.524 INFO:tasks.rados.rados.0.smithi137.stdout:814: seq_num 318 ranges {544626=699655,1892215=554429,3150041=194585} 2024-09-21T08:27:06.531 INFO:tasks.rados.rados.0.smithi137.stdout:814: writing smithi13779221-10 from 544626 to 1244281 tid 1 2024-09-21T08:27:06.538 INFO:tasks.rados.rados.0.smithi137.stdout:814: writing smithi13779221-10 from 1892215 to 2446644 tid 2 2024-09-21T08:27:06.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:06.545 INFO:tasks.rados.rados.0.smithi137.stdout:814: writing smithi13779221-10 from 3150041 to 3344626 tid 3 2024-09-21T08:27:06.545 INFO:tasks.rados.rados.0.smithi137.stdout:815: rollback oid 18 current snap is 75 2024-09-21T08:27:06.545 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 18 to 69 2024-09-21T08:27:06.545 INFO:tasks.rados.rados.0.smithi137.stdout:816: read oid 33 snap -1 2024-09-21T08:27:06.545 INFO:tasks.rados.rados.0.smithi137.stdout:816: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:27:06.546 INFO:tasks.rados.rados.0.smithi137.stdout:817: read oid 4 snap 66 2024-09-21T08:27:06.546 INFO:tasks.rados.rados.0.smithi137.stdout:817: expect (ObjNum 231 snap 47 seq_num 231) 2024-09-21T08:27:06.546 INFO:tasks.rados.rados.0.smithi137.stdout:818: delete oid 42 current snap is 75 2024-09-21T08:27:06.547 INFO:tasks.rados.rados.0.smithi137.stdout:811: finishing rollback tid 0 to smithi13779221-37 2024-09-21T08:27:06.547 INFO:tasks.rados.rados.0.smithi137.stdout:808: finishing rollback tid 1 to smithi13779221-25 2024-09-21T08:27:06.547 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 0 (ObjNum 177 snap 0 seq_num 379559296) dirty dne 2024-09-21T08:27:06.547 INFO:tasks.rados.rados.0.smithi137.stdout:811: finishing rollback tid 1 to smithi13779221-37 2024-09-21T08:27:06.547 INFO:tasks.rados.rados.0.smithi137.stdout:811: finishing rollback tid 2 to smithi13779221-37 2024-09-21T08:27:06.547 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 362 (ObjNum 243 snap 50 seq_num 243) dirty exists 2024-09-21T08:27:06.547 INFO:tasks.rados.rados.0.smithi137.stdout:809: finishing rollback tid 1 to smithi13779221-19 2024-09-21T08:27:06.547 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 100 (ObjNum 4113 snap 0 seq_num 0) dirty dne 2024-09-21T08:27:06.547 INFO:tasks.rados.rados.0.smithi137.stdout:812: expect (ObjNum 284 snap 66 seq_num 284) 2024-09-21T08:27:06.568 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:06.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:06.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:06.640 INFO:tasks.rados.rados.0.smithi137.stdout:814: finishing write tid 1 to smithi13779221-10 2024-09-21T08:27:06.640 INFO:tasks.rados.rados.0.smithi137.stdout:814: finishing write tid 2 to smithi13779221-10 2024-09-21T08:27:06.640 INFO:tasks.rados.rados.0.smithi137.stdout:814: finishing write tid 3 to smithi13779221-10 2024-09-21T08:27:06.640 INFO:tasks.rados.rados.0.smithi137.stdout:814: finishing write tid 4 to smithi13779221-10 2024-09-21T08:27:06.640 INFO:tasks.rados.rados.0.smithi137.stdout:814: finishing write tid 5 to smithi13779221-10 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:814: oid 10 updating version 0 to 225 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:814: oid 10 version 225 is already newer than 223 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:814: oid 10 updating version 225 to 226 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:814: oid 10 version 226 is already newer than 224 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 226 (ObjNum 318 snap 75 seq_num 318) dirty exists 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:814: left oid 10 (ObjNum 318 snap 75 seq_num 318) 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:815: finishing rollback tid 1 to smithi13779221-18 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 0 (ObjNum 129 snap 0 seq_num 3760782800) dirty dne 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:808: done (9 left) 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:809: done (8 left) 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:810: done (7 left) 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:811: done (6 left) 2024-09-21T08:27:06.641 INFO:tasks.rados.rados.0.smithi137.stdout:812: done (5 left) 2024-09-21T08:27:06.642 INFO:tasks.rados.rados.0.smithi137.stdout:814: done (4 left) 2024-09-21T08:27:06.642 INFO:tasks.rados.rados.0.smithi137.stdout:815: done (3 left) 2024-09-21T08:27:06.642 INFO:tasks.rados.rados.0.smithi137.stdout:818: done (2 left) 2024-09-21T08:27:06.642 INFO:tasks.rados.rados.0.smithi137.stdout:819: read oid 27 snap -1 2024-09-21T08:27:06.642 INFO:tasks.rados.rados.0.smithi137.stdout:819: expect deleted 2024-09-21T08:27:06.642 INFO:tasks.rados.rados.0.smithi137.stdout:820: snap_remove snap 73 2024-09-21T08:27:06.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:06.692 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:06.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:06.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:06.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:06.819 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:06.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:06.820 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:06.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/983254885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:06.866 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[29634]: pgmap v254: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 204 MiB data, 686 MiB used, 446 GiB / 447 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 17 op/s; 2.4 MiB/s, 4 objects/s recovering 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1757660118' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.10", "id": [5, 3, 7]}]: dispatch 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/983254885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1757660118' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.10", "id": [5, 3, 7]}]': finished 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[29634]: osdmap e233: 8 total, 8 up, 5 in 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/983254885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[41044]: pgmap v254: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 204 MiB data, 686 MiB used, 446 GiB / 447 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 17 op/s; 2.4 MiB/s, 4 objects/s recovering 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:27:06.867 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:27:06.868 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1757660118' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.10", "id": [5, 3, 7]}]: dispatch 2024-09-21T08:27:06.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:06.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/983254885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:27:06.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1757660118' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.10", "id": [5, 3, 7]}]': finished 2024-09-21T08:27:06.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:06 smithi137 ceph-mon[41044]: osdmap e233: 8 total, 8 up, 5 in 2024-09-21T08:27:06.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:06.943 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:06.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:06.943 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:06 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/983254885' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:06 smithi165 ceph-mon[32893]: pgmap v254: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 204 MiB data, 686 MiB used, 446 GiB / 447 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 17 op/s; 2.4 MiB/s, 4 objects/s recovering 2024-09-21T08:27:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:06 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:27:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:06 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:27:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:06 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1757660118' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.10", "id": [5, 3, 7]}]: dispatch 2024-09-21T08:27:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:06 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:06 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/983254885' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:27:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:06 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1757660118' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.10", "id": [5, 3, 7]}]': finished 2024-09-21T08:27:07.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:06 smithi165 ceph-mon[32893]: osdmap e233: 8 total, 8 up, 5 in 2024-09-21T08:27:07.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:07.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:07.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:07.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:07.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:07.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:07.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:07.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:07.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:07.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:07.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:07.315 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:07.415 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:07.442 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:07.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:07.443 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:07.524 INFO:tasks.rados.rados.0.smithi137.stdout:820: done (3 left) 2024-09-21T08:27:07.524 INFO:tasks.rados.rados.0.smithi137.stdout:821: snap_create 2024-09-21T08:27:07.524 INFO:tasks.rados.rados.0.smithi137.stdout:816: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:27:07.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:07.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:07.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:07.567 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:07.657 INFO:tasks.rados.rados.0.smithi137.stdout:817: expect (ObjNum 231 snap 47 seq_num 231) 2024-09-21T08:27:07.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:07.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:07.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:07.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:07.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:07.816 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:07.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:07.817 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:07.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:07.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:07.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:07.944 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:07.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T08:27:08.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:08.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:08.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:08.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:08.172 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:08.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:08.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:08.198 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:08.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:08.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:08.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:08.324 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:08.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:08.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:08.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:08.451 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:08.536 INFO:tasks.rados.rados.0.smithi137.stdout:816: done (3 left) 2024-09-21T08:27:08.536 INFO:tasks.rados.rados.0.smithi137.stdout:817: done (2 left) 2024-09-21T08:27:08.536 INFO:tasks.rados.rados.0.smithi137.stdout:819: done (1 left) 2024-09-21T08:27:08.537 INFO:tasks.rados.rados.0.smithi137.stdout:821: done (0 left) 2024-09-21T08:27:08.537 INFO:tasks.rados.rados.0.smithi137.stdout:822: write (excl) oid 46 current snap is 76 2024-09-21T08:27:08.537 INFO:tasks.rados.rados.0.smithi137.stdout:822: seq_num 320 ranges {578240=566506,1609179=558098,2804887=551750} 2024-09-21T08:27:08.542 INFO:tasks.rados.rados.0.smithi137.stdout:822: writing smithi13779221-46 from 578240 to 1144746 tid 1 2024-09-21T08:27:08.550 INFO:tasks.rados.rados.0.smithi137.stdout:822: writing smithi13779221-46 from 1609179 to 2167277 tid 2 2024-09-21T08:27:08.552 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:08.558 INFO:tasks.rados.rados.0.smithi137.stdout:822: writing smithi13779221-46 from 2804887 to 3356637 tid 3 2024-09-21T08:27:08.558 INFO:tasks.rados.rados.0.smithi137.stdout:823: copy_from oid 23 from oid 45 current snap is 76 2024-09-21T08:27:08.558 INFO:tasks.rados.rados.0.smithi137.stdout:822: finishing write tid 1 to smithi13779221-46 2024-09-21T08:27:08.558 INFO:tasks.rados.rados.0.smithi137.stdout:822: finishing write tid 2 to smithi13779221-46 2024-09-21T08:27:08.558 INFO:tasks.rados.rados.0.smithi137.stdout:824: snap_remove snap 74 2024-09-21T08:27:08.565 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:08 smithi137 ceph-mon[29634]: 3.14 scrub starts 2024-09-21T08:27:08.565 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:08 smithi137 ceph-mon[29634]: 3.14 scrub ok 2024-09-21T08:27:08.565 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:08 smithi137 ceph-mon[29634]: osdmap e234: 8 total, 8 up, 5 in 2024-09-21T08:27:08.565 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:08 smithi137 ceph-mon[41044]: 3.14 scrub starts 2024-09-21T08:27:08.565 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:08 smithi137 ceph-mon[41044]: 3.14 scrub ok 2024-09-21T08:27:08.565 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:08 smithi137 ceph-mon[41044]: osdmap e234: 8 total, 8 up, 5 in 2024-09-21T08:27:08.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:08.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:08.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:08.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:08.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:08.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:08.703 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:08.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:08.813 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:08.814 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:08.821 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:08.825 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:08.825 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:08.827 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:08.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:08.828 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:08.843 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:08.843 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:08.860 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:08.860 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:08.876 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:08.876 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:08.892 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:08.893 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:08.910 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:08.910 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:08.926 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:08.926 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:08.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:08.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:08.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:08.951 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:08 smithi165 ceph-mon[32893]: 3.14 scrub starts 2024-09-21T08:27:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:08 smithi165 ceph-mon[32893]: 3.14 scrub ok 2024-09-21T08:27:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:08 smithi165 ceph-mon[32893]: osdmap e234: 8 total, 8 up, 5 in 2024-09-21T08:27:09.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:09.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:09.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:09.074 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:09.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:09.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:09.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:09.197 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:09.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:09.318 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:09.319 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:09.319 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:09.420 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:09.441 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:09.442 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:09.442 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:09.533 INFO:tasks.rados.rados.0.smithi137.stdout:824: done (2 left) 2024-09-21T08:27:09.534 INFO:tasks.rados.rados.0.smithi137.stdout:825: rollback oid 10 current snap is 76 2024-09-21T08:27:09.534 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 10 to 66 2024-09-21T08:27:09.534 INFO:tasks.rados.rados.0.smithi137.stdout:826: write (excl) oid 21 current snap is 76 2024-09-21T08:27:09.534 INFO:tasks.rados.rados.0.smithi137.stdout:826: seq_num 321 ranges {595047=641755,1667661=418109,2548663=763022} 2024-09-21T08:27:09.541 INFO:tasks.rados.rados.0.smithi137.stdout:826: writing smithi13779221-21 from 595047 to 1236802 tid 1 2024-09-21T08:27:09.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:09.546 INFO:tasks.rados.rados.0.smithi137.stdout:826: writing smithi13779221-21 from 1667661 to 2085770 tid 2 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:826: writing smithi13779221-21 from 2548663 to 3311685 tid 3 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:827: copy_from oid 15 from oid 22 current snap is 76 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:823: finishing copy_from to smithi13779221-23 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:823: got expected ENOENT (src dne) 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:823: finishing copy_from racing read to smithi13779221-23 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:822: finishing write tid 3 to smithi13779221-46 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:822: finishing write tid 4 to smithi13779221-46 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:822: finishing write tid 5 to smithi13779221-46 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:822: oid 46 updating version 0 to 368 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:822: oid 46 updating version 368 to 369 2024-09-21T08:27:09.554 INFO:tasks.rados.rados.0.smithi137.stdout:822: oid 46 updating version 369 to 371 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:822: oid 46 version 371 is already newer than 370 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 371 (ObjNum 320 snap 76 seq_num 320) dirty exists 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:822: left oid 46 (ObjNum 320 snap 76 seq_num 320) 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:825: finishing rollback tid 0 to smithi13779221-10 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:825: finishing rollback tid 1 to smithi13779221-10 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:825: finishing rollback tid 2 to smithi13779221-10 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 232 (ObjNum 274 snap 64 seq_num 274) dirty exists 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:826: finishing write tid 1 to smithi13779221-21 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:826: finishing write tid 2 to smithi13779221-21 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:822: done (4 left) 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:823: done (3 left) 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:825: done (2 left) 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:828: read oid 28 snap 75 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:828: expect (ObjNum 314 snap 71 seq_num 314) 2024-09-21T08:27:09.555 INFO:tasks.rados.rados.0.smithi137.stdout:829: rollback oid 50 current snap is 76 2024-09-21T08:27:09.556 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 50 to 59 2024-09-21T08:27:09.556 INFO:tasks.rados.rados.0.smithi137.stdout:830: read oid 45 snap -1 2024-09-21T08:27:09.556 INFO:tasks.rados.rados.0.smithi137.stdout:830: expect deleted 2024-09-21T08:27:09.556 INFO:tasks.rados.rados.0.smithi137.stdout:831: write (excl) oid 36 current snap is 76 2024-09-21T08:27:09.556 INFO:tasks.rados.rados.0.smithi137.stdout:831: seq_num 322 ranges {611854=717004,2126143=561767,3259733=152121} 2024-09-21T08:27:09.561 INFO:tasks.rados.rados.0.smithi137.stdout:831: writing smithi13779221-36 from 611854 to 1328858 tid 1 2024-09-21T08:27:09.566 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:09.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:09.567 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:09.568 INFO:tasks.rados.rados.0.smithi137.stdout:831: writing smithi13779221-36 from 2126143 to 2687910 tid 2 2024-09-21T08:27:09.573 INFO:tasks.rados.rados.0.smithi137.stdout:831: writing smithi13779221-36 from 3259733 to 3411854 tid 3 2024-09-21T08:27:09.573 INFO:tasks.rados.rados.0.smithi137.stdout:832: write oid 44 current snap is 76 2024-09-21T08:27:09.573 INFO:tasks.rados.rados.0.smithi137.stdout:832: seq_num 323 ranges {628661=792253,2184625=421778,3403509=25152} 2024-09-21T08:27:09.580 INFO:tasks.rados.rados.0.smithi137.stdout:832: writing smithi13779221-44 from 628661 to 1420914 tid 1 2024-09-21T08:27:09.587 INFO:tasks.rados.rados.0.smithi137.stdout:832: writing smithi13779221-44 from 2184625 to 2606403 tid 2 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:832: writing smithi13779221-44 from 3403509 to 3428661 tid 3 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:833: snap_create 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:829: finishing rollback tid 0 to smithi13779221-50 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:826: finishing write tid 3 to smithi13779221-21 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:826: finishing write tid 4 to smithi13779221-21 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:829: finishing rollback tid 1 to smithi13779221-50 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:829: finishing rollback tid 2 to smithi13779221-50 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 288 (ObjNum 171 snap 34 seq_num 171) dirty exists 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:826: finishing write tid 5 to smithi13779221-21 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:826: oid 21 updating version 0 to 102 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:826: oid 21 updating version 102 to 103 2024-09-21T08:27:09.592 INFO:tasks.rados.rados.0.smithi137.stdout:826: oid 21 version 103 is already newer than 101 2024-09-21T08:27:09.593 INFO:tasks.rados.rados.0.smithi137.stdout:826: oid 21 updating version 103 to 104 2024-09-21T08:27:09.593 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 104 (ObjNum 321 snap 76 seq_num 321) dirty exists 2024-09-21T08:27:09.593 INFO:tasks.rados.rados.0.smithi137.stdout:826: left oid 21 (ObjNum 321 snap 76 seq_num 321) 2024-09-21T08:27:09.593 INFO:tasks.rados.rados.0.smithi137.stdout:828: expect (ObjNum 314 snap 71 seq_num 314) 2024-09-21T08:27:09.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:09 smithi137 ceph-mon[29634]: pgmap v257: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 204 MiB data, 686 MiB used, 446 GiB / 447 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 17 op/s; 2.4 MiB/s, 4 objects/s recovering 2024-09-21T08:27:09.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:09 smithi137 ceph-mon[29634]: 5.8 scrub starts 2024-09-21T08:27:09.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:09 smithi137 ceph-mon[29634]: 5.8 scrub ok 2024-09-21T08:27:09.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:09 smithi137 ceph-mon[29634]: osdmap e235: 8 total, 8 up, 5 in 2024-09-21T08:27:09.663 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:09 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3219936138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:09.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:09 smithi137 ceph-mon[41044]: pgmap v257: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 204 MiB data, 686 MiB used, 446 GiB / 447 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 17 op/s; 2.4 MiB/s, 4 objects/s recovering 2024-09-21T08:27:09.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:09 smithi137 ceph-mon[41044]: 5.8 scrub starts 2024-09-21T08:27:09.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:09 smithi137 ceph-mon[41044]: 5.8 scrub ok 2024-09-21T08:27:09.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:09 smithi137 ceph-mon[41044]: osdmap e235: 8 total, 8 up, 5 in 2024-09-21T08:27:09.663 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:09 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3219936138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:09.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:09.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:09.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:09.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:09.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:09.793 INFO:tasks.rados.rados.0.smithi137.stdout:831: finishing write tid 1 to smithi13779221-36 2024-09-21T08:27:09.793 INFO:tasks.rados.rados.0.smithi137.stdout:827: finishing copy_from racing read to smithi13779221-15 2024-09-21T08:27:09.793 INFO:tasks.rados.rados.0.smithi137.stdout:827: finishing copy_from to smithi13779221-15 2024-09-21T08:27:09.793 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 143 (ObjNum 220 snap 46 seq_num 220) dirty exists 2024-09-21T08:27:09.793 INFO:tasks.rados.rados.0.smithi137.stdout:831: finishing write tid 2 to smithi13779221-36 2024-09-21T08:27:09.793 INFO:tasks.rados.rados.0.smithi137.stdout:831: finishing write tid 3 to smithi13779221-36 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:831: finishing write tid 4 to smithi13779221-36 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:831: finishing write tid 5 to smithi13779221-36 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:831: oid 36 updating version 0 to 378 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:831: oid 36 version 378 is already newer than 377 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:831: oid 36 version 378 is already newer than 376 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:831: oid 36 version 378 is already newer than 375 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 378 (ObjNum 322 snap 76 seq_num 322) dirty exists 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:831: left oid 36 (ObjNum 322 snap 76 seq_num 322) 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:832: finishing write tid 1 to smithi13779221-44 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:832: finishing write tid 2 to smithi13779221-44 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:832: finishing write tid 3 to smithi13779221-44 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:832: finishing write tid 4 to smithi13779221-44 2024-09-21T08:27:09.794 INFO:tasks.rados.rados.0.smithi137.stdout:832: finishing write tid 5 to smithi13779221-44 2024-09-21T08:27:09.795 INFO:tasks.rados.rados.0.smithi137.stdout:832: oid 44 updating version 0 to 240 2024-09-21T08:27:09.795 INFO:tasks.rados.rados.0.smithi137.stdout:832: oid 44 updating version 240 to 241 2024-09-21T08:27:09.795 INFO:tasks.rados.rados.0.smithi137.stdout:832: oid 44 updating version 241 to 242 2024-09-21T08:27:09.795 INFO:tasks.rados.rados.0.smithi137.stdout:832: oid 44 version 242 is already newer than 239 2024-09-21T08:27:09.795 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 242 (ObjNum 323 snap 76 seq_num 323) dirty exists 2024-09-21T08:27:09.795 INFO:tasks.rados.rados.0.smithi137.stdout:832: left oid 44 (ObjNum 323 snap 76 seq_num 323) 2024-09-21T08:27:09.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:09.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:09.814 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:09.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:09.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:09.936 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:09.937 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:09.946 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:09.946 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:27:09.946 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:27:09.946 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:27:09.946 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:27:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:09 smithi165 ceph-mon[32893]: pgmap v257: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 204 MiB data, 686 MiB used, 446 GiB / 447 GiB avail; 17 MiB/s rd, 2.5 MiB/s wr, 17 op/s; 2.4 MiB/s, 4 objects/s recovering 2024-09-21T08:27:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:09 smithi165 ceph-mon[32893]: 5.8 scrub starts 2024-09-21T08:27:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:09 smithi165 ceph-mon[32893]: 5.8 scrub ok 2024-09-21T08:27:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:09 smithi165 ceph-mon[32893]: osdmap e235: 8 total, 8 up, 5 in 2024-09-21T08:27:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:09 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3219936138' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:10.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:10.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:10.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:10.062 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:10.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:10.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:10.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:10.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:10.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:10.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:10.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:10.312 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:10.333 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:27:10.413 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:10.436 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:10.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:10.437 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:10.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:10.544 INFO:tasks.rados.rados.0.smithi137.stdout:826: done (7 left) 2024-09-21T08:27:10.544 INFO:tasks.rados.rados.0.smithi137.stdout:827: done (6 left) 2024-09-21T08:27:10.544 INFO:tasks.rados.rados.0.smithi137.stdout:828: done (5 left) 2024-09-21T08:27:10.544 INFO:tasks.rados.rados.0.smithi137.stdout:829: done (4 left) 2024-09-21T08:27:10.544 INFO:tasks.rados.rados.0.smithi137.stdout:830: done (3 left) 2024-09-21T08:27:10.544 INFO:tasks.rados.rados.0.smithi137.stdout:831: done (2 left) 2024-09-21T08:27:10.544 INFO:tasks.rados.rados.0.smithi137.stdout:832: done (1 left) 2024-09-21T08:27:10.544 INFO:tasks.rados.rados.0.smithi137.stdout:833: done (0 left) 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:834: read oid 1 snap -1 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:834: expect (ObjNum 287 snap 66 seq_num 287) 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:835: read oid 37 snap 59 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:835: expect (ObjNum 243 snap 50 seq_num 243) 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:836: read oid 42 snap -1 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:836: expect deleted 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:837: copy_from oid 50 from oid 29 current snap is 77 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:838: rollback oid 32 current snap is 77 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 32 to 76 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:839: read oid 40 snap -1 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:839: expect (ObjNum 211 snap 43 seq_num 211) 2024-09-21T08:27:10.545 INFO:tasks.rados.rados.0.smithi137.stdout:840: snap_remove snap 69 2024-09-21T08:27:10.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:10.577 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:10.578 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:10.593 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:27:10.593 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:27:10.593 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[29634]: 6.5 scrub starts 2024-09-21T08:27:10.593 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[29634]: 6.5 scrub ok 2024-09-21T08:27:10.593 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[29634]: 3.10 scrub starts 2024-09-21T08:27:10.593 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[29634]: 3.10 scrub ok 2024-09-21T08:27:10.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[29634]: 7.e scrub starts 2024-09-21T08:27:10.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[29634]: 7.e scrub ok 2024-09-21T08:27:10.594 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[29634]: osdmap e236: 8 total, 8 up, 5 in 2024-09-21T08:27:10.594 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:27:10.594 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:27:10.594 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[41044]: 6.5 scrub starts 2024-09-21T08:27:10.594 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[41044]: 6.5 scrub ok 2024-09-21T08:27:10.594 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[41044]: 3.10 scrub starts 2024-09-21T08:27:10.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[41044]: 3.10 scrub ok 2024-09-21T08:27:10.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[41044]: 7.e scrub starts 2024-09-21T08:27:10.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[41044]: 7.e scrub ok 2024-09-21T08:27:10.595 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-mon[41044]: osdmap e236: 8 total, 8 up, 5 in 2024-09-21T08:27:10.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T08:27:10.624 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:10.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:10.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:10.700 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:10.700 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:10.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:10.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:10.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:10.825 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:10.884 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:27:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:27:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:27:10.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:10.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:10.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:10.950 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:10 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:27:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:10 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:27:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:10 smithi165 ceph-mon[32893]: 6.5 scrub starts 2024-09-21T08:27:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:10 smithi165 ceph-mon[32893]: 6.5 scrub ok 2024-09-21T08:27:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:10 smithi165 ceph-mon[32893]: 3.10 scrub starts 2024-09-21T08:27:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:10 smithi165 ceph-mon[32893]: 3.10 scrub ok 2024-09-21T08:27:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:10 smithi165 ceph-mon[32893]: 7.e scrub starts 2024-09-21T08:27:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:10 smithi165 ceph-mon[32893]: 7.e scrub ok 2024-09-21T08:27:11.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:10 smithi165 ceph-mon[32893]: osdmap e236: 8 total, 8 up, 5 in 2024-09-21T08:27:11.051 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:11.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:11.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:11.074 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:11.142 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:11.175 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:11.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:11.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:11.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:11.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:11.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:11.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:11.322 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:11.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:11.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:11.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:11.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:11.520 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:27:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:840: done (6 left) 2024-09-21T08:27:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:841: rollback oid 9 current snap is 77 2024-09-21T08:27:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 9 to 75 2024-09-21T08:27:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:842: rollback oid 2 current snap is 77 2024-09-21T08:27:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 2 to 70 2024-09-21T08:27:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:843: read oid 38 snap -1 2024-09-21T08:27:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:843: expect (ObjNum 281 snap 64 seq_num 281) 2024-09-21T08:27:11.527 INFO:tasks.rados.rados.0.smithi137.stdout:844: write oid 4 current snap is 77 2024-09-21T08:27:11.528 INFO:tasks.rados.rados.0.smithi137.stdout:844: seq_num 324 ranges {645468=467502,1726754=565436,2798226=596838} 2024-09-21T08:27:11.536 INFO:tasks.rados.rados.0.smithi137.stdout:844: writing smithi13779221-4 from 645468 to 1112970 tid 1 2024-09-21T08:27:11.544 INFO:tasks.rados.rados.0.smithi137.stdout:844: writing smithi13779221-4 from 1726754 to 2292190 tid 2 2024-09-21T08:27:11.546 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:844: writing smithi13779221-4 from 2798226 to 3395064 tid 3 2024-09-21T08:27:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:845: read oid 3 snap -1 2024-09-21T08:27:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:845: expect (ObjNum 220 snap 46 seq_num 220) 2024-09-21T08:27:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:846: read oid 19 snap 75 2024-09-21T08:27:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:846: expect deleted 2024-09-21T08:27:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:847: snap_create 2024-09-21T08:27:11.557 INFO:tasks.rados.rados.0.smithi137.stdout:838: finishing rollback tid 1 to smithi13779221-32 2024-09-21T08:27:11.557 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 0 (ObjNum 33 snap 0 seq_num 3731216216) dirty dne 2024-09-21T08:27:11.557 INFO:tasks.rados.rados.0.smithi137.stdout:835: expect (ObjNum 243 snap 50 seq_num 243) 2024-09-21T08:27:11.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:11.569 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:11.629 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:27:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:27:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:27:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:1082 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94837804043520 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:27:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:27:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 153.46718027700001, 2024-09-21T08:27:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.39751146399999998, 2024-09-21T08:27:11.629 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1082 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159528+0000", 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2749999999999998e-06 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159533+0000", 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2480000000000004e-06 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159547+0000", 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4516e-05 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159556+0000", 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9460000000000004e-06 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159617+0000", 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0999000000000003e-05 2024-09-21T08:27:11.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159625+0000", 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240000000000007e-06 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544062+0000", 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.38443652099999998 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544359+0000", 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000297378 2024-09-21T08:27:11.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548773+0000", 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044136440000000004 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548858+0000", 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5638999999999998e-05 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.557037+0000", 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081783740000000004 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:255 7.2 7:4031afd1:::smithi13779221-48:head [copy-from ver 3] snapc 4=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 146.118869883, 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015259864, 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.635 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 255 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507839+0000", 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0089999999999999e-06 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507852+0000", 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3617e-05 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507872+0000", 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9482000000000001e-05 2024-09-21T08:27:11.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507883+0000", 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1036000000000001e-05 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507941+0000", 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7819999999999999e-05 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507946+0000", 2024-09-21T08:27:11.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9080000000000003e-06 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510282+0000", 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023368170000000001 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510376+0000", 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3846999999999998e-05 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.516270+0000", 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005893934 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522776+0000", 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065057179999999997 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522902+0000", 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012641 2024-09-21T08:27:11.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522990+0000", 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7491000000000006e-05 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523050+0000", 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0075999999999999e-05 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523096+0000", 2024-09-21T08:27:11.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5698999999999997e-05 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:256 7.2 7:4031afd1:::smithi13779221-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 146.118802585, 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013379356, 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:11.642 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 256 2024-09-21T08:27:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507904+0000", 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8000000000000005e-07 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507907+0000", 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.258e-06 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507915+0000", 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.571e-06 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507920+0000", 2024-09-21T08:27:11.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1189999999999996e-06 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507998+0000", 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8181000000000002e-05 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.508004+0000", 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3960000000000003e-06 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510529+0000", 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025244989999999999 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511053+0000", 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052380199999999997 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521102+0000", 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010049361999999999 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521183+0000", 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0495999999999996e-05 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521282+0000", 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9891999999999998e-05 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:260 7.8 e93/84 7:1fd29771:::smithi13779221-22:head v 93'20, mlcod=86'17)", 2024-09-21T08:27:11.648 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 145.05775625199999, 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078947310000000003, 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568951+0000", 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1510000000000001e-06 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571221+0000", 2024-09-21T08:27:11.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022708279999999999 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571244+0000", 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2759999999999999e-05 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571257+0000", 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2669e-05 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571601+0000", 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034448900000000001 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571645+0000", 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3498999999999998e-05 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576787+0000", 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051417010000000003 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576844+0000", 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7633999999999997e-05 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:294 7.2 7:4d793d31:::smithi13779221-9:head [copy-from ver 3] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 144.864039456, 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017337353, 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.653 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 294 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762668+0000", 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.353e-06 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762673+0000", 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1569999999999999e-06 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762683+0000", 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0331e-05 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762688+0000", 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1410000000000004e-06 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766904+0000", 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042154899999999997 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767004+0000", 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100135 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.771465+0000", 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044613170000000002 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779379+0000", 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079137649999999997 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779883+0000", 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050370000000000005 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779896+0000", 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3458999999999999e-05 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779950+0000", 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3851000000000003e-05 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.780004+0000", 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3653999999999998e-05 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.658 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:295 7.2 7:4d793d31:::smithi13779221-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 144.864008738, 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012332286, 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.659 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 295 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4400000000000001e-07 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762699+0000", 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0099999999999998e-06 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762705+0000", 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0279999999999997e-06 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762710+0000", 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8210000000000001e-06 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767340+0000", 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046298770000000001 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.768012+0000", 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067169199999999999 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774900+0000", 2024-09-21T08:27:11.662 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068879680000000004 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774961+0000", 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0980999999999998e-05 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.775029+0000", 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8465000000000003e-05 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1908 7.a 7:53537bd3:::smithi13779221-39:head [append 0~0] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.076818778, 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.037489017, 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1908 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.664 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549889+0000", 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.199e-06 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549894+0000", 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9450000000000001e-06 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549902+0000", 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9459999999999998e-06 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549910+0000", 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6510000000000001e-06 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549962+0000", 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2741999999999997e-05 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549970+0000", 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0949999999999996e-06 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.556983+0000", 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.007012673 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557124+0000", 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014058199999999999 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583019+0000", 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025894830000000001 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583102+0000", 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3332999999999996e-05 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583767+0000", 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00066536999999999998 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583809+0000", 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1919999999999998e-05 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583935+0000", 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012586800000000001 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584880+0000", 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00094458400000000005 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585414+0000", 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000533892 2024-09-21T08:27:11.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.586605+0000", 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011918479999999999 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587260+0000", 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065433099999999997 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587376+0000", 2024-09-21T08:27:11.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116208 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1909 7.a 7:53537bd3:::smithi13779221-39:head [rollback 39] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 23.076722814, 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0375938790000001, 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.672 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1909 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549984+0000", 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5799999999999998e-07 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549987+0000", 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5099999999999999e-06 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549993+0000", 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6099999999999997e-06 2024-09-21T08:27:11.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549998+0000", 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4890000000000003e-06 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550039+0000", 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1155000000000002e-05 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550045+0000", 2024-09-21T08:27:11.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7130000000000002e-06 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557614+0000", 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.007568971 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557661+0000", 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6726e-05 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583163+0000", 2024-09-21T08:27:11.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025501894000000001 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583309+0000", 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014584299999999999 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584385+0000", 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010765250000000001 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584430+0000", 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5151000000000002e-05 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584564+0000", 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133652 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585131+0000", 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00056728200000000001 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585577+0000", 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044534199999999999 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587426+0000", 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001849402 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587490+0000", 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4126000000000002e-05 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.679 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587577+0000", 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6630000000000002e-05 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:1376 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94837804043520 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.997819946, 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0087414380000000007, 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.680 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1376 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628889+0000", 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9299999999999999e-06 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628895+0000", 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6189999999999996e-06 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628909+0000", 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3963e-05 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628916+0000", 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.131e-06 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628998+0000", 2024-09-21T08:27:11.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1219999999999998e-05 2024-09-21T08:27:11.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:11.684 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.629004+0000", 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7510000000000002e-06 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.634800+0000", 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005795311 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.634882+0000", 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2025000000000002e-05 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637627+0000", 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027454879999999999 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2114 7.4 e228/215 7:26d6e516:::smithi13779221-33:head v 228'62, mlcod=211'59)", 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.085765424, 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012725531, 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.686 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540942+0000", 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3549999999999999e-06 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543680+0000", 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002738379 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543701+0000", 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1072000000000001e-05 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543712+0000", 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0691e-05 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543786+0000", 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4721999999999994e-05 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543809+0000", 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2731e-05 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.689 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553613+0000", 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098038980000000001 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553666+0000", 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2682999999999997e-05 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2124 7.7 e229/215 7:e55f2143:::smithi13779221-11:head v 229'87, mlcod=224'83)", 2024-09-21T08:27:11.690 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.0939449470000007, 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008804704, 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1900000000000003e-07 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535432+0000", 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026709979999999999 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535449+0000", 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6888000000000002e-05 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535460+0000", 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0947e-05 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535516+0000", 2024-09-21T08:27:11.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5551999999999999e-05 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535538+0000", 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2135000000000001e-05 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.541520+0000", 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059823330000000003 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.541565+0000", 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5331999999999999e-05 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2177 7.2 7:43ab705f:::smithi13779221-10:head [rollback 44] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.0925486979999999, 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0080117160000000003, 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.695 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2177 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534158+0000", 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0580000000000001e-06 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534200+0000", 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1875999999999997e-05 2024-09-21T08:27:11.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534209+0000", 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0899999999999994e-06 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534215+0000", 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9499999999999998e-06 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534288+0000", 2024-09-21T08:27:11.699 INFO:tasks.rados.rados.0.smithi137.stdout:834: expect (ObjNum 287 snap 66 seq_num 287) 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3223e-05 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534294+0000", 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.519e-06 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.539853+0000", 2024-09-21T08:27:11.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055597980000000003 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.539923+0000", 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9115000000000005e-05 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540058+0000", 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013511999999999999 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:11.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540864+0000", 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00080602199999999999 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.541123+0000", 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025911100000000002 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542066+0000", 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000943541 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542123+0000", 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7108999999999998e-05 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542169+0000", 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5184e-05 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2178 7.2 7:43ab705f:::smithi13779221-10:head [append 0~0] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.0923771960000002, 2024-09-21T08:27:11.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081322689999999993, 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2178 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534329+0000", 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8299999999999995e-07 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534351+0000", 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1662999999999999e-05 2024-09-21T08:27:11.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534359+0000", 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9480000000000008e-06 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534364+0000", 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1329999999999998e-06 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534415+0000", 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1199999999999998e-05 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534421+0000", 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1200000000000001e-06 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540407+0000", 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059863010000000003 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540492+0000", 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4715000000000005e-05 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540616+0000", 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012489700000000001 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540903+0000", 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028699500000000001 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.541328+0000", 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042463800000000002 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542392+0000", 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010644199999999999 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542427+0000", 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4338999999999998e-05 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542461+0000", 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3917000000000001e-05 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2184 7.c 7:305aef79:::smithi13779221-15:head [copy-from ver 143] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.0731202839999998, 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022239622000000001, 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.710 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2184 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553588+0000", 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.021e-06 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553596+0000", 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1400000000000002e-06 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553611+0000", 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5478e-05 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553619+0000", 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9019999999999999e-06 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553685+0000", 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5977999999999997e-05 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553801+0000", 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116506 2024-09-21T08:27:11.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.565629+0000", 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011827127999999999 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.574989+0000", 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093609639999999994 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575477+0000", 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048743400000000002 2024-09-21T08:27:11.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575700+0000", 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022307499999999999 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575756+0000", 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6125999999999998e-05 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575825+0000", 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8869999999999996e-05 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.715 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2185 7.c 7:305aef79:::smithi13779221-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.0730666690000001, 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016182583, 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2185 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.716 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553640+0000", 2024-09-21T08:27:11.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.61e-07 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553643+0000", 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1020000000000001e-06 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553653+0000", 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0249e-05 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.718 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553659+0000", 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3319999999999997e-06 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553963+0000", 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000303617 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.554000+0000", 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6976e-05 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569623+0000", 2024-09-21T08:27:11.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015623076 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569721+0000", 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7398000000000005e-05 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569822+0000", 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101072 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2218 7.c 7:361ee62b:::smithi13779221-40:head [sparse-read 0~3130511,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544436+0000", 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.0822695330000001, 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078745629999999994, 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2218 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544436+0000", 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544436+0000", 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544440+0000", 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5590000000000001e-06 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544451+0000", 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1155000000000001e-05 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544473+0000", 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2599000000000002e-05 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544483+0000", 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9690000000000006e-06 2024-09-21T08:27:11.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544678+0000", 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019424500000000001 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544684+0000", 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8020000000000003e-06 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.549521+0000", 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048365730000000003 2024-09-21T08:27:11.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.549639+0000", 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117575 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.552311+0000", 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026720860000000002 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.725 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2219 7.c 7:361ee62b:::smithi13779221-40:head [sparse-read 0~3130511] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.0821327220000001, 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095045249999999998, 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2219 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.726 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544574+0000", 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.297e-06 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544579+0000", 2024-09-21T08:27:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2830000000000003e-06 2024-09-21T08:27:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544588+0000", 2024-09-21T08:27:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4849999999999998e-06 2024-09-21T08:27:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544596+0000", 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6930000000000007e-06 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544738+0000", 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014263700000000001 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544744+0000", 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6030000000000004e-06 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.552342+0000", 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075976259999999997 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.552431+0000", 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9571e-05 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554077+0000", 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016463300000000001 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.731 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2220 7.c 7:361ee62b:::smithi13779221-40:head [read 0~3546277,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 1.0821007460000001, 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011021838000000001, 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2220 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5499999999999998e-07 2024-09-21T08:27:11.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544610+0000", 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4349999999999999e-06 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544617+0000", 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2030000000000003e-06 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544623+0000", 2024-09-21T08:27:11.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0260000000000004e-06 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544805+0000", 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000181635 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544809+0000", 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5950000000000001e-06 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554139+0000", 2024-09-21T08:27:11.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.009329225 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554247+0000", 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010890700000000001 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.555627+0000", 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013792570000000001 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2233 7.8 e238/218 7:17170b1f:::smithi13779221-4:head v 238'212, mlcod=238'208)", 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.070791109000000005, 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018069034000000001, 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555917+0000", 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3089999999999998e-06 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556417+0000", 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050039099999999999 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556437+0000", 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9746e-05 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556446+0000", 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4590000000000004e-06 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556581+0000", 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013473700000000001 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556614+0000", 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3287000000000002e-05 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573937+0000", 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017322422000000001 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573984+0000", 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6683e-05 2024-09-21T08:27:11.740 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2234 7.8 e238/218 7:17170b1f:::smithi13779221-4:head v 238'213, mlcod=238'208)", 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 0.070235444999999994, 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017540864, 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:11.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556471+0000", 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3099999999999997e-07 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556477+0000", 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9680000000000003e-06 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556486+0000", 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5719999999999992e-06 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556492+0000", 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6149999999999996e-06 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.557226+0000", 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073403900000000002 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.557411+0000", 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018489400000000001 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573988+0000", 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016576941000000001 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:11.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574011+0000", 2024-09-21T08:27:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3104000000000001e-05 2024-09-21T08:27:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:11.745 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:11.745 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:27:11.745 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:27:11.766 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:11.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:11.767 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:11.838 INFO:tasks.rados.rados.0.smithi137.stdout:839: expect (ObjNum 211 snap 43 seq_num 211) 2024-09-21T08:27:11.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:11.873 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: 4.2 scrub starts 2024-09-21T08:27:11.873 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: 4.2 scrub ok 2024-09-21T08:27:11.873 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: pgmap v260: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 204 MiB data, 686 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: 4.16 scrub starts 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: 4.16 scrub ok 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: osdmap e237: 8 total, 8 up, 5 in 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: 6.1f scrub starts 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: 6.1f scrub ok 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/198215009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/198215009' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[29634]: osdmap e238: 8 total, 8 up, 5 in 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: 4.2 scrub starts 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: 4.2 scrub ok 2024-09-21T08:27:11.874 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: pgmap v260: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 204 MiB data, 686 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:27:11.875 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: 4.16 scrub starts 2024-09-21T08:27:11.875 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: 4.16 scrub ok 2024-09-21T08:27:11.875 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: osdmap e237: 8 total, 8 up, 5 in 2024-09-21T08:27:11.875 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: 6.1f scrub starts 2024-09-21T08:27:11.875 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: 6.1f scrub ok 2024-09-21T08:27:11.875 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/198215009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:11.875 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:27:11.875 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:27:11.875 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/198215009' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:11.875 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:11 smithi137 ceph-mon[41044]: osdmap e238: 8 total, 8 up, 5 in 2024-09-21T08:27:11.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:11.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:11.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:11.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:11.994 INFO:tasks.rados.rados.0.smithi137.stdout:837: finishing copy_from racing read to smithi13779221-50 2024-09-21T08:27:11.994 INFO:tasks.rados.rados.0.smithi137.stdout:837: finishing copy_from to smithi13779221-50 2024-09-21T08:27:11.994 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 311 (ObjNum 276 snap 64 seq_num 276) dirty exists 2024-09-21T08:27:11.994 INFO:tasks.rados.rados.0.smithi137.stdout:843: expect (ObjNum 281 snap 64 seq_num 281) 2024-09-21T08:27:12.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:27:12.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: 4.2 scrub starts 2024-09-21T08:27:12.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: 4.2 scrub ok 2024-09-21T08:27:12.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: pgmap v260: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 204 MiB data, 686 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:27:12.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: 4.16 scrub starts 2024-09-21T08:27:12.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: 4.16 scrub ok 2024-09-21T08:27:12.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: osdmap e237: 8 total, 8 up, 5 in 2024-09-21T08:27:12.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: 6.1f scrub starts 2024-09-21T08:27:12.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: 6.1f scrub ok 2024-09-21T08:27:12.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/198215009' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:12.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:27:12.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:27:12.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/198215009' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:12.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:11 smithi165 ceph-mon[32893]: osdmap e238: 8 total, 8 up, 5 in 2024-09-21T08:27:12.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:12.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:12.016 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:12.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:12.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:12.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:12.140 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:12.201 INFO:tasks.rados.rados.0.smithi137.stdout:842: finishing rollback tid 0 to smithi13779221-2 2024-09-21T08:27:12.201 INFO:tasks.rados.rados.0.smithi137.stdout:842: finishing rollback tid 1 to smithi13779221-2 2024-09-21T08:27:12.201 INFO:tasks.rados.rados.0.smithi137.stdout:842: finishing rollback tid 2 to smithi13779221-2 2024-09-21T08:27:12.201 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 126 (ObjNum 302 snap 70 seq_num 302) dirty exists 2024-09-21T08:27:12.201 INFO:tasks.rados.rados.0.smithi137.stdout:844: finishing write tid 1 to smithi13779221-4 2024-09-21T08:27:12.201 INFO:tasks.rados.rados.0.smithi137.stdout:841: finishing rollback tid 0 to smithi13779221-9 2024-09-21T08:27:12.201 INFO:tasks.rados.rados.0.smithi137.stdout:841: finishing rollback tid 1 to smithi13779221-9 2024-09-21T08:27:12.201 INFO:tasks.rados.rados.0.smithi137.stdout:841: finishing rollback tid 2 to smithi13779221-9 2024-09-21T08:27:12.201 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 9 v 314 (ObjNum 290 snap 66 seq_num 290) dirty exists 2024-09-21T08:27:12.201 INFO:tasks.rados.rados.0.smithi137.stdout:844: finishing write tid 2 to smithi13779221-4 2024-09-21T08:27:12.202 INFO:tasks.rados.rados.0.smithi137.stdout:845: expect (ObjNum 220 snap 46 seq_num 220) 2024-09-21T08:27:12.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:12.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:12.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:12.268 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:12.369 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:12.391 INFO:tasks.rados.rados.0.smithi137.stdout:844: finishing write tid 3 to smithi13779221-4 2024-09-21T08:27:12.391 INFO:tasks.rados.rados.0.smithi137.stdout:844: finishing write tid 4 to smithi13779221-4 2024-09-21T08:27:12.391 INFO:tasks.rados.rados.0.smithi137.stdout:844: finishing write tid 5 to smithi13779221-4 2024-09-21T08:27:12.392 INFO:tasks.rados.rados.0.smithi137.stdout:844: oid 4 updating version 0 to 255 2024-09-21T08:27:12.392 INFO:tasks.rados.rados.0.smithi137.stdout:844: oid 4 updating version 255 to 257 2024-09-21T08:27:12.392 INFO:tasks.rados.rados.0.smithi137.stdout:844: oid 4 version 257 is already newer than 254 2024-09-21T08:27:12.392 INFO:tasks.rados.rados.0.smithi137.stdout:844: oid 4 version 257 is already newer than 256 2024-09-21T08:27:12.392 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 257 (ObjNum 324 snap 77 seq_num 324) dirty exists 2024-09-21T08:27:12.392 INFO:tasks.rados.rados.0.smithi137.stdout:844: left oid 4 (ObjNum 324 snap 77 seq_num 324) 2024-09-21T08:27:12.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:12.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:12.393 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:12.494 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:12.508 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 3] out_osds: [4, 6, 1] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:27:12.508 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:27:12.509 INFO:tasks.thrashosds.thrasher:Removing osd 3, in_osds are: [0, 2, 5, 7, 3] 2024-09-21T08:27:12.509 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd out 3 2024-09-21T08:27:12.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:12.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:12.521 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:12.556 INFO:tasks.rados.rados.0.smithi137.stdout:834: done (12 left) 2024-09-21T08:27:12.556 INFO:tasks.rados.rados.0.smithi137.stdout:835: done (11 left) 2024-09-21T08:27:12.557 INFO:tasks.rados.rados.0.smithi137.stdout:836: done (10 left) 2024-09-21T08:27:12.557 INFO:tasks.rados.rados.0.smithi137.stdout:837: done (9 left) 2024-09-21T08:27:12.557 INFO:tasks.rados.rados.0.smithi137.stdout:838: done (8 left) 2024-09-21T08:27:12.557 INFO:tasks.rados.rados.0.smithi137.stdout:839: done (7 left) 2024-09-21T08:27:12.557 INFO:tasks.rados.rados.0.smithi137.stdout:841: done (6 left) 2024-09-21T08:27:12.557 INFO:tasks.rados.rados.0.smithi137.stdout:842: done (5 left) 2024-09-21T08:27:12.557 INFO:tasks.rados.rados.0.smithi137.stdout:843: done (4 left) 2024-09-21T08:27:12.557 INFO:tasks.rados.rados.0.smithi137.stdout:844: done (3 left) 2024-09-21T08:27:12.558 INFO:tasks.rados.rados.0.smithi137.stdout:845: done (2 left) 2024-09-21T08:27:12.558 INFO:tasks.rados.rados.0.smithi137.stdout:846: done (1 left) 2024-09-21T08:27:12.558 INFO:tasks.rados.rados.0.smithi137.stdout:847: done (0 left) 2024-09-21T08:27:12.558 INFO:tasks.rados.rados.0.smithi137.stdout:848: read oid 34 snap -1 2024-09-21T08:27:12.558 INFO:tasks.rados.rados.0.smithi137.stdout:848: expect (ObjNum 266 snap 59 seq_num 266) 2024-09-21T08:27:12.558 INFO:tasks.rados.rados.0.smithi137.stdout:849: read oid 35 snap 66 2024-09-21T08:27:12.558 INFO:tasks.rados.rados.0.smithi137.stdout:849: expect (ObjNum 270 snap 60 seq_num 270) 2024-09-21T08:27:12.558 INFO:tasks.rados.rados.0.smithi137.stdout:850: delete oid 24 current snap is 78 2024-09-21T08:27:12.564 INFO:tasks.rados.rados.0.smithi137.stdout:849: expect (ObjNum 270 snap 60 seq_num 270) 2024-09-21T08:27:12.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:12.643 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:12.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:12.644 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:12.721 INFO:tasks.rados.rados.0.smithi137.stdout:848: expect (ObjNum 266 snap 59 seq_num 266) 2024-09-21T08:27:12.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:12.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:12.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:12.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:848: done (2 left) 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:849: done (1 left) 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:850: done (0 left) 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:851: rollback oid 37 current snap is 78 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 37 to 76 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:852: read oid 19 snap 76 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:852: expect deleted 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:853: read oid 34 snap -1 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:853: expect (ObjNum 266 snap 59 seq_num 266) 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:854: copy_from oid 31 from oid 50 current snap is 78 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:855: rollback oid 7 current snap is 78 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 7 to 59 2024-09-21T08:27:12.803 INFO:tasks.rados.rados.0.smithi137.stdout:856: read oid 27 snap -1 2024-09-21T08:27:12.804 INFO:tasks.rados.rados.0.smithi137.stdout:856: expect deleted 2024-09-21T08:27:12.804 INFO:tasks.rados.rados.0.smithi137.stdout:857: read oid 46 snap 70 2024-09-21T08:27:12.804 INFO:tasks.rados.rados.0.smithi137.stdout:857: expect (ObjNum 239 snap 49 seq_num 239) 2024-09-21T08:27:12.804 INFO:tasks.rados.rados.0.smithi137.stdout:858: write oid 26 current snap is 78 2024-09-21T08:27:12.804 INFO:tasks.rados.rados.0.smithi137.stdout:858: seq_num 326 ranges {679082=618000,1843718=569105,2853072=626010} 2024-09-21T08:27:12.810 INFO:tasks.rados.rados.0.smithi137.stdout:858: writing smithi13779221-26 from 679082 to 1297082 tid 1 2024-09-21T08:27:12.816 INFO:tasks.rados.rados.0.smithi137.stdout:858: writing smithi13779221-26 from 1843718 to 2412823 tid 2 2024-09-21T08:27:12.822 INFO:tasks.rados.rados.0.smithi137.stdout:858: writing smithi13779221-26 from 2853072 to 3479082 tid 3 2024-09-21T08:27:12.822 INFO:tasks.rados.rados.0.smithi137.stdout:859: write (excl) oid 8 current snap is 78 2024-09-21T08:27:12.823 INFO:tasks.rados.rados.0.smithi137.stdout:859: seq_num 327 ranges {695889=409602,1618553=712763,2996848=499041} 2024-09-21T08:27:12.828 INFO:tasks.rados.rados.0.smithi137.stdout:859: writing smithi13779221-8 from 695889 to 1105491 tid 1 2024-09-21T08:27:12.835 INFO:tasks.rados.rados.0.smithi137.stdout:859: writing smithi13779221-8 from 1618553 to 2331316 tid 2 2024-09-21T08:27:12.842 INFO:tasks.rados.rados.0.smithi137.stdout:859: writing smithi13779221-8 from 2996848 to 3495889 tid 3 2024-09-21T08:27:12.842 INFO:tasks.rados.rados.0.smithi137.stdout:860: write oid 48 current snap is 78 2024-09-21T08:27:12.842 INFO:tasks.rados.rados.0.smithi137.stdout:860: seq_num 328 ranges {712696=484851,1960682=572774,3307918=204778} 2024-09-21T08:27:12.848 INFO:tasks.rados.rados.0.smithi137.stdout:860: writing smithi13779221-48 from 712696 to 1197547 tid 1 2024-09-21T08:27:12.855 INFO:tasks.rados.rados.0.smithi137.stdout:860: writing smithi13779221-48 from 1960682 to 2533456 tid 2 2024-09-21T08:27:12.861 INFO:tasks.rados.rados.0.smithi137.stdout:860: writing smithi13779221-48 from 3307918 to 3512696 tid 3 2024-09-21T08:27:12.861 INFO:tasks.rados.rados.0.smithi137.stdout:852: done (9 left) 2024-09-21T08:27:12.862 INFO:tasks.rados.rados.0.smithi137.stdout:856: done (8 left) 2024-09-21T08:27:12.862 INFO:tasks.rados.rados.0.smithi137.stdout:861: copy_from oid 40 from oid 1 current snap is 78 2024-09-21T08:27:12.862 INFO:tasks.rados.rados.0.smithi137.stdout:851: finishing rollback tid 0 to smithi13779221-37 2024-09-21T08:27:12.862 INFO:tasks.rados.rados.0.smithi137.stdout:851: finishing rollback tid 1 to smithi13779221-37 2024-09-21T08:27:12.862 INFO:tasks.rados.rados.0.smithi137.stdout:851: finishing rollback tid 2 to smithi13779221-37 2024-09-21T08:27:12.862 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 386 (ObjNum 243 snap 50 seq_num 243) dirty exists 2024-09-21T08:27:12.862 INFO:tasks.rados.rados.0.smithi137.stdout:853: expect (ObjNum 266 snap 59 seq_num 266) 2024-09-21T08:27:12.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:12.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:12.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:12.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:854: finishing copy_from racing read to smithi13779221-31 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:855: finishing rollback tid 0 to smithi13779221-7 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:855: finishing rollback tid 1 to smithi13779221-7 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:851: done (8 left) 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:853: done (7 left) 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:862: read oid 34 snap -1 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:862: expect (ObjNum 266 snap 59 seq_num 266) 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:863: read oid 5 snap -1 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:863: expect deleted 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:864: read oid 12 snap 59 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:864: expect (ObjNum 201 snap 40 seq_num 201) 2024-09-21T08:27:12.942 INFO:tasks.rados.rados.0.smithi137.stdout:865: read oid 25 snap -1 2024-09-21T08:27:12.943 INFO:tasks.rados.rados.0.smithi137.stdout:865: expect deleted 2024-09-21T08:27:12.943 INFO:tasks.rados.rados.0.smithi137.stdout:857: expect (ObjNum 239 snap 49 seq_num 239) 2024-09-21T08:27:12.971 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:12 smithi137 ceph-mon[29634]: 6.15 scrub starts 2024-09-21T08:27:12.971 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:12 smithi137 ceph-mon[29634]: 6.15 scrub ok 2024-09-21T08:27:12.971 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:12 smithi137 ceph-mon[29634]: 4.4 scrub starts 2024-09-21T08:27:12.971 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:12 smithi137 ceph-mon[29634]: 4.4 scrub ok 2024-09-21T08:27:12.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:12 smithi137 ceph-mon[41044]: 6.15 scrub starts 2024-09-21T08:27:12.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:12 smithi137 ceph-mon[41044]: 6.15 scrub ok 2024-09-21T08:27:12.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:12 smithi137 ceph-mon[41044]: 4.4 scrub starts 2024-09-21T08:27:12.972 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:12 smithi137 ceph-mon[41044]: 4.4 scrub ok 2024-09-21T08:27:12.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:12.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T08:27:13.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:13.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:13.012 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:13.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:12 smithi165 ceph-mon[32893]: 6.15 scrub starts 2024-09-21T08:27:13.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:12 smithi165 ceph-mon[32893]: 6.15 scrub ok 2024-09-21T08:27:13.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:12 smithi165 ceph-mon[32893]: 4.4 scrub starts 2024-09-21T08:27:13.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:12 smithi165 ceph-mon[32893]: 4.4 scrub ok 2024-09-21T08:27:13.072 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:13.084 INFO:tasks.rados.rados.0.smithi137.stdout:855: finishing rollback tid 2 to smithi13779221-7 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 390 (ObjNum 263 snap 56 seq_num 263) dirty exists 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:858: finishing write tid 1 to smithi13779221-26 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:854: finishing copy_from to smithi13779221-31 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 311 (ObjNum 276 snap 64 seq_num 276) dirty exists 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:858: finishing write tid 2 to smithi13779221-26 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:854: done (10 left) 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:855: done (9 left) 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:857: done (8 left) 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:866: read oid 47 snap -1 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:866: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:867: write (excl) oid 23 current snap is 78 2024-09-21T08:27:13.085 INFO:tasks.rados.rados.0.smithi137.stdout:858: finishing write tid 3 to smithi13779221-26 2024-09-21T08:27:13.086 INFO:tasks.rados.rados.0.smithi137.stdout:858: finishing write tid 4 to smithi13779221-26 2024-09-21T08:27:13.086 INFO:tasks.rados.rados.0.smithi137.stdout:858: finishing write tid 5 to smithi13779221-26 2024-09-21T08:27:13.086 INFO:tasks.rados.rados.0.smithi137.stdout:858: oid 26 updating version 0 to 281 2024-09-21T08:27:13.086 INFO:tasks.rados.rados.0.smithi137.stdout:858: oid 26 updating version 281 to 283 2024-09-21T08:27:13.086 INFO:tasks.rados.rados.0.smithi137.stdout:858: oid 26 version 283 is already newer than 280 2024-09-21T08:27:13.086 INFO:tasks.rados.rados.0.smithi137.stdout:858: oid 26 version 283 is already newer than 282 2024-09-21T08:27:13.086 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 283 (ObjNum 326 snap 78 seq_num 326) dirty exists 2024-09-21T08:27:13.086 INFO:tasks.rados.rados.0.smithi137.stdout:858: left oid 26 (ObjNum 326 snap 78 seq_num 326) 2024-09-21T08:27:13.086 INFO:tasks.rados.rados.0.smithi137.stdout:859: finishing write tid 1 to smithi13779221-8 2024-09-21T08:27:13.086 INFO:tasks.rados.rados.0.smithi137.stdout:867: seq_num 329 ranges {729503=560100,2019164=716432,3335341=194162} 2024-09-21T08:27:13.092 INFO:tasks.rados.rados.0.smithi137.stdout:867: writing smithi13779221-23 from 729503 to 1289603 tid 1 2024-09-21T08:27:13.100 INFO:tasks.rados.rados.0.smithi137.stdout:867: writing smithi13779221-23 from 2019164 to 2735596 tid 2 2024-09-21T08:27:13.105 INFO:tasks.rados.rados.0.smithi137.stdout:867: writing smithi13779221-23 from 3335341 to 3529503 tid 3 2024-09-21T08:27:13.105 INFO:tasks.rados.rados.0.smithi137.stdout:859: finishing write tid 2 to smithi13779221-8 2024-09-21T08:27:13.105 INFO:tasks.rados.rados.0.smithi137.stdout:858: done (9 left) 2024-09-21T08:27:13.105 INFO:tasks.rados.rados.0.smithi137.stdout:868: write oid 32 current snap is 78 2024-09-21T08:27:13.105 INFO:tasks.rados.rados.0.smithi137.stdout:859: finishing write tid 3 to smithi13779221-8 2024-09-21T08:27:13.105 INFO:tasks.rados.rados.0.smithi137.stdout:859: finishing write tid 4 to smithi13779221-8 2024-09-21T08:27:13.105 INFO:tasks.rados.rados.0.smithi137.stdout:859: finishing write tid 5 to smithi13779221-8 2024-09-21T08:27:13.105 INFO:tasks.rados.rados.0.smithi137.stdout:859: oid 8 updating version 0 to 259 2024-09-21T08:27:13.105 INFO:tasks.rados.rados.0.smithi137.stdout:859: oid 8 updating version 259 to 260 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:859: oid 8 version 260 is already newer than 258 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:859: oid 8 updating version 260 to 261 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 261 (ObjNum 327 snap 78 seq_num 327) dirty exists 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:859: left oid 8 (ObjNum 327 snap 78 seq_num 327) 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:860: finishing write tid 1 to smithi13779221-48 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:860: finishing write tid 2 to smithi13779221-48 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:860: finishing write tid 3 to smithi13779221-48 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:860: finishing write tid 4 to smithi13779221-48 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:860: finishing write tid 5 to smithi13779221-48 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:860: oid 48 updating version 0 to 239 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:860: oid 48 updating version 239 to 240 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:860: oid 48 version 240 is already newer than 237 2024-09-21T08:27:13.106 INFO:tasks.rados.rados.0.smithi137.stdout:860: oid 48 version 240 is already newer than 238 2024-09-21T08:27:13.107 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 240 (ObjNum 328 snap 78 seq_num 328) dirty exists 2024-09-21T08:27:13.107 INFO:tasks.rados.rados.0.smithi137.stdout:860: left oid 48 (ObjNum 328 snap 78 seq_num 328) 2024-09-21T08:27:13.107 INFO:tasks.rados.rados.0.smithi137.stdout:861: finishing copy_from racing read to smithi13779221-40 2024-09-21T08:27:13.107 INFO:tasks.rados.rados.0.smithi137.stdout:861: finishing copy_from to smithi13779221-40 2024-09-21T08:27:13.107 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 252 (ObjNum 287 snap 66 seq_num 287) dirty exists 2024-09-21T08:27:13.107 INFO:tasks.rados.rados.0.smithi137.stdout:864: expect (ObjNum 201 snap 40 seq_num 201) 2024-09-21T08:27:13.112 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:13.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:13.133 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:13.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:13.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:13.217 INFO:tasks.rados.rados.0.smithi137.stdout:862: expect (ObjNum 266 snap 59 seq_num 266) 2024-09-21T08:27:13.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:13.258 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:13.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:13.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:13.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:13.296 INFO:tasks.rados.rados.0.smithi137.stdout:868: seq_num 330 ranges {746310=635349,2077646=576443,3362764=183546} 2024-09-21T08:27:13.304 INFO:tasks.rados.rados.0.smithi137.stdout:868: writing smithi13779221-32 from 746310 to 1381659 tid 1 2024-09-21T08:27:13.311 INFO:tasks.rados.rados.0.smithi137.stdout:868: writing smithi13779221-32 from 2077646 to 2654089 tid 2 2024-09-21T08:27:13.316 INFO:tasks.rados.rados.0.smithi137.stdout:868: writing smithi13779221-32 from 3362764 to 3546310 tid 3 2024-09-21T08:27:13.316 INFO:tasks.rados.rados.0.smithi137.stdout:859: done (9 left) 2024-09-21T08:27:13.316 INFO:tasks.rados.rados.0.smithi137.stdout:860: done (8 left) 2024-09-21T08:27:13.317 INFO:tasks.rados.rados.0.smithi137.stdout:861: done (7 left) 2024-09-21T08:27:13.317 INFO:tasks.rados.rados.0.smithi137.stdout:862: done (6 left) 2024-09-21T08:27:13.317 INFO:tasks.rados.rados.0.smithi137.stdout:863: done (5 left) 2024-09-21T08:27:13.317 INFO:tasks.rados.rados.0.smithi137.stdout:864: done (4 left) 2024-09-21T08:27:13.317 INFO:tasks.rados.rados.0.smithi137.stdout:865: done (3 left) 2024-09-21T08:27:13.317 INFO:tasks.rados.rados.0.smithi137.stdout:869: copy_from oid 10 from oid 39 current snap is 78 2024-09-21T08:27:13.317 INFO:tasks.rados.rados.0.smithi137.stdout:866: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:27:13.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:13.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:13.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:13.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:13.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:13.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:13.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:13.506 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:867: finishing write tid 1 to smithi13779221-23 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:867: finishing write tid 2 to smithi13779221-23 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:866: done (3 left) 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:870: copy_from oid 36 from oid 43 current snap is 78 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:867: finishing write tid 3 to smithi13779221-23 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:867: finishing write tid 4 to smithi13779221-23 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:867: finishing write tid 5 to smithi13779221-23 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:867: oid 23 updating version 0 to 172 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:867: oid 23 version 172 is already newer than 171 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:867: oid 23 updating version 172 to 173 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:867: oid 23 updating version 173 to 174 2024-09-21T08:27:13.532 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 174 (ObjNum 329 snap 78 seq_num 329) dirty exists 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:867: left oid 23 (ObjNum 329 snap 78 seq_num 329) 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:868: finishing write tid 1 to smithi13779221-32 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:868: finishing write tid 2 to smithi13779221-32 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:869: finishing copy_from to smithi13779221-10 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:869: got expected ENOENT (src dne) 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:869: finishing copy_from racing read to smithi13779221-10 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:868: finishing write tid 3 to smithi13779221-32 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:868: finishing write tid 4 to smithi13779221-32 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:868: finishing write tid 5 to smithi13779221-32 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:868: oid 32 updating version 0 to 196 2024-09-21T08:27:13.533 INFO:tasks.rados.rados.0.smithi137.stdout:868: oid 32 version 196 is already newer than 195 2024-09-21T08:27:13.534 INFO:tasks.rados.rados.0.smithi137.stdout:868: oid 32 updating version 196 to 198 2024-09-21T08:27:13.534 INFO:tasks.rados.rados.0.smithi137.stdout:868: oid 32 version 198 is already newer than 197 2024-09-21T08:27:13.534 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 198 (ObjNum 330 snap 78 seq_num 330) dirty exists 2024-09-21T08:27:13.534 INFO:tasks.rados.rados.0.smithi137.stdout:868: left oid 32 (ObjNum 330 snap 78 seq_num 330) 2024-09-21T08:27:13.534 INFO:tasks.rados.rados.0.smithi137.stdout:867: done (3 left) 2024-09-21T08:27:13.534 INFO:tasks.rados.rados.0.smithi137.stdout:868: done (2 left) 2024-09-21T08:27:13.534 INFO:tasks.rados.rados.0.smithi137.stdout:869: done (1 left) 2024-09-21T08:27:13.534 INFO:tasks.rados.rados.0.smithi137.stdout:871: snap_remove snap 66 2024-09-21T08:27:13.572 INFO:tasks.rados.rados.0.smithi137.stdout:871: done (1 left) 2024-09-21T08:27:13.573 INFO:tasks.rados.rados.0.smithi137.stdout:872: read oid 19 snap -1 2024-09-21T08:27:13.573 INFO:tasks.rados.rados.0.smithi137.stdout:872: expect deleted 2024-09-21T08:27:13.573 INFO:tasks.rados.rados.0.smithi137.stdout:873: copy_from oid 31 from oid 1 current snap is 78 2024-09-21T08:27:13.573 INFO:tasks.rados.rados.0.smithi137.stdout:870: finishing copy_from racing read to smithi13779221-36 2024-09-21T08:27:13.573 INFO:tasks.rados.rados.0.smithi137.stdout:870: finishing copy_from to smithi13779221-36 2024-09-21T08:27:13.573 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 279 (ObjNum 156 snap 31 seq_num 156) dirty exists 2024-09-21T08:27:13.573 INFO:tasks.rados.rados.0.smithi137.stdout:870: done (2 left) 2024-09-21T08:27:13.573 INFO:tasks.rados.rados.0.smithi137.stdout:874: rollback oid 36 current snap is 78 2024-09-21T08:27:13.574 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 36 to 70 2024-09-21T08:27:13.574 INFO:tasks.rados.rados.0.smithi137.stdout:875: read oid 12 snap -1 2024-09-21T08:27:13.574 INFO:tasks.rados.rados.0.smithi137.stdout:875: expect (ObjNum 315 snap 72 seq_num 315) 2024-09-21T08:27:13.574 INFO:tasks.rados.rados.0.smithi137.stdout:876: snap_create 2024-09-21T08:27:13.580 INFO:tasks.rados.rados.0.smithi137.stdout:875: expect (ObjNum 315 snap 72 seq_num 315) 2024-09-21T08:27:13.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:13.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:13.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:13.633 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:13.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:13 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:13.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:13 smithi165 ceph-mon[32893]: 4.13 scrub starts 2024-09-21T08:27:13.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:13 smithi165 ceph-mon[32893]: 4.13 scrub ok 2024-09-21T08:27:13.653 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:13 smithi165 ceph-mon[32893]: pgmap v263: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 223 MiB data, 713 MiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 21 op/s; 1.0 MiB/s, 1 objects/s recovering 2024-09-21T08:27:13.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:13 smithi165 ceph-mon[32893]: osdmap e239: 8 total, 8 up, 5 in 2024-09-21T08:27:13.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:13 smithi165 ceph-mon[32893]: 6.c deep-scrub starts 2024-09-21T08:27:13.654 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:13 smithi165 ceph-mon[32893]: 6.c deep-scrub ok 2024-09-21T08:27:13.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:13.735 INFO:tasks.rados.rados.0.smithi137.stdout:873: finishing copy_from racing read to smithi13779221-31 2024-09-21T08:27:13.735 INFO:tasks.rados.rados.0.smithi137.stdout:874: finishing rollback tid 0 to smithi13779221-36 2024-09-21T08:27:13.735 INFO:tasks.rados.rados.0.smithi137.stdout:874: finishing rollback tid 1 to smithi13779221-36 2024-09-21T08:27:13.735 INFO:tasks.rados.rados.0.smithi137.stdout:874: finishing rollback tid 2 to smithi13779221-36 2024-09-21T08:27:13.735 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 395 (ObjNum 181 snap 36 seq_num 181) dirty exists 2024-09-21T08:27:13.736 INFO:tasks.rados.rados.0.smithi137.stdout:873: finishing copy_from to smithi13779221-31 2024-09-21T08:27:13.736 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 252 (ObjNum 287 snap 66 seq_num 287) dirty exists 2024-09-21T08:27:13.753 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:13.753 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:13.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:13.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:13.757 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:13.758 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:27:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:27:13.758 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:27:13.758 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:27:13.770 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:13.770 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:13.786 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:13.786 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:13.801 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:13.802 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:13.802 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:13.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[29634]: 4.13 scrub starts 2024-09-21T08:27:13.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[29634]: 4.13 scrub ok 2024-09-21T08:27:13.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[29634]: pgmap v263: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 223 MiB data, 713 MiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 21 op/s; 1.0 MiB/s, 1 objects/s recovering 2024-09-21T08:27:13.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[29634]: osdmap e239: 8 total, 8 up, 5 in 2024-09-21T08:27:13.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[29634]: 6.c deep-scrub starts 2024-09-21T08:27:13.803 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[29634]: 6.c deep-scrub ok 2024-09-21T08:27:13.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:13.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[41044]: 4.13 scrub starts 2024-09-21T08:27:13.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[41044]: 4.13 scrub ok 2024-09-21T08:27:13.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[41044]: pgmap v263: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 223 MiB data, 713 MiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 21 op/s; 1.0 MiB/s, 1 objects/s recovering 2024-09-21T08:27:13.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[41044]: osdmap e239: 8 total, 8 up, 5 in 2024-09-21T08:27:13.804 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[41044]: 6.c deep-scrub starts 2024-09-21T08:27:13.805 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:13 smithi137 ceph-mon[41044]: 6.c deep-scrub ok 2024-09-21T08:27:13.818 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:13.819 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:13.835 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:13.835 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:13.851 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:13.852 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:13.858 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:13.868 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:13.868 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:13.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:13.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:13.879 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:13.980 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:14.001 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:14.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:14.002 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:14.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:14.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:14.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:14.124 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:14.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:14.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:14.247 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:14.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:27:14.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:14.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:14.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:14.395 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:14.474 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29611]: 2024-09-21T08:27:14.125+0000 7f1058bb7640 -1 mon.a@0(leader).osd e240 definitely_dead 0 2024-09-21T08:27:14.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:14.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:14.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:14.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:14.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T08:27:14.567 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.3. 2024-09-21T08:27:14.572 INFO:tasks.rados.rados.0.smithi137.stdout:872: done (4 left) 2024-09-21T08:27:14.572 INFO:tasks.rados.rados.0.smithi137.stdout:873: done (3 left) 2024-09-21T08:27:14.572 INFO:tasks.rados.rados.0.smithi137.stdout:874: done (2 left) 2024-09-21T08:27:14.572 INFO:tasks.rados.rados.0.smithi137.stdout:875: done (1 left) 2024-09-21T08:27:14.572 INFO:tasks.rados.rados.0.smithi137.stdout:876: done (0 left) 2024-09-21T08:27:14.572 INFO:tasks.rados.rados.0.smithi137.stdout:877: rollback oid 15 current snap is 79 2024-09-21T08:27:14.572 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 15 to 70 2024-09-21T08:27:14.572 INFO:tasks.rados.rados.0.smithi137.stdout:878: delete oid 24 current snap is 79 2024-09-21T08:27:14.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:14.621 INFO:tasks.rados.rados.0.smithi137.stdout:878: done (1 left) 2024-09-21T08:27:14.621 INFO:tasks.rados.rados.0.smithi137.stdout:879: write oid 21 current snap is 79 2024-09-21T08:27:14.621 INFO:tasks.rados.rados.0.smithi137.stdout:879: seq_num 332 ranges {779924=785847,2078257=580112,3301257=278667} 2024-09-21T08:27:14.629 INFO:tasks.rados.rados.0.smithi137.stdout:879: writing smithi13779221-21 from 779924 to 1565771 tid 1 2024-09-21T08:27:14.635 INFO:tasks.rados.rados.0.smithi137.stdout:879: writing smithi13779221-21 from 2078257 to 2658369 tid 2 2024-09-21T08:27:14.640 INFO:tasks.rados.rados.0.smithi137.stdout:879: writing smithi13779221-21 from 3301257 to 3579924 tid 3 2024-09-21T08:27:14.641 INFO:tasks.rados.rados.0.smithi137.stdout:880: read oid 23 snap -1 2024-09-21T08:27:14.641 INFO:tasks.rados.rados.0.smithi137.stdout:880: expect (ObjNum 329 snap 78 seq_num 329) 2024-09-21T08:27:14.641 INFO:tasks.rados.rados.0.smithi137.stdout:881: write (excl) oid 41 current snap is 79 2024-09-21T08:27:14.641 INFO:tasks.rados.rados.0.smithi137.stdout:881: seq_num 333 ranges {796731=461096,1736739=723770,2928680=668051} 2024-09-21T08:27:14.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:14.642 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:14.642 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:14.647 INFO:tasks.rados.rados.0.smithi137.stdout:881: writing smithi13779221-41 from 796731 to 1257827 tid 1 2024-09-21T08:27:14.654 INFO:tasks.rados.rados.0.smithi137.stdout:881: writing smithi13779221-41 from 1736739 to 2460509 tid 2 2024-09-21T08:27:14.660 INFO:tasks.rados.rados.0.smithi137.stdout:881: writing smithi13779221-41 from 2928680 to 3596731 tid 3 2024-09-21T08:27:14.660 INFO:tasks.rados.rados.0.smithi137.stdout:882: snap_remove snap 75 2024-09-21T08:27:14.743 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:14.765 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:14.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:14.766 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:14.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[29634]: 4.1 deep-scrub starts 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[29634]: 4.1 deep-scrub ok 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[29634]: osdmap e240: 8 total, 8 up, 5 in 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/830653345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[29634]: 5.13 scrub starts 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[29634]: 5.13 scrub ok 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1637334343' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[29634]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[41044]: 4.1 deep-scrub starts 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[41044]: 4.1 deep-scrub ok 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[41044]: osdmap e240: 8 total, 8 up, 5 in 2024-09-21T08:27:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/830653345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[41044]: 5.13 scrub starts 2024-09-21T08:27:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[41044]: 5.13 scrub ok 2024-09-21T08:27:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1637334343' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-21T08:27:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:14 smithi137 ceph-mon[41044]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T08:27:14.774 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:14.805 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:14.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:14.888 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:14.889 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:14.900 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:14 smithi165 ceph-mon[32893]: 4.1 deep-scrub starts 2024-09-21T08:27:14.900 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:14 smithi165 ceph-mon[32893]: 4.1 deep-scrub ok 2024-09-21T08:27:14.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:14 smithi165 ceph-mon[32893]: osdmap e240: 8 total, 8 up, 5 in 2024-09-21T08:27:14.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:14 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/830653345' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:14.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:14 smithi165 ceph-mon[32893]: 5.13 scrub starts 2024-09-21T08:27:14.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:14 smithi165 ceph-mon[32893]: 5.13 scrub ok 2024-09-21T08:27:14.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:14 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1637334343' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["3"]}]: dispatch 2024-09-21T08:27:14.901 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:14 smithi165 ceph-mon[32893]: Client client.admin marked osd.3 out, while it was still marked up 2024-09-21T08:27:14.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:15.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:15.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:15.012 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:15.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:15.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:15.136 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:15.136 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:15.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:15.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:15.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:15.259 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:15.261 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:15.261 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:15.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:15.381 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:15.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:15.382 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:15.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:15.505 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:15.506 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:15.506 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:15.580 INFO:tasks.rados.rados.0.smithi137.stdout:882: done (4 left) 2024-09-21T08:27:15.581 INFO:tasks.rados.rados.0.smithi137.stdout:883: read oid 28 snap 70 2024-09-21T08:27:15.581 INFO:tasks.rados.rados.0.smithi137.stdout:883: expect (ObjNum 310 snap 70 seq_num 310) 2024-09-21T08:27:15.581 INFO:tasks.rados.rados.0.smithi137.stdout:884: read oid 33 snap -1 2024-09-21T08:27:15.581 INFO:tasks.rados.rados.0.smithi137.stdout:884: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:27:15.581 INFO:tasks.rados.rados.0.smithi137.stdout:885: delete oid 45 current snap is 79 2024-09-21T08:27:15.581 INFO:tasks.rados.rados.0.smithi137.stdout:880: expect (ObjNum 329 snap 78 seq_num 329) 2024-09-21T08:27:15.607 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:15.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:15.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:15.637 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:15.725 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[29634]: 6.1d scrub starts 2024-09-21T08:27:15.725 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[29634]: 6.1d scrub ok 2024-09-21T08:27:15.725 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[29634]: pgmap v266: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 223 MiB data, 713 MiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 21 op/s; 1.0 MiB/s, 1 objects/s recovering 2024-09-21T08:27:15.725 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[29634]: 4.12 scrub starts 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[29634]: 4.12 scrub ok 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1637334343' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[29634]: osdmap e241: 8 total, 8 up, 4 in 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[41044]: 6.1d scrub starts 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[41044]: 6.1d scrub ok 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[41044]: pgmap v266: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 223 MiB data, 713 MiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 21 op/s; 1.0 MiB/s, 1 objects/s recovering 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[41044]: 4.12 scrub starts 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[41044]: 4.12 scrub ok 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1637334343' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T08:27:15.726 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[41044]: osdmap e241: 8 total, 8 up, 4 in 2024-09-21T08:27:15.727 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:15.727 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:15 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:15.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:15.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:15.762 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:15.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:15.834 INFO:tasks.rados.rados.0.smithi137.stdout:881: finishing write tid 1 to smithi13779221-41 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:881: finishing write tid 2 to smithi13779221-41 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:881: finishing write tid 3 to smithi13779221-41 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:881: finishing write tid 4 to smithi13779221-41 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:881: finishing write tid 5 to smithi13779221-41 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:881: oid 41 updating version 0 to 284 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:881: oid 41 updating version 284 to 287 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:881: oid 41 version 287 is already newer than 286 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:881: oid 41 version 287 is already newer than 285 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 287 (ObjNum 333 snap 79 seq_num 333) dirty exists 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:881: left oid 41 (ObjNum 333 snap 79 seq_num 333) 2024-09-21T08:27:15.835 INFO:tasks.rados.rados.0.smithi137.stdout:884: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:27:15.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:15.884 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:15.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:15.885 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:15.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:15 smithi165 ceph-mon[32893]: 6.1d scrub starts 2024-09-21T08:27:15.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:15 smithi165 ceph-mon[32893]: 6.1d scrub ok 2024-09-21T08:27:15.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:15 smithi165 ceph-mon[32893]: pgmap v266: 158 pgs: 8 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 146 active+clean; 223 MiB data, 713 MiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 3.3 MiB/s wr, 21 op/s; 1.0 MiB/s, 1 objects/s recovering 2024-09-21T08:27:15.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:15 smithi165 ceph-mon[32893]: 4.12 scrub starts 2024-09-21T08:27:15.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:15 smithi165 ceph-mon[32893]: 4.12 scrub ok 2024-09-21T08:27:15.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:15 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1637334343' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["3"]}]': finished 2024-09-21T08:27:15.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:15 smithi165 ceph-mon[32893]: osdmap e241: 8 total, 8 up, 4 in 2024-09-21T08:27:15.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:15 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:15.933 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:15 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:15.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:879: finishing write tid 1 to smithi13779221-21 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:879: finishing write tid 2 to smithi13779221-21 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:879: finishing write tid 3 to smithi13779221-21 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:879: finishing write tid 4 to smithi13779221-21 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:879: finishing write tid 5 to smithi13779221-21 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:879: oid 21 updating version 0 to 107 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:879: oid 21 version 107 is already newer than 105 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:879: oid 21 updating version 107 to 108 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:879: oid 21 version 108 is already newer than 106 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 108 (ObjNum 332 snap 79 seq_num 332) dirty exists 2024-09-21T08:27:15.963 INFO:tasks.rados.rados.0.smithi137.stdout:879: left oid 21 (ObjNum 332 snap 79 seq_num 332) 2024-09-21T08:27:15.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:16.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:16.007 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:16.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:16.108 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:16.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:16.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:16.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:16.206 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:16.232 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:16.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:16.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:16.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:16.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:16.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:16.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:16.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:16.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:16.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:16.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:16.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:16.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:27:16.579 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:27:16.596 INFO:tasks.rados.rados.0.smithi137.stdout:879: done (6 left) 2024-09-21T08:27:16.597 INFO:tasks.rados.rados.0.smithi137.stdout:880: done (5 left) 2024-09-21T08:27:16.597 INFO:tasks.rados.rados.0.smithi137.stdout:881: done (4 left) 2024-09-21T08:27:16.597 INFO:tasks.rados.rados.0.smithi137.stdout:884: done (3 left) 2024-09-21T08:27:16.597 INFO:tasks.rados.rados.0.smithi137.stdout:885: done (2 left) 2024-09-21T08:27:16.597 INFO:tasks.rados.rados.0.smithi137.stdout:886: copy_from oid 33 from oid 2 current snap is 79 2024-09-21T08:27:16.597 INFO:tasks.rados.rados.0.smithi137.stdout:887: write (excl) oid 32 current snap is 79 2024-09-21T08:27:16.597 INFO:tasks.rados.rados.0.smithi137.stdout:887: seq_num 335 ranges {430345=727947,1570056=727439,2699879=469889} 2024-09-21T08:27:16.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:16.602 INFO:tasks.rados.rados.0.smithi137.stdout:887: writing smithi13779221-32 from 430345 to 1158292 tid 1 2024-09-21T08:27:16.610 INFO:tasks.rados.rados.0.smithi137.stdout:887: writing smithi13779221-32 from 1570056 to 2297495 tid 2 2024-09-21T08:27:16.616 INFO:tasks.rados.rados.0.smithi137.stdout:887: writing smithi13779221-32 from 2699879 to 3169768 tid 3 2024-09-21T08:27:16.617 INFO:tasks.rados.rados.0.smithi137.stdout:887: finishing write tid 1 to smithi13779221-32 2024-09-21T08:27:16.617 INFO:tasks.rados.rados.0.smithi137.stdout:877: finishing rollback tid 0 to smithi13779221-15 2024-09-21T08:27:16.617 INFO:tasks.rados.rados.0.smithi137.stdout:877: finishing rollback tid 1 to smithi13779221-15 2024-09-21T08:27:16.617 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 254 (ObjNum 129 snap 0 seq_num 3760636688) dirty dne 2024-09-21T08:27:16.617 INFO:tasks.rados.rados.0.smithi137.stdout:883: expect (ObjNum 310 snap 70 seq_num 310) 2024-09-21T08:27:16.627 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:16.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:16.629 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:16.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:16.731 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:16.732 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:16.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:16.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:16.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:16.824 INFO:tasks.rados.rados.0.smithi137.stdout:877: done (3 left) 2024-09-21T08:27:16.824 INFO:tasks.rados.rados.0.smithi137.stdout:883: done (2 left) 2024-09-21T08:27:16.824 INFO:tasks.rados.rados.0.smithi137.stdout:888: copy_from oid 48 from oid 26 current snap is 79 2024-09-21T08:27:16.824 INFO:tasks.rados.rados.0.smithi137.stdout:889: rollback oid 46 current snap is 79 2024-09-21T08:27:16.824 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 46 to 59 2024-09-21T08:27:16.824 INFO:tasks.rados.rados.0.smithi137.stdout:890: write oid 45 current snap is 79 2024-09-21T08:27:16.825 INFO:tasks.rados.rados.0.smithi137.stdout:890: seq_num 336 ranges {447152=403196,1512185=587450,2727302=681161} 2024-09-21T08:27:16.829 INFO:tasks.rados.rados.0.smithi137.stdout:890: writing smithi13779221-45 from 447152 to 850348 tid 1 2024-09-21T08:27:16.836 INFO:tasks.rados.rados.0.smithi137.stdout:890: writing smithi13779221-45 from 1512185 to 2099635 tid 2 2024-09-21T08:27:16.844 INFO:tasks.rados.rados.0.smithi137.stdout:890: writing smithi13779221-45 from 2727302 to 3408463 tid 3 2024-09-21T08:27:16.844 INFO:tasks.rados.rados.0.smithi137.stdout:891: delete oid 16 current snap is 79 2024-09-21T08:27:16.844 INFO:tasks.rados.rados.0.smithi137.stdout:887: finishing write tid 2 to smithi13779221-32 2024-09-21T08:27:16.844 INFO:tasks.rados.rados.0.smithi137.stdout:886: finishing copy_from racing read to smithi13779221-33 2024-09-21T08:27:16.844 INFO:tasks.rados.rados.0.smithi137.stdout:887: finishing write tid 3 to smithi13779221-32 2024-09-21T08:27:16.844 INFO:tasks.rados.rados.0.smithi137.stdout:887: finishing write tid 4 to smithi13779221-32 2024-09-21T08:27:16.844 INFO:tasks.rados.rados.0.smithi137.stdout:886: finishing copy_from to smithi13779221-33 2024-09-21T08:27:16.844 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 126 (ObjNum 302 snap 70 seq_num 302) dirty exists 2024-09-21T08:27:16.844 INFO:tasks.rados.rados.0.smithi137.stdout:887: finishing write tid 5 to smithi13779221-32 2024-09-21T08:27:16.844 INFO:tasks.rados.rados.0.smithi137.stdout:887: oid 32 updating version 0 to 201 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:887: oid 32 version 201 is already newer than 200 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:887: oid 32 version 201 is already newer than 199 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:887: oid 32 updating version 201 to 202 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 202 (ObjNum 335 snap 79 seq_num 335) dirty exists 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:887: left oid 32 (ObjNum 335 snap 79 seq_num 335) 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:889: finishing rollback tid 0 to smithi13779221-46 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:889: finishing rollback tid 1 to smithi13779221-46 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:889: finishing rollback tid 2 to smithi13779221-46 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 407 (ObjNum 239 snap 49 seq_num 239) dirty exists 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:890: finishing write tid 1 to smithi13779221-45 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:888: finishing copy_from racing read to smithi13779221-48 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:888: finishing copy_from to smithi13779221-48 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 283 (ObjNum 326 snap 78 seq_num 326) dirty exists 2024-09-21T08:27:16.845 INFO:tasks.rados.rados.0.smithi137.stdout:890: finishing write tid 2 to smithi13779221-45 2024-09-21T08:27:16.846 INFO:tasks.rados.rados.0.smithi137.stdout:886: done (5 left) 2024-09-21T08:27:16.846 INFO:tasks.rados.rados.0.smithi137.stdout:887: done (4 left) 2024-09-21T08:27:16.846 INFO:tasks.rados.rados.0.smithi137.stdout:888: done (3 left) 2024-09-21T08:27:16.846 INFO:tasks.rados.rados.0.smithi137.stdout:889: done (2 left) 2024-09-21T08:27:16.846 INFO:tasks.rados.rados.0.smithi137.stdout:891: done (1 left) 2024-09-21T08:27:16.846 INFO:tasks.rados.rados.0.smithi137.stdout:892: snap_remove snap 76 2024-09-21T08:27:16.852 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:16.873 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:16.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:16.874 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:16.913 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[29634]: 6.1c scrub starts 2024-09-21T08:27:16.913 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[29634]: 6.1c scrub ok 2024-09-21T08:27:16.913 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[29634]: 5.b scrub starts 2024-09-21T08:27:16.913 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[29634]: 5.b scrub ok 2024-09-21T08:27:16.914 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[29634]: osdmap e242: 8 total, 8 up, 4 in 2024-09-21T08:27:16.914 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3289192407' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:27:16.914 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[41044]: 6.1c scrub starts 2024-09-21T08:27:16.914 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[41044]: 6.1c scrub ok 2024-09-21T08:27:16.914 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[41044]: 5.b scrub starts 2024-09-21T08:27:16.914 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[41044]: 5.b scrub ok 2024-09-21T08:27:16.914 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[41044]: osdmap e242: 8 total, 8 up, 4 in 2024-09-21T08:27:16.914 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:16 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3289192407' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:27:16.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:16.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:16.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:16.996 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:16 smithi165 ceph-mon[32893]: 6.1c scrub starts 2024-09-21T08:27:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:16 smithi165 ceph-mon[32893]: 6.1c scrub ok 2024-09-21T08:27:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:16 smithi165 ceph-mon[32893]: 5.b scrub starts 2024-09-21T08:27:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:16 smithi165 ceph-mon[32893]: 5.b scrub ok 2024-09-21T08:27:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:16 smithi165 ceph-mon[32893]: osdmap e242: 8 total, 8 up, 4 in 2024-09-21T08:27:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:16 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3289192407' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:27:17.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:17.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:17.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:17.119 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:17.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:17.240 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:17.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:17.241 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:17.300 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:17.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:17.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:17.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:17.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:17.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T08:27:17.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:17.486 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:17.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:17.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:17.593 INFO:tasks.rados.rados.0.smithi137.stdout:892: done (1 left) 2024-09-21T08:27:17.593 INFO:tasks.rados.rados.0.smithi137.stdout:893: write oid 8 current snap is 79 2024-09-21T08:27:17.594 INFO:tasks.rados.rados.0.smithi137.stdout:893: seq_num 338 ranges {480766=553694,1629149=591119,2782148=420058} 2024-09-21T08:27:17.601 INFO:tasks.rados.rados.0.smithi137.stdout:893: writing smithi13779221-8 from 480766 to 1034460 tid 1 2024-09-21T08:27:17.607 INFO:tasks.rados.rados.0.smithi137.stdout:893: writing smithi13779221-8 from 1629149 to 2220268 tid 2 2024-09-21T08:27:17.613 INFO:tasks.rados.rados.0.smithi137.stdout:893: writing smithi13779221-8 from 2782148 to 3202206 tid 3 2024-09-21T08:27:17.613 INFO:tasks.rados.rados.0.smithi137.stdout:894: read oid 46 snap 70 2024-09-21T08:27:17.613 INFO:tasks.rados.rados.0.smithi137.stdout:894: expect (ObjNum 239 snap 49 seq_num 239) 2024-09-21T08:27:17.613 INFO:tasks.rados.rados.0.smithi137.stdout:895: snap_remove snap 78 2024-09-21T08:27:17.614 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:17.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:17.615 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:17.672 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:17.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:17.737 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:17.738 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:17.738 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:17 smithi165 ceph-mon[32893]: 3.5 scrub starts 2024-09-21T08:27:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:17 smithi165 ceph-mon[32893]: 3.5 scrub ok 2024-09-21T08:27:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:17 smithi165 ceph-mon[32893]: pgmap v269: 158 pgs: 5 remapped+peering, 1 active+recovering+degraded, 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 2 active+recovering, 139 active+clean; 240 MiB data, 612 MiB used, 357 GiB / 358 GiB avail; 31 MiB/s rd, 5.6 MiB/s wr, 43 op/s; 8/927 objects misplaced (0.863%); 6.7 MiB/s, 42 objects/s recovering 2024-09-21T08:27:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:17 smithi165 ceph-mon[32893]: 6.2 scrub starts 2024-09-21T08:27:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:17 smithi165 ceph-mon[32893]: 6.2 scrub ok 2024-09-21T08:27:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:17 smithi165 ceph-mon[32893]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-21T08:27:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:17 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:17 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3289192407' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:27:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:17 smithi165 ceph-mon[32893]: osdmap e243: 8 total, 8 up, 4 in 2024-09-21T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[29634]: 3.5 scrub starts 2024-09-21T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[29634]: 3.5 scrub ok 2024-09-21T08:27:17.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[29634]: pgmap v269: 158 pgs: 5 remapped+peering, 1 active+recovering+degraded, 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 2 active+recovering, 139 active+clean; 240 MiB data, 612 MiB used, 357 GiB / 358 GiB avail; 31 MiB/s rd, 5.6 MiB/s wr, 43 op/s; 8/927 objects misplaced (0.863%); 6.7 MiB/s, 42 objects/s recovering 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[29634]: 6.2 scrub starts 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[29634]: 6.2 scrub ok 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[29634]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3289192407' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[29634]: osdmap e243: 8 total, 8 up, 4 in 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[41044]: 3.5 scrub starts 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[41044]: 3.5 scrub ok 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[41044]: pgmap v269: 158 pgs: 5 remapped+peering, 1 active+recovering+degraded, 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 2 active+recovering, 139 active+clean; 240 MiB data, 612 MiB used, 357 GiB / 358 GiB avail; 31 MiB/s rd, 5.6 MiB/s wr, 43 op/s; 8/927 objects misplaced (0.863%); 6.7 MiB/s, 42 objects/s recovering 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[41044]: 6.2 scrub starts 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[41044]: 6.2 scrub ok 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[41044]: Health check failed: Degraded data redundancy: 1 pg degraded (PG_DEGRADED) 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3289192407' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:27:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:17 smithi137 ceph-mon[41044]: osdmap e243: 8 total, 8 up, 4 in 2024-09-21T08:27:17.784 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:17.784 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:17.784 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:17.785 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:17.785 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:17.785 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:17.797 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:17.798 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:17.814 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:17.814 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:17.830 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:17.830 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:17.839 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:17.847 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:17.847 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:17.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:17.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:17.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:17.863 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:17.864 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:17.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:17.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:17.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:17.983 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:18.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:18.105 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:18.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:18.106 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:18.166 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:27:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:27:18.166 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:27:18.166 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:27:18.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:18.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:18.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:18.230 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:18.330 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:18.351 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:18.352 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:18.352 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:18.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:18.475 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:18.476 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:18.476 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:18.577 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:18.599 INFO:tasks.rados.rados.0.smithi137.stdout:895: done (3 left) 2024-09-21T08:27:18.599 INFO:tasks.rados.rados.0.smithi137.stdout:896: snap_remove snap 59 2024-09-21T08:27:18.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:18.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:18.602 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:18.703 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:18.724 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:18.724 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:18.725 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:18.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:18 smithi165 ceph-mon[32893]: osdmap e244: 8 total, 8 up, 4 in 2024-09-21T08:27:18.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:18 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/16486088' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:18 smithi137 ceph-mon[29634]: osdmap e244: 8 total, 8 up, 4 in 2024-09-21T08:27:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:18 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/16486088' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:18.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:18 smithi137 ceph-mon[41044]: osdmap e244: 8 total, 8 up, 4 in 2024-09-21T08:27:18.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:18 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/16486088' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:18.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:18.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:18.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:18.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:18.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T08:27:18.948 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:19.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:19.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:19.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:19.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:19.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:19.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:19.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:19.182 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:19.224 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:27:19.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:19.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:19.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:19.250 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:19.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:19.372 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:19.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:19.373 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:19.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:19.481 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:19.496 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:19.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:19.497 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:19.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:19.601 INFO:tasks.rados.rados.0.smithi137.stdout:896: done (3 left) 2024-09-21T08:27:19.601 INFO:tasks.rados.rados.0.smithi137.stdout:897: read oid 10 snap -1 2024-09-21T08:27:19.601 INFO:tasks.rados.rados.0.smithi137.stdout:897: expect (ObjNum 274 snap 64 seq_num 274) 2024-09-21T08:27:19.601 INFO:tasks.rados.rados.0.smithi137.stdout:898: rollback oid 19 current snap is 79 2024-09-21T08:27:19.601 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 19 to 70 2024-09-21T08:27:19.601 INFO:tasks.rados.rados.0.smithi137.stdout:890: finishing write tid 3 to smithi13779221-45 2024-09-21T08:27:19.601 INFO:tasks.rados.rados.0.smithi137.stdout:890: finishing write tid 4 to smithi13779221-45 2024-09-21T08:27:19.601 INFO:tasks.rados.rados.0.smithi137.stdout:890: finishing write tid 5 to smithi13779221-45 2024-09-21T08:27:19.601 INFO:tasks.rados.rados.0.smithi137.stdout:890: oid 45 updating version 0 to 263 2024-09-21T08:27:19.601 INFO:tasks.rados.rados.0.smithi137.stdout:890: oid 45 version 263 is already newer than 262 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:890: oid 45 updating version 263 to 265 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:890: oid 45 version 265 is already newer than 264 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 265 (ObjNum 336 snap 79 seq_num 336) dirty exists 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:890: left oid 45 (ObjNum 336 snap 79 seq_num 336) 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:890: done (4 left) 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:899: read oid 26 snap -1 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:899: expect (ObjNum 326 snap 78 seq_num 326) 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:900: read oid 25 snap -1 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:900: expect deleted 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:901: delete oid 31 current snap is 79 2024-09-21T08:27:19.602 INFO:tasks.rados.rados.0.smithi137.stdout:894: expect (ObjNum 239 snap 49 seq_num 239) 2024-09-21T08:27:19.623 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:19.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:19.624 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:19.707 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:27:19.707 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:27:19.707 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:27:19.708 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:27:19.708 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:27:19.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:19.744 INFO:tasks.rados.rados.0.smithi137.stdout:893: finishing write tid 1 to smithi13779221-8 2024-09-21T08:27:19.744 INFO:tasks.rados.rados.0.smithi137.stdout:893: finishing write tid 2 to smithi13779221-8 2024-09-21T08:27:19.744 INFO:tasks.rados.rados.0.smithi137.stdout:894: done (6 left) 2024-09-21T08:27:19.744 INFO:tasks.rados.rados.0.smithi137.stdout:901: done (5 left) 2024-09-21T08:27:19.744 INFO:tasks.rados.rados.0.smithi137.stdout:902: copy_from oid 49 from oid 47 current snap is 79 2024-09-21T08:27:19.744 INFO:tasks.rados.rados.0.smithi137.stdout:893: finishing write tid 3 to smithi13779221-8 2024-09-21T08:27:19.744 INFO:tasks.rados.rados.0.smithi137.stdout:893: finishing write tid 4 to smithi13779221-8 2024-09-21T08:27:19.744 INFO:tasks.rados.rados.0.smithi137.stdout:893: finishing write tid 5 to smithi13779221-8 2024-09-21T08:27:19.744 INFO:tasks.rados.rados.0.smithi137.stdout:893: oid 8 updating version 0 to 269 2024-09-21T08:27:19.745 INFO:tasks.rados.rados.0.smithi137.stdout:893: oid 8 version 269 is already newer than 268 2024-09-21T08:27:19.745 INFO:tasks.rados.rados.0.smithi137.stdout:893: oid 8 version 269 is already newer than 267 2024-09-21T08:27:19.745 INFO:tasks.rados.rados.0.smithi137.stdout:893: oid 8 version 269 is already newer than 266 2024-09-21T08:27:19.745 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 269 (ObjNum 338 snap 79 seq_num 338) dirty exists 2024-09-21T08:27:19.745 INFO:tasks.rados.rados.0.smithi137.stdout:893: left oid 8 (ObjNum 338 snap 79 seq_num 338) 2024-09-21T08:27:19.745 INFO:tasks.rados.rados.0.smithi137.stdout:897: expect (ObjNum 274 snap 64 seq_num 274) 2024-09-21T08:27:19.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:19.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:19.746 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:19.829 INFO:tasks.rados.rados.0.smithi137.stdout:898: finishing rollback tid 1 to smithi13779221-19 2024-09-21T08:27:19.829 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 100 (ObjNum 689 snap 0 seq_num 3760597200) dirty dne 2024-09-21T08:27:19.829 INFO:tasks.rados.rados.0.smithi137.stdout:899: expect (ObjNum 326 snap 78 seq_num 326) 2024-09-21T08:27:19.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:19.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:19.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:19.870 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:19.884 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:19 smithi137 ceph-mon[29634]: pgmap v272: 158 pgs: 5 remapped+peering, 1 active+recovering+degraded, 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 2 active+recovering, 139 active+clean; 240 MiB data, 612 MiB used, 357 GiB / 358 GiB avail; 31 MiB/s rd, 5.6 MiB/s wr, 43 op/s; 8/927 objects misplaced (0.863%); 6.7 MiB/s, 42 objects/s recovering 2024-09-21T08:27:19.884 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:19 smithi137 ceph-mon[29634]: osdmap e245: 8 total, 8 up, 4 in 2024-09-21T08:27:19.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:19 smithi137 ceph-mon[41044]: pgmap v272: 158 pgs: 5 remapped+peering, 1 active+recovering+degraded, 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 2 active+recovering, 139 active+clean; 240 MiB data, 612 MiB used, 357 GiB / 358 GiB avail; 31 MiB/s rd, 5.6 MiB/s wr, 43 op/s; 8/927 objects misplaced (0.863%); 6.7 MiB/s, 42 objects/s recovering 2024-09-21T08:27:19.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:19 smithi137 ceph-mon[41044]: osdmap e245: 8 total, 8 up, 4 in 2024-09-21T08:27:19.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:19.991 INFO:tasks.rados.rados.0.smithi137.stdout:902: finishing copy_from racing read to smithi13779221-49 2024-09-21T08:27:19.991 INFO:tasks.rados.rados.0.smithi137.stdout:902: finishing copy_from to smithi13779221-49 2024-09-21T08:27:19.991 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:27:19.991 INFO:tasks.rados.rados.0.smithi137.stdout:893: done (5 left) 2024-09-21T08:27:19.991 INFO:tasks.rados.rados.0.smithi137.stdout:897: done (4 left) 2024-09-21T08:27:19.991 INFO:tasks.rados.rados.0.smithi137.stdout:898: done (3 left) 2024-09-21T08:27:19.991 INFO:tasks.rados.rados.0.smithi137.stdout:899: done (2 left) 2024-09-21T08:27:19.991 INFO:tasks.rados.rados.0.smithi137.stdout:900: done (1 left) 2024-09-21T08:27:19.991 INFO:tasks.rados.rados.0.smithi137.stdout:902: done (0 left) 2024-09-21T08:27:19.991 INFO:tasks.rados.rados.0.smithi137.stdout:903: snap_create 2024-09-21T08:27:19.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:19.993 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:19.993 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:19 smithi165 ceph-mon[32893]: pgmap v272: 158 pgs: 5 remapped+peering, 1 active+recovering+degraded, 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 2 active+recovering, 139 active+clean; 240 MiB data, 612 MiB used, 357 GiB / 358 GiB avail; 31 MiB/s rd, 5.6 MiB/s wr, 43 op/s; 8/927 objects misplaced (0.863%); 6.7 MiB/s, 42 objects/s recovering 2024-09-21T08:27:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:19 smithi165 ceph-mon[32893]: osdmap e245: 8 total, 8 up, 4 in 2024-09-21T08:27:20.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:20.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:20.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:20.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:20.217 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:20.233 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 6, 1, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:27:20.233 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:27:20.233 INFO:tasks.thrashosds.thrasher:inject_pause on osd.6 2024-09-21T08:27:20.233 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T08:27:20.233 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T08:27:20.233 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 config set filestore_inject_stall 3 2024-09-21T08:27:20.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:20.240 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:20.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:20.342 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:20.363 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:20.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:20.364 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:20.428 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T08:27:20.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:20.486 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:27:20.490 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:20.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:20.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:20.493 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:20.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:20.604 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:27:20.613 INFO:tasks.rados.rados.0.smithi137.stdout:903: done (0 left) 2024-09-21T08:27:20.613 INFO:tasks.rados.rados.0.smithi137.stdout:904: write oid 13 current snap is 80 2024-09-21T08:27:20.613 INFO:tasks.rados.rados.0.smithi137.stdout:904: seq_num 340 ranges {514380=704192,1629760=478435,2604288=442602} 2024-09-21T08:27:20.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:20.620 INFO:tasks.rados.rados.0.smithi137.stdout:904: writing smithi13779221-13 from 514380 to 1218572 tid 1 2024-09-21T08:27:20.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:20.621 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:20.625 INFO:tasks.rados.rados.0.smithi137.stdout:904: writing smithi13779221-13 from 1629760 to 2108195 tid 2 2024-09-21T08:27:20.630 INFO:tasks.rados.rados.0.smithi137.stdout:904: writing smithi13779221-13 from 2604288 to 3046890 tid 3 2024-09-21T08:27:20.631 INFO:tasks.rados.rados.0.smithi137.stdout:905: snap_remove snap 77 2024-09-21T08:27:20.721 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:20.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:20 smithi165 ceph-mon[32893]: osdmap e246: 8 total, 8 up, 4 in 2024-09-21T08:27:20.740 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:20 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3271363627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:20.744 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:20.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:20.745 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:20.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:20 smithi137 ceph-mon[29634]: osdmap e246: 8 total, 8 up, 4 in 2024-09-21T08:27:20.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:20 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3271363627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:20 smithi137 ceph-mon[41044]: osdmap e246: 8 total, 8 up, 4 in 2024-09-21T08:27:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:20 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3271363627' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:20.768 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:27:20 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:27:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:27:20.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:20.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:20.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:20.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:20.944 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:27:20.944 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T08:27:20.944 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:27:20.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:20.976 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:20.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:20.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:20.991 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:21.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:21.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:21.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:21.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:21.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:21.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:21.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:21.236 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:21.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:21.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:21.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:21.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:21.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:21.483 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:21.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:21.484 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:21.529 INFO:tasks.rados.rados.0.smithi137.stdout:905: done (1 left) 2024-09-21T08:27:21.529 INFO:tasks.rados.rados.0.smithi137.stdout:906: write (excl) oid 10 current snap is 80 2024-09-21T08:27:21.529 INFO:tasks.rados.rados.0.smithi137.stdout:904: finishing write tid 1 to smithi13779221-13 2024-09-21T08:27:21.529 INFO:tasks.rados.rados.0.smithi137.stdout:906: seq_num 341 ranges {531187=779441,2088242=738446,3431711=299476} 2024-09-21T08:27:21.536 INFO:tasks.rados.rados.0.smithi137.stdout:906: writing smithi13779221-10 from 531187 to 1310628 tid 1 2024-09-21T08:27:21.545 INFO:tasks.rados.rados.0.smithi137.stdout:906: writing smithi13779221-10 from 2088242 to 2826688 tid 2 2024-09-21T08:27:21.550 INFO:tasks.rados.rados.0.smithi137.stdout:906: writing smithi13779221-10 from 3431711 to 3731187 tid 3 2024-09-21T08:27:21.550 INFO:tasks.rados.rados.0.smithi137.stdout:907: write (excl) oid 22 current snap is 80 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:904: finishing write tid 2 to smithi13779221-13 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:904: finishing write tid 3 to smithi13779221-13 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:904: finishing write tid 4 to smithi13779221-13 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:904: finishing write tid 5 to smithi13779221-13 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:904: oid 13 updating version 0 to 415 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:904: oid 13 updating version 415 to 416 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:904: oid 13 updating version 416 to 417 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:904: oid 13 updating version 417 to 418 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 418 (ObjNum 340 snap 80 seq_num 340) dirty exists 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:904: left oid 13 (ObjNum 340 snap 80 seq_num 340) 2024-09-21T08:27:21.551 INFO:tasks.rados.rados.0.smithi137.stdout:907: seq_num 342 ranges {547994=454690,1746724=482104,2659134=465146,3605976=142018} 2024-09-21T08:27:21.556 INFO:tasks.rados.rados.0.smithi137.stdout:907: writing smithi13779221-22 from 547994 to 1002684 tid 1 2024-09-21T08:27:21.563 INFO:tasks.rados.rados.0.smithi137.stdout:907: writing smithi13779221-22 from 1746724 to 2228828 tid 2 2024-09-21T08:27:21.567 INFO:tasks.rados.rados.0.smithi137.stdout:907: writing smithi13779221-22 from 2659134 to 3124280 tid 3 2024-09-21T08:27:21.571 INFO:tasks.rados.rados.0.smithi137.stdout:907: writing smithi13779221-22 from 3605976 to 3747994 tid 4 2024-09-21T08:27:21.571 INFO:tasks.rados.rados.0.smithi137.stdout:904: done (2 left) 2024-09-21T08:27:21.571 INFO:tasks.rados.rados.0.smithi137.stdout:908: snap_remove snap 79 2024-09-21T08:27:21.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:21.599 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:21.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:21 smithi137 ceph-mon[29634]: pgmap v275: 158 pgs: 5 remapped+peering, 1 active+recovering+degraded, 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 2 active+recovering, 139 active+clean; 240 MiB data, 612 MiB used, 357 GiB / 358 GiB avail; 8/927 objects misplaced (0.863%) 2024-09-21T08:27:21.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:21 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3271363627' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:21.606 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:21.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:21.607 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:21.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:21.728 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:21.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:21.729 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:21.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:21.853 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:21.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:21.854 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:21.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:21 smithi165 ceph-mon[32893]: pgmap v275: 158 pgs: 5 remapped+peering, 1 active+recovering+degraded, 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 2 active+recovering, 139 active+clean; 240 MiB data, 612 MiB used, 357 GiB / 358 GiB avail; 8/927 objects misplaced (0.863%) 2024-09-21T08:27:21.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:21 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3271363627' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:21.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:21 smithi165 ceph-mon[32893]: osdmap e247: 8 total, 8 up, 4 in 2024-09-21T08:27:21.855 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:21 smithi165 ceph-mon[32893]: osdmap e248: 8 total, 8 up, 4 in 2024-09-21T08:27:21.856 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:27:21 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:27:21.934 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:21 smithi137 ceph-mon[29634]: osdmap e247: 8 total, 8 up, 4 in 2024-09-21T08:27:21.935 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:21 smithi137 ceph-mon[29634]: osdmap e248: 8 total, 8 up, 4 in 2024-09-21T08:27:21.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:21 smithi137 ceph-mon[41044]: pgmap v275: 158 pgs: 5 remapped+peering, 1 active+recovering+degraded, 7 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 2 active+recovering, 139 active+clean; 240 MiB data, 612 MiB used, 357 GiB / 358 GiB avail; 8/927 objects misplaced (0.863%) 2024-09-21T08:27:21.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:21 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3271363627' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:21.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:21 smithi137 ceph-mon[41044]: osdmap e247: 8 total, 8 up, 4 in 2024-09-21T08:27:21.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:21 smithi137 ceph-mon[41044]: osdmap e248: 8 total, 8 up, 4 in 2024-09-21T08:27:21.941 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:21.942 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:21.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:21.958 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:21.958 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:21.974 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:21.974 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:21.977 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:21.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:21.978 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:21.989 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:21.989 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:22.006 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:22.006 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:22.022 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:22.022 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:22.038 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:22.038 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:22.059 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:22.060 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:22.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:22.080 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:22.080 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:22.101 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:22.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:22.102 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:22.203 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:22.224 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:22.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:22.225 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:22.326 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:22.347 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:22.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:22.348 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:22.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:22.469 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:22.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:22.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:22.529 INFO:tasks.rados.rados.0.smithi137.stdout:908: done (2 left) 2024-09-21T08:27:22.529 INFO:tasks.rados.rados.0.smithi137.stdout:909: read oid 25 snap -1 2024-09-21T08:27:22.529 INFO:tasks.rados.rados.0.smithi137.stdout:909: expect deleted 2024-09-21T08:27:22.529 INFO:tasks.rados.rados.0.smithi137.stdout:910: snap_remove snap 70 2024-09-21T08:27:22.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:22.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:22.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:22.593 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:22.674 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:22 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/354793884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:22.675 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:22 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:27:22.675 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:22 smithi137 ceph-mon[29634]: osdmap e249: 8 total, 8 up, 4 in 2024-09-21T08:27:22.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:22 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/354793884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:22.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:22 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:27:22.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:22 smithi137 ceph-mon[41044]: osdmap e249: 8 total, 8 up, 4 in 2024-09-21T08:27:22.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:22.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:22.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:22.775 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T08:27:22.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:22.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:22.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:22.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:22.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:22.964 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:22.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:22.965 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:22 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/354793884' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:22 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:27:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:22 smithi165 ceph-mon[32893]: osdmap e249: 8 total, 8 up, 4 in 2024-09-21T08:27:23.023 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:23.066 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:23.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:23.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:23.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:23.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:23.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:23.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:23.212 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:23.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:23.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:23.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:23.335 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:23.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:27:23.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:23.457 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:23.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:23.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:23.534 INFO:tasks.rados.rados.0.smithi137.stdout:910: done (3 left) 2024-09-21T08:27:23.534 INFO:tasks.rados.rados.0.smithi137.stdout:911: write oid 30 current snap is 80 2024-09-21T08:27:23.534 INFO:tasks.rados.rados.0.smithi137.stdout:906: finishing write tid 1 to smithi13779221-10 2024-09-21T08:27:23.534 INFO:tasks.rados.rados.0.smithi137.stdout:906: finishing write tid 2 to smithi13779221-10 2024-09-21T08:27:23.534 INFO:tasks.rados.rados.0.smithi137.stdout:906: finishing write tid 3 to smithi13779221-10 2024-09-21T08:27:23.534 INFO:tasks.rados.rados.0.smithi137.stdout:911: seq_num 343 ranges {564801=646292,1921559=742115,3202910=561891} 2024-09-21T08:27:23.541 INFO:tasks.rados.rados.0.smithi137.stdout:911: writing smithi13779221-30 from 564801 to 1211093 tid 1 2024-09-21T08:27:23.552 INFO:tasks.rados.rados.0.smithi137.stdout:911: writing smithi13779221-30 from 1921559 to 2663674 tid 2 2024-09-21T08:27:23.559 INFO:tasks.rados.rados.0.smithi137.stdout:911: writing smithi13779221-30 from 3202910 to 3764801 tid 3 2024-09-21T08:27:23.559 INFO:tasks.rados.rados.0.smithi137.stdout:912: snap_create 2024-09-21T08:27:23.559 INFO:tasks.rados.rados.0.smithi137.stdout:906: finishing write tid 4 to smithi13779221-10 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:906: finishing write tid 5 to smithi13779221-10 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:906: oid 10 updating version 0 to 287 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:906: oid 10 version 287 is already newer than 286 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:906: oid 10 version 287 is already newer than 284 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:906: oid 10 version 287 is already newer than 285 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 287 (ObjNum 341 snap 80 seq_num 341) dirty exists 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:906: left oid 10 (ObjNum 341 snap 80 seq_num 341) 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:907: finishing write tid 1 to smithi13779221-22 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:907: finishing write tid 2 to smithi13779221-22 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:907: finishing write tid 3 to smithi13779221-22 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:907: finishing write tid 4 to smithi13779221-22 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:907: finishing write tid 5 to smithi13779221-22 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:907: finishing write tid 6 to smithi13779221-22 2024-09-21T08:27:23.560 INFO:tasks.rados.rados.0.smithi137.stdout:907: oid 22 updating version 0 to 313 2024-09-21T08:27:23.561 INFO:tasks.rados.rados.0.smithi137.stdout:907: oid 22 updating version 313 to 316 2024-09-21T08:27:23.561 INFO:tasks.rados.rados.0.smithi137.stdout:907: oid 22 updating version 316 to 317 2024-09-21T08:27:23.561 INFO:tasks.rados.rados.0.smithi137.stdout:907: oid 22 version 317 is already newer than 314 2024-09-21T08:27:23.561 INFO:tasks.rados.rados.0.smithi137.stdout:907: oid 22 version 317 is already newer than 315 2024-09-21T08:27:23.561 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 317 (ObjNum 342 snap 80 seq_num 342) dirty exists 2024-09-21T08:27:23.561 INFO:tasks.rados.rados.0.smithi137.stdout:907: left oid 22 (ObjNum 342 snap 80 seq_num 342) 2024-09-21T08:27:23.561 INFO:tasks.rados.rados.0.smithi137.stdout:911: finishing write tid 1 to smithi13779221-30 2024-09-21T08:27:23.561 INFO:tasks.rados.rados.0.smithi137.stdout:911: finishing write tid 2 to smithi13779221-30 2024-09-21T08:27:23.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:23.562 INFO:tasks.rados.rados.0.smithi137.stdout:911: finishing write tid 3 to smithi13779221-30 2024-09-21T08:27:23.562 INFO:tasks.rados.rados.0.smithi137.stdout:911: finishing write tid 4 to smithi13779221-30 2024-09-21T08:27:23.562 INFO:tasks.rados.rados.0.smithi137.stdout:911: finishing write tid 5 to smithi13779221-30 2024-09-21T08:27:23.562 INFO:tasks.rados.rados.0.smithi137.stdout:911: oid 30 updating version 0 to 425 2024-09-21T08:27:23.562 INFO:tasks.rados.rados.0.smithi137.stdout:911: oid 30 version 425 is already newer than 424 2024-09-21T08:27:23.563 INFO:tasks.rados.rados.0.smithi137.stdout:911: oid 30 updating version 425 to 426 2024-09-21T08:27:23.563 INFO:tasks.rados.rados.0.smithi137.stdout:911: oid 30 version 426 is already newer than 423 2024-09-21T08:27:23.563 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 426 (ObjNum 343 snap 80 seq_num 343) dirty exists 2024-09-21T08:27:23.563 INFO:tasks.rados.rados.0.smithi137.stdout:911: left oid 30 (ObjNum 343 snap 80 seq_num 343) 2024-09-21T08:27:23.573 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:23.573 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:23.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:23.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:23.585 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:23.674 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:23.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:23.709 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:23.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:23.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:23 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:23 smithi165 ceph-mon[32893]: pgmap v278: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 226 MiB data, 691 MiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 3.9 MiB/s wr, 18 op/s; 14 MiB/s, 21 objects/s recovering 2024-09-21T08:27:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:23 smithi165 ceph-mon[32893]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-09-21T08:27:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:23 smithi165 ceph-mon[32893]: osdmap e250: 8 total, 8 up, 4 in 2024-09-21T08:27:23.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:23 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:23.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:23 smithi137 ceph-mon[29634]: pgmap v278: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 226 MiB data, 691 MiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 3.9 MiB/s wr, 18 op/s; 14 MiB/s, 21 objects/s recovering 2024-09-21T08:27:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:23 smithi137 ceph-mon[29634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-09-21T08:27:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:23 smithi137 ceph-mon[29634]: osdmap e250: 8 total, 8 up, 4 in 2024-09-21T08:27:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:23 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:23 smithi137 ceph-mon[41044]: pgmap v278: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 226 MiB data, 691 MiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 3.9 MiB/s wr, 18 op/s; 14 MiB/s, 21 objects/s recovering 2024-09-21T08:27:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:23 smithi137 ceph-mon[41044]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 1 pg degraded) 2024-09-21T08:27:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:23 smithi137 ceph-mon[41044]: osdmap e250: 8 total, 8 up, 4 in 2024-09-21T08:27:23.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:23.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:23.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:23.834 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:23.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:23.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:23.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:23.957 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:24.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:24.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:24.080 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:24.080 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:24.181 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:24.202 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:24.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:24.203 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:24.303 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:24.324 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:24.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:24.325 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:24.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T08:27:24.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:24.448 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:24.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:24.449 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:24.530 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:27:24.535 INFO:tasks.rados.rados.0.smithi137.stdout:906: done (4 left) 2024-09-21T08:27:24.535 INFO:tasks.rados.rados.0.smithi137.stdout:907: done (3 left) 2024-09-21T08:27:24.535 INFO:tasks.rados.rados.0.smithi137.stdout:909: done (2 left) 2024-09-21T08:27:24.535 INFO:tasks.rados.rados.0.smithi137.stdout:911: done (1 left) 2024-09-21T08:27:24.535 INFO:tasks.rados.rados.0.smithi137.stdout:912: done (0 left) 2024-09-21T08:27:24.535 INFO:tasks.rados.rados.0.smithi137.stdout:913: rollback oid 32 current snap is 81 2024-09-21T08:27:24.535 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 32 to 80 2024-09-21T08:27:24.535 INFO:tasks.rados.rados.0.smithi137.stdout:914: read oid 28 snap -1 2024-09-21T08:27:24.535 INFO:tasks.rados.rados.0.smithi137.stdout:914: expect (ObjNum 314 snap 71 seq_num 314) 2024-09-21T08:27:24.535 INFO:tasks.rados.rados.0.smithi137.stdout:915: write oid 36 current snap is 81 2024-09-21T08:27:24.536 INFO:tasks.rados.rados.0.smithi137.stdout:915: seq_num 344 ranges {581608=721541,1863688=485773,3113980=604043} 2024-09-21T08:27:24.543 INFO:tasks.rados.rados.0.smithi137.stdout:915: writing smithi13779221-36 from 581608 to 1303149 tid 1 2024-09-21T08:27:24.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:24.550 INFO:tasks.rados.rados.0.smithi137.stdout:915: writing smithi13779221-36 from 1863688 to 2349461 tid 2 2024-09-21T08:27:24.558 INFO:tasks.rados.rados.0.smithi137.stdout:915: writing smithi13779221-36 from 3113980 to 3718023 tid 3 2024-09-21T08:27:24.558 INFO:tasks.rados.rados.0.smithi137.stdout:916: copy_from oid 26 from oid 5 current snap is 81 2024-09-21T08:27:24.558 INFO:tasks.rados.rados.0.smithi137.stdout:913: finishing rollback tid 0 to smithi13779221-32 2024-09-21T08:27:24.558 INFO:tasks.rados.rados.0.smithi137.stdout:913: finishing rollback tid 1 to smithi13779221-32 2024-09-21T08:27:24.558 INFO:tasks.rados.rados.0.smithi137.stdout:913: finishing rollback tid 2 to smithi13779221-32 2024-09-21T08:27:24.558 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 205 (ObjNum 335 snap 79 seq_num 335) dirty exists 2024-09-21T08:27:24.558 INFO:tasks.rados.rados.0.smithi137.stdout:914: expect (ObjNum 314 snap 71 seq_num 314) 2024-09-21T08:27:24.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:24.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:24.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:24.643 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:24.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:24.676 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:27:24.695 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:24.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:24.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:24.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:24 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/806540080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:24.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:24 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:24.698 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:24 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/806540080' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:27:24.699 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:24 smithi137 ceph-mon[29634]: osdmap e251: 8 total, 8 up, 4 in 2024-09-21T08:27:24.699 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:24 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/806540080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:24.699 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:24 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:24.699 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:24 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/806540080' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:27:24.699 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:24 smithi137 ceph-mon[41044]: osdmap e251: 8 total, 8 up, 4 in 2024-09-21T08:27:24.765 INFO:tasks.rados.rados.0.smithi137.stdout:915: finishing write tid 1 to smithi13779221-36 2024-09-21T08:27:24.765 INFO:tasks.rados.rados.0.smithi137.stdout:915: finishing write tid 2 to smithi13779221-36 2024-09-21T08:27:24.765 INFO:tasks.rados.rados.0.smithi137.stdout:916: finishing copy_from to smithi13779221-26 2024-09-21T08:27:24.765 INFO:tasks.rados.rados.0.smithi137.stdout:916: got expected ENOENT (src dne) 2024-09-21T08:27:24.765 INFO:tasks.rados.rados.0.smithi137.stdout:916: finishing copy_from racing read to smithi13779221-26 2024-09-21T08:27:24.765 INFO:tasks.rados.rados.0.smithi137.stdout:915: finishing write tid 3 to smithi13779221-36 2024-09-21T08:27:24.766 INFO:tasks.rados.rados.0.smithi137.stdout:913: done (3 left) 2024-09-21T08:27:24.766 INFO:tasks.rados.rados.0.smithi137.stdout:914: done (2 left) 2024-09-21T08:27:24.766 INFO:tasks.rados.rados.0.smithi137.stdout:916: done (1 left) 2024-09-21T08:27:24.766 INFO:tasks.rados.rados.0.smithi137.stdout:917: write oid 41 current snap is 81 2024-09-21T08:27:24.766 INFO:tasks.rados.rados.0.smithi137.stdout:915: finishing write tid 4 to smithi13779221-36 2024-09-21T08:27:24.766 INFO:tasks.rados.rados.0.smithi137.stdout:917: seq_num 345 ranges {598415=796790,1922170=745784,3141403=415315} 2024-09-21T08:27:24.772 INFO:tasks.rados.rados.0.smithi137.stdout:917: writing smithi13779221-41 from 598415 to 1395205 tid 1 2024-09-21T08:27:24.779 INFO:tasks.rados.rados.0.smithi137.stdout:917: writing smithi13779221-41 from 1922170 to 2667954 tid 2 2024-09-21T08:27:24.784 INFO:tasks.rados.rados.0.smithi137.stdout:917: writing smithi13779221-41 from 3141403 to 3556718 tid 3 2024-09-21T08:27:24.785 INFO:tasks.rados.rados.0.smithi137.stdout:918: delete oid 22 current snap is 81 2024-09-21T08:27:24.785 INFO:tasks.rados.rados.0.smithi137.stdout:915: finishing write tid 5 to smithi13779221-36 2024-09-21T08:27:24.785 INFO:tasks.rados.rados.0.smithi137.stdout:915: oid 36 updating version 0 to 431 2024-09-21T08:27:24.785 INFO:tasks.rados.rados.0.smithi137.stdout:915: oid 36 version 431 is already newer than 428 2024-09-21T08:27:24.785 INFO:tasks.rados.rados.0.smithi137.stdout:915: oid 36 version 431 is already newer than 429 2024-09-21T08:27:24.785 INFO:tasks.rados.rados.0.smithi137.stdout:915: oid 36 version 431 is already newer than 430 2024-09-21T08:27:24.785 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 431 (ObjNum 344 snap 81 seq_num 344) dirty exists 2024-09-21T08:27:24.785 INFO:tasks.rados.rados.0.smithi137.stdout:915: left oid 36 (ObjNum 344 snap 81 seq_num 344) 2024-09-21T08:27:24.785 INFO:tasks.rados.rados.0.smithi137.stdout:917: finishing write tid 1 to smithi13779221-41 2024-09-21T08:27:24.785 INFO:tasks.rados.rados.0.smithi137.stdout:917: finishing write tid 2 to smithi13779221-41 2024-09-21T08:27:24.787 INFO:tasks.rados.rados.0.smithi137.stdout:915: done (2 left) 2024-09-21T08:27:24.787 INFO:tasks.rados.rados.0.smithi137.stdout:918: done (1 left) 2024-09-21T08:27:24.787 INFO:tasks.rados.rados.0.smithi137.stdout:919: read oid 48 snap 80 2024-09-21T08:27:24.787 INFO:tasks.rados.rados.0.smithi137.stdout:919: expect (ObjNum 326 snap 78 seq_num 326) 2024-09-21T08:27:24.787 INFO:tasks.rados.rados.0.smithi137.stdout:920: read oid 27 snap -1 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:920: expect deleted 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:921: read oid 25 snap -1 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:921: expect deleted 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:922: snap_create 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:917: finishing write tid 3 to smithi13779221-41 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:917: finishing write tid 4 to smithi13779221-41 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:917: finishing write tid 5 to smithi13779221-41 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:917: oid 41 updating version 0 to 288 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:917: oid 41 updating version 288 to 290 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:917: oid 41 version 290 is already newer than 289 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:917: oid 41 updating version 290 to 291 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 291 (ObjNum 345 snap 81 seq_num 345) dirty exists 2024-09-21T08:27:24.788 INFO:tasks.rados.rados.0.smithi137.stdout:917: left oid 41 (ObjNum 345 snap 81 seq_num 345) 2024-09-21T08:27:24.793 INFO:tasks.rados.rados.0.smithi137.stdout:919: expect (ObjNum 326 snap 78 seq_num 326) 2024-09-21T08:27:24.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:24.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:24.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:24.819 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:24.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:24.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:24.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:24.942 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:24.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:24 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/806540080' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:24.976 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:24 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:24.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:24 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/806540080' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:27:24.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:24 smithi165 ceph-mon[32893]: osdmap e251: 8 total, 8 up, 4 in 2024-09-21T08:27:25.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:25.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:25.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:25.064 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:25.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:25.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:25.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:25.187 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:25.254 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:25.255 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:25.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:25.310 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:25.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:25.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:25.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:25.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:25.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:25.434 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:25.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:25.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:25.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:25.556 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:25.627 INFO:tasks.rados.rados.0.smithi137.stdout:917: done (4 left) 2024-09-21T08:27:25.627 INFO:tasks.rados.rados.0.smithi137.stdout:919: done (3 left) 2024-09-21T08:27:25.627 INFO:tasks.rados.rados.0.smithi137.stdout:920: done (2 left) 2024-09-21T08:27:25.627 INFO:tasks.rados.rados.0.smithi137.stdout:921: done (1 left) 2024-09-21T08:27:25.627 INFO:tasks.rados.rados.0.smithi137.stdout:922: done (0 left) 2024-09-21T08:27:25.627 INFO:tasks.rados.rados.0.smithi137.stdout:923: write (excl) oid 16 current snap is 82 2024-09-21T08:27:25.627 INFO:tasks.rados.rados.0.smithi137.stdout:923: seq_num 347 ranges {632029=547288,1639134=749453,2796249=437859,3653524=178505} 2024-09-21T08:27:25.633 INFO:tasks.rados.rados.0.smithi137.stdout:923: writing smithi13779221-16 from 632029 to 1179317 tid 1 2024-09-21T08:27:25.640 INFO:tasks.rados.rados.0.smithi137.stdout:923: writing smithi13779221-16 from 1639134 to 2388587 tid 2 2024-09-21T08:27:25.645 INFO:tasks.rados.rados.0.smithi137.stdout:923: writing smithi13779221-16 from 2796249 to 3234108 tid 3 2024-09-21T08:27:25.649 INFO:tasks.rados.rados.0.smithi137.stdout:923: writing smithi13779221-16 from 3653524 to 3832029 tid 4 2024-09-21T08:27:25.649 INFO:tasks.rados.rados.0.smithi137.stdout:924: write (excl) oid 25 current snap is 82 2024-09-21T08:27:25.649 INFO:tasks.rados.rados.0.smithi137.stdout:923: finishing write tid 1 to smithi13779221-16 2024-09-21T08:27:25.649 INFO:tasks.rados.rados.0.smithi137.stdout:923: finishing write tid 2 to smithi13779221-16 2024-09-21T08:27:25.649 INFO:tasks.rados.rados.0.smithi137.stdout:923: finishing write tid 3 to smithi13779221-16 2024-09-21T08:27:25.649 INFO:tasks.rados.rados.0.smithi137.stdout:924: seq_num 348 ranges {648836=622537,1981263=493111,3107319=741517} 2024-09-21T08:27:25.655 INFO:tasks.rados.rados.0.smithi137.stdout:924: writing smithi13779221-25 from 648836 to 1271373 tid 1 2024-09-21T08:27:25.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:25.662 INFO:tasks.rados.rados.0.smithi137.stdout:924: writing smithi13779221-25 from 1981263 to 2474374 tid 2 2024-09-21T08:27:25.670 INFO:tasks.rados.rados.0.smithi137.stdout:924: writing smithi13779221-25 from 3107319 to 3848836 tid 3 2024-09-21T08:27:25.670 INFO:tasks.rados.rados.0.smithi137.stdout:925: delete oid 29 current snap is 82 2024-09-21T08:27:25.670 INFO:tasks.rados.rados.0.smithi137.stdout:923: finishing write tid 4 to smithi13779221-16 2024-09-21T08:27:25.670 INFO:tasks.rados.rados.0.smithi137.stdout:923: finishing write tid 5 to smithi13779221-16 2024-09-21T08:27:25.670 INFO:tasks.rados.rados.0.smithi137.stdout:923: finishing write tid 6 to smithi13779221-16 2024-09-21T08:27:25.670 INFO:tasks.rados.rados.0.smithi137.stdout:923: oid 16 updating version 0 to 111 2024-09-21T08:27:25.670 INFO:tasks.rados.rados.0.smithi137.stdout:923: oid 16 updating version 111 to 113 2024-09-21T08:27:25.670 INFO:tasks.rados.rados.0.smithi137.stdout:923: oid 16 version 113 is already newer than 109 2024-09-21T08:27:25.670 INFO:tasks.rados.rados.0.smithi137.stdout:923: oid 16 version 113 is already newer than 110 2024-09-21T08:27:25.671 INFO:tasks.rados.rados.0.smithi137.stdout:923: oid 16 version 113 is already newer than 112 2024-09-21T08:27:25.671 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 113 (ObjNum 347 snap 82 seq_num 347) dirty exists 2024-09-21T08:27:25.671 INFO:tasks.rados.rados.0.smithi137.stdout:923: left oid 16 (ObjNum 347 snap 82 seq_num 347) 2024-09-21T08:27:25.671 INFO:tasks.rados.rados.0.smithi137.stdout:924: finishing write tid 1 to smithi13779221-25 2024-09-21T08:27:25.671 INFO:tasks.rados.rados.0.smithi137.stdout:924: finishing write tid 2 to smithi13779221-25 2024-09-21T08:27:25.672 INFO:tasks.rados.rados.0.smithi137.stdout:924: finishing write tid 3 to smithi13779221-25 2024-09-21T08:27:25.672 INFO:tasks.rados.rados.0.smithi137.stdout:923: done (2 left) 2024-09-21T08:27:25.672 INFO:tasks.rados.rados.0.smithi137.stdout:925: done (1 left) 2024-09-21T08:27:25.672 INFO:tasks.rados.rados.0.smithi137.stdout:926: snap_create 2024-09-21T08:27:25.673 INFO:tasks.rados.rados.0.smithi137.stdout:924: finishing write tid 4 to smithi13779221-25 2024-09-21T08:27:25.673 INFO:tasks.rados.rados.0.smithi137.stdout:924: finishing write tid 5 to smithi13779221-25 2024-09-21T08:27:25.673 INFO:tasks.rados.rados.0.smithi137.stdout:924: oid 25 updating version 0 to 28 2024-09-21T08:27:25.673 INFO:tasks.rados.rados.0.smithi137.stdout:924: oid 25 version 28 is already newer than 26 2024-09-21T08:27:25.673 INFO:tasks.rados.rados.0.smithi137.stdout:924: oid 25 version 28 is already newer than 27 2024-09-21T08:27:25.673 INFO:tasks.rados.rados.0.smithi137.stdout:924: oid 25 version 28 is already newer than 25 2024-09-21T08:27:25.673 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 28 (ObjNum 348 snap 82 seq_num 348) dirty exists 2024-09-21T08:27:25.673 INFO:tasks.rados.rados.0.smithi137.stdout:924: left oid 25 (ObjNum 348 snap 82 seq_num 348) 2024-09-21T08:27:25.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:25.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:25.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:25 smithi137 ceph-mon[29634]: pgmap v281: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 226 MiB data, 691 MiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 3.9 MiB/s wr, 18 op/s; 14 MiB/s, 21 objects/s recovering 2024-09-21T08:27:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:25 smithi137 ceph-mon[41044]: pgmap v281: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 226 MiB data, 691 MiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 3.9 MiB/s wr, 18 op/s; 14 MiB/s, 21 objects/s recovering 2024-09-21T08:27:25.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:25.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:25.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:25.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:25.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:25.917 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:25 smithi165 ceph-mon[32893]: pgmap v281: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 226 MiB data, 691 MiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 3.9 MiB/s wr, 18 op/s; 14 MiB/s, 21 objects/s recovering 2024-09-21T08:27:25.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:25.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:25.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:25.945 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T08:27:26.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:26.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:26.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:26.050 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:26.120 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:26.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:26.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:26.173 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:26.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:26.234 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:26.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:26.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:26.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:26.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:26.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:26.420 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:26.420 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:26.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:26.542 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:26.542 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:26.542 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:26.546 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 6, 1, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:27:26.546 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:27:26.547 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-21T08:27:26.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:27:26.640 INFO:tasks.rados.rados.0.smithi137.stdout:924: done (1 left) 2024-09-21T08:27:26.640 INFO:tasks.rados.rados.0.smithi137.stdout:926: done (0 left) 2024-09-21T08:27:26.640 INFO:tasks.rados.rados.0.smithi137.stdout:927: read oid 2 snap -1 2024-09-21T08:27:26.640 INFO:tasks.rados.rados.0.smithi137.stdout:927: expect (ObjNum 302 snap 70 seq_num 302) 2024-09-21T08:27:26.640 INFO:tasks.rados.rados.0.smithi137.stdout:928: delete oid 20 current snap is 83 2024-09-21T08:27:26.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:26.646 INFO:tasks.rados.rados.0.smithi137.stdout:928: done (1 left) 2024-09-21T08:27:26.646 INFO:tasks.rados.rados.0.smithi137.stdout:929: copy_from oid 38 from oid 19 current snap is 83 2024-09-21T08:27:26.646 INFO:tasks.rados.rados.0.smithi137.stdout:930: read oid 20 snap -1 2024-09-21T08:27:26.646 INFO:tasks.rados.rados.0.smithi137.stdout:930: expect deleted 2024-09-21T08:27:26.646 INFO:tasks.rados.rados.0.smithi137.stdout:931: read oid 6 snap 81 2024-09-21T08:27:26.646 INFO:tasks.rados.rados.0.smithi137.stdout:931: expect (ObjNum 312 snap 70 seq_num 312) 2024-09-21T08:27:26.647 INFO:tasks.rados.rados.0.smithi137.stdout:932: delete oid 4 current snap is 83 2024-09-21T08:27:26.648 INFO:tasks.rados.rados.0.smithi137.stdout:930: done (4 left) 2024-09-21T08:27:26.648 INFO:tasks.rados.rados.0.smithi137.stdout:932: done (3 left) 2024-09-21T08:27:26.648 INFO:tasks.rados.rados.0.smithi137.stdout:933: read oid 47 snap -1 2024-09-21T08:27:26.648 INFO:tasks.rados.rados.0.smithi137.stdout:933: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:27:26.648 INFO:tasks.rados.rados.0.smithi137.stdout:934: snap_remove snap 82 2024-09-21T08:27:26.668 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:26.669 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:26.754 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:26.754 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:26.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:26.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:26.795 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:26.854 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:26 smithi137 ceph-mon[29634]: osdmap e252: 8 total, 8 up, 4 in 2024-09-21T08:27:26.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:26 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:26.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:26 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:26.855 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:26 smithi137 ceph-mon[41044]: osdmap e252: 8 total, 8 up, 4 in 2024-09-21T08:27:26.855 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:26 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:26.855 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:26 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:26.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:26.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:26.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:26.918 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:26.955 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:26.955 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:26.971 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:26.971 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:26.989 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:26.989 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:26 smithi165 ceph-mon[32893]: osdmap e252: 8 total, 8 up, 4 in 2024-09-21T08:27:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:26 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:26 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:27.013 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:27.014 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:27.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:27.030 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:27.031 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:27.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:27.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:27.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:27.044 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:27.049 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:27.049 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:27.072 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:27.072 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:27.089 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:27.089 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:27.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:27.165 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:27.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:27.267 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:27.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:27.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:27.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:27.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:27.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:27.427 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:27.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T08:27:27.528 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:27.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:27.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:27.566 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:27.642 INFO:tasks.rados.rados.0.smithi137.stdout:934: done (4 left) 2024-09-21T08:27:27.642 INFO:tasks.rados.rados.0.smithi137.stdout:935: write oid 32 current snap is 83 2024-09-21T08:27:27.642 INFO:tasks.rados.rados.0.smithi137.stdout:927: expect (ObjNum 302 snap 70 seq_num 302) 2024-09-21T08:27:27.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:27.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:27.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:27.690 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:27.748 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:27 smithi165 ceph-mon[32893]: pgmap v284: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 205 MiB data, 622 MiB used, 357 GiB / 358 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-21T08:27:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:27 smithi165 ceph-mon[32893]: 4.1b scrub starts 2024-09-21T08:27:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:27 smithi165 ceph-mon[32893]: 4.1b scrub ok 2024-09-21T08:27:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:27 smithi165 ceph-mon[32893]: osdmap e253: 8 total, 8 up, 4 in 2024-09-21T08:27:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:27 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:27:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:27 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:27:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:27 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:27 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3362481631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:27.791 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:27.808 INFO:tasks.rados.rados.0.smithi137.stdout:929: finishing copy_from to smithi13779221-38 2024-09-21T08:27:27.808 INFO:tasks.rados.rados.0.smithi137.stdout:929: got expected ENOENT (src dne) 2024-09-21T08:27:27.808 INFO:tasks.rados.rados.0.smithi137.stdout:929: finishing copy_from racing read to smithi13779221-38 2024-09-21T08:27:27.808 INFO:tasks.rados.rados.0.smithi137.stdout:931: expect (ObjNum 312 snap 70 seq_num 312) 2024-09-21T08:27:27.812 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:27.812 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:27.813 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:27.870 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[29634]: pgmap v284: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 205 MiB data, 622 MiB used, 357 GiB / 358 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[29634]: 4.1b scrub starts 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[29634]: 4.1b scrub ok 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[29634]: osdmap e253: 8 total, 8 up, 4 in 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3362481631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[41044]: pgmap v284: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 205 MiB data, 622 MiB used, 357 GiB / 358 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[41044]: 4.1b scrub starts 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[41044]: 4.1b scrub ok 2024-09-21T08:27:27.871 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[41044]: osdmap e253: 8 total, 8 up, 4 in 2024-09-21T08:27:27.872 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:27:27.872 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:27:27.872 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:27.872 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:27 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3362481631' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:27.914 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:27.934 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:27.935 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:27.935 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:28.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:27:28.036 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:28.060 INFO:tasks.rados.rados.0.smithi137.stdout:933: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:27:28.060 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:28.061 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:28.061 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:28.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:28.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:28.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:28.183 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:28.269 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:27:28.271 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:27:28.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:28.296 INFO:tasks.rados.rados.0.smithi137.stdout:935: seq_num 352 ranges {716064=639886,1815191=500449,2817011=526925} 2024-09-21T08:27:28.304 INFO:tasks.rados.rados.0.smithi137.stdout:935: writing smithi13779221-32 from 716064 to 1355950 tid 1 2024-09-21T08:27:28.305 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:28.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:28.306 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:28.310 INFO:tasks.rados.rados.0.smithi137.stdout:935: writing smithi13779221-32 from 1815191 to 2315640 tid 2 2024-09-21T08:27:28.317 INFO:tasks.rados.rados.0.smithi137.stdout:935: writing smithi13779221-32 from 2817011 to 3343936 tid 3 2024-09-21T08:27:28.317 INFO:tasks.rados.rados.0.smithi137.stdout:927: done (4 left) 2024-09-21T08:27:28.317 INFO:tasks.rados.rados.0.smithi137.stdout:929: done (3 left) 2024-09-21T08:27:28.317 INFO:tasks.rados.rados.0.smithi137.stdout:931: done (2 left) 2024-09-21T08:27:28.317 INFO:tasks.rados.rados.0.smithi137.stdout:933: done (1 left) 2024-09-21T08:27:28.317 INFO:tasks.rados.rados.0.smithi137.stdout:936: read oid 35 snap -1 2024-09-21T08:27:28.317 INFO:tasks.rados.rados.0.smithi137.stdout:936: expect (ObjNum 270 snap 60 seq_num 270) 2024-09-21T08:27:28.317 INFO:tasks.rados.rados.0.smithi137.stdout:937: rollback oid 46 current snap is 83 2024-09-21T08:27:28.317 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 46 to 81 2024-09-21T08:27:28.318 INFO:tasks.rados.rados.0.smithi137.stdout:935: finishing write tid 1 to smithi13779221-32 2024-09-21T08:27:28.318 INFO:tasks.rados.rados.0.smithi137.stdout:935: finishing write tid 2 to smithi13779221-32 2024-09-21T08:27:28.318 INFO:tasks.rados.rados.0.smithi137.stdout:938: read oid 12 snap -1 2024-09-21T08:27:28.318 INFO:tasks.rados.rados.0.smithi137.stdout:938: expect (ObjNum 315 snap 72 seq_num 315) 2024-09-21T08:27:28.318 INFO:tasks.rados.rados.0.smithi137.stdout:939: write (excl) oid 48 current snap is 83 2024-09-21T08:27:28.318 INFO:tasks.rados.rados.0.smithi137.stdout:939: seq_num 353 ranges {732871=715135,1873673=644107,3244434=688437} 2024-09-21T08:27:28.324 INFO:tasks.rados.rados.0.smithi137.stdout:939: writing smithi13779221-48 from 732871 to 1448006 tid 1 2024-09-21T08:27:28.330 INFO:tasks.rados.rados.0.smithi137.stdout:939: writing smithi13779221-48 from 1873673 to 2517780 tid 2 2024-09-21T08:27:28.338 INFO:tasks.rados.rados.0.smithi137.stdout:939: writing smithi13779221-48 from 3244434 to 3932871 tid 3 2024-09-21T08:27:28.338 INFO:tasks.rados.rados.0.smithi137.stdout:940: delete oid 25 current snap is 83 2024-09-21T08:27:28.338 INFO:tasks.rados.rados.0.smithi137.stdout:937: finishing rollback tid 0 to smithi13779221-46 2024-09-21T08:27:28.338 INFO:tasks.rados.rados.0.smithi137.stdout:937: finishing rollback tid 1 to smithi13779221-46 2024-09-21T08:27:28.338 INFO:tasks.rados.rados.0.smithi137.stdout:937: finishing rollback tid 2 to smithi13779221-46 2024-09-21T08:27:28.338 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 445 (ObjNum 239 snap 49 seq_num 239) dirty exists 2024-09-21T08:27:28.338 INFO:tasks.rados.rados.0.smithi137.stdout:936: expect (ObjNum 270 snap 60 seq_num 270) 2024-09-21T08:27:28.407 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:28.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:28.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:28.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:28.489 INFO:tasks.rados.rados.0.smithi137.stdout:938: expect (ObjNum 315 snap 72 seq_num 315) 2024-09-21T08:27:28.490 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:27:28.490 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:27:28.491 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:27:28.491 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:27:28.530 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:28.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:28.551 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:28.552 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:28.638 INFO:tasks.rados.rados.0.smithi137.stdout:935: finishing write tid 3 to smithi13779221-32 2024-09-21T08:27:28.638 INFO:tasks.rados.rados.0.smithi137.stdout:935: finishing write tid 4 to smithi13779221-32 2024-09-21T08:27:28.638 INFO:tasks.rados.rados.0.smithi137.stdout:935: finishing write tid 5 to smithi13779221-32 2024-09-21T08:27:28.638 INFO:tasks.rados.rados.0.smithi137.stdout:935: oid 32 updating version 0 to 206 2024-09-21T08:27:28.638 INFO:tasks.rados.rados.0.smithi137.stdout:935: oid 32 updating version 206 to 207 2024-09-21T08:27:28.638 INFO:tasks.rados.rados.0.smithi137.stdout:935: oid 32 updating version 207 to 209 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:935: oid 32 version 209 is already newer than 208 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 209 (ObjNum 352 snap 83 seq_num 352) dirty exists 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:935: left oid 32 (ObjNum 352 snap 83 seq_num 352) 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:939: finishing write tid 1 to smithi13779221-48 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:935: done (5 left) 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:936: done (4 left) 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:937: done (3 left) 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:938: done (2 left) 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:940: done (1 left) 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:941: rollback oid 20 current snap is 83 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 20 to 81 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:939: finishing write tid 2 to smithi13779221-48 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:939: finishing write tid 3 to smithi13779221-48 2024-09-21T08:27:28.639 INFO:tasks.rados.rados.0.smithi137.stdout:939: finishing write tid 4 to smithi13779221-48 2024-09-21T08:27:28.640 INFO:tasks.rados.rados.0.smithi137.stdout:939: finishing write tid 5 to smithi13779221-48 2024-09-21T08:27:28.640 INFO:tasks.rados.rados.0.smithi137.stdout:939: oid 48 updating version 0 to 291 2024-09-21T08:27:28.640 INFO:tasks.rados.rados.0.smithi137.stdout:939: oid 48 version 291 is already newer than 289 2024-09-21T08:27:28.640 INFO:tasks.rados.rados.0.smithi137.stdout:939: oid 48 version 291 is already newer than 290 2024-09-21T08:27:28.640 INFO:tasks.rados.rados.0.smithi137.stdout:939: oid 48 version 291 is already newer than 288 2024-09-21T08:27:28.640 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 291 (ObjNum 353 snap 83 seq_num 353) dirty exists 2024-09-21T08:27:28.640 INFO:tasks.rados.rados.0.smithi137.stdout:939: left oid 48 (ObjNum 353 snap 83 seq_num 353) 2024-09-21T08:27:28.640 INFO:tasks.rados.rados.0.smithi137.stdout:939: done (1 left) 2024-09-21T08:27:28.640 INFO:tasks.rados.rados.0.smithi137.stdout:942: snap_remove snap 80 2024-09-21T08:27:28.652 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:28.673 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:28.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:28.674 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:28.692 INFO:tasks.rados.rados.0.smithi137.stdout:942: done (1 left) 2024-09-21T08:27:28.692 INFO:tasks.rados.rados.0.smithi137.stdout:943: rollback oid 7 current snap is 83 2024-09-21T08:27:28.692 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 7 to 81 2024-09-21T08:27:28.692 INFO:tasks.rados.rados.0.smithi137.stdout:941: finishing rollback tid 1 to smithi13779221-20 2024-09-21T08:27:28.692 INFO:tasks.rados.rados.0.smithi137.stdout:941: finishing rollback tid 2 to smithi13779221-20 2024-09-21T08:27:28.692 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 320 (ObjNum 301 snap 70 seq_num 301) dirty exists 2024-09-21T08:27:28.693 INFO:tasks.rados.rados.0.smithi137.stdout:941: done (1 left) 2024-09-21T08:27:28.693 INFO:tasks.rados.rados.0.smithi137.stdout:944: write oid 3 current snap is 83 2024-09-21T08:27:28.693 INFO:tasks.rados.rados.0.smithi137.stdout:944: seq_num 355 ranges {766485=465633,1990637=647776,3299280=667205} 2024-09-21T08:27:28.700 INFO:tasks.rados.rados.0.smithi137.stdout:944: writing smithi13779221-3 from 766485 to 1232118 tid 1 2024-09-21T08:27:28.707 INFO:tasks.rados.rados.0.smithi137.stdout:944: writing smithi13779221-3 from 1990637 to 2638413 tid 2 2024-09-21T08:27:28.715 INFO:tasks.rados.rados.0.smithi137.stdout:944: writing smithi13779221-3 from 3299280 to 3966485 tid 3 2024-09-21T08:27:28.715 INFO:tasks.rados.rados.0.smithi137.stdout:945: snap_create 2024-09-21T08:27:28.715 INFO:tasks.rados.rados.0.smithi137.stdout:943: finishing rollback tid 0 to smithi13779221-7 2024-09-21T08:27:28.715 INFO:tasks.rados.rados.0.smithi137.stdout:943: finishing rollback tid 1 to smithi13779221-7 2024-09-21T08:27:28.715 INFO:tasks.rados.rados.0.smithi137.stdout:943: finishing rollback tid 2 to smithi13779221-7 2024-09-21T08:27:28.715 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 451 (ObjNum 263 snap 56 seq_num 263) dirty exists 2024-09-21T08:27:28.715 INFO:tasks.rados.rados.0.smithi137.stdout:944: finishing write tid 1 to smithi13779221-3 2024-09-21T08:27:28.715 INFO:tasks.rados.rados.0.smithi137.stdout:944: finishing write tid 2 to smithi13779221-3 2024-09-21T08:27:28.718 INFO:tasks.rados.rados.0.smithi137.stdout:944: finishing write tid 3 to smithi13779221-3 2024-09-21T08:27:28.719 INFO:tasks.rados.rados.0.smithi137.stdout:944: finishing write tid 4 to smithi13779221-3 2024-09-21T08:27:28.719 INFO:tasks.rados.rados.0.smithi137.stdout:944: finishing write tid 5 to smithi13779221-3 2024-09-21T08:27:28.719 INFO:tasks.rados.rados.0.smithi137.stdout:944: oid 3 updating version 0 to 177 2024-09-21T08:27:28.719 INFO:tasks.rados.rados.0.smithi137.stdout:944: oid 3 version 177 is already newer than 175 2024-09-21T08:27:28.719 INFO:tasks.rados.rados.0.smithi137.stdout:944: oid 3 version 177 is already newer than 176 2024-09-21T08:27:28.719 INFO:tasks.rados.rados.0.smithi137.stdout:944: oid 3 updating version 177 to 178 2024-09-21T08:27:28.719 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 178 (ObjNum 355 snap 83 seq_num 355) dirty exists 2024-09-21T08:27:28.719 INFO:tasks.rados.rados.0.smithi137.stdout:944: left oid 3 (ObjNum 355 snap 83 seq_num 355) 2024-09-21T08:27:28.728 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:28.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:28.796 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:28.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:28.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:28.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:28.919 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:28.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:28.920 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:28 smithi165 ceph-mon[32893]: osdmap e254: 8 total, 8 up, 4 in 2024-09-21T08:27:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:28 smithi165 ceph-mon[32893]: 5.3 scrub starts 2024-09-21T08:27:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:28 smithi165 ceph-mon[32893]: 5.3 scrub ok 2024-09-21T08:27:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:28 smithi137 ceph-mon[29634]: osdmap e254: 8 total, 8 up, 4 in 2024-09-21T08:27:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:28 smithi137 ceph-mon[29634]: 5.3 scrub starts 2024-09-21T08:27:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:28 smithi137 ceph-mon[29634]: 5.3 scrub ok 2024-09-21T08:27:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:28 smithi137 ceph-mon[41044]: osdmap e254: 8 total, 8 up, 4 in 2024-09-21T08:27:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:28 smithi137 ceph-mon[41044]: 5.3 scrub starts 2024-09-21T08:27:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:28 smithi137 ceph-mon[41044]: 5.3 scrub ok 2024-09-21T08:27:29.020 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:29.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:29.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:29.042 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:29.104 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":287,"stamp":"2024-09-21T08:27:28.078507+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":214460861,"num_objects":289,"num_object_clones":46,"num_object_copies":867,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":289,"num_whiteouts":6,"num_read":2197,"num_read_kb":1099764,"num_write":1353,"num_write_kb":286060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":835,"num_bytes_recovered":215640067,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4483,"ondisk_log_size":4483,"up":474,"acting":474,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":474,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":637304,"kb_used_data":477848,"kb_used_omap":6,"kb_used_meta":153785,"kb_avail":374343304,"statfs":{"total":383980142592,"available":383327543296,"internally_reserved":0,"allocated":489316352,"data_stored":631341015,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6375,"internal_metadata":157476633},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-22444037,"num_objects":-14,"num_object_clones":-12,"num_object_copies":-42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-14,"num_whiteouts":-2,"num_read":35,"num_read_kb":26416,"num_write":41,"num_write_kb":12562,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001100"},"pg_stats":[{"pgid":"3.1f","version":"0'0","reported_seq":391,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628802+0000","last_change":"2024-09-21T08:25:38.378517+0000","last_active":"2024-09-21T08:27:25.628802+0000","last_peered":"2024-09-21T08:27:25.628802+0000","last_clean":"2024-09-21T08:27:25.628802+0000","last_became_active":"2024-09-21T08:23:30.549432+0000","last_became_peered":"2024-09-21T08:23:30.549432+0000","last_unstale":"2024-09-21T08:27:25.628802+0000","last_undegraded":"2024-09-21T08:27:25.628802+0000","last_fullsized":"2024-09-21T08:27:25.628802+0000","mapping_epoch":65,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":66,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.997256+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":254,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636590+0000","last_change":"2024-09-21T08:27:15.659666+0000","last_active":"2024-09-21T08:27:25.636590+0000","last_peered":"2024-09-21T08:27:25.636590+0000","last_clean":"2024-09-21T08:27:25.636590+0000","last_became_active":"2024-09-21T08:27:15.614942+0000","last_became_peered":"2024-09-21T08:27:15.614942+0000","last_unstale":"2024-09-21T08:27:25.636590+0000","last_undegraded":"2024-09-21T08:27:25.636590+0000","last_fullsized":"2024-09-21T08:27:25.636590+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:24:50.572709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.572709+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":206,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.638792+0000","last_change":"2024-09-21T08:25:48.912078+0000","last_active":"2024-09-21T08:27:25.638792+0000","last_peered":"2024-09-21T08:27:25.638792+0000","last_clean":"2024-09-21T08:27:25.638792+0000","last_became_active":"2024-09-21T08:25:48.911674+0000","last_became_peered":"2024-09-21T08:25:48.911674+0000","last_unstale":"2024-09-21T08:27:25.638792+0000","last_undegraded":"2024-09-21T08:27:25.638792+0000","last_fullsized":"2024-09-21T08:27:25.638792+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":354,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639889+0000","last_change":"2024-09-21T08:25:48.914509+0000","last_active":"2024-09-21T08:27:25.639889+0000","last_peered":"2024-09-21T08:27:25.639889+0000","last_clean":"2024-09-21T08:27:25.639889+0000","last_became_active":"2024-09-21T08:25:48.911853+0000","last_became_peered":"2024-09-21T08:25:48.911853+0000","last_unstale":"2024-09-21T08:27:25.639889+0000","last_undegraded":"2024-09-21T08:27:25.639889+0000","last_fullsized":"2024-09-21T08:27:25.639889+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1b","version":"0'0","reported_seq":291,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640421+0000","last_change":"2024-09-21T08:27:15.611319+0000","last_active":"2024-09-21T08:27:25.640421+0000","last_peered":"2024-09-21T08:27:25.640421+0000","last_clean":"2024-09-21T08:27:25.640421+0000","last_became_active":"2024-09-21T08:27:15.611005+0000","last_became_peered":"2024-09-21T08:27:15.611005+0000","last_unstale":"2024-09-21T08:27:25.640421+0000","last_undegraded":"2024-09-21T08:27:25.640421+0000","last_fullsized":"2024-09-21T08:27:25.640421+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":282,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.638247+0000","last_change":"2024-09-21T08:27:15.614080+0000","last_active":"2024-09-21T08:27:25.638247+0000","last_peered":"2024-09-21T08:27:25.638247+0000","last_clean":"2024-09-21T08:27:25.638247+0000","last_became_active":"2024-09-21T08:27:15.613763+0000","last_became_peered":"2024-09-21T08:27:15.613763+0000","last_unstale":"2024-09-21T08:27:25.638247+0000","last_undegraded":"2024-09-21T08:27:25.638247+0000","last_fullsized":"2024-09-21T08:27:25.638247+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:19.906906+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:19.906906+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":365,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636172+0000","last_change":"2024-09-21T08:27:15.692024+0000","last_active":"2024-09-21T08:27:25.636172+0000","last_peered":"2024-09-21T08:27:25.636172+0000","last_clean":"2024-09-21T08:27:25.636172+0000","last_became_active":"2024-09-21T08:27:15.611899+0000","last_became_peered":"2024-09-21T08:27:15.611899+0000","last_unstale":"2024-09-21T08:27:25.636172+0000","last_undegraded":"2024-09-21T08:27:25.636172+0000","last_fullsized":"2024-09-21T08:27:25.636172+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:25:20.941990+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:20.941990+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":211,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636118+0000","last_change":"2024-09-21T08:25:48.912528+0000","last_active":"2024-09-21T08:27:25.636118+0000","last_peered":"2024-09-21T08:27:25.636118+0000","last_clean":"2024-09-21T08:27:25.636118+0000","last_became_active":"2024-09-21T08:25:48.911583+0000","last_became_peered":"2024-09-21T08:25:48.911583+0000","last_unstale":"2024-09-21T08:27:25.636118+0000","last_undegraded":"2024-09-21T08:27:25.636118+0000","last_fullsized":"2024-09-21T08:27:25.636118+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.640831+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.640831+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.19","version":"246'187","reported_seq":1125,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640230+0000","last_change":"2024-09-21T08:27:23.653187+0000","last_active":"2024-09-21T08:27:25.640230+0000","last_peered":"2024-09-21T08:27:25.640230+0000","last_clean":"2024-09-21T08:27:25.640230+0000","last_became_active":"2024-09-21T08:26:20.935479+0000","last_became_peered":"2024-09-21T08:26:20.935479+0000","last_unstale":"2024-09-21T08:27:25.640230+0000","last_undegraded":"2024-09-21T08:27:25.640230+0000","last_fullsized":"2024-09-21T08:27:25.640230+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'167","last_scrub_stamp":"2024-09-21T08:26:26.899804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:26.899804+0000","objects_scrubbed":1,"log_size":187,"log_dups_size":0,"ondisk_log_size":187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:54.226859+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0012739579999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":18,"num_read_kb":25352,"num_write":2,"num_write_kb":3168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"b"}]},{"pgid":"3.1d","version":"0'0","reported_seq":425,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641178+0000","last_change":"2024-09-21T08:27:15.611079+0000","last_active":"2024-09-21T08:27:25.641178+0000","last_peered":"2024-09-21T08:27:25.641178+0000","last_clean":"2024-09-21T08:27:25.641178+0000","last_became_active":"2024-09-21T08:27:15.610791+0000","last_became_peered":"2024-09-21T08:27:15.610791+0000","last_unstale":"2024-09-21T08:27:25.641178+0000","last_undegraded":"2024-09-21T08:27:25.641178+0000","last_fullsized":"2024-09-21T08:27:25.641178+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.780545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.780545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:48.434470+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":351,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627769+0000","last_change":"2024-09-21T08:27:15.688594+0000","last_active":"2024-09-21T08:27:25.627769+0000","last_peered":"2024-09-21T08:27:25.627769+0000","last_clean":"2024-09-21T08:27:25.627769+0000","last_became_active":"2024-09-21T08:27:15.631743+0000","last_became_peered":"2024-09-21T08:27:15.631743+0000","last_unstale":"2024-09-21T08:27:25.627769+0000","last_undegraded":"2024-09-21T08:27:25.627769+0000","last_fullsized":"2024-09-21T08:27:25.627769+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":384,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640759+0000","last_change":"2024-09-21T08:25:22.581281+0000","last_active":"2024-09-21T08:27:25.640759+0000","last_peered":"2024-09-21T08:27:25.640759+0000","last_clean":"2024-09-21T08:27:25.640759+0000","last_became_active":"2024-09-21T08:23:34.567581+0000","last_became_peered":"2024-09-21T08:23:34.567581+0000","last_unstale":"2024-09-21T08:27:25.640759+0000","last_undegraded":"2024-09-21T08:27:25.640759+0000","last_fullsized":"2024-09-21T08:27:25.640759+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":377,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627031+0000","last_change":"2024-09-21T08:25:48.885985+0000","last_active":"2024-09-21T08:27:25.627031+0000","last_peered":"2024-09-21T08:27:25.627031+0000","last_clean":"2024-09-21T08:27:25.627031+0000","last_became_active":"2024-09-21T08:25:48.885136+0000","last_became_peered":"2024-09-21T08:25:48.885136+0000","last_unstale":"2024-09-21T08:27:25.627031+0000","last_undegraded":"2024-09-21T08:27:25.627031+0000","last_fullsized":"2024-09-21T08:27:25.627031+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.18","version":"251'186","reported_seq":1048,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.626793+0000","last_change":"2024-09-21T08:27:21.645300+0000","last_active":"2024-09-21T08:27:25.626793+0000","last_peered":"2024-09-21T08:27:25.626793+0000","last_clean":"2024-09-21T08:27:25.626793+0000","last_became_active":"2024-09-21T08:26:20.935965+0000","last_became_peered":"2024-09-21T08:26:20.935965+0000","last_unstale":"2024-09-21T08:27:25.626793+0000","last_undegraded":"2024-09-21T08:27:25.626793+0000","last_fullsized":"2024-09-21T08:27:25.626793+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"190'170","last_scrub_stamp":"2024-09-21T08:26:28.392516+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.392516+0000","objects_scrubbed":2,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:40.245103+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.11689789,"stat_sum":{"num_bytes":8653319,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":2,"num_read":14,"num_read_kb":16665,"num_write":10,"num_write_kb":6845,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":410,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639916+0000","last_change":"2024-09-21T08:25:48.912437+0000","last_active":"2024-09-21T08:27:25.639916+0000","last_peered":"2024-09-21T08:27:25.639916+0000","last_clean":"2024-09-21T08:27:25.639916+0000","last_became_active":"2024-09-21T08:25:48.912043+0000","last_became_peered":"2024-09-21T08:25:48.912043+0000","last_unstale":"2024-09-21T08:27:25.639916+0000","last_undegraded":"2024-09-21T08:27:25.639916+0000","last_fullsized":"2024-09-21T08:27:25.639916+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":203,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636746+0000","last_change":"2024-09-21T08:27:15.671454+0000","last_active":"2024-09-21T08:27:25.636746+0000","last_peered":"2024-09-21T08:27:25.636746+0000","last_clean":"2024-09-21T08:27:25.636746+0000","last_became_active":"2024-09-21T08:27:15.616942+0000","last_became_peered":"2024-09-21T08:27:15.616942+0000","last_unstale":"2024-09-21T08:27:25.636746+0000","last_undegraded":"2024-09-21T08:27:25.636746+0000","last_fullsized":"2024-09-21T08:27:25.636746+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:24:32.569846+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:32.569846+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":405,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639602+0000","last_change":"2024-09-21T08:27:15.615488+0000","last_active":"2024-09-21T08:27:25.639602+0000","last_peered":"2024-09-21T08:27:25.639602+0000","last_clean":"2024-09-21T08:27:25.639602+0000","last_became_active":"2024-09-21T08:27:15.615138+0000","last_became_peered":"2024-09-21T08:27:15.615138+0000","last_unstale":"2024-09-21T08:27:25.639602+0000","last_undegraded":"2024-09-21T08:27:25.639602+0000","last_fullsized":"2024-09-21T08:27:25.639602+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_clean_scrub_stamp":"2024-09-21T08:26:09.782307+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:33.954541+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":413,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.638909+0000","last_change":"2024-09-21T08:27:15.647024+0000","last_active":"2024-09-21T08:27:25.638909+0000","last_peered":"2024-09-21T08:27:25.638909+0000","last_clean":"2024-09-21T08:27:25.638909+0000","last_became_active":"2024-09-21T08:27:15.607178+0000","last_became_peered":"2024-09-21T08:27:15.607178+0000","last_unstale":"2024-09-21T08:27:25.638909+0000","last_undegraded":"2024-09-21T08:27:25.638909+0000","last_fullsized":"2024-09-21T08:27:25.638909+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1e","version":"0'0","reported_seq":361,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641776+0000","last_change":"2024-09-21T08:27:15.609850+0000","last_active":"2024-09-21T08:27:25.641776+0000","last_peered":"2024-09-21T08:27:25.641776+0000","last_clean":"2024-09-21T08:27:25.641776+0000","last_became_active":"2024-09-21T08:27:15.609207+0000","last_became_peered":"2024-09-21T08:27:15.609207+0000","last_unstale":"2024-09-21T08:27:25.641776+0000","last_undegraded":"2024-09-21T08:27:25.641776+0000","last_fullsized":"2024-09-21T08:27:25.641776+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.855448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.855448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":31,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627476+0000","last_change":"2024-09-21T08:27:15.639463+0000","last_active":"2024-09-21T08:27:25.627476+0000","last_peered":"2024-09-21T08:27:25.627476+0000","last_clean":"2024-09-21T08:27:25.627476+0000","last_became_active":"2024-09-21T08:27:15.636955+0000","last_became_peered":"2024-09-21T08:27:15.636955+0000","last_unstale":"2024-09-21T08:27:25.627476+0000","last_undegraded":"2024-09-21T08:27:25.627476+0000","last_fullsized":"2024-09-21T08:27:25.627476+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.997079+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":508,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.637024+0000","last_change":"2024-09-21T08:27:15.675429+0000","last_active":"2024-09-21T08:27:25.637024+0000","last_peered":"2024-09-21T08:27:25.637024+0000","last_clean":"2024-09-21T08:27:25.637024+0000","last_became_active":"2024-09-21T08:27:15.616176+0000","last_became_peered":"2024-09-21T08:27:15.616176+0000","last_unstale":"2024-09-21T08:27:25.637024+0000","last_undegraded":"2024-09-21T08:27:25.637024+0000","last_fullsized":"2024-09-21T08:27:25.637024+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:26:45.263994+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.263994+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:57.215432+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":263,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628225+0000","last_change":"2024-09-21T08:27:15.626975+0000","last_active":"2024-09-21T08:27:25.628225+0000","last_peered":"2024-09-21T08:27:25.628225+0000","last_clean":"2024-09-21T08:27:25.628225+0000","last_became_active":"2024-09-21T08:27:15.624297+0000","last_became_peered":"2024-09-21T08:27:15.624297+0000","last_unstale":"2024-09-21T08:27:25.628225+0000","last_undegraded":"2024-09-21T08:27:25.628225+0000","last_fullsized":"2024-09-21T08:27:25.628225+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:30.310628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.310628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:36.795462+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1f","version":"0'0","reported_seq":273,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641840+0000","last_change":"2024-09-21T08:27:15.610411+0000","last_active":"2024-09-21T08:27:25.641840+0000","last_peered":"2024-09-21T08:27:25.641840+0000","last_clean":"2024-09-21T08:27:25.641840+0000","last_became_active":"2024-09-21T08:27:15.610086+0000","last_became_peered":"2024-09-21T08:27:15.610086+0000","last_unstale":"2024-09-21T08:27:25.641840+0000","last_undegraded":"2024-09-21T08:27:25.641840+0000","last_fullsized":"2024-09-21T08:27:25.641840+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:10.959560+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.959560+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:26.011814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":216,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636211+0000","last_change":"2024-09-21T08:26:33.237763+0000","last_active":"2024-09-21T08:27:25.636211+0000","last_peered":"2024-09-21T08:27:25.636211+0000","last_clean":"2024-09-21T08:27:25.636211+0000","last_became_active":"2024-09-21T08:25:48.899796+0000","last_became_peered":"2024-09-21T08:25:48.899796+0000","last_unstale":"2024-09-21T08:27:25.636211+0000","last_undegraded":"2024-09-21T08:27:25.636211+0000","last_fullsized":"2024-09-21T08:27:25.636211+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:33.237691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.237691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:57.612618+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":228,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641430+0000","last_change":"2024-09-21T08:27:15.637340+0000","last_active":"2024-09-21T08:27:25.641430+0000","last_peered":"2024-09-21T08:27:25.641430+0000","last_clean":"2024-09-21T08:27:25.641430+0000","last_became_active":"2024-09-21T08:27:15.600504+0000","last_became_peered":"2024-09-21T08:27:15.600504+0000","last_unstale":"2024-09-21T08:27:25.641430+0000","last_undegraded":"2024-09-21T08:27:25.641430+0000","last_fullsized":"2024-09-21T08:27:25.641430+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:21.424414+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.424414+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":297,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.637626+0000","last_change":"2024-09-21T08:27:15.619235+0000","last_active":"2024-09-21T08:27:25.637626+0000","last_peered":"2024-09-21T08:27:25.637626+0000","last_clean":"2024-09-21T08:27:25.637626+0000","last_became_active":"2024-09-21T08:27:15.615743+0000","last_became_peered":"2024-09-21T08:27:15.615743+0000","last_unstale":"2024-09-21T08:27:25.637626+0000","last_undegraded":"2024-09-21T08:27:25.637626+0000","last_fullsized":"2024-09-21T08:27:25.637626+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.797084+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.1c","version":"73'1","reported_seq":378,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640737+0000","last_change":"2024-09-21T08:27:14.826171+0000","last_active":"2024-09-21T08:27:25.640737+0000","last_peered":"2024-09-21T08:27:25.640737+0000","last_clean":"2024-09-21T08:27:25.640737+0000","last_became_active":"2024-09-21T08:23:36.583701+0000","last_became_peered":"2024-09-21T08:23:36.583701+0000","last_unstale":"2024-09-21T08:27:25.640737+0000","last_undegraded":"2024-09-21T08:27:25.640737+0000","last_fullsized":"2024-09-21T08:27:25.640737+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:27:14.826047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.826047+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:16.123167+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.19","version":"198'2","reported_seq":252,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640047+0000","last_change":"2024-09-21T08:27:15.617663+0000","last_active":"2024-09-21T08:27:25.640047+0000","last_peered":"2024-09-21T08:27:25.640047+0000","last_clean":"2024-09-21T08:27:25.640047+0000","last_became_active":"2024-09-21T08:27:15.593659+0000","last_became_peered":"2024-09-21T08:27:15.593659+0000","last_unstale":"2024-09-21T08:27:25.640047+0000","last_undegraded":"2024-09-21T08:27:25.640047+0000","last_fullsized":"2024-09-21T08:27:25.640047+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'2","last_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_deep_scrub":"198'2","last_deep_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_clean_scrub_stamp":"2024-09-21T08:26:32.819313+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:44.254778+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":466,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628801+0000","last_change":"2024-09-21T08:27:15.656240+0000","last_active":"2024-09-21T08:27:25.628801+0000","last_peered":"2024-09-21T08:27:25.628801+0000","last_clean":"2024-09-21T08:27:25.628801+0000","last_became_active":"2024-09-21T08:27:15.624846+0000","last_became_peered":"2024-09-21T08:27:15.624846+0000","last_unstale":"2024-09-21T08:27:25.628801+0000","last_undegraded":"2024-09-21T08:27:25.628801+0000","last_fullsized":"2024-09-21T08:27:25.628801+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.1f","version":"242'14","reported_seq":486,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.642289+0000","last_change":"2024-09-21T08:27:15.681788+0000","last_active":"2024-09-21T08:27:25.642289+0000","last_peered":"2024-09-21T08:27:25.642289+0000","last_clean":"2024-09-21T08:27:25.642289+0000","last_became_active":"2024-09-21T08:27:15.613508+0000","last_became_peered":"2024-09-21T08:27:15.613508+0000","last_unstale":"2024-09-21T08:27:25.642289+0000","last_undegraded":"2024-09-21T08:27:25.642289+0000","last_fullsized":"2024-09-21T08:27:25.642289+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"98'10","last_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.594212+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.c","version":"250'155","reported_seq":1024,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.639206+0000","last_change":"2024-09-21T08:27:25.618887+0000","last_active":"2024-09-21T08:27:25.639206+0000","last_peered":"2024-09-21T08:27:25.639206+0000","last_clean":"2024-09-21T08:27:25.639206+0000","last_became_active":"2024-09-21T08:27:18.606086+0000","last_became_peered":"2024-09-21T08:27:18.606086+0000","last_unstale":"2024-09-21T08:27:25.639206+0000","last_undegraded":"2024-09-21T08:27:25.639206+0000","last_fullsized":"2024-09-21T08:27:25.639206+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"196'119","last_scrub_stamp":"2024-09-21T08:26:43.765848+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:43.765848+0000","objects_scrubbed":0,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:04.573067+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0869139059999999,"stat_sum":{"num_bytes":9798547,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":138,"num_read_kb":155238,"num_write":71,"num_write_kb":39604,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":65,"num_bytes_recovered":45887778,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"7"}]},{"pgid":"4.f","version":"73'15","reported_seq":265,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.642015+0000","last_change":"2024-09-21T08:27:15.657947+0000","last_active":"2024-09-21T08:27:25.642015+0000","last_peered":"2024-09-21T08:27:25.642015+0000","last_clean":"2024-09-21T08:27:25.642015+0000","last_became_active":"2024-09-21T08:27:15.598578+0000","last_became_peered":"2024-09-21T08:27:15.598578+0000","last_unstale":"2024-09-21T08:27:25.642015+0000","last_undegraded":"2024-09-21T08:27:25.642015+0000","last_fullsized":"2024-09-21T08:27:25.642015+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:26:46.777115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.777115+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:15.315581+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":162,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640110+0000","last_change":"2024-09-21T08:27:15.601560+0000","last_active":"2024-09-21T08:27:25.640110+0000","last_peered":"2024-09-21T08:27:25.640110+0000","last_clean":"2024-09-21T08:27:25.640110+0000","last_became_active":"2024-09-21T08:27:15.601270+0000","last_became_peered":"2024-09-21T08:27:15.601270+0000","last_unstale":"2024-09-21T08:27:25.640110+0000","last_undegraded":"2024-09-21T08:27:25.640110+0000","last_fullsized":"2024-09-21T08:27:25.640110+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:45.434055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:45.434055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.e","version":"85'9","reported_seq":427,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641639+0000","last_change":"2024-09-21T08:26:44.741364+0000","last_active":"2024-09-21T08:27:25.641639+0000","last_peered":"2024-09-21T08:27:25.641639+0000","last_clean":"2024-09-21T08:27:25.641639+0000","last_became_active":"2024-09-21T08:24:38.548789+0000","last_became_peered":"2024-09-21T08:24:38.548789+0000","last_unstale":"2024-09-21T08:27:25.641639+0000","last_undegraded":"2024-09-21T08:27:25.641639+0000","last_fullsized":"2024-09-21T08:27:25.641639+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:26:44.741275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.741275+0000","objects_scrubbed":1,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:57.665735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":386,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641512+0000","last_change":"2024-09-21T08:26:08.864503+0000","last_active":"2024-09-21T08:27:25.641512+0000","last_peered":"2024-09-21T08:27:25.641512+0000","last_clean":"2024-09-21T08:27:25.641512+0000","last_became_active":"2024-09-21T08:25:48.914453+0000","last_became_peered":"2024-09-21T08:25:48.914453+0000","last_unstale":"2024-09-21T08:27:25.641512+0000","last_undegraded":"2024-09-21T08:27:25.641512+0000","last_fullsized":"2024-09-21T08:27:25.641512+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:08.864399+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:08.864399+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:29.676635+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"179'19","reported_seq":751,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.636899+0000","last_change":"2024-09-21T08:27:21.618760+0000","last_active":"2024-09-21T08:27:25.636899+0000","last_peered":"2024-09-21T08:27:25.636899+0000","last_clean":"2024-09-21T08:27:25.636899+0000","last_became_active":"2024-09-21T08:27:15.612213+0000","last_became_peered":"2024-09-21T08:27:15.612213+0000","last_unstale":"2024-09-21T08:27:25.636899+0000","last_undegraded":"2024-09-21T08:27:25.636899+0000","last_fullsized":"2024-09-21T08:27:25.636899+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:26:51.043840+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.043840+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:10.505677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.079046090999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"1"},{"start":"49","length":"6"},{"start":"50","length":"1"}]},{"pgid":"4.0","version":"73'18","reported_seq":390,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639090+0000","last_change":"2024-09-21T08:27:15.684696+0000","last_active":"2024-09-21T08:27:25.639090+0000","last_peered":"2024-09-21T08:27:25.639090+0000","last_clean":"2024-09-21T08:27:25.639090+0000","last_became_active":"2024-09-21T08:27:15.612739+0000","last_became_peered":"2024-09-21T08:27:15.612739+0000","last_unstale":"2024-09-21T08:27:25.639090+0000","last_undegraded":"2024-09-21T08:27:25.639090+0000","last_fullsized":"2024-09-21T08:27:25.639090+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-09-21T08:26:54.003173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:54.003173+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:23.009409+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":288,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.642501+0000","last_change":"2024-09-21T08:27:15.614665+0000","last_active":"2024-09-21T08:27:25.642501+0000","last_peered":"2024-09-21T08:27:25.642501+0000","last_clean":"2024-09-21T08:27:25.642501+0000","last_became_active":"2024-09-21T08:27:15.613984+0000","last_became_peered":"2024-09-21T08:27:15.613984+0000","last_unstale":"2024-09-21T08:27:25.642501+0000","last_undegraded":"2024-09-21T08:27:25.642501+0000","last_fullsized":"2024-09-21T08:27:25.642501+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":281,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639861+0000","last_change":"2024-09-21T08:27:15.614991+0000","last_active":"2024-09-21T08:27:25.639861+0000","last_peered":"2024-09-21T08:27:25.639861+0000","last_clean":"2024-09-21T08:27:25.639861+0000","last_became_active":"2024-09-21T08:27:15.614133+0000","last_became_peered":"2024-09-21T08:27:15.614133+0000","last_unstale":"2024-09-21T08:27:25.639861+0000","last_undegraded":"2024-09-21T08:27:25.639861+0000","last_fullsized":"2024-09-21T08:27:25.639861+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":298,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636906+0000","last_change":"2024-09-21T08:27:16.341677+0000","last_active":"2024-09-21T08:27:25.636906+0000","last_peered":"2024-09-21T08:27:25.636906+0000","last_clean":"2024-09-21T08:27:25.636906+0000","last_became_active":"2024-09-21T08:27:15.597861+0000","last_became_peered":"2024-09-21T08:27:15.597861+0000","last_unstale":"2024-09-21T08:27:25.636906+0000","last_undegraded":"2024-09-21T08:27:25.636906+0000","last_fullsized":"2024-09-21T08:27:25.636906+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:16.341564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.341564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:41.520530+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.2","version":"251'255","reported_seq":1194,"reported_epoch":252,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:25.637184+0000","last_change":"2024-09-21T08:27:24.617183+0000","last_active":"2024-09-21T08:27:25.637184+0000","last_peered":"2024-09-21T08:27:25.637184+0000","last_clean":"2024-09-21T08:27:25.637184+0000","last_became_active":"2024-09-21T08:27:18.604664+0000","last_became_peered":"2024-09-21T08:27:18.604664+0000","last_unstale":"2024-09-21T08:27:25.637184+0000","last_undegraded":"2024-09-21T08:27:25.637184+0000","last_fullsized":"2024-09-21T08:27:25.637184+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"195'198","last_scrub_stamp":"2024-09-21T08:26:30.265653+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.265653+0000","objects_scrubbed":2,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:36.582480+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0035162529999999,"stat_sum":{"num_bytes":15078189,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":31,"num_read_kb":26054,"num_write":31,"num_write_kb":10197,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":10943764,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"47","length":"1"},{"start":"49","length":"5"}]},{"pgid":"4.1","version":"239'21","reported_seq":440,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.638791+0000","last_change":"2024-09-21T08:27:12.969759+0000","last_active":"2024-09-21T08:27:25.638791+0000","last_peered":"2024-09-21T08:27:25.638791+0000","last_clean":"2024-09-21T08:27:25.638791+0000","last_became_active":"2024-09-21T08:24:56.509169+0000","last_became_peered":"2024-09-21T08:24:56.509169+0000","last_unstale":"2024-09-21T08:27:25.638791+0000","last_undegraded":"2024-09-21T08:27:25.638791+0000","last_fullsized":"2024-09-21T08:27:25.638791+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21","last_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_deep_scrub":"239'21","last_deep_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.969642+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:26.500333+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":412,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628399+0000","last_change":"2024-09-21T08:25:48.919957+0000","last_active":"2024-09-21T08:27:25.628399+0000","last_peered":"2024-09-21T08:27:25.628399+0000","last_clean":"2024-09-21T08:27:25.628399+0000","last_became_active":"2024-09-21T08:25:48.893688+0000","last_became_peered":"2024-09-21T08:25:48.893688+0000","last_unstale":"2024-09-21T08:27:25.628399+0000","last_undegraded":"2024-09-21T08:27:25.628399+0000","last_fullsized":"2024-09-21T08:27:25.628399+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.997115+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":92,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.0","version":"242'13","reported_seq":243,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627192+0000","last_change":"2024-09-21T08:27:15.676151+0000","last_active":"2024-09-21T08:27:25.627192+0000","last_peered":"2024-09-21T08:27:25.627192+0000","last_clean":"2024-09-21T08:27:25.627192+0000","last_became_active":"2024-09-21T08:27:15.629588+0000","last_became_peered":"2024-09-21T08:27:15.629588+0000","last_unstale":"2024-09-21T08:27:25.627192+0000","last_undegraded":"2024-09-21T08:27:25.627192+0000","last_fullsized":"2024-09-21T08:27:25.627192+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996481+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":389,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640800+0000","last_change":"2024-09-21T08:25:48.900309+0000","last_active":"2024-09-21T08:27:25.640800+0000","last_peered":"2024-09-21T08:27:25.640800+0000","last_clean":"2024-09-21T08:27:25.640800+0000","last_became_active":"2024-09-21T08:25:48.899971+0000","last_became_peered":"2024-09-21T08:25:48.899971+0000","last_unstale":"2024-09-21T08:27:25.640800+0000","last_undegraded":"2024-09-21T08:27:25.640800+0000","last_fullsized":"2024-09-21T08:27:25.640800+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:21.561826+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:21.561826+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.1","version":"179'43","reported_seq":705,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.627837+0000","last_change":"2024-09-21T08:27:20.631674+0000","last_active":"2024-09-21T08:27:25.627837+0000","last_peered":"2024-09-21T08:27:25.627837+0000","last_clean":"2024-09-21T08:27:25.627837+0000","last_became_active":"2024-09-21T08:27:15.619710+0000","last_became_peered":"2024-09-21T08:27:15.619710+0000","last_unstale":"2024-09-21T08:27:25.627837+0000","last_undegraded":"2024-09-21T08:27:25.627837+0000","last_fullsized":"2024-09-21T08:27:25.627837+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:26:32.285475+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:32.285475+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:46.362999+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0058992469999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"1"},{"start":"49","length":"4"}]},{"pgid":"4.2","version":"73'10","reported_seq":268,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639560+0000","last_change":"2024-09-21T08:27:09.901599+0000","last_active":"2024-09-21T08:27:25.639560+0000","last_peered":"2024-09-21T08:27:25.639560+0000","last_clean":"2024-09-21T08:27:25.639560+0000","last_became_active":"2024-09-21T08:25:48.894952+0000","last_became_peered":"2024-09-21T08:25:48.894952+0000","last_unstale":"2024-09-21T08:27:25.639560+0000","last_undegraded":"2024-09-21T08:27:25.639560+0000","last_fullsized":"2024-09-21T08:27:25.639560+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:27:09.901511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:09.901511+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:14.265026+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":438,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640807+0000","last_change":"2024-09-21T08:27:16.003302+0000","last_active":"2024-09-21T08:27:25.640807+0000","last_peered":"2024-09-21T08:27:25.640807+0000","last_clean":"2024-09-21T08:27:25.640807+0000","last_became_active":"2024-09-21T08:27:15.599252+0000","last_became_peered":"2024-09-21T08:27:15.599252+0000","last_unstale":"2024-09-21T08:27:25.640807+0000","last_undegraded":"2024-09-21T08:27:25.640807+0000","last_fullsized":"2024-09-21T08:27:25.640807+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:16.003225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.003225+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:38.632224+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"242'13","reported_seq":480,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628221+0000","last_change":"2024-09-21T08:27:15.646450+0000","last_active":"2024-09-21T08:27:25.628221+0000","last_peered":"2024-09-21T08:27:25.628221+0000","last_clean":"2024-09-21T08:27:25.628221+0000","last_became_active":"2024-09-21T08:27:15.621963+0000","last_became_peered":"2024-09-21T08:27:15.621963+0000","last_unstale":"2024-09-21T08:27:25.628221+0000","last_undegraded":"2024-09-21T08:27:25.628221+0000","last_fullsized":"2024-09-21T08:27:25.628221+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"176'11","last_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_deep_scrub":"176'11","last_deep_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_clean_scrub_stamp":"2024-09-21T08:26:10.440547+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:27.434285+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":389,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.626916+0000","last_change":"2024-09-21T08:27:15.629853+0000","last_active":"2024-09-21T08:27:25.626916+0000","last_peered":"2024-09-21T08:27:25.626916+0000","last_clean":"2024-09-21T08:27:25.626916+0000","last_became_active":"2024-09-21T08:27:15.628747+0000","last_became_peered":"2024-09-21T08:27:15.628747+0000","last_unstale":"2024-09-21T08:27:25.626916+0000","last_undegraded":"2024-09-21T08:27:25.626916+0000","last_fullsized":"2024-09-21T08:27:25.626916+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.0","version":"251'134","reported_seq":1087,"reported_epoch":252,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:25.639908+0000","last_change":"2024-09-21T08:27:25.618911+0000","last_active":"2024-09-21T08:27:25.639908+0000","last_peered":"2024-09-21T08:27:25.639908+0000","last_clean":"2024-09-21T08:27:25.639908+0000","last_became_active":"2024-09-21T08:27:18.606509+0000","last_became_peered":"2024-09-21T08:27:18.606509+0000","last_unstale":"2024-09-21T08:27:25.639908+0000","last_undegraded":"2024-09-21T08:27:25.639908+0000","last_fullsized":"2024-09-21T08:27:25.639908+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'100","last_scrub_stamp":"2024-09-21T08:26:26.838731+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:26.838731+0000","objects_scrubbed":1,"log_size":134,"log_dups_size":0,"ondisk_log_size":134,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:29.421528+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.071802985,"stat_sum":{"num_bytes":3075714,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":13,"num_read_kb":14915,"num_write":11,"num_write_kb":2424,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":6477620,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"1"},{"start":"49","length":"4"}]},{"pgid":"4.3","version":"73'19","reported_seq":415,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627916+0000","last_change":"2024-09-21T08:23:32.597527+0000","last_active":"2024-09-21T08:27:25.627916+0000","last_peered":"2024-09-21T08:27:25.627916+0000","last_clean":"2024-09-21T08:27:25.627916+0000","last_became_active":"2024-09-21T08:23:32.597222+0000","last_became_peered":"2024-09-21T08:23:32.597222+0000","last_unstale":"2024-09-21T08:27:25.627916+0000","last_undegraded":"2024-09-21T08:27:25.627916+0000","last_fullsized":"2024-09-21T08:27:25.627916+0000","mapping_epoch":67,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":68,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":39,"num_read_kb":25,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":207,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.637405+0000","last_change":"2024-09-21T08:25:48.906191+0000","last_active":"2024-09-21T08:27:25.637405+0000","last_peered":"2024-09-21T08:27:25.637405+0000","last_clean":"2024-09-21T08:27:25.637405+0000","last_became_active":"2024-09-21T08:25:48.905818+0000","last_became_peered":"2024-09-21T08:25:48.905818+0000","last_unstale":"2024-09-21T08:27:25.637405+0000","last_undegraded":"2024-09-21T08:27:25.637405+0000","last_fullsized":"2024-09-21T08:27:25.637405+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:52.529672+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.529672+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":318,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627898+0000","last_change":"2024-09-21T08:25:39.339196+0000","last_active":"2024-09-21T08:27:25.627898+0000","last_peered":"2024-09-21T08:27:25.627898+0000","last_clean":"2024-09-21T08:27:25.627898+0000","last_became_active":"2024-09-21T08:24:56.499628+0000","last_became_peered":"2024-09-21T08:24:56.499628+0000","last_unstale":"2024-09-21T08:27:25.627898+0000","last_undegraded":"2024-09-21T08:27:25.627898+0000","last_fullsized":"2024-09-21T08:27:25.627898+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.339126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.339126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.997285+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":283,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636271+0000","last_change":"2024-09-21T08:26:46.264057+0000","last_active":"2024-09-21T08:27:25.636271+0000","last_peered":"2024-09-21T08:27:25.636271+0000","last_clean":"2024-09-21T08:27:25.636271+0000","last_became_active":"2024-09-21T08:25:48.885120+0000","last_became_peered":"2024-09-21T08:25:48.885120+0000","last_unstale":"2024-09-21T08:27:25.636271+0000","last_undegraded":"2024-09-21T08:27:25.636271+0000","last_fullsized":"2024-09-21T08:27:25.636271+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:46.263984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.263984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:50.138045+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.7","version":"251'106","reported_seq":1015,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.648975+0000","last_change":"2024-09-21T08:27:25.649018+0000","last_active":"2024-09-21T08:27:25.648975+0000","last_peered":"2024-09-21T08:27:25.648975+0000","last_clean":"2024-09-21T08:27:25.648975+0000","last_became_active":"2024-09-21T08:27:15.638568+0000","last_became_peered":"2024-09-21T08:27:15.638568+0000","last_unstale":"2024-09-21T08:27:25.648975+0000","last_undegraded":"2024-09-21T08:27:25.648975+0000","last_fullsized":"2024-09-21T08:27:25.648975+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:06.082149+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.019851912999999999,"stat_sum":{"num_bytes":12043155,"num_objects":5,"num_object_clones":3,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":6,"num_read_kb":3,"num_write":18,"num_write_kb":7793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":16319524,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"}]},{"pgid":"4.4","version":"242'33","reported_seq":381,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.637241+0000","last_change":"2024-09-21T08:27:15.639518+0000","last_active":"2024-09-21T08:27:25.637241+0000","last_peered":"2024-09-21T08:27:25.637241+0000","last_clean":"2024-09-21T08:27:25.637241+0000","last_became_active":"2024-09-21T08:27:15.598948+0000","last_became_peered":"2024-09-21T08:27:15.598948+0000","last_unstale":"2024-09-21T08:27:25.637241+0000","last_undegraded":"2024-09-21T08:27:25.637241+0000","last_fullsized":"2024-09-21T08:27:25.637241+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'32","last_scrub_stamp":"2024-09-21T08:27:11.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:11.407384+0000","objects_scrubbed":10,"log_size":33,"log_dups_size":0,"ondisk_log_size":33,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:24.960272+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1074,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":370,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628471+0000","last_change":"2024-09-21T08:27:15.630398+0000","last_active":"2024-09-21T08:27:25.628471+0000","last_peered":"2024-09-21T08:27:25.628471+0000","last_clean":"2024-09-21T08:27:25.628471+0000","last_became_active":"2024-09-21T08:27:15.630087+0000","last_became_peered":"2024-09-21T08:27:15.630087+0000","last_unstale":"2024-09-21T08:27:25.628471+0000","last_undegraded":"2024-09-21T08:27:25.628471+0000","last_fullsized":"2024-09-21T08:27:25.628471+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":442,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639300+0000","last_change":"2024-09-21T08:27:15.668386+0000","last_active":"2024-09-21T08:27:25.639300+0000","last_peered":"2024-09-21T08:27:25.639300+0000","last_clean":"2024-09-21T08:27:25.639300+0000","last_became_active":"2024-09-21T08:27:15.609047+0000","last_became_peered":"2024-09-21T08:27:15.609047+0000","last_unstale":"2024-09-21T08:27:25.639300+0000","last_undegraded":"2024-09-21T08:27:25.639300+0000","last_fullsized":"2024-09-21T08:27:25.639300+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":76,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":413,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628401+0000","last_change":"2024-09-21T08:27:15.631829+0000","last_active":"2024-09-21T08:27:25.628401+0000","last_peered":"2024-09-21T08:27:25.628401+0000","last_clean":"2024-09-21T08:27:25.628401+0000","last_became_active":"2024-09-21T08:27:15.631334+0000","last_became_peered":"2024-09-21T08:27:15.631334+0000","last_unstale":"2024-09-21T08:27:25.628401+0000","last_undegraded":"2024-09-21T08:27:25.628401+0000","last_fullsized":"2024-09-21T08:27:25.628401+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:45.237804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.237804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:01.540105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":341,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.642343+0000","last_change":"2024-09-21T08:27:15.614251+0000","last_active":"2024-09-21T08:27:25.642343+0000","last_peered":"2024-09-21T08:27:25.642343+0000","last_clean":"2024-09-21T08:27:25.642343+0000","last_became_active":"2024-09-21T08:27:15.613921+0000","last_became_peered":"2024-09-21T08:27:15.613921+0000","last_unstale":"2024-09-21T08:27:25.642343+0000","last_undegraded":"2024-09-21T08:27:25.642343+0000","last_fullsized":"2024-09-21T08:27:25.642343+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:45.761359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.761359+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:15.701405+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.4","version":"249'68","reported_seq":975,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.628181+0000","last_change":"2024-09-21T08:27:24.635319+0000","last_active":"2024-09-21T08:27:25.628181+0000","last_peered":"2024-09-21T08:27:25.628181+0000","last_clean":"2024-09-21T08:27:25.628181+0000","last_became_active":"2024-09-21T08:27:15.633124+0000","last_became_peered":"2024-09-21T08:27:15.633124+0000","last_unstale":"2024-09-21T08:27:25.628181+0000","last_undegraded":"2024-09-21T08:27:25.628181+0000","last_fullsized":"2024-09-21T08:27:25.628181+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:26:34.278834+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:36.393730+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0.094245047999999998,"stat_sum":{"num_bytes":5697606,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":26,"num_read_kb":35598,"num_write":5,"num_write_kb":5565,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":8122836,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"1"}]},{"pgid":"4.7","version":"73'13","reported_seq":264,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641582+0000","last_change":"2024-09-21T08:25:48.937399+0000","last_active":"2024-09-21T08:27:25.641582+0000","last_peered":"2024-09-21T08:27:25.641582+0000","last_clean":"2024-09-21T08:27:25.641582+0000","last_became_active":"2024-09-21T08:25:48.908351+0000","last_became_peered":"2024-09-21T08:25:48.908351+0000","last_unstale":"2024-09-21T08:27:25.641582+0000","last_undegraded":"2024-09-21T08:27:25.641582+0000","last_fullsized":"2024-09-21T08:27:25.641582+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":295,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636110+0000","last_change":"2024-09-21T08:26:52.225812+0000","last_active":"2024-09-21T08:27:25.636110+0000","last_peered":"2024-09-21T08:27:25.636110+0000","last_clean":"2024-09-21T08:27:25.636110+0000","last_became_active":"2024-09-21T08:25:48.911245+0000","last_became_peered":"2024-09-21T08:25:48.911245+0000","last_unstale":"2024-09-21T08:27:25.636110+0000","last_undegraded":"2024-09-21T08:27:25.636110+0000","last_fullsized":"2024-09-21T08:27:25.636110+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:52.225741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:52.225741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:08.931292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":450,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636137+0000","last_change":"2024-09-21T08:27:15.605129+0000","last_active":"2024-09-21T08:27:25.636137+0000","last_peered":"2024-09-21T08:27:25.636137+0000","last_clean":"2024-09-21T08:27:25.636137+0000","last_became_active":"2024-09-21T08:27:15.598095+0000","last_became_peered":"2024-09-21T08:27:15.598095+0000","last_unstale":"2024-09-21T08:27:25.636137+0000","last_undegraded":"2024-09-21T08:27:25.636137+0000","last_fullsized":"2024-09-21T08:27:25.636137+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:50.675673+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.675673+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":383,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.638176+0000","last_change":"2024-09-21T08:25:42.021037+0000","last_active":"2024-09-21T08:27:25.638176+0000","last_peered":"2024-09-21T08:27:25.638176+0000","last_clean":"2024-09-21T08:27:25.638176+0000","last_became_active":"2024-09-21T08:23:34.567269+0000","last_became_peered":"2024-09-21T08:23:34.567269+0000","last_unstale":"2024-09-21T08:27:25.638176+0000","last_undegraded":"2024-09-21T08:27:25.638176+0000","last_fullsized":"2024-09-21T08:27:25.638176+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:42.020928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.020928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":401,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641891+0000","last_change":"2024-09-21T08:27:15.595070+0000","last_active":"2024-09-21T08:27:25.641891+0000","last_peered":"2024-09-21T08:27:25.641891+0000","last_clean":"2024-09-21T08:27:25.641891+0000","last_became_active":"2024-09-21T08:27:15.594737+0000","last_became_peered":"2024-09-21T08:27:15.594737+0000","last_unstale":"2024-09-21T08:27:25.641891+0000","last_undegraded":"2024-09-21T08:27:25.641891+0000","last_fullsized":"2024-09-21T08:27:25.641891+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:08.831386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:08.831386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:30.539564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.5","version":"250'268","reported_seq":1143,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.642180+0000","last_change":"2024-09-21T08:27:23.654634+0000","last_active":"2024-09-21T08:27:25.642180+0000","last_peered":"2024-09-21T08:27:25.642180+0000","last_clean":"2024-09-21T08:27:25.642180+0000","last_became_active":"2024-09-21T08:26:15.915988+0000","last_became_peered":"2024-09-21T08:26:15.915988+0000","last_unstale":"2024-09-21T08:27:25.642180+0000","last_undegraded":"2024-09-21T08:27:25.642180+0000","last_fullsized":"2024-09-21T08:27:25.642180+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'245","last_scrub_stamp":"2024-09-21T08:26:30.864040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.864040+0000","objects_scrubbed":0,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:56.256974+0000","scrub_duration":0,"objects_trimmed":2,"snaptrim_duration":0.0017552170000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":11,"num_read_kb":12581,"num_write":7,"num_write_kb":1348,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"b"}]},{"pgid":"4.6","version":"73'12","reported_seq":425,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628707+0000","last_change":"2024-09-21T08:26:07.384721+0000","last_active":"2024-09-21T08:27:25.628707+0000","last_peered":"2024-09-21T08:27:25.628707+0000","last_clean":"2024-09-21T08:27:25.628707+0000","last_became_active":"2024-09-21T08:25:48.888592+0000","last_became_peered":"2024-09-21T08:25:48.888592+0000","last_unstale":"2024-09-21T08:27:25.628707+0000","last_undegraded":"2024-09-21T08:27:25.628707+0000","last_fullsized":"2024-09-21T08:27:25.628707+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:26:07.384640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:07.384640+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.997319+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":407,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627681+0000","last_change":"2024-09-21T08:27:15.615217+0000","last_active":"2024-09-21T08:27:25.627681+0000","last_peered":"2024-09-21T08:27:25.627681+0000","last_clean":"2024-09-21T08:27:25.627681+0000","last_became_active":"2024-09-21T08:27:15.614652+0000","last_became_peered":"2024-09-21T08:27:15.614652+0000","last_unstale":"2024-09-21T08:27:25.627681+0000","last_undegraded":"2024-09-21T08:27:25.627681+0000","last_fullsized":"2024-09-21T08:27:25.627681+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":658,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639557+0000","last_change":"2024-09-21T08:27:15.654062+0000","last_active":"2024-09-21T08:27:25.639557+0000","last_peered":"2024-09-21T08:27:25.639557+0000","last_clean":"2024-09-21T08:27:25.639557+0000","last_became_active":"2024-09-21T08:27:15.610881+0000","last_became_peered":"2024-09-21T08:27:15.610881+0000","last_unstale":"2024-09-21T08:27:25.639557+0000","last_undegraded":"2024-09-21T08:27:25.639557+0000","last_fullsized":"2024-09-21T08:27:25.639557+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":225,"num_read_kb":220,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":1167,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":390,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641572+0000","last_change":"2024-09-21T08:25:48.909001+0000","last_active":"2024-09-21T08:27:25.641572+0000","last_peered":"2024-09-21T08:27:25.641572+0000","last_clean":"2024-09-21T08:27:25.641572+0000","last_became_active":"2024-09-21T08:25:48.908492+0000","last_became_peered":"2024-09-21T08:25:48.908492+0000","last_unstale":"2024-09-21T08:27:25.641572+0000","last_undegraded":"2024-09-21T08:27:25.641572+0000","last_fullsized":"2024-09-21T08:27:25.641572+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.812789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.812789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":293,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640022+0000","last_change":"2024-09-21T08:27:15.609485+0000","last_active":"2024-09-21T08:27:25.640022+0000","last_peered":"2024-09-21T08:27:25.640022+0000","last_clean":"2024-09-21T08:27:25.640022+0000","last_became_active":"2024-09-21T08:27:15.609121+0000","last_became_peered":"2024-09-21T08:27:15.609121+0000","last_unstale":"2024-09-21T08:27:25.640022+0000","last_undegraded":"2024-09-21T08:27:25.640022+0000","last_fullsized":"2024-09-21T08:27:25.640022+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.6","version":"252'119","reported_seq":1023,"reported_epoch":252,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:25.649042+0000","last_change":"2024-09-21T08:27:25.649042+0000","last_active":"2024-09-21T08:27:25.649042+0000","last_peered":"2024-09-21T08:27:25.649042+0000","last_clean":"2024-09-21T08:27:25.649042+0000","last_became_active":"2024-09-21T08:27:18.603754+0000","last_became_peered":"2024-09-21T08:27:18.603754+0000","last_unstale":"2024-09-21T08:27:25.649042+0000","last_undegraded":"2024-09-21T08:27:25.649042+0000","last_fullsized":"2024-09-21T08:27:25.649042+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"219'105","last_scrub_stamp":"2024-09-21T08:26:53.224145+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.224145+0000","objects_scrubbed":2,"log_size":119,"log_dups_size":0,"ondisk_log_size":119,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:03.419612+0000","scrub_duration":2,"objects_trimmed":1,"snaptrim_duration":0.030230684000000001,"stat_sum":{"num_bytes":6605184,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":10,"num_read_kb":17710,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":8094769,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.5","version":"242'20","reported_seq":449,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641667+0000","last_change":"2024-09-21T08:27:15.666732+0000","last_active":"2024-09-21T08:27:25.641667+0000","last_peered":"2024-09-21T08:27:25.641667+0000","last_clean":"2024-09-21T08:27:25.641667+0000","last_became_active":"2024-09-21T08:27:15.612339+0000","last_became_peered":"2024-09-21T08:27:15.612339+0000","last_unstale":"2024-09-21T08:27:25.641667+0000","last_undegraded":"2024-09-21T08:27:25.641667+0000","last_fullsized":"2024-09-21T08:27:25.641667+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'17","last_scrub_stamp":"2024-09-21T08:24:48.583137+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.583137+0000","objects_scrubbed":0,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":462,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":294,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640158+0000","last_change":"2024-09-21T08:27:15.600350+0000","last_active":"2024-09-21T08:27:25.640158+0000","last_peered":"2024-09-21T08:27:25.640158+0000","last_clean":"2024-09-21T08:27:25.640158+0000","last_became_active":"2024-09-21T08:27:15.599578+0000","last_became_peered":"2024-09-21T08:27:15.599578+0000","last_unstale":"2024-09-21T08:27:25.640158+0000","last_undegraded":"2024-09-21T08:27:25.640158+0000","last_fullsized":"2024-09-21T08:27:25.640158+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.045353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.045353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:54.567105+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":676,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639238+0000","last_change":"2024-09-21T08:25:51.506619+0000","last_active":"2024-09-21T08:27:25.639238+0000","last_peered":"2024-09-21T08:27:25.639238+0000","last_clean":"2024-09-21T08:27:25.639238+0000","last_became_active":"2024-09-21T08:25:51.506298+0000","last_became_peered":"2024-09-21T08:25:51.506298+0000","last_unstale":"2024-09-21T08:27:25.639238+0000","last_undegraded":"2024-09-21T08:27:25.639238+0000","last_fullsized":"2024-09-21T08:27:25.639238+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":381,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640833+0000","last_change":"2024-09-21T08:25:41.700634+0000","last_active":"2024-09-21T08:27:25.640833+0000","last_peered":"2024-09-21T08:27:25.640833+0000","last_clean":"2024-09-21T08:27:25.640833+0000","last_became_active":"2024-09-21T08:23:34.568078+0000","last_became_peered":"2024-09-21T08:23:34.568078+0000","last_unstale":"2024-09-21T08:27:25.640833+0000","last_undegraded":"2024-09-21T08:27:25.640833+0000","last_fullsized":"2024-09-21T08:27:25.640833+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":410,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641672+0000","last_change":"2024-09-21T08:27:15.611511+0000","last_active":"2024-09-21T08:27:25.641672+0000","last_peered":"2024-09-21T08:27:25.641672+0000","last_clean":"2024-09-21T08:27:25.641672+0000","last_became_active":"2024-09-21T08:27:15.610520+0000","last_became_peered":"2024-09-21T08:27:15.610520+0000","last_unstale":"2024-09-21T08:27:25.641672+0000","last_undegraded":"2024-09-21T08:27:25.641672+0000","last_fullsized":"2024-09-21T08:27:25.641672+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"248'84","reported_seq":888,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.628482+0000","last_change":"2024-09-21T08:27:23.647342+0000","last_active":"2024-09-21T08:27:25.628482+0000","last_peered":"2024-09-21T08:27:25.628482+0000","last_clean":"2024-09-21T08:27:25.628482+0000","last_became_active":"2024-09-21T08:27:15.625303+0000","last_became_peered":"2024-09-21T08:27:15.625303+0000","last_unstale":"2024-09-21T08:27:25.628482+0000","last_undegraded":"2024-09-21T08:27:25.628482+0000","last_fullsized":"2024-09-21T08:27:25.628482+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.095687084000000006,"stat_sum":{"num_bytes":6862559,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":52,"num_read_kb":44199,"num_write":53,"num_write_kb":21779,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":31,"num_bytes_recovered":20139849,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"1"}]},{"pgid":"4.e","version":"73'11","reported_seq":263,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.642038+0000","last_change":"2024-09-21T08:27:15.630206+0000","last_active":"2024-09-21T08:27:25.642038+0000","last_peered":"2024-09-21T08:27:25.642038+0000","last_clean":"2024-09-21T08:27:25.642038+0000","last_became_active":"2024-09-21T08:27:15.599307+0000","last_became_peered":"2024-09-21T08:27:15.599307+0000","last_unstale":"2024-09-21T08:27:25.642038+0000","last_undegraded":"2024-09-21T08:27:25.642038+0000","last_fullsized":"2024-09-21T08:27:25.642038+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":361,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636234+0000","last_change":"2024-09-21T08:25:48.907133+0000","last_active":"2024-09-21T08:27:25.636234+0000","last_peered":"2024-09-21T08:27:25.636234+0000","last_clean":"2024-09-21T08:27:25.636234+0000","last_became_active":"2024-09-21T08:25:48.906832+0000","last_became_peered":"2024-09-21T08:25:48.906832+0000","last_unstale":"2024-09-21T08:27:25.636234+0000","last_undegraded":"2024-09-21T08:27:25.636234+0000","last_fullsized":"2024-09-21T08:27:25.636234+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.756905+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.756905+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":406,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641764+0000","last_change":"2024-09-21T08:27:15.609907+0000","last_active":"2024-09-21T08:27:25.641764+0000","last_peered":"2024-09-21T08:27:25.641764+0000","last_clean":"2024-09-21T08:27:25.641764+0000","last_became_active":"2024-09-21T08:27:15.609301+0000","last_became_peered":"2024-09-21T08:27:15.609301+0000","last_unstale":"2024-09-21T08:27:25.641764+0000","last_undegraded":"2024-09-21T08:27:25.641764+0000","last_fullsized":"2024-09-21T08:27:25.641764+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":273,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639565+0000","last_change":"2024-09-21T08:27:15.599794+0000","last_active":"2024-09-21T08:27:25.639565+0000","last_peered":"2024-09-21T08:27:25.639565+0000","last_clean":"2024-09-21T08:27:25.639565+0000","last_became_active":"2024-09-21T08:27:15.599430+0000","last_became_peered":"2024-09-21T08:27:15.599430+0000","last_unstale":"2024-09-21T08:27:25.639565+0000","last_undegraded":"2024-09-21T08:27:25.639565+0000","last_fullsized":"2024-09-21T08:27:25.639565+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.977956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:40.975488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"251'193","reported_seq":1039,"reported_epoch":252,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:25.637842+0000","last_change":"2024-09-21T08:27:23.620788+0000","last_active":"2024-09-21T08:27:25.637842+0000","last_peered":"2024-09-21T08:27:25.637842+0000","last_clean":"2024-09-21T08:27:25.637842+0000","last_became_active":"2024-09-21T08:24:34.508974+0000","last_became_peered":"2024-09-21T08:24:34.508974+0000","last_unstale":"2024-09-21T08:27:25.637842+0000","last_undegraded":"2024-09-21T08:27:25.637842+0000","last_fullsized":"2024-09-21T08:27:25.637842+0000","mapping_epoch":80,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":81,"parent":"0.0","parent_split_bits":0,"last_scrub":"233'158","last_scrub_stamp":"2024-09-21T08:27:09.368548+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:09.368548+0000","objects_scrubbed":7,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:18.998184+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0098113409999999995,"stat_sum":{"num_bytes":16417770,"num_objects":7,"num_object_clones":4,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":92,"num_read_kb":94302,"num_write":92,"num_write_kb":33649,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"44","length":"1"},{"start":"47","length":"1"},{"start":"49","length":"6"}]},{"pgid":"4.d","version":"73'17","reported_seq":468,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.637593+0000","last_change":"2024-09-21T08:25:48.934395+0000","last_active":"2024-09-21T08:27:25.637593+0000","last_peered":"2024-09-21T08:27:25.637593+0000","last_clean":"2024-09-21T08:27:25.637593+0000","last_became_active":"2024-09-21T08:25:48.904254+0000","last_became_peered":"2024-09-21T08:25:48.904254+0000","last_unstale":"2024-09-21T08:27:25.637593+0000","last_undegraded":"2024-09-21T08:27:25.637593+0000","last_fullsized":"2024-09-21T08:27:25.637593+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'17","last_scrub_stamp":"2024-09-21T08:25:42.978787+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.978787+0000","objects_scrubbed":8,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":235,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639988+0000","last_change":"2024-09-21T08:27:15.609239+0000","last_active":"2024-09-21T08:27:25.639988+0000","last_peered":"2024-09-21T08:27:25.639988+0000","last_clean":"2024-09-21T08:27:25.639988+0000","last_became_active":"2024-09-21T08:27:15.608916+0000","last_became_peered":"2024-09-21T08:27:15.608916+0000","last_unstale":"2024-09-21T08:27:25.639988+0000","last_undegraded":"2024-09-21T08:27:25.639988+0000","last_fullsized":"2024-09-21T08:27:25.639988+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":285,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628294+0000","last_change":"2024-09-21T08:25:48.889493+0000","last_active":"2024-09-21T08:27:25.628294+0000","last_peered":"2024-09-21T08:27:25.628294+0000","last_clean":"2024-09-21T08:27:25.628294+0000","last_became_active":"2024-09-21T08:25:48.889206+0000","last_became_peered":"2024-09-21T08:25:48.889206+0000","last_unstale":"2024-09-21T08:27:25.628294+0000","last_undegraded":"2024-09-21T08:27:25.628294+0000","last_fullsized":"2024-09-21T08:27:25.628294+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996695+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":415,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.638022+0000","last_change":"2024-09-21T08:27:15.616345+0000","last_active":"2024-09-21T08:27:25.638022+0000","last_peered":"2024-09-21T08:27:25.638022+0000","last_clean":"2024-09-21T08:27:25.638022+0000","last_became_active":"2024-09-21T08:27:15.616056+0000","last_became_peered":"2024-09-21T08:27:15.616056+0000","last_unstale":"2024-09-21T08:27:25.638022+0000","last_undegraded":"2024-09-21T08:27:25.638022+0000","last_fullsized":"2024-09-21T08:27:25.638022+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.766815+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.766815+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"247'32","reported_seq":892,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.629102+0000","last_change":"2024-09-21T08:27:22.633716+0000","last_active":"2024-09-21T08:27:25.629102+0000","last_peered":"2024-09-21T08:27:25.629102+0000","last_clean":"2024-09-21T08:27:25.629102+0000","last_became_active":"2024-09-21T08:27:15.626151+0000","last_became_peered":"2024-09-21T08:27:15.626151+0000","last_unstale":"2024-09-21T08:27:25.629102+0000","last_undegraded":"2024-09-21T08:27:25.629102+0000","last_fullsized":"2024-09-21T08:27:25.629102+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.100349283,"stat_sum":{"num_bytes":2823609,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":41,"num_read_kb":43263,"num_write":20,"num_write_kb":11169,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":13160525,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"1"}]},{"pgid":"4.c","version":"73'10","reported_seq":227,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640220+0000","last_change":"2024-09-21T08:27:15.678300+0000","last_active":"2024-09-21T08:27:25.640220+0000","last_peered":"2024-09-21T08:27:25.640220+0000","last_clean":"2024-09-21T08:27:25.640220+0000","last_became_active":"2024-09-21T08:27:15.612786+0000","last_became_peered":"2024-09-21T08:27:15.612786+0000","last_unstale":"2024-09-21T08:27:25.640220+0000","last_undegraded":"2024-09-21T08:27:25.640220+0000","last_fullsized":"2024-09-21T08:27:25.640220+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:26:33.823950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.823950+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:41.250015+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":332,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627719+0000","last_change":"2024-09-21T08:27:15.636488+0000","last_active":"2024-09-21T08:27:25.627719+0000","last_peered":"2024-09-21T08:27:25.627719+0000","last_clean":"2024-09-21T08:27:25.627719+0000","last_became_active":"2024-09-21T08:27:15.632969+0000","last_became_peered":"2024-09-21T08:27:15.632969+0000","last_unstale":"2024-09-21T08:27:25.627719+0000","last_undegraded":"2024-09-21T08:27:25.627719+0000","last_fullsized":"2024-09-21T08:27:25.627719+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996513+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.d","version":"73'8","reported_seq":439,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.637493+0000","last_change":"2024-09-21T08:24:51.689976+0000","last_active":"2024-09-21T08:27:25.637493+0000","last_peered":"2024-09-21T08:27:25.637493+0000","last_clean":"2024-09-21T08:27:25.637493+0000","last_became_active":"2024-09-21T08:23:34.571249+0000","last_became_peered":"2024-09-21T08:23:34.571249+0000","last_unstale":"2024-09-21T08:27:25.637493+0000","last_undegraded":"2024-09-21T08:27:25.637493+0000","last_fullsized":"2024-09-21T08:27:25.637493+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:24:51.689901+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.689901+0000","objects_scrubbed":1,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":217,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.638086+0000","last_change":"2024-09-21T08:26:51.222803+0000","last_active":"2024-09-21T08:27:25.638086+0000","last_peered":"2024-09-21T08:27:25.638086+0000","last_clean":"2024-09-21T08:27:25.638086+0000","last_became_active":"2024-09-21T08:25:48.903536+0000","last_became_peered":"2024-09-21T08:25:48.903536+0000","last_unstale":"2024-09-21T08:27:25.638086+0000","last_undegraded":"2024-09-21T08:27:25.638086+0000","last_fullsized":"2024-09-21T08:27:25.638086+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:51.222707+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.222707+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:05.138581+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.8","version":"248'236","reported_seq":1202,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.636552+0000","last_change":"2024-09-21T08:27:24.617195+0000","last_active":"2024-09-21T08:27:25.636552+0000","last_peered":"2024-09-21T08:27:25.636552+0000","last_clean":"2024-09-21T08:27:25.636552+0000","last_became_active":"2024-09-21T08:27:18.606247+0000","last_became_peered":"2024-09-21T08:27:18.606247+0000","last_unstale":"2024-09-21T08:27:25.636552+0000","last_undegraded":"2024-09-21T08:27:25.636552+0000","last_fullsized":"2024-09-21T08:27:25.636552+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'178","last_scrub_stamp":"2024-09-21T08:26:28.217842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.217842+0000","objects_scrubbed":4,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:49.926242+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.080511866000000001,"stat_sum":{"num_bytes":12840111,"num_objects":4,"num_object_clones":1,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":42,"num_read_kb":50801,"num_write":27,"num_write_kb":11347,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":23089101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"},{"start":"47","length":"1"},{"start":"49","length":"4"}]},{"pgid":"4.b","version":"73'9","reported_seq":483,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628015+0000","last_change":"2024-09-21T08:27:15.651487+0000","last_active":"2024-09-21T08:27:25.628015+0000","last_peered":"2024-09-21T08:27:25.628015+0000","last_clean":"2024-09-21T08:27:25.628015+0000","last_became_active":"2024-09-21T08:27:15.623299+0000","last_became_peered":"2024-09-21T08:27:15.623299+0000","last_unstale":"2024-09-21T08:27:25.628015+0000","last_undegraded":"2024-09-21T08:27:25.628015+0000","last_fullsized":"2024-09-21T08:27:25.628015+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:25:22.369811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.369811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.997166+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":422,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640430+0000","last_change":"2024-09-21T08:27:15.610382+0000","last_active":"2024-09-21T08:27:25.640430+0000","last_peered":"2024-09-21T08:27:25.640430+0000","last_clean":"2024-09-21T08:27:25.640430+0000","last_became_active":"2024-09-21T08:27:15.610075+0000","last_became_peered":"2024-09-21T08:27:15.610075+0000","last_unstale":"2024-09-21T08:27:25.640430+0000","last_undegraded":"2024-09-21T08:27:25.640430+0000","last_fullsized":"2024-09-21T08:27:25.640430+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.665444+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.665444+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":413,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636822+0000","last_change":"2024-09-21T08:27:15.613838+0000","last_active":"2024-09-21T08:27:25.636822+0000","last_peered":"2024-09-21T08:27:25.636822+0000","last_clean":"2024-09-21T08:27:25.636822+0000","last_became_active":"2024-09-21T08:27:15.612757+0000","last_became_peered":"2024-09-21T08:27:15.612757+0000","last_unstale":"2024-09-21T08:27:25.636822+0000","last_undegraded":"2024-09-21T08:27:25.636822+0000","last_fullsized":"2024-09-21T08:27:25.636822+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,7],"acting":[2,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":371,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627185+0000","last_change":"2024-09-21T08:23:36.583517+0000","last_active":"2024-09-21T08:27:25.627185+0000","last_peered":"2024-09-21T08:27:25.627185+0000","last_clean":"2024-09-21T08:27:25.627185+0000","last_became_active":"2024-09-21T08:23:36.583180+0000","last_became_peered":"2024-09-21T08:23:36.583180+0000","last_unstale":"2024-09-21T08:27:25.627185+0000","last_undegraded":"2024-09-21T08:27:25.627185+0000","last_fullsized":"2024-09-21T08:27:25.627185+0000","mapping_epoch":71,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":72,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996945+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.9","version":"245'214","reported_seq":1127,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.627388+0000","last_change":"2024-09-21T08:27:20.631231+0000","last_active":"2024-09-21T08:27:25.627388+0000","last_peered":"2024-09-21T08:27:25.627388+0000","last_clean":"2024-09-21T08:27:25.627388+0000","last_became_active":"2024-09-21T08:26:15.917755+0000","last_became_peered":"2024-09-21T08:26:15.917755+0000","last_unstale":"2024-09-21T08:27:25.627388+0000","last_undegraded":"2024-09-21T08:27:25.627388+0000","last_fullsized":"2024-09-21T08:27:25.627388+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":182,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":214,"log_dups_size":0,"ondisk_log_size":214,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:47.572333+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0057892799999999999,"stat_sum":{"num_bytes":18946156,"num_objects":9,"num_object_clones":5,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":56,"num_read_kb":75456,"num_write":21,"num_write_kb":5356,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"44","length":"1"},{"start":"47","length":"1"}]},{"pgid":"4.a","version":"73'19","reported_seq":275,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639437+0000","last_change":"2024-09-21T08:25:48.964103+0000","last_active":"2024-09-21T08:27:25.639437+0000","last_peered":"2024-09-21T08:27:25.639437+0000","last_clean":"2024-09-21T08:27:25.639437+0000","last_became_active":"2024-09-21T08:25:48.895647+0000","last_became_peered":"2024-09-21T08:25:48.895647+0000","last_unstale":"2024-09-21T08:27:25.639437+0000","last_undegraded":"2024-09-21T08:27:25.639437+0000","last_fullsized":"2024-09-21T08:27:25.639437+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":407,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641426+0000","last_change":"2024-09-21T08:27:15.613148+0000","last_active":"2024-09-21T08:27:25.641426+0000","last_peered":"2024-09-21T08:27:25.641426+0000","last_clean":"2024-09-21T08:27:25.641426+0000","last_became_active":"2024-09-21T08:27:15.612852+0000","last_became_peered":"2024-09-21T08:27:15.612852+0000","last_unstale":"2024-09-21T08:27:25.641426+0000","last_undegraded":"2024-09-21T08:27:25.641426+0000","last_fullsized":"2024-09-21T08:27:25.641426+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.740674+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.740674+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":383,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636643+0000","last_change":"2024-09-21T08:27:15.362144+0000","last_active":"2024-09-21T08:27:25.636643+0000","last_peered":"2024-09-21T08:27:25.636643+0000","last_clean":"2024-09-21T08:27:25.636643+0000","last_became_active":"2024-09-21T08:23:34.564182+0000","last_became_peered":"2024-09-21T08:23:34.564182+0000","last_unstale":"2024-09-21T08:27:25.636643+0000","last_undegraded":"2024-09-21T08:27:25.636643+0000","last_fullsized":"2024-09-21T08:27:25.636643+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:15.362045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:15.362045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:16.792076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":401,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.638800+0000","last_change":"2024-09-21T08:27:15.611787+0000","last_active":"2024-09-21T08:27:25.638800+0000","last_peered":"2024-09-21T08:27:25.638800+0000","last_clean":"2024-09-21T08:27:25.638800+0000","last_became_active":"2024-09-21T08:27:15.611496+0000","last_became_peered":"2024-09-21T08:27:15.611496+0000","last_unstale":"2024-09-21T08:27:25.638800+0000","last_undegraded":"2024-09-21T08:27:25.638800+0000","last_fullsized":"2024-09-21T08:27:25.638800+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.a","version":"248'68","reported_seq":143,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.638145+0000","last_change":"2024-09-21T08:27:23.620745+0000","last_active":"2024-09-21T08:27:25.638145+0000","last_peered":"2024-09-21T08:27:25.638145+0000","last_clean":"2024-09-21T08:27:25.638145+0000","last_became_active":"2024-09-21T08:27:15.598294+0000","last_became_peered":"2024-09-21T08:27:15.598294+0000","last_unstale":"2024-09-21T08:27:25.638145+0000","last_undegraded":"2024-09-21T08:27:25.638145+0000","last_fullsized":"2024-09-21T08:27:25.638145+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"204'60","last_scrub_stamp":"2024-09-21T08:26:44.214809+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.214809+0000","objects_scrubbed":1,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:03.504068+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.068662825999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":35,"num_read_kb":32896,"num_write":30,"num_write_kb":11916,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":8736584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"1"},{"start":"49","length":"6"},{"start":"50","length":"1"}]},{"pgid":"4.9","version":"73'12","reported_seq":267,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627795+0000","last_change":"2024-09-21T08:27:15.667015+0000","last_active":"2024-09-21T08:27:25.627795+0000","last_peered":"2024-09-21T08:27:25.627795+0000","last_clean":"2024-09-21T08:27:25.627795+0000","last_became_active":"2024-09-21T08:27:15.629692+0000","last_became_peered":"2024-09-21T08:27:15.629692+0000","last_unstale":"2024-09-21T08:27:25.627795+0000","last_undegraded":"2024-09-21T08:27:25.627795+0000","last_fullsized":"2024-09-21T08:27:25.627795+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:41.463820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.463820+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.997197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":401,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641860+0000","last_change":"2024-09-21T08:26:51.769124+0000","last_active":"2024-09-21T08:27:25.641860+0000","last_peered":"2024-09-21T08:27:25.641860+0000","last_clean":"2024-09-21T08:27:25.641860+0000","last_became_active":"2024-09-21T08:25:48.896222+0000","last_became_peered":"2024-09-21T08:25:48.896222+0000","last_unstale":"2024-09-21T08:27:25.641860+0000","last_undegraded":"2024-09-21T08:27:25.641860+0000","last_fullsized":"2024-09-21T08:27:25.641860+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:51.769044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.769044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:06.214839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":397,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636191+0000","last_change":"2024-09-21T08:27:08.331225+0000","last_active":"2024-09-21T08:27:25.636191+0000","last_peered":"2024-09-21T08:27:25.636191+0000","last_clean":"2024-09-21T08:27:25.636191+0000","last_became_active":"2024-09-21T08:25:48.912993+0000","last_became_peered":"2024-09-21T08:25:48.912993+0000","last_unstale":"2024-09-21T08:27:25.636191+0000","last_undegraded":"2024-09-21T08:27:25.636191+0000","last_fullsized":"2024-09-21T08:27:25.636191+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:08.331128+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:08.331128+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:20.977455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":299,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640966+0000","last_change":"2024-09-21T08:27:15.595003+0000","last_active":"2024-09-21T08:27:25.640966+0000","last_peered":"2024-09-21T08:27:25.640966+0000","last_clean":"2024-09-21T08:27:25.640966+0000","last_became_active":"2024-09-21T08:27:15.594658+0000","last_became_peered":"2024-09-21T08:27:15.594658+0000","last_unstale":"2024-09-21T08:27:25.640966+0000","last_undegraded":"2024-09-21T08:27:25.640966+0000","last_fullsized":"2024-09-21T08:27:25.640966+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.b","version":"252'437","reported_seq":1425,"reported_epoch":252,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:25.640585+0000","last_change":"2024-09-21T08:27:22.616236+0000","last_active":"2024-09-21T08:27:25.640585+0000","last_peered":"2024-09-21T08:27:25.640585+0000","last_clean":"2024-09-21T08:27:25.640585+0000","last_became_active":"2024-09-21T08:25:51.507525+0000","last_became_peered":"2024-09-21T08:25:51.507525+0000","last_unstale":"2024-09-21T08:27:25.640585+0000","last_undegraded":"2024-09-21T08:27:25.640585+0000","last_fullsized":"2024-09-21T08:27:25.640585+0000","mapping_epoch":157,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":158,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:24:33.484514+0000","objects_scrubbed":0,"log_size":437,"log_dups_size":0,"ondisk_log_size":437,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054099890999999997,"stat_sum":{"num_bytes":40451376,"num_objects":19,"num_object_clones":11,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":351,"num_read_kb":328052,"num_write":255,"num_write_kb":74312,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":46,"num_bytes_recovered":31871318,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,0],"acting":[5,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"44","length":"1"},{"start":"47","length":"1"},{"start":"49","length":"1"}]},{"pgid":"4.8","version":"73'15","reported_seq":501,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639660+0000","last_change":"2024-09-21T08:27:15.651284+0000","last_active":"2024-09-21T08:27:25.639660+0000","last_peered":"2024-09-21T08:27:25.639660+0000","last_clean":"2024-09-21T08:27:25.639660+0000","last_became_active":"2024-09-21T08:27:15.602245+0000","last_became_peered":"2024-09-21T08:27:15.602245+0000","last_unstale":"2024-09-21T08:27:25.639660+0000","last_undegraded":"2024-09-21T08:27:25.639660+0000","last_fullsized":"2024-09-21T08:27:25.639660+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":425,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.642513+0000","last_change":"2024-09-21T08:25:48.975855+0000","last_active":"2024-09-21T08:27:25.642513+0000","last_peered":"2024-09-21T08:27:25.642513+0000","last_clean":"2024-09-21T08:27:25.642513+0000","last_became_active":"2024-09-21T08:25:48.897974+0000","last_became_peered":"2024-09-21T08:25:48.897974+0000","last_unstale":"2024-09-21T08:27:25.642513+0000","last_undegraded":"2024-09-21T08:27:25.642513+0000","last_fullsized":"2024-09-21T08:27:25.642513+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.9","version":"242'13","reported_seq":488,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639573+0000","last_change":"2024-09-21T08:27:15.680924+0000","last_active":"2024-09-21T08:27:25.639573+0000","last_peered":"2024-09-21T08:27:25.639573+0000","last_clean":"2024-09-21T08:27:25.639573+0000","last_became_active":"2024-09-21T08:27:15.612940+0000","last_became_peered":"2024-09-21T08:27:15.612940+0000","last_unstale":"2024-09-21T08:27:25.639573+0000","last_undegraded":"2024-09-21T08:27:25.639573+0000","last_fullsized":"2024-09-21T08:27:25.639573+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'10","last_scrub_stamp":"2024-09-21T08:25:42.708458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.708458+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":378,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639798+0000","last_change":"2024-09-21T08:24:56.500567+0000","last_active":"2024-09-21T08:27:25.639798+0000","last_peered":"2024-09-21T08:27:25.639798+0000","last_clean":"2024-09-21T08:27:25.639798+0000","last_became_active":"2024-09-21T08:24:56.500287+0000","last_became_peered":"2024-09-21T08:24:56.500287+0000","last_unstale":"2024-09-21T08:27:25.639798+0000","last_undegraded":"2024-09-21T08:27:25.639798+0000","last_fullsized":"2024-09-21T08:27:25.639798+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,2],"acting":[5,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"180'50","reported_seq":808,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.636559+0000","last_change":"2024-09-21T08:27:21.619106+0000","last_active":"2024-09-21T08:27:25.636559+0000","last_peered":"2024-09-21T08:27:25.636559+0000","last_clean":"2024-09-21T08:27:25.636559+0000","last_became_active":"2024-09-21T08:27:15.616665+0000","last_became_peered":"2024-09-21T08:27:15.616665+0000","last_unstale":"2024-09-21T08:27:25.636559+0000","last_undegraded":"2024-09-21T08:27:25.636559+0000","last_fullsized":"2024-09-21T08:27:25.636559+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:26:50.014780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:50.014780+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:02.246679+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.078521815999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"1"},{"start":"49","length":"6"},{"start":"50","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":330,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.626828+0000","last_change":"2024-09-21T08:27:15.630070+0000","last_active":"2024-09-21T08:27:25.626828+0000","last_peered":"2024-09-21T08:27:25.626828+0000","last_clean":"2024-09-21T08:27:25.626828+0000","last_became_active":"2024-09-21T08:27:15.629103+0000","last_became_peered":"2024-09-21T08:27:15.629103+0000","last_unstale":"2024-09-21T08:27:25.626828+0000","last_undegraded":"2024-09-21T08:27:25.626828+0000","last_fullsized":"2024-09-21T08:27:25.626828+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:09.170818+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:09.170818+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:18.528926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":337,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.642426+0000","last_change":"2024-09-21T08:27:15.688702+0000","last_active":"2024-09-21T08:27:25.642426+0000","last_peered":"2024-09-21T08:27:25.642426+0000","last_clean":"2024-09-21T08:27:25.642426+0000","last_became_active":"2024-09-21T08:27:15.614401+0000","last_became_peered":"2024-09-21T08:27:15.614401+0000","last_unstale":"2024-09-21T08:27:25.642426+0000","last_undegraded":"2024-09-21T08:27:25.642426+0000","last_fullsized":"2024-09-21T08:27:25.642426+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":414,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641496+0000","last_change":"2024-09-21T08:27:15.608038+0000","last_active":"2024-09-21T08:27:25.641496+0000","last_peered":"2024-09-21T08:27:25.641496+0000","last_clean":"2024-09-21T08:27:25.641496+0000","last_became_active":"2024-09-21T08:27:15.607700+0000","last_became_peered":"2024-09-21T08:27:15.607700+0000","last_unstale":"2024-09-21T08:27:25.641496+0000","last_undegraded":"2024-09-21T08:27:25.641496+0000","last_fullsized":"2024-09-21T08:27:25.641496+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.795325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:13.911980+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":389,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641991+0000","last_change":"2024-09-21T08:27:10.851586+0000","last_active":"2024-09-21T08:27:25.641991+0000","last_peered":"2024-09-21T08:27:25.641991+0000","last_clean":"2024-09-21T08:27:25.641991+0000","last_became_active":"2024-09-21T08:24:56.502937+0000","last_became_peered":"2024-09-21T08:24:56.502937+0000","last_unstale":"2024-09-21T08:27:25.641991+0000","last_undegraded":"2024-09-21T08:27:25.641991+0000","last_fullsized":"2024-09-21T08:27:25.641991+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:10.851510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.851510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:37.463269+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"246'314","reported_seq":1257,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641303+0000","last_change":"2024-09-21T08:27:23.653196+0000","last_active":"2024-09-21T08:27:25.641303+0000","last_peered":"2024-09-21T08:27:25.641303+0000","last_clean":"2024-09-21T08:27:25.641303+0000","last_became_active":"2024-09-21T08:26:20.938955+0000","last_became_peered":"2024-09-21T08:26:20.938955+0000","last_unstale":"2024-09-21T08:27:25.641303+0000","last_undegraded":"2024-09-21T08:27:25.641303+0000","last_fullsized":"2024-09-21T08:27:25.641303+0000","mapping_epoch":186,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":187,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'262","last_scrub_stamp":"2024-09-21T08:26:29.875830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.875830+0000","objects_scrubbed":3,"log_size":314,"log_dups_size":0,"ondisk_log_size":314,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:46.339942+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00053789300000000001,"stat_sum":{"num_bytes":11714545,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":59,"num_read_kb":58973,"num_write":19,"num_write_kb":10627,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"b"}]},{"pgid":"4.16","version":"73'9","reported_seq":451,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628756+0000","last_change":"2024-09-21T08:27:15.724400+0000","last_active":"2024-09-21T08:27:25.628756+0000","last_peered":"2024-09-21T08:27:25.628756+0000","last_clean":"2024-09-21T08:27:25.628756+0000","last_became_active":"2024-09-21T08:27:15.636161+0000","last_became_peered":"2024-09-21T08:27:15.636161+0000","last_unstale":"2024-09-21T08:27:25.628756+0000","last_undegraded":"2024-09-21T08:27:25.628756+0000","last_fullsized":"2024-09-21T08:27:25.628756+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:10.183035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.183035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:21.552603+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":413,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639324+0000","last_change":"2024-09-21T08:27:15.615261+0000","last_active":"2024-09-21T08:27:25.639324+0000","last_peered":"2024-09-21T08:27:25.639324+0000","last_clean":"2024-09-21T08:27:25.639324+0000","last_became_active":"2024-09-21T08:27:15.614284+0000","last_became_peered":"2024-09-21T08:27:15.614284+0000","last_unstale":"2024-09-21T08:27:25.639324+0000","last_undegraded":"2024-09-21T08:27:25.639324+0000","last_fullsized":"2024-09-21T08:27:25.639324+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:46.719545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:46.719545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":169,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.642329+0000","last_change":"2024-09-21T08:27:15.614441+0000","last_active":"2024-09-21T08:27:25.642329+0000","last_peered":"2024-09-21T08:27:25.642329+0000","last_clean":"2024-09-21T08:27:25.642329+0000","last_became_active":"2024-09-21T08:27:15.613911+0000","last_became_peered":"2024-09-21T08:27:15.613911+0000","last_unstale":"2024-09-21T08:27:25.642329+0000","last_undegraded":"2024-09-21T08:27:25.642329+0000","last_fullsized":"2024-09-21T08:27:25.642329+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:05.814753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.814753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":385,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.637741+0000","last_change":"2024-09-21T08:24:53.746385+0000","last_active":"2024-09-21T08:27:25.637741+0000","last_peered":"2024-09-21T08:27:25.637741+0000","last_clean":"2024-09-21T08:27:25.637741+0000","last_became_active":"2024-09-21T08:24:38.539845+0000","last_became_peered":"2024-09-21T08:24:38.539845+0000","last_unstale":"2024-09-21T08:27:25.637741+0000","last_undegraded":"2024-09-21T08:27:25.637741+0000","last_fullsized":"2024-09-21T08:27:25.637741+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.746295+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.746295+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,0],"acting":[2,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"251'121","reported_seq":976,"reported_epoch":252,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:27:25.628448+0000","last_change":"2024-09-21T08:27:24.635351+0000","last_active":"2024-09-21T08:27:25.628448+0000","last_peered":"2024-09-21T08:27:25.628448+0000","last_clean":"2024-09-21T08:27:25.628448+0000","last_became_active":"2024-09-21T08:26:17.924805+0000","last_became_peered":"2024-09-21T08:26:17.924805+0000","last_unstale":"2024-09-21T08:27:25.628448+0000","last_undegraded":"2024-09-21T08:27:25.628448+0000","last_fullsized":"2024-09-21T08:27:25.628448+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'102","last_scrub_stamp":"2024-09-21T08:26:33.255256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.255256+0000","objects_scrubbed":1,"log_size":121,"log_dups_size":0,"ondisk_log_size":121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:00.893544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.031898016000000001,"stat_sum":{"num_bytes":4806035,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":12,"num_read_kb":12624,"num_write":9,"num_write_kb":2434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"44","length":"1"}]},{"pgid":"4.15","version":"73'9","reported_seq":460,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640961+0000","last_change":"2024-09-21T08:27:15.663095+0000","last_active":"2024-09-21T08:27:25.640961+0000","last_peered":"2024-09-21T08:27:25.640961+0000","last_clean":"2024-09-21T08:27:25.640961+0000","last_became_active":"2024-09-21T08:27:15.608540+0000","last_became_peered":"2024-09-21T08:27:15.608540+0000","last_unstale":"2024-09-21T08:27:25.640961+0000","last_undegraded":"2024-09-21T08:27:25.640961+0000","last_fullsized":"2024-09-21T08:27:25.640961+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,0],"acting":[5,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":407,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627395+0000","last_change":"2024-09-21T08:27:15.629055+0000","last_active":"2024-09-21T08:27:25.627395+0000","last_peered":"2024-09-21T08:27:25.627395+0000","last_clean":"2024-09-21T08:27:25.627395+0000","last_became_active":"2024-09-21T08:27:15.628709+0000","last_became_peered":"2024-09-21T08:27:15.628709+0000","last_unstale":"2024-09-21T08:27:25.627395+0000","last_undegraded":"2024-09-21T08:27:25.627395+0000","last_fullsized":"2024-09-21T08:27:25.627395+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996977+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.14","version":"242'13","reported_seq":358,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640775+0000","last_change":"2024-09-21T08:27:15.641626+0000","last_active":"2024-09-21T08:27:25.640775+0000","last_peered":"2024-09-21T08:27:25.640775+0000","last_clean":"2024-09-21T08:27:25.640775+0000","last_became_active":"2024-09-21T08:27:15.611096+0000","last_became_peered":"2024-09-21T08:27:15.611096+0000","last_unstale":"2024-09-21T08:27:25.640775+0000","last_undegraded":"2024-09-21T08:27:25.640775+0000","last_fullsized":"2024-09-21T08:27:25.640775+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":356,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.637970+0000","last_change":"2024-09-21T08:25:39.989977+0000","last_active":"2024-09-21T08:27:25.637970+0000","last_peered":"2024-09-21T08:27:25.637970+0000","last_clean":"2024-09-21T08:27:25.637970+0000","last_became_active":"2024-09-21T08:24:38.542854+0000","last_became_peered":"2024-09-21T08:24:38.542854+0000","last_unstale":"2024-09-21T08:27:25.637970+0000","last_undegraded":"2024-09-21T08:27:25.637970+0000","last_fullsized":"2024-09-21T08:27:25.637970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.989838+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.989838+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,0],"acting":[2,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"179'63","reported_seq":935,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641693+0000","last_change":"2024-09-21T08:27:23.653530+0000","last_active":"2024-09-21T08:27:25.641693+0000","last_peered":"2024-09-21T08:27:25.641693+0000","last_clean":"2024-09-21T08:27:25.641693+0000","last_became_active":"2024-09-21T08:26:17.919628+0000","last_became_peered":"2024-09-21T08:26:17.919628+0000","last_unstale":"2024-09-21T08:27:25.641693+0000","last_undegraded":"2024-09-21T08:27:25.641693+0000","last_fullsized":"2024-09-21T08:27:25.641693+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'63","last_scrub_stamp":"2024-09-21T08:26:28.855711+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.855711+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:44.329097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00075049600000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"b"}]},{"pgid":"4.14","version":"104'15","reported_seq":262,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640833+0000","last_change":"2024-09-21T08:27:15.625934+0000","last_active":"2024-09-21T08:27:25.640833+0000","last_peered":"2024-09-21T08:27:25.640833+0000","last_clean":"2024-09-21T08:27:25.640833+0000","last_became_active":"2024-09-21T08:27:15.599161+0000","last_became_peered":"2024-09-21T08:27:15.599161+0000","last_unstale":"2024-09-21T08:27:25.640833+0000","last_undegraded":"2024-09-21T08:27:25.640833+0000","last_fullsized":"2024-09-21T08:27:25.640833+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":415,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640413+0000","last_change":"2024-09-21T08:24:56.709528+0000","last_active":"2024-09-21T08:27:25.640413+0000","last_peered":"2024-09-21T08:27:25.640413+0000","last_clean":"2024-09-21T08:27:25.640413+0000","last_became_active":"2024-09-21T08:24:56.504665+0000","last_became_peered":"2024-09-21T08:24:56.504665+0000","last_unstale":"2024-09-21T08:27:25.640413+0000","last_undegraded":"2024-09-21T08:27:25.640413+0000","last_fullsized":"2024-09-21T08:27:25.640413+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:56.709437+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.709437+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"242'11","reported_seq":470,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641189+0000","last_change":"2024-09-21T08:27:15.679761+0000","last_active":"2024-09-21T08:27:25.641189+0000","last_peered":"2024-09-21T08:27:25.641189+0000","last_clean":"2024-09-21T08:27:25.641189+0000","last_became_active":"2024-09-21T08:27:15.612180+0000","last_became_peered":"2024-09-21T08:27:15.612180+0000","last_unstale":"2024-09-21T08:27:25.641189+0000","last_undegraded":"2024-09-21T08:27:25.641189+0000","last_fullsized":"2024-09-21T08:27:25.641189+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":397,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.629176+0000","last_change":"2024-09-21T08:27:15.624841+0000","last_active":"2024-09-21T08:27:25.629176+0000","last_peered":"2024-09-21T08:27:25.629176+0000","last_clean":"2024-09-21T08:27:25.629176+0000","last_became_active":"2024-09-21T08:27:15.624330+0000","last_became_peered":"2024-09-21T08:27:15.624330+0000","last_unstale":"2024-09-21T08:27:25.629176+0000","last_undegraded":"2024-09-21T08:27:25.629176+0000","last_fullsized":"2024-09-21T08:27:25.629176+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:48.251366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:48.251366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:52.188221+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,5],"acting":[0,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.10","version":"252'110","reported_seq":170,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.650755+0000","last_change":"2024-09-21T08:27:21.618434+0000","last_active":"2024-09-21T08:27:25.650755+0000","last_peered":"2024-09-21T08:27:25.650755+0000","last_clean":"2024-09-21T08:27:25.650755+0000","last_became_active":"2024-09-21T08:27:18.605054+0000","last_became_peered":"2024-09-21T08:27:18.605054+0000","last_unstale":"2024-09-21T08:27:25.650755+0000","last_undegraded":"2024-09-21T08:27:25.650755+0000","last_fullsized":"2024-09-21T08:27:25.650755+0000","mapping_epoch":244,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":245,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'94","last_scrub_stamp":"2024-09-21T08:26:27.208234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.208234+0000","objects_scrubbed":1,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:33.767179+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.079927446999999999,"stat_sum":{"num_bytes":6605217,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":13,"num_read_kb":13072,"num_write":12,"num_write_kb":3043,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":8719784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"1"},{"start":"49","length":"5"}]},{"pgid":"4.13","version":"73'11","reported_seq":266,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641031+0000","last_change":"2024-09-21T08:27:11.919390+0000","last_active":"2024-09-21T08:27:25.641031+0000","last_peered":"2024-09-21T08:27:25.641031+0000","last_clean":"2024-09-21T08:27:25.641031+0000","last_became_active":"2024-09-21T08:25:48.914638+0000","last_became_peered":"2024-09-21T08:25:48.914638+0000","last_unstale":"2024-09-21T08:27:25.641031+0000","last_undegraded":"2024-09-21T08:27:25.641031+0000","last_fullsized":"2024-09-21T08:27:25.641031+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:27:11.919267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:11.919267+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:18.384851+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":359,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.640923+0000","last_change":"2024-09-21T08:27:06.889245+0000","last_active":"2024-09-21T08:27:25.640923+0000","last_peered":"2024-09-21T08:27:25.640923+0000","last_clean":"2024-09-21T08:27:25.640923+0000","last_became_active":"2024-09-21T08:24:56.503232+0000","last_became_peered":"2024-09-21T08:24:56.503232+0000","last_unstale":"2024-09-21T08:27:25.640923+0000","last_undegraded":"2024-09-21T08:27:25.640923+0000","last_fullsized":"2024-09-21T08:27:25.640923+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:06.889127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:06.889127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:16.183116+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":275,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639711+0000","last_change":"2024-09-21T08:27:15.599907+0000","last_active":"2024-09-21T08:27:25.639711+0000","last_peered":"2024-09-21T08:27:25.639711+0000","last_clean":"2024-09-21T08:27:25.639711+0000","last_became_active":"2024-09-21T08:27:15.599131+0000","last_became_peered":"2024-09-21T08:27:15.599131+0000","last_unstale":"2024-09-21T08:27:25.639711+0000","last_undegraded":"2024-09-21T08:27:25.639711+0000","last_fullsized":"2024-09-21T08:27:25.639711+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":280,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627269+0000","last_change":"2024-09-21T08:27:15.630524+0000","last_active":"2024-09-21T08:27:25.627269+0000","last_peered":"2024-09-21T08:27:25.627269+0000","last_clean":"2024-09-21T08:27:25.627269+0000","last_became_active":"2024-09-21T08:27:15.630119+0000","last_became_peered":"2024-09-21T08:27:15.630119+0000","last_unstale":"2024-09-21T08:27:25.627269+0000","last_undegraded":"2024-09-21T08:27:25.627269+0000","last_fullsized":"2024-09-21T08:27:25.627269+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996664+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.11","version":"246'80","reported_seq":847,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.638903+0000","last_change":"2024-09-21T08:27:23.652922+0000","last_active":"2024-09-21T08:27:25.638903+0000","last_peered":"2024-09-21T08:27:25.638903+0000","last_clean":"2024-09-21T08:27:25.638903+0000","last_became_active":"2024-09-21T08:27:15.614880+0000","last_became_peered":"2024-09-21T08:27:15.614880+0000","last_unstale":"2024-09-21T08:27:25.638903+0000","last_undegraded":"2024-09-21T08:27:25.638903+0000","last_fullsized":"2024-09-21T08:27:25.638903+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'44","last_scrub_stamp":"2024-09-21T08:26:27.882963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.882963+0000","objects_scrubbed":1,"log_size":80,"log_dups_size":0,"ondisk_log_size":80,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:52.969441+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.001677478,"stat_sum":{"num_bytes":3243784,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":13,"num_write_kb":6260,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":11110430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,0],"acting":[7,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"b"}]},{"pgid":"4.12","version":"73'9","reported_seq":259,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.636807+0000","last_change":"2024-09-21T08:27:14.398090+0000","last_active":"2024-09-21T08:27:25.636807+0000","last_peered":"2024-09-21T08:27:25.636807+0000","last_clean":"2024-09-21T08:27:25.636807+0000","last_became_active":"2024-09-21T08:25:48.910041+0000","last_became_peered":"2024-09-21T08:25:48.910041+0000","last_unstale":"2024-09-21T08:27:25.636807+0000","last_undegraded":"2024-09-21T08:27:25.636807+0000","last_fullsized":"2024-09-21T08:27:25.636807+0000","mapping_epoch":154,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":155,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:14.398007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.398007+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:32.485081+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0,5],"acting":[2,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":419,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641520+0000","last_change":"2024-09-21T08:27:15.612042+0000","last_active":"2024-09-21T08:27:25.641520+0000","last_peered":"2024-09-21T08:27:25.641520+0000","last_clean":"2024-09-21T08:27:25.641520+0000","last_became_active":"2024-09-21T08:27:15.611740+0000","last_became_peered":"2024-09-21T08:27:15.611740+0000","last_unstale":"2024-09-21T08:27:25.641520+0000","last_undegraded":"2024-09-21T08:27:25.641520+0000","last_fullsized":"2024-09-21T08:27:25.641520+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:49.790900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:49.790900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:09.364200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,2],"acting":[7,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":286,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628325+0000","last_change":"2024-09-21T08:27:15.621528+0000","last_active":"2024-09-21T08:27:25.628325+0000","last_peered":"2024-09-21T08:27:25.628325+0000","last_clean":"2024-09-21T08:27:25.628325+0000","last_became_active":"2024-09-21T08:27:15.621255+0000","last_became_peered":"2024-09-21T08:27:15.621255+0000","last_unstale":"2024-09-21T08:27:25.628325+0000","last_undegraded":"2024-09-21T08:27:25.628325+0000","last_fullsized":"2024-09-21T08:27:25.628325+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:13.940546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:13.940546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:26.113028+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,7],"acting":[0,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":389,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627108+0000","last_change":"2024-09-21T08:27:15.629626+0000","last_active":"2024-09-21T08:27:25.627108+0000","last_peered":"2024-09-21T08:27:25.627108+0000","last_clean":"2024-09-21T08:27:25.627108+0000","last_became_active":"2024-09-21T08:27:15.628666+0000","last_became_peered":"2024-09-21T08:27:15.628666+0000","last_unstale":"2024-09-21T08:27:25.627108+0000","last_undegraded":"2024-09-21T08:27:25.627108+0000","last_fullsized":"2024-09-21T08:27:25.627108+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.12","version":"244'259","reported_seq":1152,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.627570+0000","last_change":"2024-09-21T08:27:19.643231+0000","last_active":"2024-09-21T08:27:25.627570+0000","last_peered":"2024-09-21T08:27:25.627570+0000","last_clean":"2024-09-21T08:27:25.627570+0000","last_became_active":"2024-09-21T08:26:17.923264+0000","last_became_peered":"2024-09-21T08:26:17.923264+0000","last_unstale":"2024-09-21T08:27:25.627570+0000","last_undegraded":"2024-09-21T08:27:25.627570+0000","last_fullsized":"2024-09-21T08:27:25.627570+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'200","last_scrub_stamp":"2024-09-21T08:26:27.417028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.417028+0000","objects_scrubbed":3,"log_size":259,"log_dups_size":0,"ondisk_log_size":259,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":12,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:44.043654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.040766243000000001,"stat_sum":{"num_bytes":24352642,"num_objects":12,"num_object_clones":8,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":35,"num_read_kb":41025,"num_write":23,"num_write_kb":12948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"44","length":"1"}]},{"pgid":"4.11","version":"73'11","reported_seq":367,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639466+0000","last_change":"2024-09-21T08:27:15.695254+0000","last_active":"2024-09-21T08:27:25.639466+0000","last_peered":"2024-09-21T08:27:25.639466+0000","last_clean":"2024-09-21T08:27:25.639466+0000","last_became_active":"2024-09-21T08:27:15.613527+0000","last_became_peered":"2024-09-21T08:27:15.613527+0000","last_unstale":"2024-09-21T08:27:25.639466+0000","last_undegraded":"2024-09-21T08:27:25.639466+0000","last_fullsized":"2024-09-21T08:27:25.639466+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,0],"acting":[7,5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":410,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641335+0000","last_change":"2024-09-21T08:27:15.600484+0000","last_active":"2024-09-21T08:27:25.641335+0000","last_peered":"2024-09-21T08:27:25.641335+0000","last_clean":"2024-09-21T08:27:25.641335+0000","last_became_active":"2024-09-21T08:27:15.600174+0000","last_became_peered":"2024-09-21T08:27:15.600174+0000","last_unstale":"2024-09-21T08:27:25.641335+0000","last_undegraded":"2024-09-21T08:27:25.641335+0000","last_fullsized":"2024-09-21T08:27:25.641335+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":393,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639412+0000","last_change":"2024-09-21T08:24:58.688115+0000","last_active":"2024-09-21T08:27:25.639412+0000","last_peered":"2024-09-21T08:27:25.639412+0000","last_clean":"2024-09-21T08:27:25.639412+0000","last_became_active":"2024-09-21T08:24:56.503123+0000","last_became_peered":"2024-09-21T08:24:56.503123+0000","last_unstale":"2024-09-21T08:27:25.639412+0000","last_undegraded":"2024-09-21T08:27:25.639412+0000","last_fullsized":"2024-09-21T08:27:25.639412+0000","mapping_epoch":102,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":103,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:58.688046+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:58.688046+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":286,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.627814+0000","last_change":"2024-09-21T08:27:15.636837+0000","last_active":"2024-09-21T08:27:25.627814+0000","last_peered":"2024-09-21T08:27:25.627814+0000","last_clean":"2024-09-21T08:27:25.627814+0000","last_became_active":"2024-09-21T08:27:15.633344+0000","last_became_peered":"2024-09-21T08:27:15.633344+0000","last_unstale":"2024-09-21T08:27:25.627814+0000","last_undegraded":"2024-09-21T08:27:25.627814+0000","last_fullsized":"2024-09-21T08:27:25.627814+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:23:35.559396+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,5],"acting":[0,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"7.13","version":"252'28","reported_seq":854,"reported_epoch":252,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:27:25.672265+0000","last_change":"2024-09-21T08:27:19.643609+0000","last_active":"2024-09-21T08:27:25.672265+0000","last_peered":"2024-09-21T08:27:25.672265+0000","last_clean":"2024-09-21T08:27:25.672265+0000","last_became_active":"2024-09-21T08:26:17.925392+0000","last_became_peered":"2024-09-21T08:26:17.925392+0000","last_unstale":"2024-09-21T08:27:25.672265+0000","last_undegraded":"2024-09-21T08:27:25.672265+0000","last_fullsized":"2024-09-21T08:27:25.672265+0000","mapping_epoch":183,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":184,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'22","last_scrub_stamp":"2024-09-21T08:26:29.353795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.353795+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:46.774327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.038501516999999999,"stat_sum":{"num_bytes":3848836,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":1,"num_write":5,"num_write_kb":1815,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"3d","length":"1"},{"start":"44","length":"1"},{"start":"47","length":"1"}]},{"pgid":"4.10","version":"73'4","reported_seq":207,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639950+0000","last_change":"2024-09-21T08:27:15.639635+0000","last_active":"2024-09-21T08:27:25.639950+0000","last_peered":"2024-09-21T08:27:25.639950+0000","last_clean":"2024-09-21T08:27:25.639950+0000","last_became_active":"2024-09-21T08:27:15.601741+0000","last_became_peered":"2024-09-21T08:27:15.601741+0000","last_unstale":"2024-09-21T08:27:25.639950+0000","last_undegraded":"2024-09-21T08:27:25.639950+0000","last_fullsized":"2024-09-21T08:27:25.639950+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-09-21T08:26:29.821849+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.821849+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:30.080046+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":413,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628775+0000","last_change":"2024-09-21T08:27:15.630156+0000","last_active":"2024-09-21T08:27:25.628775+0000","last_peered":"2024-09-21T08:27:25.628775+0000","last_clean":"2024-09-21T08:27:25.628775+0000","last_became_active":"2024-09-21T08:27:15.629795+0000","last_became_peered":"2024-09-21T08:27:15.629795+0000","last_unstale":"2024-09-21T08:27:25.628775+0000","last_undegraded":"2024-09-21T08:27:25.628775+0000","last_fullsized":"2024-09-21T08:27:25.628775+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:23:29.515888+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.996766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5,2],"acting":[0,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":365,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639674+0000","last_change":"2024-09-21T08:27:15.611268+0000","last_active":"2024-09-21T08:27:25.639674+0000","last_peered":"2024-09-21T08:27:25.639674+0000","last_clean":"2024-09-21T08:27:25.639674+0000","last_became_active":"2024-09-21T08:27:15.610949+0000","last_became_peered":"2024-09-21T08:27:15.610949+0000","last_unstale":"2024-09-21T08:27:25.639674+0000","last_undegraded":"2024-09-21T08:27:25.639674+0000","last_fullsized":"2024-09-21T08:27:25.639674+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0,5],"acting":[7,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":407,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.639774+0000","last_change":"2024-09-21T08:27:15.598339+0000","last_active":"2024-09-21T08:27:25.639774+0000","last_peered":"2024-09-21T08:27:25.639774+0000","last_clean":"2024-09-21T08:27:25.639774+0000","last_became_active":"2024-09-21T08:27:15.598023+0000","last_became_peered":"2024-09-21T08:27:15.598023+0000","last_unstale":"2024-09-21T08:27:25.639774+0000","last_undegraded":"2024-09-21T08:27:25.639774+0000","last_fullsized":"2024-09-21T08:27:25.639774+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.736816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.736816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:50.643683+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,5],"acting":[7,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.1d","version":"0'0","reported_seq":259,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641091+0000","last_change":"2024-09-21T08:27:15.600770+0000","last_active":"2024-09-21T08:27:25.641091+0000","last_peered":"2024-09-21T08:27:25.641091+0000","last_clean":"2024-09-21T08:27:25.641091+0000","last_became_active":"2024-09-21T08:27:15.599097+0000","last_became_peered":"2024-09-21T08:27:15.599097+0000","last_unstale":"2024-09-21T08:27:25.641091+0000","last_undegraded":"2024-09-21T08:27:25.641091+0000","last_fullsized":"2024-09-21T08:27:25.641091+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:13.999162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:13.999162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:28:42.693513+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":295,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.626677+0000","last_change":"2024-09-21T08:27:15.630279+0000","last_active":"2024-09-21T08:27:25.626677+0000","last_peered":"2024-09-21T08:27:25.626677+0000","last_clean":"2024-09-21T08:27:25.626677+0000","last_became_active":"2024-09-21T08:27:15.629171+0000","last_became_peered":"2024-09-21T08:27:15.629171+0000","last_unstale":"2024-09-21T08:27:25.626677+0000","last_undegraded":"2024-09-21T08:27:25.626677+0000","last_fullsized":"2024-09-21T08:27:25.626677+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.997047+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,7],"acting":[0,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":415,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.641664+0000","last_change":"2024-09-21T08:27:15.676612+0000","last_active":"2024-09-21T08:27:25.641664+0000","last_peered":"2024-09-21T08:27:25.641664+0000","last_clean":"2024-09-21T08:27:25.641664+0000","last_became_active":"2024-09-21T08:27:15.609135+0000","last_became_peered":"2024-09-21T08:27:15.609135+0000","last_unstale":"2024-09-21T08:27:25.641664+0000","last_undegraded":"2024-09-21T08:27:25.641664+0000","last_fullsized":"2024-09-21T08:27:25.641664+0000","mapping_epoch":241,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,7],"acting":[5,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":383,"reported_epoch":252,"state":"active+clean","last_fresh":"2024-09-21T08:27:25.628920+0000","last_change":"2024-09-21T08:25:25.350265+0000","last_active":"2024-09-21T08:27:25.628920+0000","last_peered":"2024-09-21T08:27:25.628920+0000","last_clean":"2024-09-21T08:27:25.628920+0000","last_became_active":"2024-09-21T08:23:34.572855+0000","last_became_peered":"2024-09-21T08:23:34.572855+0000","last_unstale":"2024-09-21T08:27:25.628920+0000","last_undegraded":"2024-09-21T08:27:25.628920+0000","last_fullsized":"2024-09-21T08:27:25.628920+0000","mapping_epoch":69,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.350197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.350197+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:27:28.997226+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":26,"stat_sum":{"num_bytes":213864354,"num_objects":90,"num_object_clones":46,"num_object_copies":270,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":90,"num_whiteouts":6,"num_read":1061,"num_read_kb":1098782,"num_write":738,"num_write_kb":283599,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":328,"num_bytes_recovered":212673882,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":634380288,"data_stored":780201208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3829,"ondisk_log_size":3829,"up":78,"acting":78,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":52,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1261,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":94,"ondisk_log_size":94,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":675,"num_read_kb":432,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":443,"num_bytes_recovered":8906,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":659456,"data_stored":17520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":414,"ondisk_log_size":414,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":4144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":7675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":225,"num_read_kb":220,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1243,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":2951840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":3,"acting":3,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070516,"num_pgs":121,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":128700,"kb_used_data":101124,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93616452,"statfs":{"total":95995035648,"available":95863246848,"internally_reserved":0,"allocated":103550976,"data_stored":133098815,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397562,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659197,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":146004,"kb_used_data":103976,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93599148,"statfs":{"total":95995035648,"available":95845527552,"internally_reserved":0,"allocated":106471424,"data_stored":133625185,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986243,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182471,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411405,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":178744,"kb_used_data":135080,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93566408,"statfs":{"total":95995035648,"available":95812001792,"internally_reserved":0,"allocated":138321920,"data_stored":176695661,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":0,"seq":55834574929,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738454,"num_pgs":121,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":183856,"kb_used_data":137668,"kb_used_omap":1,"kb_used_meta":43646,"kb_avail":93561296,"statfs":{"total":95995035648,"available":95806767104,"internally_reserved":0,"allocated":140972032,"data_stored":187921354,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":44693957},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":620,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":3262,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":3042,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":140836864,"data_stored":180614099,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":45531136,"data_stored":53470237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":140140544,"data_stored":174640519,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70848512,"data_stored":84468350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":103608320,"data_stored":126963768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":105320448,"data_stored":132173820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:27:29.110 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T08:27:29.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_0 pg_num 36 2024-09-21T08:27:29.143 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:29.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:29.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:29.165 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:29.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T08:27:29.266 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:29.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:29.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:29.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:29.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:29.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:29.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:29.412 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:29.471 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:29.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:29.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:29.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:29.534 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:29.585 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:29.636 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:29.656 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:29.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:29.657 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:29.696 INFO:tasks.rados.rados.0.smithi137.stdout:943: done (2 left) 2024-09-21T08:27:29.696 INFO:tasks.rados.rados.0.smithi137.stdout:944: done (1 left) 2024-09-21T08:27:29.697 INFO:tasks.rados.rados.0.smithi137.stdout:945: done (0 left) 2024-09-21T08:27:29.697 INFO:tasks.rados.rados.0.smithi137.stdout:946: delete oid 8 current snap is 84 2024-09-21T08:27:29.698 INFO:tasks.rados.rados.0.smithi137.stdout:946: done (0 left) 2024-09-21T08:27:29.698 INFO:tasks.rados.rados.0.smithi137.stdout:947: delete oid 12 current snap is 84 2024-09-21T08:27:29.702 INFO:tasks.rados.rados.0.smithi137.stdout:947: done (0 left) 2024-09-21T08:27:29.702 INFO:tasks.rados.rados.0.smithi137.stdout:948: delete oid 7 current snap is 84 2024-09-21T08:27:29.704 INFO:tasks.rados.rados.0.smithi137.stdout:948: done (0 left) 2024-09-21T08:27:29.704 INFO:tasks.rados.rados.0.smithi137.stdout:949: snap_create 2024-09-21T08:27:29.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:29 smithi137 ceph-mon[29634]: pgmap v287: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 205 MiB data, 622 MiB used, 357 GiB / 358 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-21T08:27:29.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:29 smithi137 ceph-mon[29634]: from='client.15336 v1:172.21.15.137:0/4155850971' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:27:29.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:29 smithi137 ceph-mon[29634]: 3.9 scrub starts 2024-09-21T08:27:29.742 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:29 smithi137 ceph-mon[29634]: 3.9 scrub ok 2024-09-21T08:27:29.742 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:29 smithi137 ceph-mon[29634]: osdmap e255: 8 total, 8 up, 4 in 2024-09-21T08:27:29.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:29 smithi137 ceph-mon[41044]: pgmap v287: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 205 MiB data, 622 MiB used, 357 GiB / 358 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-21T08:27:29.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:29 smithi137 ceph-mon[41044]: from='client.15336 v1:172.21.15.137:0/4155850971' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:27:29.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:29 smithi137 ceph-mon[41044]: 3.9 scrub starts 2024-09-21T08:27:29.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:29 smithi137 ceph-mon[41044]: 3.9 scrub ok 2024-09-21T08:27:29.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:29 smithi137 ceph-mon[41044]: osdmap e255: 8 total, 8 up, 4 in 2024-09-21T08:27:29.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:29.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:29.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:29.780 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:29.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:27:29.874 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:29 smithi165 ceph-mon[32893]: pgmap v287: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 205 MiB data, 622 MiB used, 357 GiB / 358 GiB avail; 6.4 MiB/s rd, 3.1 MiB/s wr, 18 op/s 2024-09-21T08:27:29.874 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:29 smithi165 ceph-mon[32893]: from='client.15336 v1:172.21.15.137:0/4155850971' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:27:29.875 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:29 smithi165 ceph-mon[32893]: 3.9 scrub starts 2024-09-21T08:27:29.875 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:29 smithi165 ceph-mon[32893]: 3.9 scrub ok 2024-09-21T08:27:29.875 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:29 smithi165 ceph-mon[32893]: osdmap e255: 8 total, 8 up, 4 in 2024-09-21T08:27:29.881 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:29.902 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:29.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:29.903 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:29.980 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:27:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:27:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:27:29.980 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:27:29.980 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:27:30.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:30.026 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:30.026 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:30.027 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:30.127 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:30.151 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:30.152 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:30.152 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:30.164 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:30.253 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:30.274 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:30.275 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:30.275 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:30.376 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:30.396 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:30.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:30.397 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:30.498 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:30.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:30.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:30.520 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:30.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:30.637 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:30.637 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:30.642 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:30.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:30.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:30.654 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:30.655 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:30.670 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:30.671 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:30.686 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:30.687 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:30.698 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:27:30.707 INFO:tasks.rados.rados.0.smithi137.stdout:949: done (0 left) 2024-09-21T08:27:30.707 INFO:tasks.rados.rados.0.smithi137.stdout:950: copy_from oid 22 from oid 47 current snap is 85 2024-09-21T08:27:30.707 INFO:tasks.rados.rados.0.smithi137.stdout:951: read oid 27 snap 84 2024-09-21T08:27:30.707 INFO:tasks.rados.rados.0.smithi137.stdout:951: expect deleted 2024-09-21T08:27:30.707 INFO:tasks.rados.rados.0.smithi137.stdout:952: write (excl) oid 44 current snap is 85 2024-09-21T08:27:30.707 INFO:tasks.rados.rados.0.smithi137.stdout:952: seq_num 359 ranges {433713=482982,1424565=609148} 2024-09-21T08:27:30.707 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 7 pg_num to 36 2024-09-21T08:27:30.708 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:30.708 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:30.712 INFO:tasks.rados.rados.0.smithi137.stdout:952: writing smithi13779221-44 from 433713 to 916695 tid 1 2024-09-21T08:27:30.719 INFO:tasks.rados.rados.0.smithi137.stdout:952: writing smithi13779221-44 from 1424565 to 2033713 tid 2 2024-09-21T08:27:30.719 INFO:tasks.rados.rados.0.smithi137.stdout:951: done (2 left) 2024-09-21T08:27:30.719 INFO:tasks.rados.rados.0.smithi137.stdout:953: write oid 5 current snap is 85 2024-09-21T08:27:30.719 INFO:tasks.rados.rados.0.smithi137.stdout:953: seq_num 360 ranges {450520=558231,1483047=515125} 2024-09-21T08:27:30.724 INFO:tasks.rados.rados.0.smithi137.stdout:953: writing smithi13779221-5 from 450520 to 1008751 tid 1 2024-09-21T08:27:30.730 INFO:tasks.rados.rados.0.smithi137.stdout:953: writing smithi13779221-5 from 1483047 to 1998172 tid 2 2024-09-21T08:27:30.731 INFO:tasks.rados.rados.0.smithi137.stdout:954: snap_remove snap 81 2024-09-21T08:27:30.736 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:30.736 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:30.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:30.754 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:30.755 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:30.766 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:30.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:30.767 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:30.771 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:30.772 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:30.803 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T08:27:30.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:30.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:30.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:30.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:30.910 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:27:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:27:30.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: 5.18 scrub starts 2024-09-21T08:27:30.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: 5.18 scrub ok 2024-09-21T08:27:30.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: osdmap e256: 8 total, 8 up, 4 in 2024-09-21T08:27:30.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3347473738' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:30.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: 3.1 scrub starts 2024-09-21T08:27:30.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: 3.1 scrub ok 2024-09-21T08:27:30.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:30.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:30.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: pgmap v290: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 205 MiB data, 622 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:30.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/293672237' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-09-21T08:27:30.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: from='client.45239 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-09-21T08:27:30.911 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3418212797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:30.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: 5.18 scrub starts 2024-09-21T08:27:30.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: 5.18 scrub ok 2024-09-21T08:27:30.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: osdmap e256: 8 total, 8 up, 4 in 2024-09-21T08:27:30.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3347473738' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:30.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: 3.1 scrub starts 2024-09-21T08:27:30.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: 3.1 scrub ok 2024-09-21T08:27:30.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:30.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:30.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: pgmap v290: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 205 MiB data, 622 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:30.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/293672237' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-09-21T08:27:30.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: from='client.45239 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-09-21T08:27:30.912 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:30 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3418212797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:30.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: 5.18 scrub starts 2024-09-21T08:27:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: 5.18 scrub ok 2024-09-21T08:27:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: osdmap e256: 8 total, 8 up, 4 in 2024-09-21T08:27:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3347473738' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: 3.1 scrub starts 2024-09-21T08:27:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: 3.1 scrub ok 2024-09-21T08:27:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:27:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: pgmap v290: 158 pgs: 15 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 137 active+clean; 205 MiB data, 622 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/293672237' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-09-21T08:27:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: from='client.45239 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]: dispatch 2024-09-21T08:27:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:30 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3418212797' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:31.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:31.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:31.015 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:31.052 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:31.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:31.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:31.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:31.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:31.238 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:31.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:31.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:31.260 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:31.361 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:31.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:31.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:31.384 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:31.485 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:31.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:31.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:31.508 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:31.535 INFO:tasks.rados.rados.0.smithi137.stdout:954: done (3 left) 2024-09-21T08:27:31.535 INFO:tasks.rados.rados.0.smithi137.stdout:955: snap_remove snap 83 2024-09-21T08:27:31.566 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:31.566 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:31.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:31.631 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:31.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:31.632 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:31.733 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:31.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-mon[32893]: 6.1b scrub starts 2024-09-21T08:27:31.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-mon[32893]: 6.1b scrub ok 2024-09-21T08:27:31.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:27:31.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:27:31.742 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3347473738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:31.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-mon[32893]: from='client.45239 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-21T08:27:31.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-mon[32893]: osdmap e257: 8 total, 8 up, 4 in 2024-09-21T08:27:31.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-mon[32893]: 3.a scrub starts 2024-09-21T08:27:31.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-mon[32893]: 3.a scrub ok 2024-09-21T08:27:31.743 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-mon[32893]: osdmap e258: 8 total, 8 up, 4 in 2024-09-21T08:27:31.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:31.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:31.755 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:31.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[29634]: 6.1b scrub starts 2024-09-21T08:27:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[29634]: 6.1b scrub ok 2024-09-21T08:27:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:27:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:27:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3347473738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[29634]: from='client.45239 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-21T08:27:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[29634]: osdmap e257: 8 total, 8 up, 4 in 2024-09-21T08:27:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[29634]: 3.a scrub starts 2024-09-21T08:27:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[29634]: 3.a scrub ok 2024-09-21T08:27:31.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[29634]: osdmap e258: 8 total, 8 up, 4 in 2024-09-21T08:27:31.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[41044]: 6.1b scrub starts 2024-09-21T08:27:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[41044]: 6.1b scrub ok 2024-09-21T08:27:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:27:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:27:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3347473738' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[41044]: from='client.45239 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "36"}]': finished 2024-09-21T08:27:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[41044]: osdmap e257: 8 total, 8 up, 4 in 2024-09-21T08:27:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[41044]: 3.a scrub starts 2024-09-21T08:27:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[41044]: 3.a scrub ok 2024-09-21T08:27:31.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:31 smithi137 ceph-mon[41044]: osdmap e258: 8 total, 8 up, 4 in 2024-09-21T08:27:31.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:31.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:31.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:31.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:31.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:32.002 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:32.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:32.003 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:32.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:27:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:27:32.103 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:32.125 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:32.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:32.126 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:32.170 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T08:27:32.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:32.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:32.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:32.249 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:32.350 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:32.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:32.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:32.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:32.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:32.626 INFO:tasks.rados.rados.0.smithi137.stdout:955: done (3 left) 2024-09-21T08:27:32.626 INFO:tasks.rados.rados.0.smithi137.stdout:956: snap_create 2024-09-21T08:27:32.626 INFO:tasks.rados.rados.0.smithi137.stdout:950: finishing copy_from racing read to smithi13779221-22 2024-09-21T08:27:32.626 INFO:tasks.rados.rados.0.smithi137.stdout:950: finishing copy_from to smithi13779221-22 2024-09-21T08:27:32.626 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:27:32.626 INFO:tasks.rados.rados.0.smithi137.stdout:953: finishing write tid 1 to smithi13779221-5 2024-09-21T08:27:32.626 INFO:tasks.rados.rados.0.smithi137.stdout:952: finishing write tid 1 to smithi13779221-44 2024-09-21T08:27:32.626 INFO:tasks.rados.rados.0.smithi137.stdout:952: finishing write tid 2 to smithi13779221-44 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:952: finishing write tid 3 to smithi13779221-44 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:952: finishing write tid 4 to smithi13779221-44 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:952: oid 44 updating version 0 to 246 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:952: oid 44 version 246 is already newer than 245 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:952: oid 44 version 246 is already newer than 244 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 246 (ObjNum 359 snap 85 seq_num 359) dirty exists 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:952: left oid 44 (ObjNum 359 snap 85 seq_num 359) 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:953: finishing write tid 2 to smithi13779221-5 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:953: finishing write tid 3 to smithi13779221-5 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:953: finishing write tid 4 to smithi13779221-5 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:953: oid 5 updating version 0 to 316 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:953: oid 5 updating version 316 to 318 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:953: oid 5 version 318 is already newer than 317 2024-09-21T08:27:32.627 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 318 (ObjNum 360 snap 85 seq_num 360) dirty exists 2024-09-21T08:27:32.628 INFO:tasks.rados.rados.0.smithi137.stdout:953: left oid 5 (ObjNum 360 snap 85 seq_num 360) 2024-09-21T08:27:32.652 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:32.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:32.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:32.674 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:32.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:32.796 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:32.797 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:32.797 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:32.898 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:32.914 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[29634]: 4.17 scrub starts 2024-09-21T08:27:32.914 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[29634]: 4.17 scrub ok 2024-09-21T08:27:32.914 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[29634]: 6.f scrub starts 2024-09-21T08:27:32.914 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[29634]: 6.f scrub ok 2024-09-21T08:27:32.914 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:32.914 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[29634]: pgmap v293: 158 pgs: 14 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 139 active+clean; 173 MiB data, 563 MiB used, 357 GiB / 358 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[29634]: osdmap e259: 8 total, 8 up, 4 in 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[41044]: 4.17 scrub starts 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[41044]: 4.17 scrub ok 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[41044]: 6.f scrub starts 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[41044]: 6.f scrub ok 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[41044]: pgmap v293: 158 pgs: 14 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 139 active+clean; 173 MiB data, 563 MiB used, 357 GiB / 358 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:32.915 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:32.916 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-21T08:27:32.916 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:32 smithi137 ceph-mon[41044]: osdmap e259: 8 total, 8 up, 4 in 2024-09-21T08:27:32.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:32.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:32.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:32 smithi165 ceph-mon[32893]: 4.17 scrub starts 2024-09-21T08:27:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:32 smithi165 ceph-mon[32893]: 4.17 scrub ok 2024-09-21T08:27:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:32 smithi165 ceph-mon[32893]: 6.f scrub starts 2024-09-21T08:27:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:32 smithi165 ceph-mon[32893]: 6.f scrub ok 2024-09-21T08:27:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:32 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:32 smithi165 ceph-mon[32893]: pgmap v293: 158 pgs: 14 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 139 active+clean; 173 MiB data, 563 MiB used, 357 GiB / 358 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-21T08:27:33.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:32 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:33.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:32 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:33.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:32 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "36"}]': finished 2024-09-21T08:27:33.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:32 smithi165 ceph-mon[32893]: osdmap e259: 8 total, 8 up, 4 in 2024-09-21T08:27:33.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:33.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:33.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:33.043 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:33.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:33.162 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:33.162 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:33.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:33.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:33.167 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:33.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:33.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:33.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:33.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:33.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:33.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:33.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:33.412 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:33.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:33.514 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:27:33.536 INFO:tasks.rados.rados.0.smithi137.stdout:950: done (3 left) 2024-09-21T08:27:33.536 INFO:tasks.rados.rados.0.smithi137.stdout:952: done (2 left) 2024-09-21T08:27:33.536 INFO:tasks.rados.rados.0.smithi137.stdout:953: done (1 left) 2024-09-21T08:27:33.536 INFO:tasks.rados.rados.0.smithi137.stdout:956: done (0 left) 2024-09-21T08:27:33.536 INFO:tasks.rados.rados.0.smithi137.stdout:957: write (excl) oid 15 current snap is 86 2024-09-21T08:27:33.536 INFO:tasks.rados.rados.0.smithi137.stdout:957: seq_num 361 ranges {467327=633480,1825176=242151} 2024-09-21T08:27:33.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:33.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:33.539 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:33.541 INFO:tasks.rados.rados.0.smithi137.stdout:957: writing smithi13779221-15 from 467327 to 1100807 tid 1 2024-09-21T08:27:33.547 INFO:tasks.rados.rados.0.smithi137.stdout:957: writing smithi13779221-15 from 1825176 to 2067327 tid 2 2024-09-21T08:27:33.548 INFO:tasks.rados.rados.0.smithi137.stdout:958: read oid 42 snap -1 2024-09-21T08:27:33.548 INFO:tasks.rados.rados.0.smithi137.stdout:958: expect deleted 2024-09-21T08:27:33.548 INFO:tasks.rados.rados.0.smithi137.stdout:959: write oid 41 current snap is 86 2024-09-21T08:27:33.548 INFO:tasks.rados.rados.0.smithi137.stdout:959: seq_num 362 ranges {484134=708729,1883658=200476} 2024-09-21T08:27:33.556 INFO:tasks.rados.rados.0.smithi137.stdout:959: writing smithi13779221-41 from 484134 to 1192863 tid 1 2024-09-21T08:27:33.561 INFO:tasks.rados.rados.0.smithi137.stdout:959: writing smithi13779221-41 from 1883658 to 2084134 tid 2 2024-09-21T08:27:33.561 INFO:tasks.rados.rados.0.smithi137.stdout:960: read oid 43 snap -1 2024-09-21T08:27:33.561 INFO:tasks.rados.rados.0.smithi137.stdout:960: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:27:33.561 INFO:tasks.rados.rados.0.smithi137.stdout:961: write oid 49 current snap is 86 2024-09-21T08:27:33.561 INFO:tasks.rados.rados.0.smithi137.stdout:961: seq_num 363 ranges {500941=783978,1942140=158801} 2024-09-21T08:27:33.568 INFO:tasks.rados.rados.0.smithi137.stdout:961: writing smithi13779221-49 from 500941 to 1284919 tid 1 2024-09-21T08:27:33.574 INFO:tasks.rados.rados.0.smithi137.stdout:961: writing smithi13779221-49 from 1942140 to 2100941 tid 2 2024-09-21T08:27:33.574 INFO:tasks.rados.rados.0.smithi137.stdout:962: read oid 37 snap -1 2024-09-21T08:27:33.574 INFO:tasks.rados.rados.0.smithi137.stdout:962: expect (ObjNum 243 snap 50 seq_num 243) 2024-09-21T08:27:33.574 INFO:tasks.rados.rados.0.smithi137.stdout:963: rollback oid 27 current snap is 86 2024-09-21T08:27:33.574 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 27 to 84 2024-09-21T08:27:33.574 INFO:tasks.rados.rados.0.smithi137.stdout:957: finishing write tid 1 to smithi13779221-15 2024-09-21T08:27:33.574 INFO:tasks.rados.rados.0.smithi137.stdout:958: done (6 left) 2024-09-21T08:27:33.574 INFO:tasks.rados.rados.0.smithi137.stdout:964: copy_from oid 9 from oid 23 current snap is 86 2024-09-21T08:27:33.574 INFO:tasks.rados.rados.0.smithi137.stdout:957: finishing write tid 2 to smithi13779221-15 2024-09-21T08:27:33.574 INFO:tasks.rados.rados.0.smithi137.stdout:957: finishing write tid 3 to smithi13779221-15 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:957: finishing write tid 4 to smithi13779221-15 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:957: oid 15 updating version 0 to 256 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:957: oid 15 updating version 256 to 257 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:957: oid 15 version 257 is already newer than 255 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 257 (ObjNum 361 snap 86 seq_num 361) dirty exists 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:957: left oid 15 (ObjNum 361 snap 86 seq_num 361) 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:959: finishing write tid 1 to smithi13779221-41 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:957: done (6 left) 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:965: delete oid 18 current snap is 86 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:959: finishing write tid 2 to smithi13779221-41 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:959: finishing write tid 3 to smithi13779221-41 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:959: finishing write tid 4 to smithi13779221-41 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:959: oid 41 updating version 0 to 292 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:959: oid 41 updating version 292 to 294 2024-09-21T08:27:33.575 INFO:tasks.rados.rados.0.smithi137.stdout:959: oid 41 version 294 is already newer than 293 2024-09-21T08:27:33.576 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 294 (ObjNum 362 snap 86 seq_num 362) dirty exists 2024-09-21T08:27:33.576 INFO:tasks.rados.rados.0.smithi137.stdout:959: left oid 41 (ObjNum 362 snap 86 seq_num 362) 2024-09-21T08:27:33.576 INFO:tasks.rados.rados.0.smithi137.stdout:960: expect (ObjNum 156 snap 31 seq_num 156) 2024-09-21T08:27:33.640 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:33.662 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:33.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:33.662 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:33.705 INFO:tasks.rados.rados.0.smithi137.stdout:961: finishing write tid 1 to smithi13779221-49 2024-09-21T08:27:33.706 INFO:tasks.rados.rados.0.smithi137.stdout:959: done (6 left) 2024-09-21T08:27:33.706 INFO:tasks.rados.rados.0.smithi137.stdout:960: done (5 left) 2024-09-21T08:27:33.706 INFO:tasks.rados.rados.0.smithi137.stdout:965: done (4 left) 2024-09-21T08:27:33.706 INFO:tasks.rados.rados.0.smithi137.stdout:966: snap_remove snap 85 2024-09-21T08:27:33.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:33.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T08:27:33.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:33.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:33.787 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:33.820 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:33.888 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:33.909 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:33.910 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:33.910 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:34.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:34.032 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:34.033 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:34.033 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:34.134 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:34.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:34.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:34.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:34.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:27:34.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:34.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:34.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:34.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:34.325 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:34.325 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:34.351 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:34.351 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:34.363 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:34.374 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:34.374 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:34.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:34.395 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:34.395 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:34.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:34.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:34.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:34.417 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:34.418 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:34.434 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:34.435 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:34.451 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:34.451 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:34.467 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:34.468 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:34.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:34.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:34.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:34.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:966: done (4 left) 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:967: write (excl) oid 44 current snap is 86 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:963: finishing rollback tid 1 to smithi13779221-27 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 0 (ObjNum 129 snap 0 seq_num 3760572080) dirty dne 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:961: finishing write tid 2 to smithi13779221-49 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:961: finishing write tid 3 to smithi13779221-49 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:961: finishing write tid 4 to smithi13779221-49 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:961: oid 49 updating version 0 to 326 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:961: oid 49 updating version 326 to 328 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:961: oid 49 version 328 is already newer than 327 2024-09-21T08:27:34.540 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 328 (ObjNum 363 snap 86 seq_num 363) dirty exists 2024-09-21T08:27:34.541 INFO:tasks.rados.rados.0.smithi137.stdout:961: left oid 49 (ObjNum 363 snap 86 seq_num 363) 2024-09-21T08:27:34.541 INFO:tasks.rados.rados.0.smithi137.stdout:964: finishing copy_from racing read to smithi13779221-9 2024-09-21T08:27:34.541 INFO:tasks.rados.rados.0.smithi137.stdout:962: expect (ObjNum 243 snap 50 seq_num 243) 2024-09-21T08:27:34.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[29634]: 5.1 scrub starts 2024-09-21T08:27:34.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[29634]: 5.1 scrub ok 2024-09-21T08:27:34.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[29634]: 5.f deep-scrub starts 2024-09-21T08:27:34.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[29634]: 5.f deep-scrub ok 2024-09-21T08:27:34.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[29634]: osdmap e260: 8 total, 8 up, 4 in 2024-09-21T08:27:34.618 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:34.619 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:34.619 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3685594377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:34.619 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[41044]: 5.1 scrub starts 2024-09-21T08:27:34.619 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[41044]: 5.1 scrub ok 2024-09-21T08:27:34.619 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[41044]: 5.f deep-scrub starts 2024-09-21T08:27:34.619 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[41044]: 5.f deep-scrub ok 2024-09-21T08:27:34.619 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[41044]: osdmap e260: 8 total, 8 up, 4 in 2024-09-21T08:27:34.619 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:34.619 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:34.619 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3685594377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:34.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:34.630 INFO:tasks.rados.rados.0.smithi137.stdout:964: finishing copy_from to smithi13779221-9 2024-09-21T08:27:34.630 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 9 v 174 (ObjNum 329 snap 78 seq_num 329) dirty exists 2024-09-21T08:27:34.630 INFO:tasks.rados.rados.0.smithi137.stdout:967: seq_num 365 ranges {534555=650829,1659104=475451} 2024-09-21T08:27:34.636 INFO:tasks.rados.rados.0.smithi137.stdout:967: writing smithi13779221-44 from 534555 to 1185384 tid 1 2024-09-21T08:27:34.642 INFO:tasks.rados.rados.0.smithi137.stdout:967: writing smithi13779221-44 from 1659104 to 2134555 tid 2 2024-09-21T08:27:34.642 INFO:tasks.rados.rados.0.smithi137.stdout:961: done (4 left) 2024-09-21T08:27:34.642 INFO:tasks.rados.rados.0.smithi137.stdout:962: done (3 left) 2024-09-21T08:27:34.642 INFO:tasks.rados.rados.0.smithi137.stdout:963: done (2 left) 2024-09-21T08:27:34.642 INFO:tasks.rados.rados.0.smithi137.stdout:964: done (1 left) 2024-09-21T08:27:34.642 INFO:tasks.rados.rados.0.smithi137.stdout:968: snap_create 2024-09-21T08:27:34.643 INFO:tasks.rados.rados.0.smithi137.stdout:967: finishing write tid 1 to smithi13779221-44 2024-09-21T08:27:34.644 INFO:tasks.rados.rados.0.smithi137.stdout:967: finishing write tid 2 to smithi13779221-44 2024-09-21T08:27:34.644 INFO:tasks.rados.rados.0.smithi137.stdout:967: finishing write tid 3 to smithi13779221-44 2024-09-21T08:27:34.644 INFO:tasks.rados.rados.0.smithi137.stdout:967: finishing write tid 4 to smithi13779221-44 2024-09-21T08:27:34.644 INFO:tasks.rados.rados.0.smithi137.stdout:967: oid 44 updating version 0 to 247 2024-09-21T08:27:34.644 INFO:tasks.rados.rados.0.smithi137.stdout:967: oid 44 updating version 247 to 249 2024-09-21T08:27:34.645 INFO:tasks.rados.rados.0.smithi137.stdout:967: oid 44 version 249 is already newer than 248 2024-09-21T08:27:34.645 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 249 (ObjNum 365 snap 86 seq_num 365) dirty exists 2024-09-21T08:27:34.645 INFO:tasks.rados.rados.0.smithi137.stdout:967: left oid 44 (ObjNum 365 snap 86 seq_num 365) 2024-09-21T08:27:34.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:34.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:34.649 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:34.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:34.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:34.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:34.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:34.825 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:34 smithi165 ceph-mon[32893]: 5.1 scrub starts 2024-09-21T08:27:34.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:34 smithi165 ceph-mon[32893]: 5.1 scrub ok 2024-09-21T08:27:34.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:34 smithi165 ceph-mon[32893]: 5.f deep-scrub starts 2024-09-21T08:27:34.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:34 smithi165 ceph-mon[32893]: 5.f deep-scrub ok 2024-09-21T08:27:34.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:34 smithi165 ceph-mon[32893]: osdmap e260: 8 total, 8 up, 4 in 2024-09-21T08:27:34.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:34 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:34.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:34 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]: dispatch 2024-09-21T08:27:34.826 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3685594377' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:34.872 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:34.894 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:34.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:34.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:34.995 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:34.999 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:34.999 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:27:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:27:35.000 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:27:35.000 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:27:35.016 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:35.017 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:35.017 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:35.118 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:35.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:35.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:35.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:35.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:35.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:35.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:35.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:35.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:35.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:35.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:35.391 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:35.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:35.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:35.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:35.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:35.545 INFO:tasks.rados.rados.0.smithi137.stdout:967: done (1 left) 2024-09-21T08:27:35.545 INFO:tasks.rados.rados.0.smithi137.stdout:968: done (0 left) 2024-09-21T08:27:35.545 INFO:tasks.rados.rados.0.smithi137.stdout:969: delete oid 46 current snap is 87 2024-09-21T08:27:35.551 INFO:tasks.rados.rados.0.smithi137.stdout:969: done (0 left) 2024-09-21T08:27:35.551 INFO:tasks.rados.rados.0.smithi137.stdout:970: snap_remove snap 86 2024-09-21T08:27:35.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:35.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[29634]: 7.0 scrub starts 2024-09-21T08:27:35.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[29634]: 7.0 scrub ok 2024-09-21T08:27:35.623 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[29634]: pgmap v296: 168 pgs: 10 unknown, 14 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 139 active+clean; 173 MiB data, 563 MiB used, 357 GiB / 358 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[29634]: 7.1a scrub starts 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[29634]: 7.1a scrub ok 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[29634]: osdmap e261: 8 total, 8 up, 4 in 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[41044]: 7.0 scrub starts 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[41044]: 7.0 scrub ok 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[41044]: pgmap v296: 168 pgs: 10 unknown, 14 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 139 active+clean; 173 MiB data, 563 MiB used, 357 GiB / 358 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[41044]: 7.1a scrub starts 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[41044]: 7.1a scrub ok 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-21T08:27:35.624 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:35 smithi137 ceph-mon[41044]: osdmap e261: 8 total, 8 up, 4 in 2024-09-21T08:27:35.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:35.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:35.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:35.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T08:27:35.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:35.760 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:35.760 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:35.761 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:35 smithi165 ceph-mon[32893]: 7.0 scrub starts 2024-09-21T08:27:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:35 smithi165 ceph-mon[32893]: 7.0 scrub ok 2024-09-21T08:27:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:35 smithi165 ceph-mon[32893]: pgmap v296: 168 pgs: 10 unknown, 14 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 139 active+clean; 173 MiB data, 563 MiB used, 357 GiB / 358 GiB avail; 14 MiB/s rd, 2.0 MiB/s wr, 20 op/s 2024-09-21T08:27:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:35 smithi165 ceph-mon[32893]: 7.1a scrub starts 2024-09-21T08:27:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:35 smithi165 ceph-mon[32893]: 7.1a scrub ok 2024-09-21T08:27:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:35 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "36"}]': finished 2024-09-21T08:27:35.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:35 smithi165 ceph-mon[32893]: osdmap e261: 8 total, 8 up, 4 in 2024-09-21T08:27:35.861 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:35.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:35.883 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:35.883 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:35.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:36.005 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:36.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:36.006 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:36.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:36.122 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:36.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:36.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:36.130 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:36.142 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:36.231 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:36.252 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:36.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:36.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:36.375 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:36.376 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:36.376 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:36.477 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:36.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:36.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:36.500 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:36.541 INFO:tasks.rados.rados.0.smithi137.stdout:970: done (0 left) 2024-09-21T08:27:36.541 INFO:tasks.rados.rados.0.smithi137.stdout:971: delete oid 12 current snap is 87 2024-09-21T08:27:36.545 INFO:tasks.rados.rados.0.smithi137.stdout:971: done (0 left) 2024-09-21T08:27:36.545 INFO:tasks.rados.rados.0.smithi137.stdout:972: snap_create 2024-09-21T08:27:36.600 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:36.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:36.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:36.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:36.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[29634]: 7.c scrub starts 2024-09-21T08:27:36.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[29634]: 7.c scrub ok 2024-09-21T08:27:36.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[29634]: 7.a scrub starts 2024-09-21T08:27:36.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[29634]: 7.a scrub ok 2024-09-21T08:27:36.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[29634]: osdmap e262: 8 total, 8 up, 4 in 2024-09-21T08:27:36.660 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[29634]: osdmap e263: 8 total, 8 up, 4 in 2024-09-21T08:27:36.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[41044]: 7.c scrub starts 2024-09-21T08:27:36.660 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[41044]: 7.c scrub ok 2024-09-21T08:27:36.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[41044]: 7.a scrub starts 2024-09-21T08:27:36.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[41044]: 7.a scrub ok 2024-09-21T08:27:36.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[41044]: osdmap e262: 8 total, 8 up, 4 in 2024-09-21T08:27:36.661 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:36 smithi137 ceph-mon[41044]: osdmap e263: 8 total, 8 up, 4 in 2024-09-21T08:27:36.723 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:36.745 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:36.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:36.746 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:36.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:36.867 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:36.867 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:27:36.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:27:36.867 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:27:36.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.867 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:336 7.9 7:9d4d36a0:::smithi13779221-27:head [copy-from ver 7] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:27:36.867 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:27:36.867 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 168.29494836999999, 2024-09-21T08:27:36.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050439811000000001, 2024-09-21T08:27:36.867 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 336 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569727+0000", 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4899999999999999e-06 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569733+0000", 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9120000000000003e-06 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569741+0000", 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4999999999999999e-06 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569749+0000", 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2099999999999996e-06 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.583356+0000", 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013607731 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584178+0000", 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082153700000000001 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.610330+0000", 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026152526999999998 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.617047+0000", 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067167470000000003 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.619683+0000", 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026361039999999998 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620040+0000", 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035616499999999998 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620096+0000", 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6922000000000002e-05 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620165+0000", 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8966000000000003e-05 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:337 7.9 7:9d4d36a0:::smithi13779221-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 168.28700853800001, 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.038882593999999999, 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.873 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 337 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577668+0000", 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2960000000000001e-06 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577674+0000", 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5479999999999998e-06 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577687+0000", 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2829e-05 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577698+0000", 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0832999999999999e-05 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584334+0000", 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066355900000000002 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584379+0000", 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5575999999999999e-05 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616393+0000", 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032013592 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616468+0000", 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5150999999999999e-05 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616548+0000", 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0179000000000006e-05 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.877 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1594 7.16 7:6a4067f0:::smithi13779221-23:head [write 433680~466703] snapc 32=[32] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.326885+0000", 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 79.537789361999998, 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.62075412799999996, 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1594 2024-09-21T08:27:36.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.326885+0000", 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.326885+0000", 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.326887+0000", 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9489999999999999e-06 2024-09-21T08:27:36.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327091+0000", 2024-09-21T08:27:36.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020473400000000001 2024-09-21T08:27:36.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327103+0000", 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.153e-05 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327108+0000", 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4489999999999998e-06 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327157+0000", 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8429e-05 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:36.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327164+0000", 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6899999999999992e-06 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.925242+0000", 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59807755799999995 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928049+0000", 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028070149999999999 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.944633+0000", 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016583665000000001 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.944714+0000", 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1816000000000004e-05 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.944826+0000", 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111823 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946814+0000", 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001987815 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946867+0000", 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3052e-05 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947560+0000", 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069266300000000005 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947595+0000", 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5394e-05 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947639+0000", 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3545999999999999e-05 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1595 7.16 7:6a4067f0:::smithi13779221-23:head [write 1577796~455884] snapc 32=[32] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:27:36.885 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.332973+0000", 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 79.531700964999999, 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61484140200000004, 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1595 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.332973+0000", 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.332973+0000", 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.332975+0000", 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5680000000000001e-06 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333220+0000", 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024565600000000002 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333230+0000", 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0223e-05 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333236+0000", 2024-09-21T08:27:36.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5230000000000003e-06 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333274+0000", 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8232999999999998e-05 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333280+0000", 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2040000000000002e-06 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928405+0000", 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59512471499999997 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928464+0000", 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9123000000000003e-05 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.945609+0000", 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017145059000000001 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.945660+0000", 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0562999999999997e-05 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.945757+0000", 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6949999999999998e-05 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947123+0000", 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001366173 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947230+0000", 2024-09-21T08:27:36.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010691700000000001 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947760+0000", 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053026199999999999 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947791+0000", 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0923000000000002e-05 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947814+0000", 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3309999999999999e-05 2024-09-21T08:27:36.892 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1596 7.16 7:6a4067f0:::smithi13779221-23:head [setxattr _header (57) in=7b,truncate 2033680] snapc 32=[32] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:27:36.893 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.333246+0000", 2024-09-21T08:27:36.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:36.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:36.894 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 79.531427887999996, 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61472346600000005, 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1596 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333246+0000", 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333246+0000", 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333247+0000", 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1299999999999998e-07 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333251+0000", 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1699999999999999e-06 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333256+0000", 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3340000000000001e-06 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333262+0000", 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2780000000000001e-06 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333310+0000", 2024-09-21T08:27:36.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8572000000000002e-05 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333320+0000", 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8099999999999992e-06 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928508+0000", 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59518784499999999 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928553+0000", 2024-09-21T08:27:36.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4545000000000001e-05 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946225+0000", 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017672832999999999 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946285+0000", 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9466999999999998e-05 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946398+0000", 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011340000000000001 2024-09-21T08:27:36.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947240+0000", 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00084159800000000004 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947726+0000", 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048660799999999997 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947904+0000", 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017706899999999999 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947931+0000", 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7742e-05 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947970+0000", 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8281999999999997e-05 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.901 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1597 7.16 7:6a4067f0:::smithi13779221-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.333275+0000", 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 79.531398584000002, 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61483585500000004, 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1597 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333275+0000", 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333275+0000", 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333276+0000", 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2099999999999996e-07 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333278+0000", 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4229999999999999e-06 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333285+0000", 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0820000000000001e-06 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333291+0000", 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2089999999999996e-06 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333334+0000", 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3482999999999997e-05 2024-09-21T08:27:36.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333338+0000", 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0169999999999998e-06 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928654+0000", 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59531576200000003 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928713+0000", 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8971999999999997e-05 2024-09-21T08:27:36.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946717+0000", 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018003894999999999 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946799+0000", 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2521000000000005e-05 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.948007+0000", 2024-09-21T08:27:36.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012079180000000001 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.948053+0000", 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5661000000000002e-05 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.948111+0000", 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8291000000000001e-05 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.908 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1742 7.d 7:b31d99ce:::smithi13779221-21:head [copy-from ver 99] snapc 39=[39] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T08:27:36.908 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:30.594911+0000", 2024-09-21T08:27:36.908 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 66.269762643000007, 2024-09-21T08:27:36.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021953376, 2024-09-21T08:27:36.909 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.909 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.909 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.909 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1742 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594911+0000", 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594911+0000", 2024-09-21T08:27:36.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594913+0000", 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2190000000000001e-06 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594919+0000", 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2049999999999999e-06 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594930+0000", 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1416e-05 2024-09-21T08:27:36.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595001+0000", 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0678000000000003e-05 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595054+0000", 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2966000000000001e-05 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595167+0000", 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113167 2024-09-21T08:27:36.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.605259+0000", 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010091829 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.610873+0000", 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056143 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616680+0000", 2024-09-21T08:27:36.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058072159999999996 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616794+0000", 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113318 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616826+0000", 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2264999999999997e-05 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616865+0000", 2024-09-21T08:27:36.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8797000000000001e-05 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1910 7.f 7:f1c8dcbe:::smithi13779221-1:head [write 423609~701524] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.544612+0000", 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 48.320061860999999, 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0451588970000001, 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1910 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.544612+0000", 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.544612+0000", 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.544614+0000", 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7349999999999999e-06 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545147+0000", 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053351400000000002 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545158+0000", 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0241000000000001e-05 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545168+0000", 2024-09-21T08:27:36.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0047000000000001e-05 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545223+0000", 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5858e-05 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545235+0000", 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1525000000000001e-05 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.568108+0000", 2024-09-21T08:27:36.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.022872655 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.568937+0000", 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082958499999999998 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585576+0000", 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016639025000000002 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585673+0000", 2024-09-21T08:27:36.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6496999999999994e-05 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585866+0000", 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000193019 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589564+0000", 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036982180000000001 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589584+0000", 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0097000000000001e-05 2024-09-21T08:27:36.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589637+0000", 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2794000000000003e-05 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589704+0000", 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7438e-05 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589771+0000", 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6649000000000005e-05 2024-09-21T08:27:36.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1911 7.f 7:f1c8dcbe:::smithi13779221-1:head [write 1628332~472089] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.545246+0000", 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 48.319427933, 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0460972529999999, 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.922 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1911 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545246+0000", 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545246+0000", 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545247+0000", 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5799999999999998e-07 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545571+0000", 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000323703 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545581+0000", 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0515e-05 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545588+0000", 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8689999999999998e-06 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545632+0000", 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4149e-05 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545641+0000", 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4509999999999992e-06 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569011+0000", 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0233707439999999 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569071+0000", 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9499000000000001e-05 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587283+0000", 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018211871000000001 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587400+0000", 2024-09-21T08:27:36.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117216 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587560+0000", 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016029999999999999 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589591+0000", 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002030853 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589853+0000", 2024-09-21T08:27:36.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026147699999999998 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591174+0000", 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001321046 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591220+0000", 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6112999999999997e-05 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591343+0000", 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012348900000000001 2024-09-21T08:27:36.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1912 7.f 7:f1c8dcbe:::smithi13779221-1:head [write 2648987~174622] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.549709+0000", 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 48.314965151999999, 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.041840002, 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1912 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549709+0000", 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549709+0000", 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549712+0000", 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4740000000000001e-06 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549841+0000", 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012874699999999999 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549873+0000", 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2208000000000001e-05 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549885+0000", 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1878e-05 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549977+0000", 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2044000000000002e-05 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549988+0000", 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0723999999999999e-05 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569102+0000", 2024-09-21T08:27:36.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.019114064 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569150+0000", 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7576e-05 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.588268+0000", 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019118133999999998 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.588389+0000", 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012087999999999999 2024-09-21T08:27:36.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.588544+0000", 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015529300000000001 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589784+0000", 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001239696 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589910+0000", 2024-09-21T08:27:36.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126747 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591404+0000", 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014933500000000001 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591452+0000", 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8714000000000001e-05 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591549+0000", 2024-09-21T08:27:36.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6472999999999996e-05 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1913 7.f 7:f1c8dcbe:::smithi13779221-1:head [setxattr _header (56) in=7b,truncate 2823609] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.550029+0000", 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 48.314644891999997, 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0418481909999999, 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.936 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1913 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550029+0000", 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550029+0000", 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550030+0000", 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1100000000000004e-07 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550036+0000", 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5999999999999997e-06 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550044+0000", 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6090000000000001e-06 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550051+0000", 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5150000000000003e-06 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550105+0000", 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4258999999999997e-05 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550113+0000", 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7549999999999994e-06 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569285+0000", 2024-09-21T08:27:36.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0191723500000001 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569337+0000", 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2369e-05 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589107+0000", 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019769201 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589204+0000", 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7133999999999995e-05 2024-09-21T08:27:36.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589314+0000", 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110308 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589972+0000", 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000658079 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.590152+0000", 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017941499999999999 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591636+0000", 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014841979999999999 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591704+0000", 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8051000000000003e-05 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591877+0000", 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000173437 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.942 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1914 7.f 7:f1c8dcbe:::smithi13779221-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.550141+0000", 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 48.314533165, 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.042047948, 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1914 2024-09-21T08:27:36.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550141+0000", 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550141+0000", 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550142+0000", 2024-09-21T08:27:36.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1099999999999995e-07 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550145+0000", 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2550000000000002e-06 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550153+0000", 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0439999999999996e-06 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550158+0000", 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2290000000000002e-06 2024-09-21T08:27:36.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550217+0000", 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9219999999999999e-05 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550224+0000", 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.199e-06 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569368+0000", 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.019143197 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569416+0000", 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8168999999999999e-05 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.590013+0000", 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020597133 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.590102+0000", 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9364999999999996e-05 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591921+0000", 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018190389999999999 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.592009+0000", 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8096000000000005e-05 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.592189+0000", 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017929099999999999 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2031 7.12 7:4d793d31:::smithi13779221-9:head [copy-from ver 248] snapc 47=[47,46,45,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e224)", 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:57.568773+0000", 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 39.295901200000003, 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020237121, 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.949 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2031 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.568773+0000", 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.568773+0000", 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.568774+0000", 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.254e-06 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.568779+0000", 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0699999999999997e-06 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.568787+0000", 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3119999999999994e-06 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.568795+0000", 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3139999999999998e-06 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.568856+0000", 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1344000000000001e-05 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.568970+0000", 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113564 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.577747+0000", 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0087774129999999995 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.587225+0000", 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094779919999999993 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.587467+0000", 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024246399999999999 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.588882+0000", 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001414474 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.588952+0000", 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9808000000000007e-05 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:57.589010+0000", 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8112000000000002e-05 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2114 7.4 7:26d6e516:::smithi13779221-33:head [copy-from ver 279] snapc 4a=[4a,49,48,47,45,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.523396+0000", 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 35.341277706, 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030629719, 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.955 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2114 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523396+0000", 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523396+0000", 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523397+0000", 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4199999999999996e-07 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523401+0000", 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5970000000000002e-06 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523406+0000", 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0159999999999999e-06 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523411+0000", 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4070000000000003e-06 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523463+0000", 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1997999999999999e-05 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523469+0000", 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6810000000000001e-06 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.528702+0000", 2024-09-21T08:27:36.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052329919999999997 2024-09-21T08:27:36.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.528799+0000", 2024-09-21T08:27:36.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6323000000000006e-05 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540822+0000", 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012022812000000001 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.547580+0000", 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067587519999999998 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553592+0000", 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0060113479999999997 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553788+0000", 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019646900000000001 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553858+0000", 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9536999999999995e-05 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.554026+0000", 2024-09-21T08:27:36.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000168245 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2115 7.4 7:26d6e516:::smithi13779221-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e228)", 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.523423+0000", 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 35.341251266999997, 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020869774000000001, 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.962 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2115 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523423+0000", 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523423+0000", 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523423+0000", 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4999999999999998e-07 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523426+0000", 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.633e-06 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523472+0000", 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5775999999999997e-05 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523476+0000", 2024-09-21T08:27:36.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6460000000000001e-06 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.529003+0000", 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055263200000000004 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.529053+0000", 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0130000000000003e-05 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.544083+0000", 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015030646 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.544186+0000", 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102518 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.544292+0000", 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106655 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2215 7.12 7:4ccee2c4:::smithi13779221-50:head [copy-from ver 311] snapc 4e=[4e,4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:27:36.966 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544476+0000", 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 26.320197919999998, 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026654285999999999, 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2215 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544476+0000", 2024-09-21T08:27:36.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544476+0000", 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544477+0000", 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6400000000000002e-07 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544480+0000", 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2179999999999999e-06 2024-09-21T08:27:36.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544485+0000", 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7620000000000003e-06 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544489+0000", 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7180000000000004e-06 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544550+0000", 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0439000000000002e-05 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544557+0000", 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1069999999999999e-06 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.547211+0000", 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026542789999999998 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.547316+0000", 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104938 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.562697+0000", 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015381183 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.568461+0000", 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057634590000000003 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.569259+0000", 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00079830400000000001 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.571009+0000", 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017497599999999999 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.571062+0000", 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2991999999999997e-05 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.571130+0000", 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8563000000000001e-05 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2216 7.12 7:4ccee2c4:::smithi13779221-50:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544502+0000", 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 26.320172111000002, 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022611549000000002, 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2216 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544502+0000", 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544502+0000", 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544502+0000", 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5199999999999996e-07 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544505+0000", 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6230000000000001e-06 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544511+0000", 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8579999999999996e-06 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544516+0000", 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1649999999999997e-06 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544596+0000", 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9684000000000005e-05 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544602+0000", 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2249999999999997e-06 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.547457+0000", 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028550049999999999 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:36.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.547501+0000", 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3945e-05 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.566950+0000", 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019448564000000002 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.567030+0000", 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0321999999999994e-05 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.567113+0000", 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3505999999999996e-05 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2222 7.12 7:4d793d31:::smithi13779221-9:head [append 0~0] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.527713+0000", 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 25.336960662999999, 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022209310999999999, 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.978 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2222 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527713+0000", 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527713+0000", 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527716+0000", 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6129999999999998e-06 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527723+0000", 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9059999999999996e-06 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527739+0000", 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6589e-05 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527748+0000", 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3119999999999994e-06 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527857+0000", 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010948800000000001 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527864+0000", 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2579999999999998e-06 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.533682+0000", 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058173829999999998 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.533784+0000", 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102514 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.533993+0000", 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020875399999999999 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.541706+0000", 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077127029999999996 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.543228+0000", 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001522203 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.549783+0000", 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065552730000000003 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.549860+0000", 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6810000000000005e-05 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.549923+0000", 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2504999999999999e-05 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2223 7.12 7:4d793d31:::smithi13779221-9:head [rollback 4d] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:27:36.984 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.527776+0000", 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 25.336897854, 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027805044000000001, 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2223 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527776+0000", 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527776+0000", 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527777+0000", 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5000000000000001e-07 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527780+0000", 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4580000000000002e-06 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527789+0000", 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1780000000000003e-06 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527794+0000", 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6319999999999996e-06 2024-09-21T08:27:36.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527912+0000", 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117972 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527918+0000", 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0680000000000002e-06 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.539337+0000", 2024-09-21T08:27:36.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011418991999999999 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.539460+0000", 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012270400000000001 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.539688+0000", 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022830499999999999 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.541712+0000", 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020237190000000002 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.544647+0000", 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002934688 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555467+0000", 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010820299 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555525+0000", 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8349e-05 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555581+0000", 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5829999999999999e-05 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2224 7.12 7:4d793d31:::smithi13779221-9:head [append 0~0] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:27:36.991 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.527802+0000", 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 25.336871877, 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027898684999999999, 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2224 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:36.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527802+0000", 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527802+0000", 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527803+0000", 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6400000000000001e-07 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:36.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527806+0000", 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.056e-06 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527811+0000", 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5879999999999997e-06 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527816+0000", 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9930000000000003e-06 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527939+0000", 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012283699999999999 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527945+0000", 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2900000000000002e-06 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.540300+0000", 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012355457 2024-09-21T08:27:36.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.540404+0000", 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104356 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.540576+0000", 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017131899999999999 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.541717+0000", 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001140907 2024-09-21T08:27:36.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.544877+0000", 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031601649999999999 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555618+0000", 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010741550000000001 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555665+0000", 2024-09-21T08:27:36.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6968000000000002e-05 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555701+0000", 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5435000000000002e-05 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:36.998 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:27:36.998 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:36 smithi165 ceph-mon[32893]: 7.c scrub starts 2024-09-21T08:27:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:36 smithi165 ceph-mon[32893]: 7.c scrub ok 2024-09-21T08:27:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:36 smithi165 ceph-mon[32893]: 7.a scrub starts 2024-09-21T08:27:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:36 smithi165 ceph-mon[32893]: 7.a scrub ok 2024-09-21T08:27:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:36 smithi165 ceph-mon[32893]: osdmap e262: 8 total, 8 up, 4 in 2024-09-21T08:27:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:36 smithi165 ceph-mon[32893]: osdmap e263: 8 total, 8 up, 4 in 2024-09-21T08:27:37.019 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:37.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:37.020 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:37.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:37.142 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:37.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:37.143 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:37.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:37.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:37.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:37.266 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:37.366 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:37.387 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:37.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:37.388 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:37.489 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:37.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:37.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:37.511 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:37.537 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:27:37.542 INFO:tasks.rados.rados.0.smithi137.stdout:972: done (0 left) 2024-09-21T08:27:37.542 INFO:tasks.rados.rados.0.smithi137.stdout:973: copy_from oid 17 from oid 11 current snap is 88 2024-09-21T08:27:37.542 INFO:tasks.rados.rados.0.smithi137.stdout:974: delete oid 3 current snap is 88 2024-09-21T08:27:37.549 INFO:tasks.rados.rados.0.smithi137.stdout:974: done (1 left) 2024-09-21T08:27:37.549 INFO:tasks.rados.rados.0.smithi137.stdout:975: write (excl) oid 35 current snap is 88 2024-09-21T08:27:37.549 INFO:tasks.rados.rados.0.smithi137.stdout:975: seq_num 369 ranges {601783=551825,1776679=425104} 2024-09-21T08:27:37.556 INFO:tasks.rados.rados.0.smithi137.stdout:975: writing smithi13779221-35 from 601783 to 1153608 tid 1 2024-09-21T08:27:37.563 INFO:tasks.rados.rados.0.smithi137.stdout:975: writing smithi13779221-35 from 1776679 to 2201783 tid 2 2024-09-21T08:27:37.563 INFO:tasks.rados.rados.0.smithi137.stdout:976: copy_from oid 28 from oid 9 current snap is 88 2024-09-21T08:27:37.563 INFO:tasks.rados.rados.0.smithi137.stdout:973: finishing copy_from racing read to smithi13779221-17 2024-09-21T08:27:37.563 INFO:tasks.rados.rados.0.smithi137.stdout:973: finishing copy_from to smithi13779221-17 2024-09-21T08:27:37.563 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 99 (ObjNum 200 snap 40 seq_num 200) dirty exists 2024-09-21T08:27:37.563 INFO:tasks.rados.rados.0.smithi137.stdout:975: finishing write tid 1 to smithi13779221-35 2024-09-21T08:27:37.563 INFO:tasks.rados.rados.0.smithi137.stdout:973: done (2 left) 2024-09-21T08:27:37.563 INFO:tasks.rados.rados.0.smithi137.stdout:977: snap_create 2024-09-21T08:27:37.565 INFO:tasks.rados.rados.0.smithi137.stdout:975: finishing write tid 2 to smithi13779221-35 2024-09-21T08:27:37.565 INFO:tasks.rados.rados.0.smithi137.stdout:975: finishing write tid 3 to smithi13779221-35 2024-09-21T08:27:37.565 INFO:tasks.rados.rados.0.smithi137.stdout:975: finishing write tid 4 to smithi13779221-35 2024-09-21T08:27:37.565 INFO:tasks.rados.rados.0.smithi137.stdout:975: oid 35 updating version 0 to 252 2024-09-21T08:27:37.565 INFO:tasks.rados.rados.0.smithi137.stdout:975: oid 35 version 252 is already newer than 250 2024-09-21T08:27:37.565 INFO:tasks.rados.rados.0.smithi137.stdout:975: oid 35 version 252 is already newer than 251 2024-09-21T08:27:37.566 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 252 (ObjNum 369 snap 88 seq_num 369) dirty exists 2024-09-21T08:27:37.566 INFO:tasks.rados.rados.0.smithi137.stdout:975: left oid 35 (ObjNum 369 snap 88 seq_num 369) 2024-09-21T08:27:37.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:37.629 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 6, 1, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:27:37.629 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:27:37.629 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T08:27:37.629 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 1.000000 2024-09-21T08:27:37.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 1 2024-09-21T08:27:37.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:37.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:37.635 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:37.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:37.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:37.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:37.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:37.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:37.880 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:37.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:37.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:37.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[29634]: 7.1a scrub starts 2024-09-21T08:27:37.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[29634]: 7.1a scrub ok 2024-09-21T08:27:37.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[29634]: pgmap v299: 168 pgs: 4 remapped+peering, 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 147 active+clean; 144 MiB data, 496 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:37.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[29634]: 7.3 scrub starts 2024-09-21T08:27:37.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[29634]: 7.3 scrub ok 2024-09-21T08:27:37.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3169249615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3169249615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[29634]: osdmap e264: 8 total, 8 up, 4 in 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[41044]: 7.1a scrub starts 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[41044]: 7.1a scrub ok 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[41044]: pgmap v299: 168 pgs: 4 remapped+peering, 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 147 active+clean; 144 MiB data, 496 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[41044]: 7.3 scrub starts 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[41044]: 7.3 scrub ok 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3169249615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3169249615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:27:37.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:37 smithi137 ceph-mon[41044]: osdmap e264: 8 total, 8 up, 4 in 2024-09-21T08:27:37.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:38.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:38.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:38.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:37 smithi165 ceph-mon[32893]: 7.1a scrub starts 2024-09-21T08:27:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:37 smithi165 ceph-mon[32893]: 7.1a scrub ok 2024-09-21T08:27:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:37 smithi165 ceph-mon[32893]: pgmap v299: 168 pgs: 4 remapped+peering, 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 147 active+clean; 144 MiB data, 496 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:37 smithi165 ceph-mon[32893]: 7.3 scrub starts 2024-09-21T08:27:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:37 smithi165 ceph-mon[32893]: 7.3 scrub ok 2024-09-21T08:27:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3169249615' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:27:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:37 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3169249615' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:27:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:37 smithi165 ceph-mon[32893]: osdmap e264: 8 total, 8 up, 4 in 2024-09-21T08:27:38.050 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:27:38.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:38.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:38.126 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:38.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:38.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:38.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:38.250 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:38.350 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:38.372 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:38.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:38.373 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:38.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:38.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:38.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:38.496 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:38.546 INFO:tasks.rados.rados.0.smithi137.stdout:975: done (2 left) 2024-09-21T08:27:38.546 INFO:tasks.rados.rados.0.smithi137.stdout:977: done (1 left) 2024-09-21T08:27:38.546 INFO:tasks.rados.rados.0.smithi137.stdout:978: copy_from oid 20 from oid 15 current snap is 89 2024-09-21T08:27:38.546 INFO:tasks.rados.rados.0.smithi137.stdout:979: read oid 22 snap -1 2024-09-21T08:27:38.547 INFO:tasks.rados.rados.0.smithi137.stdout:979: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:27:38.547 INFO:tasks.rados.rados.0.smithi137.stdout:980: snap_create 2024-09-21T08:27:38.554 INFO:tasks.rados.rados.0.smithi137.stdout:979: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:27:38.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:38.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:38.619 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:38.619 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:38.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T08:27:38.715 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:38.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:38.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:38.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:38.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:38.775 INFO:tasks.rados.rados.0.smithi137.stdout:978: finishing copy_from racing read to smithi13779221-20 2024-09-21T08:27:38.775 INFO:tasks.rados.rados.0.smithi137.stdout:978: finishing copy_from to smithi13779221-20 2024-09-21T08:27:38.775 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 257 (ObjNum 361 snap 86 seq_num 361) dirty exists 2024-09-21T08:27:38.775 INFO:tasks.rados.rados.0.smithi137.stdout:976: finishing copy_from racing read to smithi13779221-28 2024-09-21T08:27:38.775 INFO:tasks.rados.rados.0.smithi137.stdout:976: finishing copy_from to smithi13779221-28 2024-09-21T08:27:38.775 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 174 (ObjNum 329 snap 78 seq_num 329) dirty exists 2024-09-21T08:27:38.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[29634]: 7.21 scrub starts 2024-09-21T08:27:38.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[29634]: 7.21 scrub ok 2024-09-21T08:27:38.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[29634]: 5.0 deep-scrub starts 2024-09-21T08:27:38.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[29634]: 5.0 deep-scrub ok 2024-09-21T08:27:38.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:27:38.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:27:38.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[29634]: 7.e scrub starts 2024-09-21T08:27:38.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[29634]: 7.e scrub ok 2024-09-21T08:27:38.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[29634]: osdmap e265: 8 total, 8 up, 4 in 2024-09-21T08:27:38.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[41044]: 7.21 scrub starts 2024-09-21T08:27:38.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[41044]: 7.21 scrub ok 2024-09-21T08:27:38.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[41044]: 5.0 deep-scrub starts 2024-09-21T08:27:38.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[41044]: 5.0 deep-scrub ok 2024-09-21T08:27:38.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:27:38.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:27:38.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[41044]: 7.e scrub starts 2024-09-21T08:27:38.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[41044]: 7.e scrub ok 2024-09-21T08:27:38.840 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:38 smithi137 ceph-mon[41044]: osdmap e265: 8 total, 8 up, 4 in 2024-09-21T08:27:38.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:38.863 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:38.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:38.864 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:38.965 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:38.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:38.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:38.989 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:38 smithi165 ceph-mon[32893]: 7.21 scrub starts 2024-09-21T08:27:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:38 smithi165 ceph-mon[32893]: 7.21 scrub ok 2024-09-21T08:27:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:38 smithi165 ceph-mon[32893]: 5.0 deep-scrub starts 2024-09-21T08:27:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:38 smithi165 ceph-mon[32893]: 5.0 deep-scrub ok 2024-09-21T08:27:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:38 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:27:39.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:38 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:27:39.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:38 smithi165 ceph-mon[32893]: 7.e scrub starts 2024-09-21T08:27:39.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:38 smithi165 ceph-mon[32893]: 7.e scrub ok 2024-09-21T08:27:39.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:38 smithi165 ceph-mon[32893]: osdmap e265: 8 total, 8 up, 4 in 2024-09-21T08:27:39.089 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:39.111 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:39.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:39.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:39.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:39.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:39.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:39.234 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:39.238 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:39.239 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:39.239 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:39.239 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:39.244 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:39.245 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:39.262 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:39.262 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:39.280 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:39.280 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:39.288 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:39.294 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:39.303 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:39.303 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:39.323 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:39.324 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:39.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:39.340 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:39.340 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:39.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:39.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:39.357 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:39.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:39.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:39.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:39.482 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:39.550 INFO:tasks.rados.rados.0.smithi137.stdout:976: done (3 left) 2024-09-21T08:27:39.550 INFO:tasks.rados.rados.0.smithi137.stdout:978: done (2 left) 2024-09-21T08:27:39.550 INFO:tasks.rados.rados.0.smithi137.stdout:979: done (1 left) 2024-09-21T08:27:39.550 INFO:tasks.rados.rados.0.smithi137.stdout:980: done (0 left) 2024-09-21T08:27:39.550 INFO:tasks.rados.rados.0.smithi137.stdout:981: rollback oid 27 current snap is 90 2024-09-21T08:27:39.550 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 27 to 87 2024-09-21T08:27:39.550 INFO:tasks.rados.rados.0.smithi137.stdout:982: snap_remove snap 88 2024-09-21T08:27:39.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:39.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:39.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:39.606 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:39.707 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:39.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:39.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:39.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:39.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:39 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:27:39.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:39 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:27:39.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:39 smithi137 ceph-mon[29634]: pgmap v302: 168 pgs: 4 remapped+peering, 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 147 active+clean; 144 MiB data, 496 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:39.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:39 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/309378290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:39.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:39 smithi137 ceph-mon[29634]: osdmap e266: 8 total, 8 up, 4 in 2024-09-21T08:27:39.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:39 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:27:39.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:39 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:27:39.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:39 smithi137 ceph-mon[41044]: pgmap v302: 168 pgs: 4 remapped+peering, 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 147 active+clean; 144 MiB data, 496 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:39.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:39 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/309378290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:39.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:39 smithi137 ceph-mon[41044]: osdmap e266: 8 total, 8 up, 4 in 2024-09-21T08:27:39.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:39.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:39.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:39.851 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:39.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:39.973 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:39.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:39 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:27:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:39 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:27:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:39 smithi165 ceph-mon[32893]: pgmap v302: 168 pgs: 4 remapped+peering, 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 147 active+clean; 144 MiB data, 496 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:39 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/309378290' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:39 smithi165 ceph-mon[32893]: osdmap e266: 8 total, 8 up, 4 in 2024-09-21T08:27:40.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:40.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:40.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:40.097 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:40.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:40.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:27:40.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:40.220 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:40.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:40.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:40.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:40.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:40.343 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:40.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:40.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:40.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:40.466 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:40.547 INFO:tasks.rados.rados.0.smithi137.stdout:982: done (1 left) 2024-09-21T08:27:40.548 INFO:tasks.rados.rados.0.smithi137.stdout:983: rollback oid 45 current snap is 90 2024-09-21T08:27:40.548 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 45 to 87 2024-09-21T08:27:40.548 INFO:tasks.rados.rados.0.smithi137.stdout:984: write (excl) oid 11 current snap is 90 2024-09-21T08:27:40.548 INFO:tasks.rados.rados.0.smithi137.stdout:984: seq_num 370 ranges {618590=627074,1835161=383429} 2024-09-21T08:27:40.551 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.2 primary-affinity to 1 (8655362) 2024-09-21T08:27:40.555 INFO:tasks.rados.rados.0.smithi137.stdout:984: writing smithi13779221-11 from 618590 to 1245664 tid 1 2024-09-21T08:27:40.563 INFO:tasks.rados.rados.0.smithi137.stdout:984: writing smithi13779221-11 from 1835161 to 2218590 tid 2 2024-09-21T08:27:40.563 INFO:tasks.rados.rados.0.smithi137.stdout:985: write (excl) oid 49 current snap is 90 2024-09-21T08:27:40.563 INFO:tasks.rados.rados.0.smithi137.stdout:981: finishing rollback tid 1 to smithi13779221-27 2024-09-21T08:27:40.563 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 0 (ObjNum 465 snap 0 seq_num 379559584) dirty dne 2024-09-21T08:27:40.563 INFO:tasks.rados.rados.0.smithi137.stdout:983: finishing rollback tid 0 to smithi13779221-45 2024-09-21T08:27:40.563 INFO:tasks.rados.rados.0.smithi137.stdout:983: finishing rollback tid 1 to smithi13779221-45 2024-09-21T08:27:40.563 INFO:tasks.rados.rados.0.smithi137.stdout:983: finishing rollback tid 2 to smithi13779221-45 2024-09-21T08:27:40.563 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 274 (ObjNum 336 snap 79 seq_num 336) dirty exists 2024-09-21T08:27:40.563 INFO:tasks.rados.rados.0.smithi137.stdout:985: seq_num 371 ranges {635397=702323,1893643=341754} 2024-09-21T08:27:40.567 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:40.570 INFO:tasks.rados.rados.0.smithi137.stdout:985: writing smithi13779221-49 from 635397 to 1337720 tid 1 2024-09-21T08:27:40.577 INFO:tasks.rados.rados.0.smithi137.stdout:985: writing smithi13779221-49 from 1893643 to 2235397 tid 2 2024-09-21T08:27:40.577 INFO:tasks.rados.rados.0.smithi137.stdout:981: done (3 left) 2024-09-21T08:27:40.577 INFO:tasks.rados.rados.0.smithi137.stdout:983: done (2 left) 2024-09-21T08:27:40.577 INFO:tasks.rados.rados.0.smithi137.stdout:986: write oid 14 current snap is 90 2024-09-21T08:27:40.577 INFO:tasks.rados.rados.0.smithi137.stdout:984: finishing write tid 1 to smithi13779221-11 2024-09-21T08:27:40.577 INFO:tasks.rados.rados.0.smithi137.stdout:984: finishing write tid 2 to smithi13779221-11 2024-09-21T08:27:40.578 INFO:tasks.rados.rados.0.smithi137.stdout:984: finishing write tid 3 to smithi13779221-11 2024-09-21T08:27:40.578 INFO:tasks.rados.rados.0.smithi137.stdout:984: finishing write tid 4 to smithi13779221-11 2024-09-21T08:27:40.578 INFO:tasks.rados.rados.0.smithi137.stdout:984: oid 11 updating version 0 to 212 2024-09-21T08:27:40.578 INFO:tasks.rados.rados.0.smithi137.stdout:984: oid 11 version 212 is already newer than 210 2024-09-21T08:27:40.578 INFO:tasks.rados.rados.0.smithi137.stdout:984: oid 11 version 212 is already newer than 211 2024-09-21T08:27:40.578 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 212 (ObjNum 370 snap 90 seq_num 370) dirty exists 2024-09-21T08:27:40.578 INFO:tasks.rados.rados.0.smithi137.stdout:984: left oid 11 (ObjNum 370 snap 90 seq_num 370) 2024-09-21T08:27:40.578 INFO:tasks.rados.rados.0.smithi137.stdout:985: finishing write tid 1 to smithi13779221-49 2024-09-21T08:27:40.578 INFO:tasks.rados.rados.0.smithi137.stdout:986: seq_num 372 ranges {652204=777572,1952125=300079} 2024-09-21T08:27:40.585 INFO:tasks.rados.rados.0.smithi137.stdout:986: writing smithi13779221-14 from 652204 to 1429776 tid 1 2024-09-21T08:27:40.591 INFO:tasks.rados.rados.0.smithi137.stdout:986: writing smithi13779221-14 from 1952125 to 2252204 tid 2 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:985: finishing write tid 2 to smithi13779221-49 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:985: finishing write tid 3 to smithi13779221-49 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:985: finishing write tid 4 to smithi13779221-49 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:985: oid 49 updating version 0 to 337 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:985: oid 49 updating version 337 to 338 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:985: oid 49 version 338 is already newer than 336 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 338 (ObjNum 371 snap 90 seq_num 371) dirty exists 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:985: left oid 49 (ObjNum 371 snap 90 seq_num 371) 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:986: finishing write tid 1 to smithi13779221-14 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:984: done (2 left) 2024-09-21T08:27:40.592 INFO:tasks.rados.rados.0.smithi137.stdout:985: done (1 left) 2024-09-21T08:27:40.593 INFO:tasks.rados.rados.0.smithi137.stdout:987: snap_remove snap 87 2024-09-21T08:27:40.597 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:40.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:40.598 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:40.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:40.722 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:40.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:40.723 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:40.823 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:40.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:40.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:40.846 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:40.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:40 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/478911885' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:27:40.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:40 smithi137 ceph-mon[29634]: from='client.45269 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:27:40.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:40 smithi137 ceph-mon[29634]: from='client.45269 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T08:27:40.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:40 smithi137 ceph-mon[29634]: osdmap e267: 8 total, 8 up, 4 in 2024-09-21T08:27:40.936 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:27:40 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:27:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:27:40.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:40 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/478911885' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:27:40.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:40 smithi137 ceph-mon[41044]: from='client.45269 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:27:40.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:40 smithi137 ceph-mon[41044]: from='client.45269 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T08:27:40.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:40 smithi137 ceph-mon[41044]: osdmap e267: 8 total, 8 up, 4 in 2024-09-21T08:27:40.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:40.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:40.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:40.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:40 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/478911885' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:27:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:40 smithi165 ceph-mon[32893]: from='client.45269 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:27:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:40 smithi165 ceph-mon[32893]: from='client.45269 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 1.0}]': finished 2024-09-21T08:27:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:40 smithi165 ceph-mon[32893]: osdmap e267: 8 total, 8 up, 4 in 2024-09-21T08:27:41.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:41.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:41.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:41.091 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:41.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:41.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:41.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:41.214 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:41.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:41.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:41.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:41.337 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:41.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:41.446 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:41.446 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:41.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:41.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:41.460 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:41.535 INFO:tasks.rados.rados.0.smithi137.stdout:987: done (1 left) 2024-09-21T08:27:41.535 INFO:tasks.rados.rados.0.smithi137.stdout:988: delete oid 28 current snap is 90 2024-09-21T08:27:41.536 INFO:tasks.rados.rados.0.smithi137.stdout:986: finishing write tid 2 to smithi13779221-14 2024-09-21T08:27:41.536 INFO:tasks.rados.rados.0.smithi137.stdout:986: finishing write tid 3 to smithi13779221-14 2024-09-21T08:27:41.536 INFO:tasks.rados.rados.0.smithi137.stdout:986: finishing write tid 4 to smithi13779221-14 2024-09-21T08:27:41.536 INFO:tasks.rados.rados.0.smithi137.stdout:986: oid 14 updating version 0 to 242 2024-09-21T08:27:41.536 INFO:tasks.rados.rados.0.smithi137.stdout:986: oid 14 version 242 is already newer than 240 2024-09-21T08:27:41.536 INFO:tasks.rados.rados.0.smithi137.stdout:986: oid 14 version 242 is already newer than 241 2024-09-21T08:27:41.536 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 242 (ObjNum 372 snap 90 seq_num 372) dirty exists 2024-09-21T08:27:41.536 INFO:tasks.rados.rados.0.smithi137.stdout:986: left oid 14 (ObjNum 372 snap 90 seq_num 372) 2024-09-21T08:27:41.540 INFO:tasks.rados.rados.0.smithi137.stdout:986: done (1 left) 2024-09-21T08:27:41.540 INFO:tasks.rados.rados.0.smithi137.stdout:988: done (0 left) 2024-09-21T08:27:41.540 INFO:tasks.rados.rados.0.smithi137.stdout:989: snap_remove snap 89 2024-09-21T08:27:41.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:41.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:41.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:41.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:41.659 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:41.684 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:41.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:41.706 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:41.706 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:41.807 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:41.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:41.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:41.829 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:41.915 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:41 smithi137 ceph-mon[29634]: pgmap v305: 168 pgs: 4 remapped+peering, 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 147 active+clean; 144 MiB data, 496 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:41.915 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:41 smithi137 ceph-mon[29634]: osdmap e268: 8 total, 8 up, 4 in 2024-09-21T08:27:41.915 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:41 smithi137 ceph-mon[41044]: pgmap v305: 168 pgs: 4 remapped+peering, 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 147 active+clean; 144 MiB data, 496 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:41.915 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:41 smithi137 ceph-mon[41044]: osdmap e268: 8 total, 8 up, 4 in 2024-09-21T08:27:41.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:41.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:41.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:41.952 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:41 smithi165 ceph-mon[32893]: pgmap v305: 168 pgs: 4 remapped+peering, 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 147 active+clean; 144 MiB data, 496 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:41 smithi165 ceph-mon[32893]: osdmap e268: 8 total, 8 up, 4 in 2024-09-21T08:27:42.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:27:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:27:42.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:42.075 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:42.076 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:42.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:42.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:42.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:42.198 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:42.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:42.302 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:27:42.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:42.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:42.322 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:42.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:42.445 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:42.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:42.446 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:42.538 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:27:42.538 INFO:tasks.rados.rados.0.smithi137.stdout:989: done (0 left) 2024-09-21T08:27:42.538 INFO:tasks.rados.rados.0.smithi137.stdout:990: snap_create 2024-09-21T08:27:42.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:42.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:42.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:42.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:42.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T08:27:42.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:42.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:42.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:42.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:42.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:42.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:42.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:42.832 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:42.834 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:42 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4149088306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:42.834 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:42 smithi137 ceph-mon[29634]: from='client.45275 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:42.834 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:42 smithi137 ceph-mon[29634]: from='client.45275 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:42.834 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:42 smithi137 ceph-mon[29634]: osdmap e269: 8 total, 8 up, 4 in 2024-09-21T08:27:42.834 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:42 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4149088306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:42.834 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:42 smithi137 ceph-mon[41044]: from='client.45275 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:42.834 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:42 smithi137 ceph-mon[41044]: from='client.45275 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:42.834 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:42 smithi137 ceph-mon[41044]: osdmap e269: 8 total, 8 up, 4 in 2024-09-21T08:27:42.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:42.956 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:42.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:42.957 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:42 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4149088306' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:42 smithi165 ceph-mon[32893]: from='client.45275 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:42 smithi165 ceph-mon[32893]: from='client.45275 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:42 smithi165 ceph-mon[32893]: osdmap e269: 8 total, 8 up, 4 in 2024-09-21T08:27:43.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:43.083 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:43.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:43.084 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:43.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:43.205 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:43.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:43.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:43.210 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:43.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:43.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:43.333 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:43.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:43.458 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:43.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:43.459 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:43.539 INFO:tasks.rados.rados.0.smithi137.stdout:990: done (0 left) 2024-09-21T08:27:43.539 INFO:tasks.rados.rados.0.smithi137.stdout:991: delete oid 14 current snap is 91 2024-09-21T08:27:43.544 INFO:tasks.rados.rados.0.smithi137.stdout:991: done (0 left) 2024-09-21T08:27:43.545 INFO:tasks.rados.rados.0.smithi137.stdout:992: delete oid 43 current snap is 91 2024-09-21T08:27:43.550 INFO:tasks.rados.rados.0.smithi137.stdout:992: done (0 left) 2024-09-21T08:27:43.551 INFO:tasks.rados.rados.0.smithi137.stdout:993: snap_remove snap 84 2024-09-21T08:27:43.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:43.586 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:43.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:43.587 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:43.687 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:43.712 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:43 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:43.712 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:43 smithi137 ceph-mon[29634]: pgmap v308: 168 pgs: 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 137 MiB data, 490 MiB used, 357 GiB / 358 GiB avail; 3.1 MiB/s, 6 objects/s recovering 2024-09-21T08:27:43.712 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:43 smithi137 ceph-mon[29634]: osdmap e270: 8 total, 8 up, 4 in 2024-09-21T08:27:43.712 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:43 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:43.713 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:43 smithi137 ceph-mon[41044]: pgmap v308: 168 pgs: 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 137 MiB data, 490 MiB used, 357 GiB / 358 GiB avail; 3.1 MiB/s, 6 objects/s recovering 2024-09-21T08:27:43.713 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:43 smithi137 ceph-mon[41044]: osdmap e270: 8 total, 8 up, 4 in 2024-09-21T08:27:43.713 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:43.713 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:43.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:43.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:43.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:43.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:43.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:43.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:43.842 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:43.914 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:43.943 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:43.964 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:43.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:43.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:43 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:43 smithi165 ceph-mon[32893]: pgmap v308: 168 pgs: 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 137 MiB data, 490 MiB used, 357 GiB / 358 GiB avail; 3.1 MiB/s, 6 objects/s recovering 2024-09-21T08:27:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:43 smithi165 ceph-mon[32893]: osdmap e270: 8 total, 8 up, 4 in 2024-09-21T08:27:44.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:44.092 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:44.093 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:44.093 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:44.194 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:44.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:44.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:44.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:44.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:44.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:44.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:44.346 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:44.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T08:27:44.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:44.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:44.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:44.469 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:44.539 INFO:tasks.rados.rados.0.smithi137.stdout:993: done (0 left) 2024-09-21T08:27:44.539 INFO:tasks.rados.rados.0.smithi137.stdout:994: write (excl) oid 35 current snap is 91 2024-09-21T08:27:44.539 INFO:tasks.rados.rados.0.smithi137.stdout:994: seq_num 376 ranges {719432=794921,2186053=133379} 2024-09-21T08:27:44.547 INFO:tasks.rados.rados.0.smithi137.stdout:994: writing smithi13779221-35 from 719432 to 1514353 tid 1 2024-09-21T08:27:44.551 INFO:tasks.rados.rados.0.smithi137.stdout:994: writing smithi13779221-35 from 2186053 to 2319432 tid 2 2024-09-21T08:27:44.552 INFO:tasks.rados.rados.0.smithi137.stdout:995: snap_create 2024-09-21T08:27:44.556 INFO:tasks.rados.rados.0.smithi137.stdout:994: finishing write tid 1 to smithi13779221-35 2024-09-21T08:27:44.556 INFO:tasks.rados.rados.0.smithi137.stdout:994: finishing write tid 2 to smithi13779221-35 2024-09-21T08:27:44.556 INFO:tasks.rados.rados.0.smithi137.stdout:994: finishing write tid 3 to smithi13779221-35 2024-09-21T08:27:44.557 INFO:tasks.rados.rados.0.smithi137.stdout:994: finishing write tid 4 to smithi13779221-35 2024-09-21T08:27:44.557 INFO:tasks.rados.rados.0.smithi137.stdout:994: oid 35 updating version 0 to 253 2024-09-21T08:27:44.557 INFO:tasks.rados.rados.0.smithi137.stdout:994: oid 35 updating version 253 to 255 2024-09-21T08:27:44.557 INFO:tasks.rados.rados.0.smithi137.stdout:994: oid 35 version 255 is already newer than 254 2024-09-21T08:27:44.557 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 255 (ObjNum 376 snap 91 seq_num 376) dirty exists 2024-09-21T08:27:44.557 INFO:tasks.rados.rados.0.smithi137.stdout:994: left oid 35 (ObjNum 376 snap 91 seq_num 376) 2024-09-21T08:27:44.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:44.594 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:44.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:44.595 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:44.696 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:44.718 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:44.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:44.719 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:44.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:44.820 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:44.820 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:44.837 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:44.837 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:44.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:44.846 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:44.846 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:44.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:44.855 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:44.855 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:44.871 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:44.871 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:44.896 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:44.896 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:44.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:44.913 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:44.929 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:44.930 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:44.946 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:44.946 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:44.947 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:44.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:44.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:44.972 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:45.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:45.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:45.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:45.096 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:45.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:27:45.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:45.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:45.223 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:45.223 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:45.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:45.344 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:45.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:45.345 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:45.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:45.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:45.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:45.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:45.547 INFO:tasks.rados.rados.0.smithi137.stdout:994: done (1 left) 2024-09-21T08:27:45.547 INFO:tasks.rados.rados.0.smithi137.stdout:995: done (0 left) 2024-09-21T08:27:45.547 INFO:tasks.rados.rados.0.smithi137.stdout:996: snap_create 2024-09-21T08:27:45.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:45.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:45.597 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:45.597 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:45.698 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:45.722 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:45.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:45.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:45.742 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 389.26509757899998, 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27973036899999998, 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.743 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473701+0000", 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1320000000000001e-06 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473716+0000", 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5622999999999999e-05 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473730+0000", 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3259e-05 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473740+0000", 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0746e-05 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473802+0000", 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2043000000000002e-05 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473810+0000", 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1500000000000002e-06 2024-09-21T08:27:45.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752339+0000", 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27852951199999998 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752485+0000", 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014628699999999999 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752609+0000", 2024-09-21T08:27:45.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012376499999999999 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753145+0000", 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053571000000000001 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753326+0000", 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018102799999999999 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753367+0000", 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1253e-05 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753400+0000", 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3065999999999998e-05 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753429+0000", 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8795000000000001e-05 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.749 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:324 7.b 7:d7506421:::smithi13779221-13:head [write 658841~471511] snapc 5=[] ondisk+write+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.585914+0000", 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 178.15288208499999, 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013763888, 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 324 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.750 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.585914+0000", 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.585914+0000", 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.585916+0000", 2024-09-21T08:27:45.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9099999999999999e-06 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.586227+0000", 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000311043 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.586239+0000", 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.15e-05 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.586249+0000", 2024-09-21T08:27:45.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0713999999999999e-05 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.593696+0000", 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0074462340000000004 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.593795+0000", 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9533999999999999e-05 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.593950+0000", 2024-09-21T08:27:45.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015500999999999999 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.595433+0000", 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014831639999999999 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.598311+0000", 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028782439999999999 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.599578+0000", 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001266238 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.599631+0000", 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3742999999999997e-05 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.599678+0000", 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6554000000000002e-05 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.755 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:422 7.d 7:b31d99ce:::smithi13779221-21:head [write 443718~615603] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 169.233820597, 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034885076000000001, 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 422 2024-09-21T08:27:45.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504978+0000", 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0090000000000002e-06 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506268+0000", 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001290444 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506279+0000", 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0526999999999999e-05 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506288+0000", 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5470000000000002e-06 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506343+0000", 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5287000000000001e-05 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506354+0000", 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0798999999999999e-05 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.525308+0000", 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018953616 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529390+0000", 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040819029999999996 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533318+0000", 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039282190000000002 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533380+0000", 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1890000000000005e-05 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533709+0000", 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032948400000000001 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536916+0000", 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032063399999999998 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539553+0000", 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026370909999999998 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539658+0000", 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105585 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539723+0000", 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4343000000000004e-05 2024-09-21T08:27:45.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539861+0000", 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137992 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:423 7.d 7:b31d99ce:::smithi13779221-21:head [write 1564138~470341] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 169.232430929, 2024-09-21T08:27:45.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033782404000000002, 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 423 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506366+0000", 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0449999999999999e-06 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507012+0000", 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064568899999999996 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507021+0000", 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2550000000000003e-06 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507029+0000", 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2559999999999996e-06 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507079+0000", 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0957e-05 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507088+0000", 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3650000000000004e-06 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529457+0000", 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022369185999999999 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529526+0000", 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8683999999999995e-05 2024-09-21T08:27:45.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535157+0000", 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056315849999999997 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535244+0000", 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6840000000000002e-05 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535398+0000", 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015404399999999999 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539569+0000", 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004170361 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539613+0000", 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4101999999999999e-05 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539985+0000", 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037279300000000001 2024-09-21T08:27:45.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540037+0000", 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1345999999999999e-05 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540148+0000", 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110896 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.770 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:424 7.d 7:b31d99ce:::smithi13779221-21:head [write 2633066~610652] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 169.225441915, 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027015847999999999, 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 424 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513356+0000", 2024-09-21T08:27:45.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8789999999999999e-06 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513753+0000", 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039670999999999999 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513764+0000", 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.154e-05 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513772+0000", 2024-09-21T08:27:45.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9149999999999996e-06 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513833+0000", 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1072000000000001e-05 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513843+0000", 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8349999999999999e-06 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529644+0000", 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015801084 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529688+0000", 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3568e-05 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536031+0000", 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063434659999999999 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536135+0000", 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103599 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536291+0000", 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155668 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539575+0000", 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032840450000000002 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539925+0000", 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034988899999999998 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540200+0000", 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027585000000000001 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540250+0000", 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9413000000000003e-05 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540370+0000", 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120315 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:425 7.d 7:b31d99ce:::smithi13779221-21:head [setxattr _header (56) in=7b,truncate 3243718] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 169.224939275, 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030130239, 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.778 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 425 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513858+0000", 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9199999999999999e-07 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513863+0000", 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7650000000000001e-06 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513869+0000", 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9880000000000001e-06 2024-09-21T08:27:45.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513874+0000", 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.643e-06 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515915+0000", 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020407279999999999 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515925+0000", 2024-09-21T08:27:45.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0051000000000001e-05 2024-09-21T08:27:45.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529722+0000", 2024-09-21T08:27:45.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013796780999999999 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529804+0000", 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1663999999999996e-05 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540423+0000", 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010619635000000001 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540671+0000", 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024766600000000001 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540795+0000", 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012383899999999999 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541324+0000", 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052910200000000004 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.542654+0000", 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013304370000000001 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543785+0000", 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001130603 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543841+0000", 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5939000000000001e-05 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543987+0000", 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014640599999999999 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:426 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 169.22298033000001, 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028482668999999999, 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:45.786 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 426 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515818+0000", 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7999999999999999e-06 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515825+0000", 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0790000000000003e-06 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515840+0000", 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4765e-05 2024-09-21T08:27:45.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515857+0000", 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7414999999999998e-05 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515968+0000", 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111161 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515975+0000", 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7499999999999997e-06 2024-09-21T08:27:45.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529841+0000", 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013866387000000001 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529951+0000", 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010953100000000001 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541110+0000", 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011159591 2024-09-21T08:27:45.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541190+0000", 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9243e-05 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544032+0000", 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028424050000000001 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544118+0000", 2024-09-21T08:27:45.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5798999999999996e-05 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544299+0000", 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180743 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:430 7.d 7:b0b32695:::smithi13779221-19:head [delete] snapc a=[9] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 169.222809652, 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028508957000000001, 2024-09-21T08:27:45.792 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 430 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1900000000000001e-07 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515991+0000", 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6899999999999998e-06 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515997+0000", 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2999999999999998e-06 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516003+0000", 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0220000000000001e-06 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516045+0000", 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1069000000000003e-05 2024-09-21T08:27:45.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.796 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:45 smithi137 ceph-mon[29634]: pgmap v311: 168 pgs: 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 137 MiB data, 490 MiB used, 357 GiB / 358 GiB avail; 3.1 MiB/s, 6 objects/s recovering 2024-09-21T08:27:45.796 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:45 smithi137 ceph-mon[29634]: osdmap e271: 8 total, 8 up, 4 in 2024-09-21T08:27:45.796 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:45 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1709630387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:45.796 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:45 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:45.796 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:45 smithi137 ceph-mon[41044]: pgmap v311: 168 pgs: 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 137 MiB data, 490 MiB used, 357 GiB / 358 GiB avail; 3.1 MiB/s, 6 objects/s recovering 2024-09-21T08:27:45.796 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:45 smithi137 ceph-mon[41044]: osdmap e271: 8 total, 8 up, 4 in 2024-09-21T08:27:45.796 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:45 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1709630387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:45.796 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:45 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:45.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516052+0000", 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9890000000000003e-06 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529980+0000", 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013927999999999999 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.532305+0000", 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023250580000000001 2024-09-21T08:27:45.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541672+0000", 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093675080000000001 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541749+0000", 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6829000000000006e-05 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541934+0000", 2024-09-21T08:27:45.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018531 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543560+0000", 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00162525 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543661+0000", 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101604 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544339+0000", 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067745299999999995 2024-09-21T08:27:45.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544394+0000", 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5515e-05 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544496+0000", 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010144100000000001 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.801 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:552 7.d 7:b0b32695:::smithi13779221-19:head [copy-from ver 34] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:09.528772+0000", 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.210024621, 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022405184000000002, 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 552 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528772+0000", 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528772+0000", 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528776+0000", 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3510000000000004e-06 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528788+0000", 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2542e-05 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528810+0000", 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1467e-05 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528822+0000", 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1973e-05 2024-09-21T08:27:45.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528913+0000", 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1458000000000004e-05 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528922+0000", 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0780000000000002e-06 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.531438+0000", 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002515541 2024-09-21T08:27:45.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.531531+0000", 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3175000000000002e-05 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.539639+0000", 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081080649999999994 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.547323+0000", 2024-09-21T08:27:45.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076836079999999998 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.550791+0000", 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034681719999999998 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.551056+0000", 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000264966 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.551116+0000", 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9905000000000003e-05 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.551177+0000", 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0883e-05 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:553 7.d 7:b0b32695:::smithi13779221-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:09.528845+0000", 2024-09-21T08:27:45.808 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 156.20995151599999, 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015306544, 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 553 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528845+0000", 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528845+0000", 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528846+0000", 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9899999999999999e-07 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528857+0000", 2024-09-21T08:27:45.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1755e-05 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528864+0000", 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0389999999999999e-06 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528871+0000", 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1229999999999997e-06 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528987+0000", 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116597 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528992+0000", 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1250000000000001e-06 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.531699+0000", 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027068890000000001 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:45.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.532428+0000", 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00072873100000000004 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.543914+0000", 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011486322 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.544028+0000", 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114063 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.813 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.544151+0000", 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123001 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:571 7.1 7:88bf4ad2:::smithi13779221-6:head [copy-from ver 13] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 155.191742998, 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037746406000000003, 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.814 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 571 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:27:45.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547056+0000", 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.146e-06 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547067+0000", 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0609e-05 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547090+0000", 2024-09-21T08:27:45.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2714999999999999e-05 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547100+0000", 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9920000000000002e-06 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547203+0000", 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102828 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547210+0000", 2024-09-21T08:27:45.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3629999999999996e-06 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553577+0000", 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063672829999999996 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553669+0000", 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1856999999999998e-05 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:27:45.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573941+0000", 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020272168 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.583019+0000", 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090777630000000008 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584643+0000", 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016245319999999999 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584686+0000", 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2259999999999999e-05 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584748+0000", 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2049000000000002e-05 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584800+0000", 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1841000000000001e-05 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.820 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:572 7.1 7:88bf4ad2:::smithi13779221-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 155.19167396899999, 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032210599999999999, 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 572 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.821 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547123+0000", 2024-09-21T08:27:45.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5300000000000003e-07 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547138+0000", 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4718999999999999e-05 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547145+0000", 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2520000000000002e-06 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547151+0000", 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7649999999999999e-06 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547263+0000", 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112001 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547269+0000", 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8919999999999997e-06 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553824+0000", 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065557499999999999 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553873+0000", 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8727999999999998e-05 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579133+0000", 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025259964999999999 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579227+0000", 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3517999999999995e-05 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579333+0000", 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010615699999999999 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:576 7.2 7:4031afd1:::smithi13779221-48:head [append 0~0] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:27:45.826 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547160+0000", 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 155.19163625799999, 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026627913, 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 576 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.827 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547160+0000", 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547160+0000", 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547161+0000", 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.44e-07 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547163+0000", 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5790000000000002e-06 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547169+0000", 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7610000000000004e-06 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547174+0000", 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7849999999999999e-06 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547266+0000", 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1978999999999995e-05 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547272+0000", 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3629999999999999e-06 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.562646+0000", 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015374095000000001 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.562751+0000", 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010446 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.562953+0000", 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000202151 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564215+0000", 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012625589999999999 2024-09-21T08:27:45.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567784+0000", 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035684559999999998 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573649+0000", 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058654110000000001 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573729+0000", 2024-09-21T08:27:45.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9555999999999995e-05 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573788+0000", 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9113999999999997e-05 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:577 7.2 7:4031afd1:::smithi13779221-48:head [rollback 14] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 155.19160960599999, 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026796044000000002, 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.833 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 577 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3e-07 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547190+0000", 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3980000000000002e-06 2024-09-21T08:27:45.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547196+0000", 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5239999999999997e-06 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547201+0000", 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5839999999999996e-06 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547293+0000", 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2613000000000003e-05 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547298+0000", 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6500000000000004e-06 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564248+0000", 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01694959 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564348+0000", 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100893 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564547+0000", 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000198278 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565560+0000", 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001013391 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567845+0000", 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022853090000000001 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573898+0000", 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.006052094 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573944+0000", 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6928000000000002e-05 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573983+0000", 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8161999999999998e-05 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.839 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:578 7.2 7:4031afd1:::smithi13779221-48:head [append 0~0] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 155.19159034500001, 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026976508, 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.840 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 578 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8100000000000003e-07 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547209+0000", 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4549999999999998e-06 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547214+0000", 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2889999999999997e-06 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547219+0000", 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7419999999999997e-06 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547317+0000", 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8029000000000006e-05 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547321+0000", 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2880000000000003e-06 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566078+0000", 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018756951000000001 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566167+0000", 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9068999999999997e-05 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566318+0000", 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015045999999999999 2024-09-21T08:27:45.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567184+0000", 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00086585299999999998 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.568942+0000", 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001758103 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574047+0000", 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051054719999999998 2024-09-21T08:27:45.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574143+0000", 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5616000000000001e-05 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574182+0000", 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.96e-05 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.846 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:830 7.b 7:d8f73c7b:::smithi13779221-38:head [sparse-read 0~2050454,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.705969+0000", 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 136.03282747099999, 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017749276000000001, 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 830 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705969+0000", 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705969+0000", 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705973+0000", 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7360000000000001e-06 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705988+0000", 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.413e-05 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706011+0000", 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3484000000000001e-05 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706023+0000", 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1899000000000001e-05 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706110+0000", 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7439000000000001e-05 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706117+0000", 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2190000000000001e-06 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.719472+0000", 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013354927000000001 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.719550+0000", 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8486999999999996e-05 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.723718+0000", 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041679550000000001 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:831 7.b 7:d8f73c7b:::smithi13779221-38:head [read 0~2050454,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:27:45.851 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 136.03274435500001, 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018156002000000001, 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 831 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706053+0000", 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0299999999999997e-07 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706058+0000", 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0490000000000003e-06 2024-09-21T08:27:45.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706071+0000", 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3542e-05 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706077+0000", 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0730000000000001e-06 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706181+0000", 2024-09-21T08:27:45.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103132 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706185+0000", 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.831e-06 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.723755+0000", 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017570065999999999 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.723835+0000", 2024-09-21T08:27:45.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9320000000000006e-05 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.724208+0000", 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037308600000000001 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:832 7.b 7:d8f73c7b:::smithi13779221-38:head [sparse-read 0~2050454] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.706101+0000", 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 136.03269492199999, 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018300291, 2024-09-21T08:27:45.856 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 832 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706101+0000", 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706101+0000", 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706103+0000", 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.508e-06 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706108+0000", 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2979999999999999e-06 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706118+0000", 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0033e-05 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706128+0000", 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4360000000000008e-06 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706208+0000", 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9973000000000002e-05 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706212+0000", 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9280000000000001e-06 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.724233+0000", 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018020673000000001 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.724281+0000", 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8254000000000003e-05 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.724402+0000", 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120188 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1014 7.b 7:d6c5a045:::smithi13779221-46:head [copy-from ver 47] snapc 22=[22,21,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:40.781122+0000", 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.957674385, 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018020263000000002, 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.861 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1014 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781122+0000", 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781122+0000", 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781124+0000", 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5119999999999998e-06 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781133+0000", 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5169999999999999e-06 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781148+0000", 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5481e-05 2024-09-21T08:27:45.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781158+0000", 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2869999999999993e-06 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781222+0000", 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4428e-05 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781227+0000", 2024-09-21T08:27:45.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0690000000000001e-06 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.784918+0000", 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036913620000000001 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.785019+0000", 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100597 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.790943+0000", 2024-09-21T08:27:45.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059236510000000003 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.797751+0000", 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068083559999999998 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.798733+0000", 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00098198100000000004 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.799028+0000", 2024-09-21T08:27:45.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029540499999999999 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.799088+0000", 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9206999999999997e-05 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.799142+0000", 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4410000000000003e-05 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1015 7.b 7:d6c5a045:::smithi13779221-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e147)", 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:40.781176+0000", 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 124.957619827, 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014394074999999999, 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1015 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:45.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781176+0000", 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781176+0000", 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781177+0000", 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1800000000000005e-07 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781180+0000", 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1520000000000001e-06 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781189+0000", 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4789999999999996e-06 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781195+0000", 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8900000000000004e-06 2024-09-21T08:27:45.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781293+0000", 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8461000000000005e-05 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781297+0000", 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.45e-06 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.785184+0000", 2024-09-21T08:27:45.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038876869999999999 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.785229+0000", 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4678999999999997e-05 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.795302+0000", 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010073267 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.795422+0000", 2024-09-21T08:27:45.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000119663 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.795570+0000", 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000148629 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:45.873 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:27:45.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:45.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:45.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:45.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:45.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:45 smithi165 ceph-mon[32893]: pgmap v311: 168 pgs: 11 active+clean+snaptrim_wait, 4 active+clean+snaptrim, 153 active+clean; 137 MiB data, 490 MiB used, 357 GiB / 358 GiB avail; 3.1 MiB/s, 6 objects/s recovering 2024-09-21T08:27:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:45 smithi165 ceph-mon[32893]: osdmap e271: 8 total, 8 up, 4 in 2024-09-21T08:27:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:45 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1709630387' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:45 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:27:46.022 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:46.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:46.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:46.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:46.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:46.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:46.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:46.246 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:46.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:46.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:46.268 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:46.340 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 6, 1, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:27:46.340 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:27:46.340 INFO:tasks.thrashosds.thrasher:inject_pause on osd.4 2024-09-21T08:27:46.340 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T08:27:46.340 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T08:27:46.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 config set filestore_inject_stall 3 2024-09-21T08:27:46.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:46.407 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:46.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:46.408 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:46.508 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:46.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:46.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:46.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:46.550 INFO:tasks.rados.rados.0.smithi137.stdout:996: done (0 left) 2024-09-21T08:27:46.551 INFO:tasks.rados.rados.0.smithi137.stdout:997: write oid 41 current snap is 93 2024-09-21T08:27:46.551 INFO:tasks.rados.rados.0.smithi137.stdout:997: seq_num 377 ranges {736239=470170,1728182=571782} 2024-09-21T08:27:46.557 INFO:tasks.rados.rados.0.smithi137.stdout:997: writing smithi13779221-41 from 736239 to 1206409 tid 1 2024-09-21T08:27:46.563 INFO:tasks.rados.rados.0.smithi137.stdout:997: writing smithi13779221-41 from 1728182 to 2299964 tid 2 2024-09-21T08:27:46.563 INFO:tasks.rados.rados.0.smithi137.stdout:998: write (excl) oid 32 current snap is 93 2024-09-21T08:27:46.563 INFO:tasks.rados.rados.0.smithi137.stdout:998: seq_num 378 ranges {753046=545419,1786664=431793} 2024-09-21T08:27:46.570 INFO:tasks.rados.rados.0.smithi137.stdout:998: writing smithi13779221-32 from 753046 to 1298465 tid 1 2024-09-21T08:27:46.576 INFO:tasks.rados.rados.0.smithi137.stdout:998: writing smithi13779221-32 from 1786664 to 2218457 tid 2 2024-09-21T08:27:46.576 INFO:tasks.rados.rados.0.smithi137.stdout:999: read oid 7 snap -1 2024-09-21T08:27:46.576 INFO:tasks.rados.rados.0.smithi137.stdout:999: expect deleted 2024-09-21T08:27:46.576 INFO:tasks.rados.rados.0.smithi137.stdout:1000: snap_remove snap 91 2024-09-21T08:27:46.609 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:27:46.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:46.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:46.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:46.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:46.665 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:46.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:46 smithi165 ceph-mon[32893]: osdmap e272: 8 total, 8 up, 4 in 2024-09-21T08:27:46.730 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:46 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:46.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:46.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:46.776 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:46.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:46.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:46.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:46.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:46.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:47.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:47.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:46 smithi137 ceph-mon[29634]: osdmap e272: 8 total, 8 up, 4 in 2024-09-21T08:27:47.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:46 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:47.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:46 smithi137 ceph-mon[41044]: osdmap e272: 8 total, 8 up, 4 in 2024-09-21T08:27:47.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:46 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:47.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:47.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:47.021 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:47.090 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:27:47.090 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "" 2024-09-21T08:27:47.090 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:27:47.096 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T08:27:47.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:47.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:47.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:47.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:47.259 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:47.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:47.281 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:47.281 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:47.382 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:47.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:47.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:47.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:47.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:47.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:47.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:47.528 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:47.556 INFO:tasks.rados.rados.0.smithi137.stdout:1000: done (3 left) 2024-09-21T08:27:47.556 INFO:tasks.rados.rados.0.smithi137.stdout:1001: read oid 42 snap 92 2024-09-21T08:27:47.556 INFO:tasks.rados.rados.0.smithi137.stdout:1001: expect deleted 2024-09-21T08:27:47.556 INFO:tasks.rados.rados.0.smithi137.stdout:1002: snap_remove snap 90 2024-09-21T08:27:47.610 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:47.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:47.651 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:47.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:47.652 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:47.734 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:47 smithi137 ceph-mon[29634]: pgmap v314: 168 pgs: 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 151 active+clean; 113 MiB data, 423 MiB used, 357 GiB / 358 GiB avail; 1.5 KiB/s rd, 227 KiB/s wr, 3 op/s 2024-09-21T08:27:47.734 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:47 smithi137 ceph-mon[29634]: osdmap e273: 8 total, 8 up, 4 in 2024-09-21T08:27:47.734 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:47 smithi137 ceph-mon[41044]: pgmap v314: 168 pgs: 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 151 active+clean; 113 MiB data, 423 MiB used, 357 GiB / 358 GiB avail; 1.5 KiB/s rd, 227 KiB/s wr, 3 op/s 2024-09-21T08:27:47.734 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:47 smithi137 ceph-mon[41044]: osdmap e273: 8 total, 8 up, 4 in 2024-09-21T08:27:47.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:47.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:47.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:47.776 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:47.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:47.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:47.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:47.899 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:47.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:47 smithi165 ceph-mon[32893]: pgmap v314: 168 pgs: 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 151 active+clean; 113 MiB data, 423 MiB used, 357 GiB / 358 GiB avail; 1.5 KiB/s rd, 227 KiB/s wr, 3 op/s 2024-09-21T08:27:47.969 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:47 smithi165 ceph-mon[32893]: osdmap e273: 8 total, 8 up, 4 in 2024-09-21T08:27:47.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:48.022 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:48.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:48.023 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:48.110 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:48.111 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:27:48.111 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:27:48.111 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:27:48.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:48.146 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:48.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:48.147 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:48.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:48.269 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:48.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:48.270 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:48.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:48.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:48.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:48.392 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:48.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:48.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:48.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:48.515 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:48.561 INFO:tasks.rados.rados.0.smithi137.stdout:1002: done (4 left) 2024-09-21T08:27:48.561 INFO:tasks.rados.rados.0.smithi137.stdout:1003: read oid 47 snap 92 2024-09-21T08:27:48.561 INFO:tasks.rados.rados.0.smithi137.stdout:1003: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:27:48.561 INFO:tasks.rados.rados.0.smithi137.stdout:1004: delete oid 10 current snap is 93 2024-09-21T08:27:48.561 INFO:tasks.rados.rados.0.smithi137.stdout:997: finishing write tid 1 to smithi13779221-41 2024-09-21T08:27:48.561 INFO:tasks.rados.rados.0.smithi137.stdout:997: finishing write tid 2 to smithi13779221-41 2024-09-21T08:27:48.561 INFO:tasks.rados.rados.0.smithi137.stdout:997: finishing write tid 3 to smithi13779221-41 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:997: finishing write tid 4 to smithi13779221-41 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:997: oid 41 updating version 0 to 297 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:997: oid 41 updating version 297 to 298 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:997: oid 41 version 298 is already newer than 296 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 298 (ObjNum 377 snap 93 seq_num 377) dirty exists 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:997: left oid 41 (ObjNum 377 snap 93 seq_num 377) 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:998: finishing write tid 1 to smithi13779221-32 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:998: finishing write tid 2 to smithi13779221-32 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:998: finishing write tid 3 to smithi13779221-32 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:998: finishing write tid 4 to smithi13779221-32 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:998: oid 32 updating version 0 to 215 2024-09-21T08:27:48.562 INFO:tasks.rados.rados.0.smithi137.stdout:998: oid 32 version 215 is already newer than 213 2024-09-21T08:27:48.563 INFO:tasks.rados.rados.0.smithi137.stdout:998: oid 32 version 215 is already newer than 214 2024-09-21T08:27:48.563 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 215 (ObjNum 378 snap 93 seq_num 378) dirty exists 2024-09-21T08:27:48.563 INFO:tasks.rados.rados.0.smithi137.stdout:998: left oid 32 (ObjNum 378 snap 93 seq_num 378) 2024-09-21T08:27:48.563 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:27:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:997: done (5 left) 2024-09-21T08:27:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:998: done (4 left) 2024-09-21T08:27:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:999: done (3 left) 2024-09-21T08:27:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:1001: done (2 left) 2024-09-21T08:27:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:1004: done (1 left) 2024-09-21T08:27:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:1005: rollback oid 40 current snap is 93 2024-09-21T08:27:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 40 to 92 2024-09-21T08:27:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:1006: copy_from oid 48 from oid 4 current snap is 93 2024-09-21T08:27:48.567 INFO:tasks.rados.rados.0.smithi137.stdout:1007: snap_create 2024-09-21T08:27:48.568 INFO:tasks.rados.rados.0.smithi137.stdout:1006: finishing copy_from to smithi13779221-48 2024-09-21T08:27:48.568 INFO:tasks.rados.rados.0.smithi137.stdout:1006: got expected ENOENT (src dne) 2024-09-21T08:27:48.568 INFO:tasks.rados.rados.0.smithi137.stdout:1006: finishing copy_from racing read to smithi13779221-48 2024-09-21T08:27:48.571 INFO:tasks.rados.rados.0.smithi137.stdout:1003: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:27:48.616 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:48.640 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:48.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:48.641 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:48.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:27:48.672 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:48 smithi137 ceph-mon[29634]: osdmap e274: 8 total, 8 up, 4 in 2024-09-21T08:27:48.672 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:48 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3954110200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:48.672 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:48 smithi137 ceph-mon[29634]: from='client.45287 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:48.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:48 smithi137 ceph-mon[41044]: osdmap e274: 8 total, 8 up, 4 in 2024-09-21T08:27:48.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:48 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3954110200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:48.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:48 smithi137 ceph-mon[41044]: from='client.45287 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:48.705 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T08:27:48.742 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:48.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:48.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:48.765 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:48.788 INFO:tasks.rados.rados.0.smithi137.stdout:1005: finishing rollback tid 0 to smithi13779221-40 2024-09-21T08:27:48.788 INFO:tasks.rados.rados.0.smithi137.stdout:1005: finishing rollback tid 1 to smithi13779221-40 2024-09-21T08:27:48.788 INFO:tasks.rados.rados.0.smithi137.stdout:1005: finishing rollback tid 2 to smithi13779221-40 2024-09-21T08:27:48.788 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 260 (ObjNum 287 snap 66 seq_num 287) dirty exists 2024-09-21T08:27:48.866 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:48.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:48.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:48.890 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:48.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:49.013 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:48 smithi165 ceph-mon[32893]: osdmap e274: 8 total, 8 up, 4 in 2024-09-21T08:27:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:48 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3954110200' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:48 smithi165 ceph-mon[32893]: from='client.45287 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:27:49.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:49.014 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:49.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:49.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:49.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:49.138 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:49.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:49.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:49.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:49.263 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:49.328 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:49.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:49.388 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:49.389 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:49.389 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:49.490 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:49.513 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:49.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:49.514 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:49.572 INFO:tasks.rados.rados.0.smithi137.stdout:1003: done (3 left) 2024-09-21T08:27:49.572 INFO:tasks.rados.rados.0.smithi137.stdout:1005: done (2 left) 2024-09-21T08:27:49.573 INFO:tasks.rados.rados.0.smithi137.stdout:1006: done (1 left) 2024-09-21T08:27:49.573 INFO:tasks.rados.rados.0.smithi137.stdout:1007: done (0 left) 2024-09-21T08:27:49.573 INFO:tasks.rados.rados.0.smithi137.stdout:1008: rollback oid 37 current snap is 94 2024-09-21T08:27:49.573 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 37 to 92 2024-09-21T08:27:49.573 INFO:tasks.rados.rados.0.smithi137.stdout:1009: read oid 32 snap -1 2024-09-21T08:27:49.573 INFO:tasks.rados.rados.0.smithi137.stdout:1009: expect (ObjNum 378 snap 93 seq_num 378) 2024-09-21T08:27:49.573 INFO:tasks.rados.rados.0.smithi137.stdout:1010: rollback oid 25 current snap is 94 2024-09-21T08:27:49.573 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 25 to 92 2024-09-21T08:27:49.573 INFO:tasks.rados.rados.0.smithi137.stdout:1011: write (excl) oid 29 current snap is 94 2024-09-21T08:27:49.573 INFO:tasks.rados.rados.0.smithi137.stdout:1011: seq_num 380 ranges {786660=695917,1903628=435462} 2024-09-21T08:27:49.580 INFO:tasks.rados.rados.0.smithi137.stdout:1011: writing smithi13779221-29 from 786660 to 1482577 tid 1 2024-09-21T08:27:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1011: writing smithi13779221-29 from 1903628 to 2339090 tid 2 2024-09-21T08:27:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1012: write oid 20 current snap is 94 2024-09-21T08:27:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1008: finishing rollback tid 0 to smithi13779221-37 2024-09-21T08:27:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1012: seq_num 381 ranges {403467=487519,1562110=579120} 2024-09-21T08:27:49.591 INFO:tasks.rados.rados.0.smithi137.stdout:1012: writing smithi13779221-20 from 403467 to 890986 tid 1 2024-09-21T08:27:49.598 INFO:tasks.rados.rados.0.smithi137.stdout:1012: writing smithi13779221-20 from 1562110 to 2141230 tid 2 2024-09-21T08:27:49.599 INFO:tasks.rados.rados.0.smithi137.stdout:1013: delete oid 26 current snap is 94 2024-09-21T08:27:49.599 INFO:tasks.rados.rados.0.smithi137.stdout:1009: expect (ObjNum 378 snap 93 seq_num 378) 2024-09-21T08:27:49.599 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:49.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:49.636 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:49.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:49.637 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:49.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:49.761 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:49.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:49.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:49.783 INFO:tasks.rados.rados.0.smithi137.stdout:1008: finishing rollback tid 1 to smithi13779221-37 2024-09-21T08:27:49.783 INFO:tasks.rados.rados.0.smithi137.stdout:1010: finishing rollback tid 1 to smithi13779221-25 2024-09-21T08:27:49.783 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 0 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-21T08:27:49.783 INFO:tasks.rados.rados.0.smithi137.stdout:1008: finishing rollback tid 2 to smithi13779221-37 2024-09-21T08:27:49.783 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 469 (ObjNum 243 snap 50 seq_num 243) dirty exists 2024-09-21T08:27:49.783 INFO:tasks.rados.rados.0.smithi137.stdout:1011: finishing write tid 1 to smithi13779221-29 2024-09-21T08:27:49.783 INFO:tasks.rados.rados.0.smithi137.stdout:1008: done (5 left) 2024-09-21T08:27:49.783 INFO:tasks.rados.rados.0.smithi137.stdout:1009: done (4 left) 2024-09-21T08:27:49.783 INFO:tasks.rados.rados.0.smithi137.stdout:1010: done (3 left) 2024-09-21T08:27:49.783 INFO:tasks.rados.rados.0.smithi137.stdout:1013: done (2 left) 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1014: write oid 50 current snap is 94 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1011: finishing write tid 2 to smithi13779221-29 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1011: finishing write tid 3 to smithi13779221-29 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1011: finishing write tid 4 to smithi13779221-29 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1011: oid 29 updating version 0 to 480 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1011: oid 29 version 480 is already newer than 478 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1011: oid 29 version 480 is already newer than 479 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 480 (ObjNum 380 snap 94 seq_num 380) dirty exists 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1011: left oid 29 (ObjNum 380 snap 94 seq_num 380) 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1012: finishing write tid 1 to smithi13779221-20 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1012: finishing write tid 2 to smithi13779221-20 2024-09-21T08:27:49.784 INFO:tasks.rados.rados.0.smithi137.stdout:1014: seq_num 383 ranges {437081=638017,1679074=582789} 2024-09-21T08:27:49.789 INFO:tasks.rados.rados.0.smithi137.stdout:1014: writing smithi13779221-50 from 437081 to 1075098 tid 1 2024-09-21T08:27:49.798 INFO:tasks.rados.rados.0.smithi137.stdout:1014: writing smithi13779221-50 from 1679074 to 2261863 tid 2 2024-09-21T08:27:49.798 INFO:tasks.rados.rados.0.smithi137.stdout:1011: done (2 left) 2024-09-21T08:27:49.798 INFO:tasks.rados.rados.0.smithi137.stdout:1015: copy_from oid 47 from oid 27 current snap is 94 2024-09-21T08:27:49.798 INFO:tasks.rados.rados.0.smithi137.stdout:1012: finishing write tid 3 to smithi13779221-20 2024-09-21T08:27:49.798 INFO:tasks.rados.rados.0.smithi137.stdout:1012: finishing write tid 4 to smithi13779221-20 2024-09-21T08:27:49.798 INFO:tasks.rados.rados.0.smithi137.stdout:1012: oid 20 updating version 0 to 352 2024-09-21T08:27:49.798 INFO:tasks.rados.rados.0.smithi137.stdout:1012: oid 20 version 352 is already newer than 350 2024-09-21T08:27:49.798 INFO:tasks.rados.rados.0.smithi137.stdout:1012: oid 20 version 352 is already newer than 351 2024-09-21T08:27:49.798 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 352 (ObjNum 381 snap 94 seq_num 381) dirty exists 2024-09-21T08:27:49.798 INFO:tasks.rados.rados.0.smithi137.stdout:1012: left oid 20 (ObjNum 381 snap 94 seq_num 381) 2024-09-21T08:27:49.799 INFO:tasks.rados.rados.0.smithi137.stdout:1014: finishing write tid 1 to smithi13779221-50 2024-09-21T08:27:49.799 INFO:tasks.rados.rados.0.smithi137.stdout:1012: done (2 left) 2024-09-21T08:27:49.799 INFO:tasks.rados.rados.0.smithi137.stdout:1016: copy_from oid 16 from oid 5 current snap is 94 2024-09-21T08:27:49.799 INFO:tasks.rados.rados.0.smithi137.stdout:1017: rollback oid 29 current snap is 94 2024-09-21T08:27:49.799 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 29 to 93 2024-09-21T08:27:49.799 INFO:tasks.rados.rados.0.smithi137.stdout:1018: write (excl) oid 10 current snap is 94 2024-09-21T08:27:49.799 INFO:tasks.rados.rados.0.smithi137.stdout:1018: seq_num 384 ranges {453888=713266,1737556=442800} 2024-09-21T08:27:49.806 INFO:tasks.rados.rados.0.smithi137.stdout:1018: writing smithi13779221-10 from 453888 to 1167154 tid 1 2024-09-21T08:27:49.813 INFO:tasks.rados.rados.0.smithi137.stdout:1018: writing smithi13779221-10 from 1737556 to 2180356 tid 2 2024-09-21T08:27:49.813 INFO:tasks.rados.rados.0.smithi137.stdout:1019: rollback oid 43 current snap is 94 2024-09-21T08:27:49.813 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 43 to 93 2024-09-21T08:27:49.813 INFO:tasks.rados.rados.0.smithi137.stdout:1015: finishing copy_from to smithi13779221-47 2024-09-21T08:27:49.813 INFO:tasks.rados.rados.0.smithi137.stdout:1015: got expected ENOENT (src dne) 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1015: finishing copy_from racing read to smithi13779221-47 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1017: finishing rollback tid 0 to smithi13779221-29 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1017: finishing rollback tid 1 to smithi13779221-29 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 482 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1014: finishing write tid 2 to smithi13779221-50 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1014: finishing write tid 3 to smithi13779221-50 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1014: finishing write tid 4 to smithi13779221-50 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1014: oid 50 updating version 0 to 319 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1014: oid 50 updating version 319 to 321 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1014: oid 50 version 321 is already newer than 320 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 321 (ObjNum 383 snap 94 seq_num 383) dirty exists 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1014: left oid 50 (ObjNum 383 snap 94 seq_num 383) 2024-09-21T08:27:49.814 INFO:tasks.rados.rados.0.smithi137.stdout:1016: finishing copy_from racing read to smithi13779221-16 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1016: finishing copy_from to smithi13779221-16 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 318 (ObjNum 360 snap 85 seq_num 360) dirty exists 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1018: finishing write tid 1 to smithi13779221-10 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1020: read oid 25 snap -1 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1020: expect deleted 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1014: done (6 left) 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1015: done (5 left) 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1016: done (4 left) 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1017: done (3 left) 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1021: rollback oid 6 current snap is 94 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 6 to 93 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1022: read oid 11 snap -1 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1022: expect (ObjNum 370 snap 90 seq_num 370) 2024-09-21T08:27:49.815 INFO:tasks.rados.rados.0.smithi137.stdout:1023: read oid 13 snap 93 2024-09-21T08:27:49.816 INFO:tasks.rados.rados.0.smithi137.stdout:1023: expect (ObjNum 340 snap 80 seq_num 340) 2024-09-21T08:27:49.816 INFO:tasks.rados.rados.0.smithi137.stdout:1024: copy_from oid 30 from oid 47 current snap is 94 2024-09-21T08:27:49.816 INFO:tasks.rados.rados.0.smithi137.stdout:1025: rollback oid 39 current snap is 94 2024-09-21T08:27:49.816 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 39 to 93 2024-09-21T08:27:49.816 INFO:tasks.rados.rados.0.smithi137.stdout:1026: snap_remove snap 92 2024-09-21T08:27:49.822 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:49 smithi137 ceph-mon[29634]: pgmap v317: 168 pgs: 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 151 active+clean; 113 MiB data, 423 MiB used, 357 GiB / 358 GiB avail; 1.5 KiB/s rd, 227 KiB/s wr, 3 op/s 2024-09-21T08:27:49.822 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:49 smithi137 ceph-mon[29634]: from='client.45287 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:27:49.822 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:49 smithi137 ceph-mon[29634]: osdmap e275: 8 total, 8 up, 4 in 2024-09-21T08:27:49.822 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:49 smithi137 ceph-mon[29634]: 7.22 scrub starts 2024-09-21T08:27:49.822 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:49 smithi137 ceph-mon[29634]: 7.22 scrub ok 2024-09-21T08:27:49.822 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:49 smithi137 ceph-mon[41044]: pgmap v317: 168 pgs: 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 151 active+clean; 113 MiB data, 423 MiB used, 357 GiB / 358 GiB avail; 1.5 KiB/s rd, 227 KiB/s wr, 3 op/s 2024-09-21T08:27:49.822 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:49 smithi137 ceph-mon[41044]: from='client.45287 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:27:49.822 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:49 smithi137 ceph-mon[41044]: osdmap e275: 8 total, 8 up, 4 in 2024-09-21T08:27:49.822 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:49 smithi137 ceph-mon[41044]: 7.22 scrub starts 2024-09-21T08:27:49.822 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:49 smithi137 ceph-mon[41044]: 7.22 scrub ok 2024-09-21T08:27:49.849 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:49.849 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:27:49.850 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:27:49.850 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:27:49.850 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:27:49.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:49.884 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:49.885 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:49.986 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:50.008 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:50.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:50.009 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:49 smithi165 ceph-mon[32893]: pgmap v317: 168 pgs: 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 151 active+clean; 113 MiB data, 423 MiB used, 357 GiB / 358 GiB avail; 1.5 KiB/s rd, 227 KiB/s wr, 3 op/s 2024-09-21T08:27:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:49 smithi165 ceph-mon[32893]: from='client.45287 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:27:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:49 smithi165 ceph-mon[32893]: osdmap e275: 8 total, 8 up, 4 in 2024-09-21T08:27:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:49 smithi165 ceph-mon[32893]: 7.22 scrub starts 2024-09-21T08:27:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:49 smithi165 ceph-mon[32893]: 7.22 scrub ok 2024-09-21T08:27:50.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:50.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:50.132 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:50.133 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:50.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:50.255 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:50.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:50.256 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:50.357 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:50.377 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:50.378 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:50.378 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:50.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:50.482 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:50.482 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:50.498 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:50.498 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:50.499 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:50.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:50.500 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:50.516 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:50.516 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:50.532 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:50.532 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:50.549 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:50.549 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:50.566 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:50.567 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:1026: done (8 left) 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:1027: rollback oid 35 current snap is 94 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 35 to 93 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:1019: finishing rollback tid 1 to smithi13779221-43 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:1018: finishing write tid 2 to smithi13779221-10 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:1025: finishing rollback tid 1 to smithi13779221-39 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 0 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:1018: finishing write tid 3 to smithi13779221-10 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:1018: finishing write tid 4 to smithi13779221-10 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:1018: oid 10 updating version 0 to 293 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:1018: oid 10 updating version 293 to 294 2024-09-21T08:27:50.576 INFO:tasks.rados.rados.0.smithi137.stdout:1018: oid 10 updating version 294 to 295 2024-09-21T08:27:50.577 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 295 (ObjNum 384 snap 94 seq_num 384) dirty exists 2024-09-21T08:27:50.577 INFO:tasks.rados.rados.0.smithi137.stdout:1018: left oid 10 (ObjNum 384 snap 94 seq_num 384) 2024-09-21T08:27:50.577 INFO:tasks.rados.rados.0.smithi137.stdout:1022: expect (ObjNum 370 snap 90 seq_num 370) 2024-09-21T08:27:50.592 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:50.592 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:50.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:50.620 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:50.620 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:50.623 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:50.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:50.624 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:50.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[41044]: 7.1c scrub starts 2024-09-21T08:27:50.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[41044]: 7.1c scrub ok 2024-09-21T08:27:50.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[41044]: 7.20 scrub starts 2024-09-21T08:27:50.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[41044]: 7.20 scrub ok 2024-09-21T08:27:50.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[41044]: osdmap e276: 8 total, 8 up, 4 in 2024-09-21T08:27:50.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[41044]: 7.19 scrub starts 2024-09-21T08:27:50.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[41044]: 7.19 scrub ok 2024-09-21T08:27:50.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1171236535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:50.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[29634]: 7.1c scrub starts 2024-09-21T08:27:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[29634]: 7.1c scrub ok 2024-09-21T08:27:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[29634]: 7.20 scrub starts 2024-09-21T08:27:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[29634]: 7.20 scrub ok 2024-09-21T08:27:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[29634]: osdmap e276: 8 total, 8 up, 4 in 2024-09-21T08:27:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[29634]: 7.19 scrub starts 2024-09-21T08:27:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[29634]: 7.19 scrub ok 2024-09-21T08:27:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1171236535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:50.710 INFO:tasks.rados.rados.0.smithi137.stdout:1021: finishing rollback tid 0 to smithi13779221-6 2024-09-21T08:27:50.710 INFO:tasks.rados.rados.0.smithi137.stdout:1021: finishing rollback tid 1 to smithi13779221-6 2024-09-21T08:27:50.710 INFO:tasks.rados.rados.0.smithi137.stdout:1018: done (8 left) 2024-09-21T08:27:50.711 INFO:tasks.rados.rados.0.smithi137.stdout:1019: done (7 left) 2024-09-21T08:27:50.711 INFO:tasks.rados.rados.0.smithi137.stdout:1020: done (6 left) 2024-09-21T08:27:50.711 INFO:tasks.rados.rados.0.smithi137.stdout:1022: done (5 left) 2024-09-21T08:27:50.711 INFO:tasks.rados.rados.0.smithi137.stdout:1025: done (4 left) 2024-09-21T08:27:50.711 INFO:tasks.rados.rados.0.smithi137.stdout:1028: write (excl) oid 11 current snap is 94 2024-09-21T08:27:50.711 INFO:tasks.rados.rados.0.smithi137.stdout:1021: finishing rollback tid 2 to smithi13779221-6 2024-09-21T08:27:50.711 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 246 (ObjNum 312 snap 70 seq_num 312) dirty exists 2024-09-21T08:27:50.711 INFO:tasks.rados.rados.0.smithi137.stdout:1023: expect (ObjNum 340 snap 80 seq_num 340) 2024-09-21T08:27:50.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:50.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:50.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:50.746 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:50.846 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:50.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:50.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:50.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:50.931 INFO:tasks.rados.rados.0.smithi137.stdout:1024: finishing copy_from racing read to smithi13779221-30 2024-09-21T08:27:50.931 INFO:tasks.rados.rados.0.smithi137.stdout:1024: finishing copy_from to smithi13779221-30 2024-09-21T08:27:50.931 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:27:50.931 INFO:tasks.rados.rados.0.smithi137.stdout:1028: seq_num 385 ranges {470695=788515,1679685=586458} 2024-09-21T08:27:50.931 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:27:50 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:27:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:27:50.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T08:27:50.937 INFO:tasks.rados.rados.0.smithi137.stdout:1028: writing smithi13779221-11 from 470695 to 1259210 tid 1 2024-09-21T08:27:50.943 INFO:tasks.rados.rados.0.smithi137.stdout:1028: writing smithi13779221-11 from 1679685 to 2266143 tid 2 2024-09-21T08:27:50.943 INFO:tasks.rados.rados.0.smithi137.stdout:1021: done (4 left) 2024-09-21T08:27:50.943 INFO:tasks.rados.rados.0.smithi137.stdout:1023: done (3 left) 2024-09-21T08:27:50.944 INFO:tasks.rados.rados.0.smithi137.stdout:1024: done (2 left) 2024-09-21T08:27:50.944 INFO:tasks.rados.rados.0.smithi137.stdout:1029: read oid 39 snap -1 2024-09-21T08:27:50.944 INFO:tasks.rados.rados.0.smithi137.stdout:1029: expect deleted 2024-09-21T08:27:50.944 INFO:tasks.rados.rados.0.smithi137.stdout:1030: write (excl) oid 44 current snap is 94 2024-09-21T08:27:50.944 INFO:tasks.rados.rados.0.smithi137.stdout:1027: finishing rollback tid 0 to smithi13779221-35 2024-09-21T08:27:50.944 INFO:tasks.rados.rados.0.smithi137.stdout:1030: seq_num 386 ranges {487502=463764,1738167=446469} 2024-09-21T08:27:50.948 INFO:tasks.rados.rados.0.smithi137.stdout:1030: writing smithi13779221-44 from 487502 to 951266 tid 1 2024-09-21T08:27:50.956 INFO:tasks.rados.rados.0.smithi137.stdout:1030: writing smithi13779221-44 from 1738167 to 2184636 tid 2 2024-09-21T08:27:50.956 INFO:tasks.rados.rados.0.smithi137.stdout:1031: rollback oid 23 current snap is 94 2024-09-21T08:27:50.956 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 23 to 93 2024-09-21T08:27:50.956 INFO:tasks.rados.rados.0.smithi137.stdout:1027: finishing rollback tid 1 to smithi13779221-35 2024-09-21T08:27:50.956 INFO:tasks.rados.rados.0.smithi137.stdout:1027: finishing rollback tid 2 to smithi13779221-35 2024-09-21T08:27:50.956 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 258 (ObjNum 376 snap 91 seq_num 376) dirty exists 2024-09-21T08:27:50.956 INFO:tasks.rados.rados.0.smithi137.stdout:1028: finishing write tid 1 to smithi13779221-11 2024-09-21T08:27:50.956 INFO:tasks.rados.rados.0.smithi137.stdout:1028: finishing write tid 2 to smithi13779221-11 2024-09-21T08:27:50.956 INFO:tasks.rados.rados.0.smithi137.stdout:1028: finishing write tid 3 to smithi13779221-11 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1028: finishing write tid 4 to smithi13779221-11 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1028: oid 11 updating version 0 to 218 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1028: oid 11 version 218 is already newer than 217 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1028: oid 11 version 218 is already newer than 216 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 218 (ObjNum 385 snap 94 seq_num 385) dirty exists 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1028: left oid 11 (ObjNum 385 snap 94 seq_num 385) 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1030: finishing write tid 1 to smithi13779221-44 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1027: done (4 left) 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1028: done (3 left) 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1029: done (2 left) 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1032: write (excl) oid 28 current snap is 94 2024-09-21T08:27:50.957 INFO:tasks.rados.rados.0.smithi137.stdout:1032: seq_num 387 ranges {504309=539013,1796649=590127} 2024-09-21T08:27:50.961 INFO:tasks.rados.rados.0.smithi137.stdout:1032: writing smithi13779221-28 from 504309 to 1043322 tid 1 2024-09-21T08:27:50.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:50.969 INFO:tasks.rados.rados.0.smithi137.stdout:1032: writing smithi13779221-28 from 1796649 to 2386776 tid 2 2024-09-21T08:27:50.969 INFO:tasks.rados.rados.0.smithi137.stdout:1033: write (excl) oid 20 current snap is 94 2024-09-21T08:27:50.970 INFO:tasks.rados.rados.0.smithi137.stdout:1030: finishing write tid 2 to smithi13779221-44 2024-09-21T08:27:50.970 INFO:tasks.rados.rados.0.smithi137.stdout:1033: seq_num 388 ranges {521116=730615,1971484=549632} 2024-09-21T08:27:50.975 INFO:tasks.rados.rados.0.smithi137.stdout:1033: writing smithi13779221-20 from 521116 to 1251731 tid 1 2024-09-21T08:27:50.983 INFO:tasks.rados.rados.0.smithi137.stdout:1033: writing smithi13779221-20 from 1971484 to 2521116 tid 2 2024-09-21T08:27:50.983 INFO:tasks.rados.rados.0.smithi137.stdout:1034: snap_create 2024-09-21T08:27:50.983 INFO:tasks.rados.rados.0.smithi137.stdout:1031: finishing rollback tid 0 to smithi13779221-23 2024-09-21T08:27:50.983 INFO:tasks.rados.rados.0.smithi137.stdout:1030: finishing write tid 3 to smithi13779221-44 2024-09-21T08:27:50.983 INFO:tasks.rados.rados.0.smithi137.stdout:1031: finishing rollback tid 1 to smithi13779221-23 2024-09-21T08:27:50.983 INFO:tasks.rados.rados.0.smithi137.stdout:1030: finishing write tid 4 to smithi13779221-44 2024-09-21T08:27:50.983 INFO:tasks.rados.rados.0.smithi137.stdout:1030: oid 44 updating version 0 to 259 2024-09-21T08:27:50.983 INFO:tasks.rados.rados.0.smithi137.stdout:1030: oid 44 updating version 259 to 260 2024-09-21T08:27:50.983 INFO:tasks.rados.rados.0.smithi137.stdout:1030: oid 44 updating version 260 to 261 2024-09-21T08:27:50.983 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 261 (ObjNum 386 snap 94 seq_num 386) dirty exists 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1030: left oid 44 (ObjNum 386 snap 94 seq_num 386) 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1031: finishing rollback tid 2 to smithi13779221-23 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 177 (ObjNum 329 snap 78 seq_num 329) dirty exists 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1032: finishing write tid 1 to smithi13779221-28 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1032: finishing write tid 2 to smithi13779221-28 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1032: finishing write tid 3 to smithi13779221-28 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1032: finishing write tid 4 to smithi13779221-28 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1032: oid 28 updating version 0 to 257 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1032: oid 28 updating version 257 to 258 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1032: oid 28 version 258 is already newer than 256 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 258 (ObjNum 387 snap 94 seq_num 387) dirty exists 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1032: left oid 28 (ObjNum 387 snap 94 seq_num 387) 2024-09-21T08:27:50.984 INFO:tasks.rados.rados.0.smithi137.stdout:1033: finishing write tid 1 to smithi13779221-20 2024-09-21T08:27:50.986 INFO:tasks.rados.rados.0.smithi137.stdout:1033: finishing write tid 2 to smithi13779221-20 2024-09-21T08:27:50.986 INFO:tasks.rados.rados.0.smithi137.stdout:1033: finishing write tid 3 to smithi13779221-20 2024-09-21T08:27:50.986 INFO:tasks.rados.rados.0.smithi137.stdout:1033: finishing write tid 4 to smithi13779221-20 2024-09-21T08:27:50.986 INFO:tasks.rados.rados.0.smithi137.stdout:1033: oid 20 updating version 0 to 356 2024-09-21T08:27:50.986 INFO:tasks.rados.rados.0.smithi137.stdout:1033: oid 20 updating version 356 to 357 2024-09-21T08:27:50.986 INFO:tasks.rados.rados.0.smithi137.stdout:1033: oid 20 version 357 is already newer than 355 2024-09-21T08:27:50.986 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 357 (ObjNum 388 snap 94 seq_num 388) dirty exists 2024-09-21T08:27:50.986 INFO:tasks.rados.rados.0.smithi137.stdout:1033: left oid 20 (ObjNum 388 snap 94 seq_num 388) 2024-09-21T08:27:50.996 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:50.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:50.996 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:50 smithi165 ceph-mon[32893]: 7.1c scrub starts 2024-09-21T08:27:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:50 smithi165 ceph-mon[32893]: 7.1c scrub ok 2024-09-21T08:27:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:50 smithi165 ceph-mon[32893]: 7.20 scrub starts 2024-09-21T08:27:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:50 smithi165 ceph-mon[32893]: 7.20 scrub ok 2024-09-21T08:27:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:50 smithi165 ceph-mon[32893]: osdmap e276: 8 total, 8 up, 4 in 2024-09-21T08:27:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:50 smithi165 ceph-mon[32893]: 7.19 scrub starts 2024-09-21T08:27:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:50 smithi165 ceph-mon[32893]: 7.19 scrub ok 2024-09-21T08:27:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:50 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1171236535' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:51.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:51.123 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:51.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:51.124 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:51.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:51.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:51.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:51.247 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:51.282 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:27:51.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:51.370 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:51.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:51.371 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:51.471 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:51.497 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:51.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:51.498 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:51.537 INFO:tasks.rados.rados.0.smithi137.stdout:1030: done (4 left) 2024-09-21T08:27:51.537 INFO:tasks.rados.rados.0.smithi137.stdout:1031: done (3 left) 2024-09-21T08:27:51.537 INFO:tasks.rados.rados.0.smithi137.stdout:1032: done (2 left) 2024-09-21T08:27:51.537 INFO:tasks.rados.rados.0.smithi137.stdout:1033: done (1 left) 2024-09-21T08:27:51.537 INFO:tasks.rados.rados.0.smithi137.stdout:1034: done (0 left) 2024-09-21T08:27:51.537 INFO:tasks.rados.rados.0.smithi137.stdout:1035: write (excl) oid 23 current snap is 95 2024-09-21T08:27:51.538 INFO:tasks.rados.rados.0.smithi137.stdout:1035: seq_num 389 ranges {537923=405864,1513613=593796} 2024-09-21T08:27:51.542 INFO:tasks.rados.rados.0.smithi137.stdout:1035: writing smithi13779221-23 from 537923 to 943787 tid 1 2024-09-21T08:27:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:1035: writing smithi13779221-23 from 1513613 to 2107409 tid 2 2024-09-21T08:27:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:1036: write oid 22 current snap is 95 2024-09-21T08:27:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:1036: seq_num 390 ranges {554730=481113,1572095=737454} 2024-09-21T08:27:51.556 INFO:tasks.rados.rados.0.smithi137.stdout:1036: writing smithi13779221-22 from 554730 to 1035843 tid 1 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1036: writing smithi13779221-22 from 1572095 to 2309549 tid 2 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1037: read oid 10 snap 94 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1037: expect (ObjNum 384 snap 94 seq_num 384) 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1038: rollback oid 5 current snap is 95 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 5 to 94 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1035: finishing write tid 1 to smithi13779221-23 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1035: finishing write tid 2 to smithi13779221-23 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1035: finishing write tid 3 to smithi13779221-23 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1035: finishing write tid 4 to smithi13779221-23 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1035: oid 23 updating version 0 to 179 2024-09-21T08:27:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1035: oid 23 version 179 is already newer than 178 2024-09-21T08:27:51.565 INFO:tasks.rados.rados.0.smithi137.stdout:1035: oid 23 updating version 179 to 180 2024-09-21T08:27:51.565 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 180 (ObjNum 389 snap 95 seq_num 389) dirty exists 2024-09-21T08:27:51.565 INFO:tasks.rados.rados.0.smithi137.stdout:1035: left oid 23 (ObjNum 389 snap 95 seq_num 389) 2024-09-21T08:27:51.565 INFO:tasks.rados.rados.0.smithi137.stdout:1035: done (3 left) 2024-09-21T08:27:51.565 INFO:tasks.rados.rados.0.smithi137.stdout:1039: delete oid 15 current snap is 95 2024-09-21T08:27:51.568 INFO:tasks.rados.rados.0.smithi137.stdout:1039: done (3 left) 2024-09-21T08:27:51.568 INFO:tasks.rados.rados.0.smithi137.stdout:1040: write oid 31 current snap is 95 2024-09-21T08:27:51.568 INFO:tasks.rados.rados.0.smithi137.stdout:1040: seq_num 392 ranges {588344=631611,1689059=741123} 2024-09-21T08:27:51.575 INFO:tasks.rados.rados.0.smithi137.stdout:1040: writing smithi13779221-31 from 588344 to 1219955 tid 1 2024-09-21T08:27:51.583 INFO:tasks.rados.rados.0.smithi137.stdout:1040: writing smithi13779221-31 from 1689059 to 2430182 tid 2 2024-09-21T08:27:51.583 INFO:tasks.rados.rados.0.smithi137.stdout:1041: snap_remove snap 93 2024-09-21T08:27:51.583 INFO:tasks.rados.rados.0.smithi137.stdout:1036: finishing write tid 1 to smithi13779221-22 2024-09-21T08:27:51.599 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:51.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:51.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:51.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:51.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:51.746 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:51.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:51.747 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:51.778 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:51.801 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:51.849 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:51.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:51.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:51.874 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:51.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[29634]: pgmap v320: 168 pgs: 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 151 active+clean; 113 MiB data, 423 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:51.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[29634]: 7.23 scrub starts 2024-09-21T08:27:51.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[29634]: 7.23 scrub ok 2024-09-21T08:27:51.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[29634]: 7.1 scrub starts 2024-09-21T08:27:51.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[29634]: 7.1 scrub ok 2024-09-21T08:27:51.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[29634]: 7.2 scrub starts 2024-09-21T08:27:51.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[29634]: 7.2 scrub ok 2024-09-21T08:27:51.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[29634]: osdmap e277: 8 total, 8 up, 4 in 2024-09-21T08:27:51.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:51.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[29634]: osdmap e278: 8 total, 8 up, 4 in 2024-09-21T08:27:51.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[41044]: pgmap v320: 168 pgs: 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 151 active+clean; 113 MiB data, 423 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:51.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[41044]: 7.23 scrub starts 2024-09-21T08:27:51.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[41044]: 7.23 scrub ok 2024-09-21T08:27:51.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[41044]: 7.1 scrub starts 2024-09-21T08:27:51.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[41044]: 7.1 scrub ok 2024-09-21T08:27:51.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[41044]: 7.2 scrub starts 2024-09-21T08:27:51.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[41044]: 7.2 scrub ok 2024-09-21T08:27:51.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[41044]: osdmap e277: 8 total, 8 up, 4 in 2024-09-21T08:27:51.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:51.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:51 smithi137 ceph-mon[41044]: osdmap e278: 8 total, 8 up, 4 in 2024-09-21T08:27:51.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:51.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:51.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:51.997 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:52.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:27:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-mon[32893]: pgmap v320: 168 pgs: 12 active+clean+snaptrim_wait, 5 active+clean+snaptrim, 151 active+clean; 113 MiB data, 423 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-mon[32893]: 7.23 scrub starts 2024-09-21T08:27:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-mon[32893]: 7.23 scrub ok 2024-09-21T08:27:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-mon[32893]: 7.1 scrub starts 2024-09-21T08:27:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-mon[32893]: 7.1 scrub ok 2024-09-21T08:27:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-mon[32893]: 7.2 scrub starts 2024-09-21T08:27:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-mon[32893]: 7.2 scrub ok 2024-09-21T08:27:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-mon[32893]: osdmap e277: 8 total, 8 up, 4 in 2024-09-21T08:27:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:51 smithi165 ceph-mon[32893]: osdmap e278: 8 total, 8 up, 4 in 2024-09-21T08:27:52.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:52.120 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:52.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:52.121 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:52.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:52.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:52.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:52.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:52.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:52.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:52.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:52.375 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:52.476 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:52.500 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:52.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:52.501 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:52.586 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:27:52.587 INFO:tasks.rados.rados.0.smithi137.stdout:1041: done (4 left) 2024-09-21T08:27:52.587 INFO:tasks.rados.rados.0.smithi137.stdout:1042: rollback oid 2 current snap is 95 2024-09-21T08:27:52.587 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 2 to 94 2024-09-21T08:27:52.587 INFO:tasks.rados.rados.0.smithi137.stdout:1043: write (excl) oid 36 current snap is 95 2024-09-21T08:27:52.588 INFO:tasks.rados.rados.0.smithi137.stdout:1043: seq_num 393 ranges {605151=706860,2031188=573963} 2024-09-21T08:27:52.596 INFO:tasks.rados.rados.0.smithi137.stdout:1043: writing smithi13779221-36 from 605151 to 1312011 tid 1 2024-09-21T08:27:52.601 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:52.605 INFO:tasks.rados.rados.0.smithi137.stdout:1043: writing smithi13779221-36 from 2031188 to 2605151 tid 2 2024-09-21T08:27:52.605 INFO:tasks.rados.rados.0.smithi137.stdout:1044: read oid 28 snap -1 2024-09-21T08:27:52.605 INFO:tasks.rados.rados.0.smithi137.stdout:1044: expect (ObjNum 387 snap 94 seq_num 387) 2024-09-21T08:27:52.606 INFO:tasks.rados.rados.0.smithi137.stdout:1045: write oid 45 current snap is 95 2024-09-21T08:27:52.608 INFO:tasks.rados.rados.0.smithi137.stdout:1036: finishing write tid 2 to smithi13779221-22 2024-09-21T08:27:52.608 INFO:tasks.rados.rados.0.smithi137.stdout:1036: finishing write tid 3 to smithi13779221-22 2024-09-21T08:27:52.608 INFO:tasks.rados.rados.0.smithi137.stdout:1036: finishing write tid 4 to smithi13779221-22 2024-09-21T08:27:52.608 INFO:tasks.rados.rados.0.smithi137.stdout:1036: oid 22 updating version 0 to 319 2024-09-21T08:27:52.608 INFO:tasks.rados.rados.0.smithi137.stdout:1036: oid 22 updating version 319 to 321 2024-09-21T08:27:52.608 INFO:tasks.rados.rados.0.smithi137.stdout:1036: oid 22 version 321 is already newer than 320 2024-09-21T08:27:52.609 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 321 (ObjNum 390 snap 95 seq_num 390) dirty exists 2024-09-21T08:27:52.609 INFO:tasks.rados.rados.0.smithi137.stdout:1036: left oid 22 (ObjNum 390 snap 95 seq_num 390) 2024-09-21T08:27:52.609 INFO:tasks.rados.rados.0.smithi137.stdout:1038: finishing rollback tid 0 to smithi13779221-5 2024-09-21T08:27:52.609 INFO:tasks.rados.rados.0.smithi137.stdout:1037: expect (ObjNum 384 snap 94 seq_num 384) 2024-09-21T08:27:52.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:52.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:52.630 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:52.689 INFO:tasks.rados.rados.0.smithi137.stdout:1038: finishing rollback tid 1 to smithi13779221-5 2024-09-21T08:27:52.689 INFO:tasks.rados.rados.0.smithi137.stdout:1038: finishing rollback tid 2 to smithi13779221-5 2024-09-21T08:27:52.689 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 324 (ObjNum 360 snap 85 seq_num 360) dirty exists 2024-09-21T08:27:52.689 INFO:tasks.rados.rados.0.smithi137.stdout:1040: finishing write tid 1 to smithi13779221-31 2024-09-21T08:27:52.689 INFO:tasks.rados.rados.0.smithi137.stdout:1045: seq_num 394 ranges {621958=782109,2089670=532288} 2024-09-21T08:27:52.697 INFO:tasks.rados.rados.0.smithi137.stdout:1045: writing smithi13779221-45 from 621958 to 1404067 tid 1 2024-09-21T08:27:52.704 INFO:tasks.rados.rados.0.smithi137.stdout:1045: writing smithi13779221-45 from 2089670 to 2621958 tid 2 2024-09-21T08:27:52.705 INFO:tasks.rados.rados.0.smithi137.stdout:1036: done (7 left) 2024-09-21T08:27:52.705 INFO:tasks.rados.rados.0.smithi137.stdout:1037: done (6 left) 2024-09-21T08:27:52.705 INFO:tasks.rados.rados.0.smithi137.stdout:1038: done (5 left) 2024-09-21T08:27:52.705 INFO:tasks.rados.rados.0.smithi137.stdout:1046: copy_from oid 25 from oid 17 current snap is 95 2024-09-21T08:27:52.705 INFO:tasks.rados.rados.0.smithi137.stdout:1040: finishing write tid 2 to smithi13779221-31 2024-09-21T08:27:52.705 INFO:tasks.rados.rados.0.smithi137.stdout:1040: finishing write tid 3 to smithi13779221-31 2024-09-21T08:27:52.705 INFO:tasks.rados.rados.0.smithi137.stdout:1040: finishing write tid 4 to smithi13779221-31 2024-09-21T08:27:52.705 INFO:tasks.rados.rados.0.smithi137.stdout:1040: oid 31 updating version 0 to 324 2024-09-21T08:27:52.706 INFO:tasks.rados.rados.0.smithi137.stdout:1040: oid 31 version 324 is already newer than 323 2024-09-21T08:27:52.706 INFO:tasks.rados.rados.0.smithi137.stdout:1040: oid 31 version 324 is already newer than 322 2024-09-21T08:27:52.706 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 324 (ObjNum 392 snap 95 seq_num 392) dirty exists 2024-09-21T08:27:52.706 INFO:tasks.rados.rados.0.smithi137.stdout:1040: left oid 31 (ObjNum 392 snap 95 seq_num 392) 2024-09-21T08:27:52.706 INFO:tasks.rados.rados.0.smithi137.stdout:1042: finishing rollback tid 0 to smithi13779221-2 2024-09-21T08:27:52.706 INFO:tasks.rados.rados.0.smithi137.stdout:1042: finishing rollback tid 1 to smithi13779221-2 2024-09-21T08:27:52.706 INFO:tasks.rados.rados.0.smithi137.stdout:1040: done (5 left) 2024-09-21T08:27:52.706 INFO:tasks.rados.rados.0.smithi137.stdout:1047: read oid 29 snap -1 2024-09-21T08:27:52.706 INFO:tasks.rados.rados.0.smithi137.stdout:1047: expect deleted 2024-09-21T08:27:52.707 INFO:tasks.rados.rados.0.smithi137.stdout:1048: rollback oid 13 current snap is 95 2024-09-21T08:27:52.707 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 13 to 94 2024-09-21T08:27:52.707 INFO:tasks.rados.rados.0.smithi137.stdout:1042: finishing rollback tid 2 to smithi13779221-2 2024-09-21T08:27:52.707 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 138 (ObjNum 302 snap 70 seq_num 302) dirty exists 2024-09-21T08:27:52.707 INFO:tasks.rados.rados.0.smithi137.stdout:1043: finishing write tid 1 to smithi13779221-36 2024-09-21T08:27:52.707 INFO:tasks.rados.rados.0.smithi137.stdout:1043: finishing write tid 2 to smithi13779221-36 2024-09-21T08:27:52.707 INFO:tasks.rados.rados.0.smithi137.stdout:1043: finishing write tid 3 to smithi13779221-36 2024-09-21T08:27:52.707 INFO:tasks.rados.rados.0.smithi137.stdout:1042: done (6 left) 2024-09-21T08:27:52.707 INFO:tasks.rados.rados.0.smithi137.stdout:1049: snap_create 2024-09-21T08:27:52.707 INFO:tasks.rados.rados.0.smithi137.stdout:1043: finishing write tid 4 to smithi13779221-36 2024-09-21T08:27:52.708 INFO:tasks.rados.rados.0.smithi137.stdout:1043: oid 36 updating version 0 to 477 2024-09-21T08:27:52.708 INFO:tasks.rados.rados.0.smithi137.stdout:1043: oid 36 updating version 477 to 479 2024-09-21T08:27:52.708 INFO:tasks.rados.rados.0.smithi137.stdout:1043: oid 36 version 479 is already newer than 478 2024-09-21T08:27:52.708 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 479 (ObjNum 393 snap 95 seq_num 393) dirty exists 2024-09-21T08:27:52.708 INFO:tasks.rados.rados.0.smithi137.stdout:1043: left oid 36 (ObjNum 393 snap 95 seq_num 393) 2024-09-21T08:27:52.708 INFO:tasks.rados.rados.0.smithi137.stdout:1044: expect (ObjNum 387 snap 94 seq_num 387) 2024-09-21T08:27:52.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:52.755 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:52.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:52.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[29634]: 7.1f scrub starts 2024-09-21T08:27:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[29634]: 7.1f scrub ok 2024-09-21T08:27:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[29634]: 5.d scrub starts 2024-09-21T08:27:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[29634]: 5.d scrub ok 2024-09-21T08:27:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4041946025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[29634]: from='client.45299 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[41044]: 7.1f scrub starts 2024-09-21T08:27:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[41044]: 7.1f scrub ok 2024-09-21T08:27:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[41044]: 5.d scrub starts 2024-09-21T08:27:52.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[41044]: 5.d scrub ok 2024-09-21T08:27:52.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4041946025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:52.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:52 smithi137 ceph-mon[41044]: from='client.45299 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1045: finishing write tid 1 to smithi13779221-45 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1045: finishing write tid 2 to smithi13779221-45 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1045: finishing write tid 3 to smithi13779221-45 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1045: finishing write tid 4 to smithi13779221-45 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1045: oid 45 updating version 0 to 276 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1045: oid 45 updating version 276 to 277 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1045: oid 45 version 277 is already newer than 275 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 277 (ObjNum 394 snap 95 seq_num 394) dirty exists 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1045: left oid 45 (ObjNum 394 snap 95 seq_num 394) 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1048: finishing rollback tid 0 to smithi13779221-13 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1048: finishing rollback tid 1 to smithi13779221-13 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:1048: finishing rollback tid 2 to smithi13779221-13 2024-09-21T08:27:52.787 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 486 (ObjNum 340 snap 80 seq_num 340) dirty exists 2024-09-21T08:27:52.788 INFO:tasks.rados.rados.0.smithi137.stdout:1046: finishing copy_from racing read to smithi13779221-25 2024-09-21T08:27:52.788 INFO:tasks.rados.rados.0.smithi137.stdout:1046: finishing copy_from to smithi13779221-25 2024-09-21T08:27:52.788 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 99 (ObjNum 200 snap 40 seq_num 200) dirty exists 2024-09-21T08:27:52.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:52.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:52.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:52.880 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:52.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:52 smithi165 ceph-mon[32893]: 7.1f scrub starts 2024-09-21T08:27:52.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:52 smithi165 ceph-mon[32893]: 7.1f scrub ok 2024-09-21T08:27:52.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:52 smithi165 ceph-mon[32893]: 5.d scrub starts 2024-09-21T08:27:52.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:52 smithi165 ceph-mon[32893]: 5.d scrub ok 2024-09-21T08:27:52.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:52 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4041946025' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:52.915 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:52 smithi165 ceph-mon[32893]: from='client.45299 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:27:52.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:52.998 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 6, 1, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:27:52.998 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:27:52.998 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T08:27:52.998 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.692965 2024-09-21T08:27:52.998 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 0.6929648520469458 2024-09-21T08:27:53.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:53.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:53.004 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:53.011 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:27:53.011 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:27:53.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:53.130 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:53.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:53.131 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:53.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:53.254 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:53.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:53.255 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:53.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:53.381 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:53.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:53.382 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:53.483 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:53.504 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:53.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:53.505 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:53.583 INFO:tasks.rados.rados.0.smithi137.stdout:1043: done (6 left) 2024-09-21T08:27:53.583 INFO:tasks.rados.rados.0.smithi137.stdout:1044: done (5 left) 2024-09-21T08:27:53.584 INFO:tasks.rados.rados.0.smithi137.stdout:1045: done (4 left) 2024-09-21T08:27:53.584 INFO:tasks.rados.rados.0.smithi137.stdout:1046: done (3 left) 2024-09-21T08:27:53.584 INFO:tasks.rados.rados.0.smithi137.stdout:1047: done (2 left) 2024-09-21T08:27:53.584 INFO:tasks.rados.rados.0.smithi137.stdout:1048: done (1 left) 2024-09-21T08:27:53.584 INFO:tasks.rados.rados.0.smithi137.stdout:1049: done (0 left) 2024-09-21T08:27:53.584 INFO:tasks.rados.rados.0.smithi137.stdout:1050: snap_create 2024-09-21T08:27:53.606 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:53.627 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:53.628 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:53.628 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:53.729 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:53.752 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:53.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:53.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:53.756 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:27:53.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:53.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[29634]: pgmap v323: 168 pgs: 13 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 104 MiB data, 401 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:53.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[29634]: 7.b scrub starts 2024-09-21T08:27:53.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[29634]: 7.b scrub ok 2024-09-21T08:27:53.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:27:53.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:27:53.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[29634]: from='client.45299 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:53.761 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[29634]: osdmap e279: 8 total, 8 up, 4 in 2024-09-21T08:27:53.761 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:53.761 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[41044]: pgmap v323: 168 pgs: 13 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 104 MiB data, 401 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:53.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[41044]: 7.b scrub starts 2024-09-21T08:27:53.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[41044]: 7.b scrub ok 2024-09-21T08:27:53.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:27:53.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:27:53.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[41044]: from='client.45299 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:53.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:53 smithi137 ceph-mon[41044]: osdmap e279: 8 total, 8 up, 4 in 2024-09-21T08:27:53.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:53 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:27:53.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:53 smithi165 ceph-mon[32893]: pgmap v323: 168 pgs: 13 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 104 MiB data, 401 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:53.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:53 smithi165 ceph-mon[32893]: 7.b scrub starts 2024-09-21T08:27:53.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:53 smithi165 ceph-mon[32893]: 7.b scrub ok 2024-09-21T08:27:53.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:53 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:27:53.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:53 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:27:53.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:53 smithi165 ceph-mon[32893]: from='client.45299 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:27:53.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:53 smithi165 ceph-mon[32893]: osdmap e279: 8 total, 8 up, 4 in 2024-09-21T08:27:53.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:53.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:53.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:53.877 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:53.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:53.999 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:54.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:54.000 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:54.040 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:54.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T08:27:54.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:54.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:54.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:54.123 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:54.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:54.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:54.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:54.245 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:54.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:54.368 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:54.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:54.369 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:54.469 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:54.491 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:54.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:54.492 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:54.585 INFO:tasks.rados.rados.0.smithi137.stdout:1050: done (0 left) 2024-09-21T08:27:54.585 INFO:tasks.rados.rados.0.smithi137.stdout:1051: write oid 14 current snap is 97 2024-09-21T08:27:54.585 INFO:tasks.rados.rados.0.smithi137.stdout:1051: seq_num 395 ranges {638765=457358,1748152=604803} 2024-09-21T08:27:54.591 INFO:tasks.rados.rados.0.smithi137.stdout:1051: writing smithi13779221-14 from 638765 to 1096123 tid 1 2024-09-21T08:27:54.592 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:54.599 INFO:tasks.rados.rados.0.smithi137.stdout:1051: writing smithi13779221-14 from 1748152 to 2352955 tid 2 2024-09-21T08:27:54.600 INFO:tasks.rados.rados.0.smithi137.stdout:1052: write (excl) oid 31 current snap is 97 2024-09-21T08:27:54.600 INFO:tasks.rados.rados.0.smithi137.stdout:1052: seq_num 396 ranges {655572=648960,1922987=732585} 2024-09-21T08:27:54.608 INFO:tasks.rados.rados.0.smithi137.stdout:1052: writing smithi13779221-31 from 655572 to 1304532 tid 1 2024-09-21T08:27:54.609 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:54.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:54.618 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:54.618 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1052: writing smithi13779221-31 from 1922987 to 2655572 tid 2 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1051: finishing write tid 1 to smithi13779221-14 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1051: finishing write tid 2 to smithi13779221-14 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1051: finishing write tid 3 to smithi13779221-14 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1051: finishing write tid 4 to smithi13779221-14 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1051: oid 14 updating version 0 to 247 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1051: oid 14 updating version 247 to 249 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1051: oid 14 version 249 is already newer than 248 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 249 (ObjNum 395 snap 97 seq_num 395) dirty exists 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1051: left oid 14 (ObjNum 395 snap 97 seq_num 395) 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1052: finishing write tid 1 to smithi13779221-31 2024-09-21T08:27:54.619 INFO:tasks.rados.rados.0.smithi137.stdout:1051: done (1 left) 2024-09-21T08:27:54.620 INFO:tasks.rados.rados.0.smithi137.stdout:1053: copy_from oid 8 from oid 30 current snap is 97 2024-09-21T08:27:54.620 INFO:tasks.rados.rados.0.smithi137.stdout:1054: write oid 42 current snap is 97 2024-09-21T08:27:54.620 INFO:tasks.rados.rados.0.smithi137.stdout:1054: seq_num 397 ranges {672379=724209,1865116=608472} 2024-09-21T08:27:54.627 INFO:tasks.rados.rados.0.smithi137.stdout:1054: writing smithi13779221-42 from 672379 to 1396588 tid 1 2024-09-21T08:27:54.635 INFO:tasks.rados.rados.0.smithi137.stdout:1054: writing smithi13779221-42 from 1865116 to 2473588 tid 2 2024-09-21T08:27:54.636 INFO:tasks.rados.rados.0.smithi137.stdout:1052: finishing write tid 2 to smithi13779221-31 2024-09-21T08:27:54.636 INFO:tasks.rados.rados.0.smithi137.stdout:1052: finishing write tid 3 to smithi13779221-31 2024-09-21T08:27:54.636 INFO:tasks.rados.rados.0.smithi137.stdout:1052: finishing write tid 4 to smithi13779221-31 2024-09-21T08:27:54.637 INFO:tasks.rados.rados.0.smithi137.stdout:1052: oid 31 updating version 0 to 326 2024-09-21T08:27:54.637 INFO:tasks.rados.rados.0.smithi137.stdout:1052: oid 31 version 326 is already newer than 325 2024-09-21T08:27:54.637 INFO:tasks.rados.rados.0.smithi137.stdout:1052: oid 31 updating version 326 to 327 2024-09-21T08:27:54.637 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 327 (ObjNum 396 snap 97 seq_num 396) dirty exists 2024-09-21T08:27:54.637 INFO:tasks.rados.rados.0.smithi137.stdout:1052: left oid 31 (ObjNum 396 snap 97 seq_num 396) 2024-09-21T08:27:54.637 INFO:tasks.rados.rados.0.smithi137.stdout:1053: finishing copy_from racing read to smithi13779221-8 2024-09-21T08:27:54.637 INFO:tasks.rados.rados.0.smithi137.stdout:1052: done (2 left) 2024-09-21T08:27:54.637 INFO:tasks.rados.rados.0.smithi137.stdout:1055: read oid 17 snap -1 2024-09-21T08:27:54.637 INFO:tasks.rados.rados.0.smithi137.stdout:1055: expect (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:27:54.637 INFO:tasks.rados.rados.0.smithi137.stdout:1056: snap_remove snap 96 2024-09-21T08:27:54.650 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:54.720 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:54.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:54.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:54.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:54 smithi165 ceph-mon[32893]: osdmap e280: 8 total, 8 up, 4 in 2024-09-21T08:27:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:54 smithi165 ceph-mon[32893]: 3.b scrub starts 2024-09-21T08:27:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:54 smithi165 ceph-mon[32893]: 3.b scrub ok 2024-09-21T08:27:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:54 smithi165 ceph-mon[32893]: osdmap e281: 8 total, 8 up, 4 in 2024-09-21T08:27:54.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:54.864 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:54.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:54.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:54.888 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:54 smithi137 ceph-mon[29634]: osdmap e280: 8 total, 8 up, 4 in 2024-09-21T08:27:54.888 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:54 smithi137 ceph-mon[29634]: 3.b scrub starts 2024-09-21T08:27:54.888 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:54 smithi137 ceph-mon[29634]: 3.b scrub ok 2024-09-21T08:27:54.888 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:54 smithi137 ceph-mon[29634]: osdmap e281: 8 total, 8 up, 4 in 2024-09-21T08:27:54.888 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:54 smithi137 ceph-mon[41044]: osdmap e280: 8 total, 8 up, 4 in 2024-09-21T08:27:54.889 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:54 smithi137 ceph-mon[41044]: 3.b scrub starts 2024-09-21T08:27:54.889 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:54 smithi137 ceph-mon[41044]: 3.b scrub ok 2024-09-21T08:27:54.889 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:54 smithi137 ceph-mon[41044]: osdmap e281: 8 total, 8 up, 4 in 2024-09-21T08:27:54.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:54.988 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:54.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:54.989 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:55.089 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:55.111 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:55.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:55.112 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:55.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:55.234 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:55.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:55.235 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:55.334 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:27:55.335 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:27:55.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:55.353 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:27:55.353 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:27:55.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:55.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:55.358 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:55.369 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:27:55.369 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:27:55.385 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:27:55.385 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:27:55.402 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:27:55.402 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:27:55.407 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:27:55.418 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:27:55.419 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:27:55.435 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:27:55.435 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:27:55.453 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:27:55.453 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:27:55.460 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:55.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:55.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:55.483 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:55.572 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:55.572 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:27:55.572 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:27:55.572 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:27:55.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:55.600 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.2 primary-affinity to 0.692965 (8454142) 2024-09-21T08:27:55.603 INFO:tasks.rados.rados.0.smithi137.stdout:1056: done (3 left) 2024-09-21T08:27:55.603 INFO:tasks.rados.rados.0.smithi137.stdout:1057: write oid 48 current snap is 97 2024-09-21T08:27:55.603 INFO:tasks.rados.rados.0.smithi137.stdout:1057: seq_num 398 ranges {689186=799458,1923598=752130} 2024-09-21T08:27:55.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:55.612 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:55.612 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:55.613 INFO:tasks.rados.rados.0.smithi137.stdout:1057: writing smithi13779221-48 from 689186 to 1488644 tid 1 2024-09-21T08:27:55.623 INFO:tasks.rados.rados.0.smithi137.stdout:1057: writing smithi13779221-48 from 1923598 to 2675728 tid 2 2024-09-21T08:27:55.623 INFO:tasks.rados.rados.0.smithi137.stdout:1054: finishing write tid 1 to smithi13779221-42 2024-09-21T08:27:55.623 INFO:tasks.rados.rados.0.smithi137.stdout:1053: finishing copy_from to smithi13779221-8 2024-09-21T08:27:55.623 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:27:55.623 INFO:tasks.rados.rados.0.smithi137.stdout:1055: expect (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:27:55.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:55.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:55.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:55.736 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[29634]: pgmap v326: 168 pgs: 13 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 104 MiB data, 401 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[29634]: 3.4 scrub starts 2024-09-21T08:27:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[29634]: 3.4 scrub ok 2024-09-21T08:27:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3494716601' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6929648520469458}]: dispatch 2024-09-21T08:27:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[29634]: 4.3 scrub starts 2024-09-21T08:27:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[29634]: 4.3 scrub ok 2024-09-21T08:27:55.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/367082299' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:55.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[41044]: pgmap v326: 168 pgs: 13 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 104 MiB data, 401 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:55.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[41044]: 3.4 scrub starts 2024-09-21T08:27:55.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[41044]: 3.4 scrub ok 2024-09-21T08:27:55.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3494716601' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6929648520469458}]: dispatch 2024-09-21T08:27:55.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[41044]: 4.3 scrub starts 2024-09-21T08:27:55.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[41044]: 4.3 scrub ok 2024-09-21T08:27:55.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:55 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/367082299' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:55.772 INFO:tasks.rados.rados.0.smithi137.stdout:1054: finishing write tid 2 to smithi13779221-42 2024-09-21T08:27:55.772 INFO:tasks.rados.rados.0.smithi137.stdout:1054: finishing write tid 3 to smithi13779221-42 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:1054: finishing write tid 4 to smithi13779221-42 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:1054: oid 42 updating version 0 to 269 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:1054: oid 42 updating version 269 to 271 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:1054: oid 42 version 271 is already newer than 270 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 271 (ObjNum 397 snap 97 seq_num 397) dirty exists 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:1054: left oid 42 (ObjNum 397 snap 97 seq_num 397) 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:1053: done (3 left) 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:1054: done (2 left) 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:1055: done (1 left) 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:1058: rollback oid 39 current snap is 97 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 39 to 95 2024-09-21T08:27:55.773 INFO:tasks.rados.rados.0.smithi137.stdout:1059: delete oid 46 current snap is 97 2024-09-21T08:27:55.774 INFO:tasks.rados.rados.0.smithi137.stdout:1058: finishing rollback tid 1 to smithi13779221-39 2024-09-21T08:27:55.774 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 0 (ObjNum 81 snap 0 seq_num 379559200) dirty dne 2024-09-21T08:27:55.774 INFO:tasks.rados.rados.0.smithi137.stdout:1058: done (2 left) 2024-09-21T08:27:55.774 INFO:tasks.rados.rados.0.smithi137.stdout:1059: done (1 left) 2024-09-21T08:27:55.774 INFO:tasks.rados.rados.0.smithi137.stdout:1060: copy_from oid 37 from oid 47 current snap is 97 2024-09-21T08:27:55.774 INFO:tasks.rados.rados.0.smithi137.stdout:1061: read oid 5 snap -1 2024-09-21T08:27:55.774 INFO:tasks.rados.rados.0.smithi137.stdout:1061: expect (ObjNum 360 snap 85 seq_num 360) 2024-09-21T08:27:55.775 INFO:tasks.rados.rados.0.smithi137.stdout:1062: snap_remove snap 94 2024-09-21T08:27:55.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:55.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:55.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:55.860 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:55.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:55.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:55.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:55.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:55 smithi165 ceph-mon[32893]: pgmap v326: 168 pgs: 13 active+clean+snaptrim_wait, 6 active+clean+snaptrim, 149 active+clean; 104 MiB data, 401 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:55 smithi165 ceph-mon[32893]: 3.4 scrub starts 2024-09-21T08:27:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:55 smithi165 ceph-mon[32893]: 3.4 scrub ok 2024-09-21T08:27:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:55 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3494716601' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6929648520469458}]: dispatch 2024-09-21T08:27:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:55 smithi165 ceph-mon[32893]: 4.3 scrub starts 2024-09-21T08:27:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:55 smithi165 ceph-mon[32893]: 4.3 scrub ok 2024-09-21T08:27:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:55 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/367082299' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:27:56.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:56.107 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:56.108 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:56.108 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:56.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T08:27:56.209 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:27:56.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:56.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:56.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:27:56.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:56.360 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:56.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:56.361 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:56.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:56.483 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:56.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:56.484 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:56.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:56.604 INFO:tasks.rados.rados.0.smithi137.stdout:1062: done (3 left) 2024-09-21T08:27:56.604 INFO:tasks.rados.rados.0.smithi137.stdout:1063: read oid 4 snap -1 2024-09-21T08:27:56.605 INFO:tasks.rados.rados.0.smithi137.stdout:1063: expect deleted 2024-09-21T08:27:56.605 INFO:tasks.rados.rados.0.smithi137.stdout:1064: write (excl) oid 44 current snap is 97 2024-09-21T08:27:56.605 INFO:tasks.rados.rados.0.smithi137.stdout:1064: seq_num 400 ranges {722800=549956,2040562=682238} 2024-09-21T08:27:56.613 INFO:tasks.rados.rados.0.smithi137.stdout:1064: writing smithi13779221-44 from 722800 to 1272756 tid 1 2024-09-21T08:27:56.614 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:56.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:56.615 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:56.622 INFO:tasks.rados.rados.0.smithi137.stdout:1064: writing smithi13779221-44 from 2040562 to 2722800 tid 2 2024-09-21T08:27:56.622 INFO:tasks.rados.rados.0.smithi137.stdout:1065: write oid 36 current snap is 97 2024-09-21T08:27:56.622 INFO:tasks.rados.rados.0.smithi137.stdout:1065: seq_num 401 ranges {739607=625205,1982691=499457} 2024-09-21T08:27:56.629 INFO:tasks.rados.rados.0.smithi137.stdout:1065: writing smithi13779221-36 from 739607 to 1364812 tid 1 2024-09-21T08:27:56.636 INFO:tasks.rados.rados.0.smithi137.stdout:1065: writing smithi13779221-36 from 1982691 to 2482148 tid 2 2024-09-21T08:27:56.636 INFO:tasks.rados.rados.0.smithi137.stdout:1066: read oid 25 snap -1 2024-09-21T08:27:56.636 INFO:tasks.rados.rados.0.smithi137.stdout:1066: expect (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:27:56.636 INFO:tasks.rados.rados.0.smithi137.stdout:1067: snap_create 2024-09-21T08:27:56.636 INFO:tasks.rados.rados.0.smithi137.stdout:1061: expect (ObjNum 360 snap 85 seq_num 360) 2024-09-21T08:27:56.702 INFO:tasks.rados.rados.0.smithi137.stdout:1060: finishing copy_from racing read to smithi13779221-37 2024-09-21T08:27:56.702 INFO:tasks.rados.rados.0.smithi137.stdout:1060: finishing copy_from to smithi13779221-37 2024-09-21T08:27:56.702 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:27:56.702 INFO:tasks.rados.rados.0.smithi137.stdout:1064: finishing write tid 1 to smithi13779221-44 2024-09-21T08:27:56.702 INFO:tasks.rados.rados.0.smithi137.stdout:1057: finishing write tid 1 to smithi13779221-48 2024-09-21T08:27:56.702 INFO:tasks.rados.rados.0.smithi137.stdout:1057: finishing write tid 2 to smithi13779221-48 2024-09-21T08:27:56.702 INFO:tasks.rados.rados.0.smithi137.stdout:1057: finishing write tid 3 to smithi13779221-48 2024-09-21T08:27:56.702 INFO:tasks.rados.rados.0.smithi137.stdout:1057: finishing write tid 4 to smithi13779221-48 2024-09-21T08:27:56.702 INFO:tasks.rados.rados.0.smithi137.stdout:1057: oid 48 updating version 0 to 298 2024-09-21T08:27:56.702 INFO:tasks.rados.rados.0.smithi137.stdout:1057: oid 48 version 298 is already newer than 297 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1057: oid 48 version 298 is already newer than 296 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 298 (ObjNum 398 snap 97 seq_num 398) dirty exists 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1057: left oid 48 (ObjNum 398 snap 97 seq_num 398) 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1064: finishing write tid 2 to smithi13779221-44 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1064: finishing write tid 3 to smithi13779221-44 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1064: finishing write tid 4 to smithi13779221-44 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1064: oid 44 updating version 0 to 264 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1064: oid 44 version 264 is already newer than 263 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1064: oid 44 version 264 is already newer than 262 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 264 (ObjNum 400 snap 97 seq_num 400) dirty exists 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1064: left oid 44 (ObjNum 400 snap 97 seq_num 400) 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1065: finishing write tid 1 to smithi13779221-36 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1065: finishing write tid 2 to smithi13779221-36 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1065: finishing write tid 3 to smithi13779221-36 2024-09-21T08:27:56.703 INFO:tasks.rados.rados.0.smithi137.stdout:1066: expect (ObjNum 200 snap 40 seq_num 200) 2024-09-21T08:27:56.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:56.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:56.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:56.741 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:56 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3494716601' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6929648520469458}]': finished 2024-09-21T08:27:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:56 smithi137 ceph-mon[29634]: osdmap e282: 8 total, 8 up, 4 in 2024-09-21T08:27:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:56 smithi137 ceph-mon[29634]: 7.1e scrub starts 2024-09-21T08:27:56.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:56 smithi137 ceph-mon[29634]: 7.1e scrub ok 2024-09-21T08:27:56.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:56 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3494716601' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6929648520469458}]': finished 2024-09-21T08:27:56.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:56 smithi137 ceph-mon[41044]: osdmap e282: 8 total, 8 up, 4 in 2024-09-21T08:27:56.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:56 smithi137 ceph-mon[41044]: 7.1e scrub starts 2024-09-21T08:27:56.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:56 smithi137 ceph-mon[41044]: 7.1e scrub ok 2024-09-21T08:27:56.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:56.855 INFO:tasks.rados.rados.0.smithi137.stdout:1065: finishing write tid 4 to smithi13779221-36 2024-09-21T08:27:56.855 INFO:tasks.rados.rados.0.smithi137.stdout:1065: oid 36 updating version 0 to 489 2024-09-21T08:27:56.855 INFO:tasks.rados.rados.0.smithi137.stdout:1065: oid 36 version 489 is already newer than 488 2024-09-21T08:27:56.855 INFO:tasks.rados.rados.0.smithi137.stdout:1065: oid 36 version 489 is already newer than 487 2024-09-21T08:27:56.855 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 489 (ObjNum 401 snap 97 seq_num 401) dirty exists 2024-09-21T08:27:56.855 INFO:tasks.rados.rados.0.smithi137.stdout:1065: left oid 36 (ObjNum 401 snap 97 seq_num 401) 2024-09-21T08:27:56.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:56.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:56.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:56.966 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:56.989 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:56.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:56.989 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:56 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3494716601' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.6929648520469458}]': finished 2024-09-21T08:27:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:56 smithi165 ceph-mon[32893]: osdmap e282: 8 total, 8 up, 4 in 2024-09-21T08:27:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:56 smithi165 ceph-mon[32893]: 7.1e scrub starts 2024-09-21T08:27:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:56 smithi165 ceph-mon[32893]: 7.1e scrub ok 2024-09-21T08:27:57.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:57.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:57.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:57.114 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:57.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:57.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:57.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:57.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:57.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:57.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:57.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:57.361 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:57.462 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:57.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:57.488 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:57.561 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:57.563 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:57.588 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:57.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:57.614 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:57.615 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:57.617 INFO:tasks.rados.rados.0.smithi137.stdout:1057: done (7 left) 2024-09-21T08:27:57.617 INFO:tasks.rados.rados.0.smithi137.stdout:1060: done (6 left) 2024-09-21T08:27:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:1061: done (5 left) 2024-09-21T08:27:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:1063: done (4 left) 2024-09-21T08:27:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:1064: done (3 left) 2024-09-21T08:27:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:1065: done (2 left) 2024-09-21T08:27:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:1066: done (1 left) 2024-09-21T08:27:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:1067: done (0 left) 2024-09-21T08:27:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:1068: snap_remove snap 95 2024-09-21T08:27:57.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:57.736 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[29634]: pgmap v329: 168 pgs: 10 peering, 11 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 145 active+clean; 112 MiB data, 411 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:57.736 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[29634]: 4.18 scrub starts 2024-09-21T08:27:57.736 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[29634]: 4.18 scrub ok 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[29634]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[29634]: osdmap e283: 8 total, 8 up, 4 in 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[29634]: 6.18 scrub starts 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[29634]: 6.18 scrub ok 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[41044]: pgmap v329: 168 pgs: 10 peering, 11 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 145 active+clean; 112 MiB data, 411 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[41044]: 4.18 scrub starts 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[41044]: 4.18 scrub ok 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[41044]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[41044]: osdmap e283: 8 total, 8 up, 4 in 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[41044]: 6.18 scrub starts 2024-09-21T08:27:57.737 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:57 smithi137 ceph-mon[41044]: 6.18 scrub ok 2024-09-21T08:27:57.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:57.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:57.741 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:57.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:57.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:57.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:57.864 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:57.964 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:57.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:57.992 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:57.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:57 smithi165 ceph-mon[32893]: pgmap v329: 168 pgs: 10 peering, 11 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 145 active+clean; 112 MiB data, 411 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:57 smithi165 ceph-mon[32893]: 4.18 scrub starts 2024-09-21T08:27:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:57 smithi165 ceph-mon[32893]: 4.18 scrub ok 2024-09-21T08:27:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:57 smithi165 ceph-mon[32893]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T08:27:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:57 smithi165 ceph-mon[32893]: osdmap e283: 8 total, 8 up, 4 in 2024-09-21T08:27:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:57 smithi165 ceph-mon[32893]: 6.18 scrub starts 2024-09-21T08:27:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:57 smithi165 ceph-mon[32893]: 6.18 scrub ok 2024-09-21T08:27:58.093 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:58.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:58.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:58.119 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:58.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:58.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:58.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:58.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:58.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:58.358 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:27:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:27:58.358 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:27:58.358 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:27:58.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:58.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:58.375 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:58.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:58.498 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:58.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:58.499 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:58.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:27:58.619 INFO:tasks.rados.rados.0.smithi137.stdout:1068: done (0 left) 2024-09-21T08:27:58.619 INFO:tasks.rados.rados.0.smithi137.stdout:1069: snap_create 2024-09-21T08:27:58.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:58.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:58.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:27:58.730 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:27:58.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:58.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:58.755 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:27:58.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:27:58.876 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:58.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:58.877 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:27:58.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:58.987 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:58 smithi137 ceph-mon[29634]: osdmap e284: 8 total, 8 up, 4 in 2024-09-21T08:27:58.987 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:58 smithi137 ceph-mon[29634]: 6.13 scrub starts 2024-09-21T08:27:58.987 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:58 smithi137 ceph-mon[29634]: 6.13 scrub ok 2024-09-21T08:27:58.987 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:58 smithi137 ceph-mon[41044]: osdmap e284: 8 total, 8 up, 4 in 2024-09-21T08:27:58.987 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:58 smithi137 ceph-mon[41044]: 6.13 scrub starts 2024-09-21T08:27:58.988 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:58 smithi137 ceph-mon[41044]: 6.13 scrub ok 2024-09-21T08:27:59.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:59.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:59.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:58 smithi165 ceph-mon[32893]: osdmap e284: 8 total, 8 up, 4 in 2024-09-21T08:27:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:58 smithi165 ceph-mon[32893]: 6.13 scrub starts 2024-09-21T08:27:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:58 smithi165 ceph-mon[32893]: 6.13 scrub ok 2024-09-21T08:27:59.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T08:27:59.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:59.127 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:59.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:59.128 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:59.229 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:59.251 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:59.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:59.252 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:59.314 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:27:59.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:27:59.378 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:59.379 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:59.379 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:27:59.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:59.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:27:59.502 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:59.503 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:59.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:59.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:27:59.616 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:27:59.620 INFO:tasks.rados.rados.0.smithi137.stdout:1069: done (0 left) 2024-09-21T08:27:59.620 INFO:tasks.rados.rados.0.smithi137.stdout:1070: rollback oid 17 current snap is 99 2024-09-21T08:27:59.620 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 17 to 98 2024-09-21T08:27:59.621 INFO:tasks.rados.rados.0.smithi137.stdout:1071: read oid 42 snap -1 2024-09-21T08:27:59.621 INFO:tasks.rados.rados.0.smithi137.stdout:1071: expect (ObjNum 397 snap 97 seq_num 397) 2024-09-21T08:27:59.621 INFO:tasks.rados.rados.0.smithi137.stdout:1072: write oid 29 current snap is 99 2024-09-21T08:27:59.621 INFO:tasks.rados.rados.0.smithi137.stdout:1072: seq_num 402 ranges {756414=700454,2041173=715241} 2024-09-21T08:27:59.629 INFO:tasks.rados.rados.0.smithi137.stdout:1072: writing smithi13779221-29 from 756414 to 1456868 tid 1 2024-09-21T08:27:59.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:59.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:59.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:27:59.639 INFO:tasks.rados.rados.0.smithi137.stdout:1072: writing smithi13779221-29 from 2041173 to 2756414 tid 2 2024-09-21T08:27:59.639 INFO:tasks.rados.rados.0.smithi137.stdout:1073: write (excl) oid 46 current snap is 99 2024-09-21T08:27:59.639 INFO:tasks.rados.rados.0.smithi137.stdout:1073: seq_num 403 ranges {773221=492056,1816008=503126} 2024-09-21T08:27:59.646 INFO:tasks.rados.rados.0.smithi137.stdout:1073: writing smithi13779221-46 from 773221 to 1265277 tid 1 2024-09-21T08:27:59.652 INFO:tasks.rados.rados.0.smithi137.stdout:1073: writing smithi13779221-46 from 1816008 to 2319134 tid 2 2024-09-21T08:27:59.652 INFO:tasks.rados.rados.0.smithi137.stdout:1074: read oid 36 snap -1 2024-09-21T08:27:59.652 INFO:tasks.rados.rados.0.smithi137.stdout:1074: expect (ObjNum 401 snap 97 seq_num 401) 2024-09-21T08:27:59.652 INFO:tasks.rados.rados.0.smithi137.stdout:1075: write oid 8 current snap is 99 2024-09-21T08:27:59.652 INFO:tasks.rados.rados.0.smithi137.stdout:1075: seq_num 404 ranges {790028=567305,1874490=763137} 2024-09-21T08:27:59.659 INFO:tasks.rados.rados.0.smithi137.stdout:1075: writing smithi13779221-8 from 790028 to 1357333 tid 1 2024-09-21T08:27:59.667 INFO:tasks.rados.rados.0.smithi137.stdout:1075: writing smithi13779221-8 from 1874490 to 2637627 tid 2 2024-09-21T08:27:59.667 INFO:tasks.rados.rados.0.smithi137.stdout:1076: snap_remove snap 97 2024-09-21T08:27:59.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:27:59.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:27:59.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:59.763 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:27:59.844 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[29634]: 3.d scrub starts 2024-09-21T08:27:59.844 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[29634]: 3.d scrub ok 2024-09-21T08:27:59.844 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[29634]: pgmap v332: 168 pgs: 10 peering, 11 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 145 active+clean; 112 MiB data, 411 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:59.844 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[29634]: osdmap e285: 8 total, 8 up, 4 in 2024-09-21T08:27:59.844 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3687705135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:27:59.844 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:59.844 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3687705135' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:27:59.844 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[29634]: osdmap e286: 8 total, 8 up, 4 in 2024-09-21T08:27:59.844 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[41044]: 3.d scrub starts 2024-09-21T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[41044]: 3.d scrub ok 2024-09-21T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[41044]: pgmap v332: 168 pgs: 10 peering, 11 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 145 active+clean; 112 MiB data, 411 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[41044]: osdmap e285: 8 total, 8 up, 4 in 2024-09-21T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3687705135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3687705135' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:27:59.845 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:27:59 smithi137 ceph-mon[41044]: osdmap e286: 8 total, 8 up, 4 in 2024-09-21T08:27:59.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:27:59.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:27:59.886 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:27:59.886 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:27:59.960 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:27:59.960 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:1082 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94837804043520 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 201.78462861400001, 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.39751146399999998, 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:27:59.961 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1082 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159528+0000", 2024-09-21T08:27:59.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2749999999999998e-06 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159533+0000", 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2480000000000004e-06 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159547+0000", 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4516e-05 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159556+0000", 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9460000000000004e-06 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159617+0000", 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0999000000000003e-05 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159625+0000", 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240000000000007e-06 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544062+0000", 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.38443652099999998 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544359+0000", 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000297378 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548773+0000", 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044136440000000004 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548858+0000", 2024-09-21T08:27:59.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5638999999999998e-05 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.557037+0000", 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081783740000000004 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:255 7.2 7:4031afd1:::smithi13779221-48:head [copy-from ver 3] snapc 4=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 194.43631822, 2024-09-21T08:27:59.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015259864, 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 255 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507839+0000", 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0089999999999999e-06 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:59.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507852+0000", 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3617e-05 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507872+0000", 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9482000000000001e-05 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507883+0000", 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1036000000000001e-05 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507941+0000", 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7819999999999999e-05 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507946+0000", 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9080000000000003e-06 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510282+0000", 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023368170000000001 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:59.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510376+0000", 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3846999999999998e-05 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.516270+0000", 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005893934 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522776+0000", 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065057179999999997 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:59.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522902+0000", 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012641 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522990+0000", 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7491000000000006e-05 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523050+0000", 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0075999999999999e-05 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523096+0000", 2024-09-21T08:27:59.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5698999999999997e-05 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:256 7.2 7:4031afd1:::smithi13779221-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 194.436250922, 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013379356, 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:59.973 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 256 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507904+0000", 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8000000000000005e-07 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507907+0000", 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.258e-06 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507915+0000", 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.571e-06 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507920+0000", 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1189999999999996e-06 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507998+0000", 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8181000000000002e-05 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.508004+0000", 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3960000000000003e-06 2024-09-21T08:27:59.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510529+0000", 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025244989999999999 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511053+0000", 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052380199999999997 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521102+0000", 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010049361999999999 2024-09-21T08:27:59.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521183+0000", 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0495999999999996e-05 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521282+0000", 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9891999999999998e-05 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.978 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:260 7.8 e93/84 7:1fd29771:::smithi13779221-22:head v 93'20, mlcod=86'17)", 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 193.37520458899999, 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078947310000000003, 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568951+0000", 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1510000000000001e-06 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571221+0000", 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022708279999999999 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:59.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571244+0000", 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2759999999999999e-05 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571257+0000", 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2669e-05 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571601+0000", 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034448900000000001 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:59.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571645+0000", 2024-09-21T08:27:59.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3498999999999998e-05 2024-09-21T08:27:59.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:59.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576787+0000", 2024-09-21T08:27:59.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051417010000000003 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576844+0000", 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7633999999999997e-05 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:294 7.2 7:4d793d31:::smithi13779221-9:head [copy-from ver 3] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 193.181487793, 2024-09-21T08:27:59.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017337353, 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 294 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762668+0000", 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.353e-06 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762673+0000", 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1569999999999999e-06 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762683+0000", 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0331e-05 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762688+0000", 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1410000000000004e-06 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766904+0000", 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042154899999999997 2024-09-21T08:27:59.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767004+0000", 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100135 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.771465+0000", 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044613170000000002 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779379+0000", 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079137649999999997 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779883+0000", 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050370000000000005 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779896+0000", 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3458999999999999e-05 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779950+0000", 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3851000000000003e-05 2024-09-21T08:27:59.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.780004+0000", 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3653999999999998e-05 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:295 7.2 7:4d793d31:::smithi13779221-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 193.181457075, 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012332286, 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:59.989 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 295 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:27:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:59.991 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4400000000000001e-07 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762699+0000", 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0099999999999998e-06 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762705+0000", 2024-09-21T08:27:59.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0279999999999997e-06 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762710+0000", 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8210000000000001e-06 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767340+0000", 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046298770000000001 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.768012+0000", 2024-09-21T08:27:59.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067169199999999999 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774900+0000", 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068879680000000004 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774961+0000", 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0980999999999998e-05 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:27:59.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.775029+0000", 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8465000000000003e-05 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1908 7.a 7:53537bd3:::smithi13779221-39:head [append 0~0] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 71.394267115000005, 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.037489017, 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:27:59.995 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1908 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:27:59.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549889+0000", 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.199e-06 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549894+0000", 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9450000000000001e-06 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549902+0000", 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9459999999999998e-06 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549910+0000", 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6510000000000001e-06 2024-09-21T08:27:59.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549962+0000", 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2741999999999997e-05 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549970+0000", 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0949999999999996e-06 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.556983+0000", 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.007012673 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557124+0000", 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014058199999999999 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583019+0000", 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025894830000000001 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583102+0000", 2024-09-21T08:27:59.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3332999999999996e-05 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583767+0000", 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00066536999999999998 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583809+0000", 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1919999999999998e-05 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:28:00.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583935+0000", 2024-09-21T08:28:00.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012586800000000001 2024-09-21T08:28:00.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:00.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584880+0000", 2024-09-21T08:28:00.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00094458400000000005 2024-09-21T08:28:00.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585414+0000", 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000533892 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.586605+0000", 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011918479999999999 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587260+0000", 2024-09-21T08:28:00.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065433099999999997 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587376+0000", 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116208 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1909 7.a 7:53537bd3:::smithi13779221-39:head [rollback 39] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 71.394171150999995, 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0375938790000001, 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.003 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1909 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:28:00.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549984+0000", 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5799999999999998e-07 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549987+0000", 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5099999999999999e-06 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549993+0000", 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6099999999999997e-06 2024-09-21T08:28:00.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549998+0000", 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4890000000000003e-06 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550039+0000", 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1155000000000002e-05 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550045+0000", 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7130000000000002e-06 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557614+0000", 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.007568971 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557661+0000", 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6726e-05 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583163+0000", 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025501894000000001 2024-09-21T08:28:00.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583309+0000", 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014584299999999999 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584385+0000", 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010765250000000001 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584430+0000", 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5151000000000002e-05 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584564+0000", 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133652 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585131+0000", 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00056728200000000001 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585577+0000", 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044534199999999999 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587426+0000", 2024-09-21T08:28:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001849402 2024-09-21T08:28:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587490+0000", 2024-09-21T08:28:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4126000000000002e-05 2024-09-21T08:28:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587577+0000", 2024-09-21T08:28:00.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6630000000000002e-05 2024-09-21T08:28:00.011 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.011 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.011 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:1376 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94837804043520 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 69.315268282999995, 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0087414380000000007, 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1376 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628889+0000", 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9299999999999999e-06 2024-09-21T08:28:00.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628895+0000", 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6189999999999996e-06 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628909+0000", 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3963e-05 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628916+0000", 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.131e-06 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628998+0000", 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1219999999999998e-05 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.629004+0000", 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7510000000000002e-06 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.634800+0000", 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005795311 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.634882+0000", 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2025000000000002e-05 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637627+0000", 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027454879999999999 2024-09-21T08:28:00.016 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.017 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.017 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.017 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2114 7.4 e228/215 7:26d6e516:::smithi13779221-33:head v 228'62, mlcod=211'59)", 2024-09-21T08:28:00.017 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:28:00.018 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 58.403213761000004, 2024-09-21T08:28:00.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012725531, 2024-09-21T08:28:00.018 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.018 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.018 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:00.018 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:28:00.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540942+0000", 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3549999999999999e-06 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543680+0000", 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002738379 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543701+0000", 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1072000000000001e-05 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543712+0000", 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0691e-05 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543786+0000", 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4721999999999994e-05 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543809+0000", 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2731e-05 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553613+0000", 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098038980000000001 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553666+0000", 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2682999999999997e-05 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2124 7.7 e229/215 7:e55f2143:::smithi13779221-11:head v 229'87, mlcod=224'83)", 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 57.411393283999999, 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008804704, 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:00.022 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1900000000000003e-07 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535432+0000", 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026709979999999999 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535449+0000", 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6888000000000002e-05 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535460+0000", 2024-09-21T08:28:00.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0947e-05 2024-09-21T08:28:00.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535516+0000", 2024-09-21T08:28:00.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5551999999999999e-05 2024-09-21T08:28:00.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535538+0000", 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2135000000000001e-05 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.541520+0000", 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059823330000000003 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.541565+0000", 2024-09-21T08:28:00.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5331999999999999e-05 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2177 7.2 7:43ab705f:::smithi13779221-10:head [rollback 44] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.409997035000004, 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0080117160000000003, 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.027 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2177 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.028 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534158+0000", 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0580000000000001e-06 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534200+0000", 2024-09-21T08:28:00.029 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1875999999999997e-05 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534209+0000", 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0899999999999994e-06 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534215+0000", 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9499999999999998e-06 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.030 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:00.031 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:59 smithi165 ceph-mon[32893]: 3.d scrub starts 2024-09-21T08:28:00.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:59 smithi165 ceph-mon[32893]: 3.d scrub ok 2024-09-21T08:28:00.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:59 smithi165 ceph-mon[32893]: pgmap v332: 168 pgs: 10 peering, 11 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 145 active+clean; 112 MiB data, 411 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:00.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:59 smithi165 ceph-mon[32893]: osdmap e285: 8 total, 8 up, 4 in 2024-09-21T08:28:00.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:59 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3687705135' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:28:00.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:59 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:00.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:59 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3687705135' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:28:00.032 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:27:59 smithi165 ceph-mon[32893]: osdmap e286: 8 total, 8 up, 4 in 2024-09-21T08:28:00.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534288+0000", 2024-09-21T08:28:00.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3223e-05 2024-09-21T08:28:00.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:00.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534294+0000", 2024-09-21T08:28:00.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.519e-06 2024-09-21T08:28:00.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.539853+0000", 2024-09-21T08:28:00.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055597980000000003 2024-09-21T08:28:00.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.539923+0000", 2024-09-21T08:28:00.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9115000000000005e-05 2024-09-21T08:28:00.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:00.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540058+0000", 2024-09-21T08:28:00.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013511999999999999 2024-09-21T08:28:00.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540864+0000", 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00080602199999999999 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.541123+0000", 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025911100000000002 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542066+0000", 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000943541 2024-09-21T08:28:00.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542123+0000", 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7108999999999998e-05 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542169+0000", 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5184e-05 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2178 7.2 7:43ab705f:::smithi13779221-10:head [append 0~0] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:28:00.037 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.409825533000003, 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081322689999999993, 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2178 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534329+0000", 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8299999999999995e-07 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534351+0000", 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1662999999999999e-05 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534359+0000", 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9480000000000008e-06 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534364+0000", 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1329999999999998e-06 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534415+0000", 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1199999999999998e-05 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534421+0000", 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1200000000000001e-06 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.041 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540407+0000", 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059863010000000003 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540492+0000", 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4715000000000005e-05 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540616+0000", 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012489700000000001 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540903+0000", 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028699500000000001 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.541328+0000", 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042463800000000002 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542392+0000", 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010644199999999999 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542427+0000", 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4338999999999998e-05 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542461+0000", 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3917000000000001e-05 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.044 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2184 7.c 7:305aef79:::smithi13779221-15:head [copy-from ver 143] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.390568621, 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022239622000000001, 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2184 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553588+0000", 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.021e-06 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553596+0000", 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1400000000000002e-06 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553611+0000", 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5478e-05 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553619+0000", 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9019999999999999e-06 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553685+0000", 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5977999999999997e-05 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553801+0000", 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116506 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.565629+0000", 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011827127999999999 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.574989+0000", 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093609639999999994 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575477+0000", 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048743400000000002 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575700+0000", 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022307499999999999 2024-09-21T08:28:00.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575756+0000", 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6125999999999998e-05 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575825+0000", 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8869999999999996e-05 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.051 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2185 7.c 7:305aef79:::smithi13779221-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:28:00.051 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:28:00.051 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 50.390515006000001, 2024-09-21T08:28:00.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016182583, 2024-09-21T08:28:00.051 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2185 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553640+0000", 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.61e-07 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553643+0000", 2024-09-21T08:28:00.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1020000000000001e-06 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553653+0000", 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0249e-05 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553659+0000", 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3319999999999997e-06 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553963+0000", 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000303617 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.554000+0000", 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6976e-05 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569623+0000", 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015623076 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569721+0000", 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7398000000000005e-05 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569822+0000", 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101072 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2219 7.c 7:361ee62b:::smithi13779221-40:head [sparse-read 0~3130511] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:28:00.056 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 49.399581058999999, 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095045249999999998, 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2219 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:28:00.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544574+0000", 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.297e-06 2024-09-21T08:28:00.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544579+0000", 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2830000000000003e-06 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544588+0000", 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4849999999999998e-06 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544596+0000", 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6930000000000007e-06 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544738+0000", 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014263700000000001 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:00.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544744+0000", 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6030000000000004e-06 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.552342+0000", 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075976259999999997 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.552431+0000", 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9571e-05 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554077+0000", 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016463300000000001 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2220 7.c 7:361ee62b:::smithi13779221-40:head [read 0~3546277,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 49.399549082999997, 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011021838000000001, 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.062 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2220 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5499999999999998e-07 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544610+0000", 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4349999999999999e-06 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544617+0000", 2024-09-21T08:28:00.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2030000000000003e-06 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544623+0000", 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0260000000000004e-06 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544805+0000", 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000181635 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544809+0000", 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5950000000000001e-06 2024-09-21T08:28:00.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554139+0000", 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.009329225 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554247+0000", 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010890700000000001 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.555627+0000", 2024-09-21T08:28:00.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013792570000000001 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2233 7.8 e238/218 7:17170b1f:::smithi13779221-4:head v 238'212, mlcod=238'208)", 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 48.388239446, 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018069034000000001, 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555917+0000", 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3089999999999998e-06 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556417+0000", 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050039099999999999 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556437+0000", 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9746e-05 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556446+0000", 2024-09-21T08:28:00.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4590000000000004e-06 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556581+0000", 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013473700000000001 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556614+0000", 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3287000000000002e-05 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573937+0000", 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017322422000000001 2024-09-21T08:28:00.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573984+0000", 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6683e-05 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2234 7.8 e238/218 7:17170b1f:::smithi13779221-4:head v 238'213, mlcod=238'208)", 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 48.387683782000003, 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017540864, 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.071 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556471+0000", 2024-09-21T08:28:00.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3099999999999997e-07 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556477+0000", 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9680000000000003e-06 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556486+0000", 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5719999999999992e-06 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556492+0000", 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6149999999999996e-06 2024-09-21T08:28:00.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.557226+0000", 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073403900000000002 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.557411+0000", 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018489400000000001 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573988+0000", 2024-09-21T08:28:00.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016576941000000001 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574011+0000", 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3104000000000001e-05 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2285 7.c 7:361ee62b:::smithi13779221-40:head [copy-from ver 252] snapc 4f=[4f,4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 47.082637294000001, 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010853527, 2024-09-21T08:28:00.075 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2285 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5999999999999999e-07 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861521+0000", 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7299999999999999e-06 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861524+0000", 2024-09-21T08:28:00.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9510000000000002e-06 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861526+0000", 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5739999999999998e-06 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861577+0000", 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0764999999999999e-05 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861629+0000", 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1937000000000001e-05 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.863963+0000", 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002334175 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.869705+0000", 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057416610000000003 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:00.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872022+0000", 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023169670000000001 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872295+0000", 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027316900000000003 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872331+0000", 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6276999999999998e-05 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:00.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:00.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872370+0000", 2024-09-21T08:28:00.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8961000000000002e-05 2024-09-21T08:28:00.081 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.081 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.081 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.081 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:00.081 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:00.081 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:28:00.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:00.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:00.082 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:00.183 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:00.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:00.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:00.210 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:00.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:00.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:00.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:00.337 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:00.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:00.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:00.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:00.460 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:00.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:00.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:00.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:00.588 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:00.639 INFO:tasks.rados.rados.0.smithi137.stdout:1076: done (6 left) 2024-09-21T08:28:00.639 INFO:tasks.rados.rados.0.smithi137.stdout:1077: copy_from oid 19 from oid 21 current snap is 99 2024-09-21T08:28:00.639 INFO:tasks.rados.rados.0.smithi137.stdout:1070: finishing rollback tid 0 to smithi13779221-17 2024-09-21T08:28:00.639 INFO:tasks.rados.rados.0.smithi137.stdout:1070: finishing rollback tid 1 to smithi13779221-17 2024-09-21T08:28:00.639 INFO:tasks.rados.rados.0.smithi137.stdout:1071: expect (ObjNum 397 snap 97 seq_num 397) 2024-09-21T08:28:00.689 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:00.710 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:00.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:00.711 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:00.730 INFO:tasks.rados.rados.0.smithi137.stdout:1070: finishing rollback tid 2 to smithi13779221-17 2024-09-21T08:28:00.730 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 267 (ObjNum 200 snap 40 seq_num 200) dirty exists 2024-09-21T08:28:00.730 INFO:tasks.rados.rados.0.smithi137.stdout:1072: finishing write tid 1 to smithi13779221-29 2024-09-21T08:28:00.730 INFO:tasks.rados.rados.0.smithi137.stdout:1072: finishing write tid 2 to smithi13779221-29 2024-09-21T08:28:00.730 INFO:tasks.rados.rados.0.smithi137.stdout:1070: done (6 left) 2024-09-21T08:28:00.730 INFO:tasks.rados.rados.0.smithi137.stdout:1071: done (5 left) 2024-09-21T08:28:00.730 INFO:tasks.rados.rados.0.smithi137.stdout:1078: snap_remove snap 98 2024-09-21T08:28:00.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:00.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:00.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:00.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:00.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:00.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:00.963 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:00.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:00 smithi165 ceph-mon[32893]: 2.1 scrub starts 2024-09-21T08:28:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:00 smithi165 ceph-mon[32893]: 2.1 scrub ok 2024-09-21T08:28:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:01.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:00 smithi137 ceph-mon[29634]: 2.1 scrub starts 2024-09-21T08:28:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:00 smithi137 ceph-mon[29634]: 2.1 scrub ok 2024-09-21T08:28:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:01.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:01.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:28:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:28:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:28:01.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:00 smithi137 ceph-mon[41044]: 2.1 scrub starts 2024-09-21T08:28:01.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:00 smithi137 ceph-mon[41044]: 2.1 scrub ok 2024-09-21T08:28:01.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:01.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:01.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:01.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:01.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:01.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:01.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:01.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:01.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:01.213 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:01.314 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:01.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:01.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:01.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:01.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T08:28:01.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:01.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:01.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:01.460 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:01.502 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:01.537 INFO:tasks.rados.rados.0.smithi137.stdout:1078: done (5 left) 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1079: rollback oid 22 current snap is 99 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1079: done (5 left) 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1080: copy_from oid 45 from oid 10 current snap is 99 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1072: finishing write tid 3 to smithi13779221-29 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1081: write (excl) oid 39 current snap is 99 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1072: finishing write tid 4 to smithi13779221-29 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1072: oid 29 updating version 0 to 491 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1072: oid 29 version 491 is already newer than 490 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1072: oid 29 updating version 491 to 492 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 492 (ObjNum 402 snap 99 seq_num 402) dirty exists 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1072: left oid 29 (ObjNum 402 snap 99 seq_num 402) 2024-09-21T08:28:01.538 INFO:tasks.rados.rados.0.smithi137.stdout:1073: finishing write tid 1 to smithi13779221-46 2024-09-21T08:28:01.539 INFO:tasks.rados.rados.0.smithi137.stdout:1073: finishing write tid 2 to smithi13779221-46 2024-09-21T08:28:01.539 INFO:tasks.rados.rados.0.smithi137.stdout:1074: expect (ObjNum 401 snap 97 seq_num 401) 2024-09-21T08:28:01.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:01.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:01.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:01.583 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:01.659 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:01.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:01.697 INFO:tasks.rados.rados.0.smithi137.stdout:1073: finishing write tid 3 to smithi13779221-46 2024-09-21T08:28:01.697 INFO:tasks.rados.rados.0.smithi137.stdout:1073: finishing write tid 4 to smithi13779221-46 2024-09-21T08:28:01.697 INFO:tasks.rados.rados.0.smithi137.stdout:1073: oid 46 updating version 0 to 493 2024-09-21T08:28:01.698 INFO:tasks.rados.rados.0.smithi137.stdout:1073: oid 46 updating version 493 to 495 2024-09-21T08:28:01.698 INFO:tasks.rados.rados.0.smithi137.stdout:1073: oid 46 version 495 is already newer than 494 2024-09-21T08:28:01.698 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 495 (ObjNum 403 snap 99 seq_num 403) dirty exists 2024-09-21T08:28:01.698 INFO:tasks.rados.rados.0.smithi137.stdout:1073: left oid 46 (ObjNum 403 snap 99 seq_num 403) 2024-09-21T08:28:01.698 INFO:tasks.rados.rados.0.smithi137.stdout:1075: finishing write tid 1 to smithi13779221-8 2024-09-21T08:28:01.698 INFO:tasks.rados.rados.0.smithi137.stdout:1075: finishing write tid 2 to smithi13779221-8 2024-09-21T08:28:01.698 INFO:tasks.rados.rados.0.smithi137.stdout:1075: finishing write tid 3 to smithi13779221-8 2024-09-21T08:28:01.698 INFO:tasks.rados.rados.0.smithi137.stdout:1081: seq_num 405 ranges {406835=642554,1532972=506795,2772312=34523} 2024-09-21T08:28:01.703 INFO:tasks.rados.rados.0.smithi137.stdout:1081: writing smithi13779221-39 from 406835 to 1049389 tid 1 2024-09-21T08:28:01.705 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:01.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:01.706 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:01.708 INFO:tasks.rados.rados.0.smithi137.stdout:1081: writing smithi13779221-39 from 1532972 to 2039767 tid 2 2024-09-21T08:28:01.713 INFO:tasks.rados.rados.0.smithi137.stdout:1081: writing smithi13779221-39 from 2772312 to 2806835 tid 3 2024-09-21T08:28:01.713 INFO:tasks.rados.rados.0.smithi137.stdout:1072: done (6 left) 2024-09-21T08:28:01.713 INFO:tasks.rados.rados.0.smithi137.stdout:1073: done (5 left) 2024-09-21T08:28:01.713 INFO:tasks.rados.rados.0.smithi137.stdout:1074: done (4 left) 2024-09-21T08:28:01.713 INFO:tasks.rados.rados.0.smithi137.stdout:1082: write (excl) oid 18 current snap is 99 2024-09-21T08:28:01.713 INFO:tasks.rados.rados.0.smithi137.stdout:1075: finishing write tid 4 to smithi13779221-8 2024-09-21T08:28:01.714 INFO:tasks.rados.rados.0.smithi137.stdout:1075: oid 8 updating version 0 to 278 2024-09-21T08:28:01.714 INFO:tasks.rados.rados.0.smithi137.stdout:1075: oid 8 updating version 278 to 280 2024-09-21T08:28:01.714 INFO:tasks.rados.rados.0.smithi137.stdout:1075: oid 8 version 280 is already newer than 279 2024-09-21T08:28:01.714 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 280 (ObjNum 404 snap 99 seq_num 404) dirty exists 2024-09-21T08:28:01.714 INFO:tasks.rados.rados.0.smithi137.stdout:1075: left oid 8 (ObjNum 404 snap 99 seq_num 404) 2024-09-21T08:28:01.714 INFO:tasks.rados.rados.0.smithi137.stdout:1082: seq_num 406 ranges {423642=717803,1875101=766806} 2024-09-21T08:28:01.719 INFO:tasks.rados.rados.0.smithi137.stdout:1082: writing smithi13779221-18 from 423642 to 1141445 tid 1 2024-09-21T08:28:01.728 INFO:tasks.rados.rados.0.smithi137.stdout:1082: writing smithi13779221-18 from 1875101 to 2641907 tid 2 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1075: done (4 left) 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1083: snap_create 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1077: finishing copy_from racing read to smithi13779221-19 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1077: finishing copy_from to smithi13779221-19 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 108 (ObjNum 332 snap 79 seq_num 332) dirty exists 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1080: finishing copy_from racing read to smithi13779221-45 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1080: finishing copy_from to smithi13779221-45 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 295 (ObjNum 384 snap 94 seq_num 384) dirty exists 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1081: finishing write tid 1 to smithi13779221-39 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1081: finishing write tid 2 to smithi13779221-39 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1081: finishing write tid 3 to smithi13779221-39 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1081: finishing write tid 4 to smithi13779221-39 2024-09-21T08:28:01.729 INFO:tasks.rados.rados.0.smithi137.stdout:1081: finishing write tid 5 to smithi13779221-39 2024-09-21T08:28:01.730 INFO:tasks.rados.rados.0.smithi137.stdout:1081: oid 39 updating version 0 to 223 2024-09-21T08:28:01.730 INFO:tasks.rados.rados.0.smithi137.stdout:1081: oid 39 version 223 is already newer than 222 2024-09-21T08:28:01.730 INFO:tasks.rados.rados.0.smithi137.stdout:1081: oid 39 version 223 is already newer than 220 2024-09-21T08:28:01.730 INFO:tasks.rados.rados.0.smithi137.stdout:1081: oid 39 version 223 is already newer than 221 2024-09-21T08:28:01.730 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 223 (ObjNum 405 snap 99 seq_num 405) dirty exists 2024-09-21T08:28:01.730 INFO:tasks.rados.rados.0.smithi137.stdout:1081: left oid 39 (ObjNum 405 snap 99 seq_num 405) 2024-09-21T08:28:01.730 INFO:tasks.rados.rados.0.smithi137.stdout:1082: finishing write tid 1 to smithi13779221-18 2024-09-21T08:28:01.732 INFO:tasks.rados.rados.0.smithi137.stdout:1082: finishing write tid 2 to smithi13779221-18 2024-09-21T08:28:01.732 INFO:tasks.rados.rados.0.smithi137.stdout:1082: finishing write tid 3 to smithi13779221-18 2024-09-21T08:28:01.732 INFO:tasks.rados.rados.0.smithi137.stdout:1082: finishing write tid 4 to smithi13779221-18 2024-09-21T08:28:01.732 INFO:tasks.rados.rados.0.smithi137.stdout:1082: oid 18 updating version 0 to 138 2024-09-21T08:28:01.732 INFO:tasks.rados.rados.0.smithi137.stdout:1082: oid 18 version 138 is already newer than 136 2024-09-21T08:28:01.732 INFO:tasks.rados.rados.0.smithi137.stdout:1082: oid 18 version 138 is already newer than 137 2024-09-21T08:28:01.732 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 138 (ObjNum 406 snap 99 seq_num 406) dirty exists 2024-09-21T08:28:01.732 INFO:tasks.rados.rados.0.smithi137.stdout:1082: left oid 18 (ObjNum 406 snap 99 seq_num 406) 2024-09-21T08:28:01.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:01 smithi165 ceph-mon[32893]: pgmap v335: 168 pgs: 10 peering, 11 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 145 active+clean; 112 MiB data, 411 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:01.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:01 smithi165 ceph-mon[32893]: osdmap e287: 8 total, 8 up, 4 in 2024-09-21T08:28:01.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:01 smithi165 ceph-mon[32893]: osdmap e288: 8 total, 8 up, 4 in 2024-09-21T08:28:01.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:01.828 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:01.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:01.829 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:01.855 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7] out_osds: [4, 6, 1, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:28:01.855 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:28:01.855 INFO:tasks.thrashosds.thrasher:Adding osd 1 2024-09-21T08:28:01.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd in 1 2024-09-21T08:28:01.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:01 smithi137 ceph-mon[29634]: pgmap v335: 168 pgs: 10 peering, 11 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 145 active+clean; 112 MiB data, 411 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:01.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:01 smithi137 ceph-mon[29634]: osdmap e287: 8 total, 8 up, 4 in 2024-09-21T08:28:01.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:01 smithi137 ceph-mon[29634]: osdmap e288: 8 total, 8 up, 4 in 2024-09-21T08:28:01.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:01 smithi137 ceph-mon[41044]: pgmap v335: 168 pgs: 10 peering, 11 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 145 active+clean; 112 MiB data, 411 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:01.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:01 smithi137 ceph-mon[41044]: osdmap e287: 8 total, 8 up, 4 in 2024-09-21T08:28:01.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:01 smithi137 ceph-mon[41044]: osdmap e288: 8 total, 8 up, 4 in 2024-09-21T08:28:01.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:01.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:01.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:01.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:02.050 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:02.050 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:02.053 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:02.068 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:02.068 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:02.074 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:02.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:02.075 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:02.091 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:02.091 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:02.102 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:28:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:28:02.106 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:28:02.106 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:28:02.107 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:28:02.107 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:28:02.109 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:02.110 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:02.131 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:02.131 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:02.133 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:02.148 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:02.148 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:02.170 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:02.171 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:02.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:02.187 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:02.187 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:02.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:02.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:02.198 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:02.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:02.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:02.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:02.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:02.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:02.445 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:02.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:02.446 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:02.547 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:02.568 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:02.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:02.569 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:02.647 INFO:tasks.rados.rados.0.smithi137.stdout:1077: done (4 left) 2024-09-21T08:28:02.647 INFO:tasks.rados.rados.0.smithi137.stdout:1080: done (3 left) 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:1081: done (2 left) 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:1082: done (1 left) 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:1083: done (0 left) 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:1084: rollback oid 44 current snap is 100 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 44 to 99 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:1085: read oid 19 snap 99 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:1085: expect (ObjNum 332 snap 79 seq_num 332) 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:1086: read oid 2 snap -1 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:1086: expect (ObjNum 302 snap 70 seq_num 302) 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:1087: rollback oid 14 current snap is 100 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 14 to 99 2024-09-21T08:28:02.648 INFO:tasks.rados.rados.0.smithi137.stdout:1088: snap_create 2024-09-21T08:28:02.654 INFO:tasks.rados.rados.0.smithi137.stdout:1084: finishing rollback tid 0 to smithi13779221-44 2024-09-21T08:28:02.655 INFO:tasks.rados.rados.0.smithi137.stdout:1084: finishing rollback tid 1 to smithi13779221-44 2024-09-21T08:28:02.655 INFO:tasks.rados.rados.0.smithi137.stdout:1084: finishing rollback tid 2 to smithi13779221-44 2024-09-21T08:28:02.655 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 270 (ObjNum 400 snap 97 seq_num 400) dirty exists 2024-09-21T08:28:02.659 INFO:tasks.rados.rados.0.smithi137.stdout:1085: expect (ObjNum 332 snap 79 seq_num 332) 2024-09-21T08:28:02.670 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:02.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:02.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:02.691 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:02.729 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:02 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1751564878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:02.729 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:02 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1751564878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:02 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1751564878' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:02.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:28:02.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:02.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:02.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:02.815 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:02.825 INFO:tasks.rados.rados.0.smithi137.stdout:1086: expect (ObjNum 302 snap 70 seq_num 302) 2024-09-21T08:28:02.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T08:28:02.916 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:02.937 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:02.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:02.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:02.994 INFO:tasks.rados.rados.0.smithi137.stdout:1087: finishing rollback tid 0 to smithi13779221-14 2024-09-21T08:28:02.995 INFO:tasks.rados.rados.0.smithi137.stdout:1087: finishing rollback tid 1 to smithi13779221-14 2024-09-21T08:28:02.995 INFO:tasks.rados.rados.0.smithi137.stdout:1087: finishing rollback tid 2 to smithi13779221-14 2024-09-21T08:28:02.995 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 252 (ObjNum 395 snap 97 seq_num 395) dirty exists 2024-09-21T08:28:03.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:03.060 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:03.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:03.061 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:03.109 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:03.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:03.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:03.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:03.185 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:03.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29611]: 2024-09-21T08:28:03.008+0000 7f1058bb7640 -1 mon.a@0(leader).osd e289 definitely_dead 0 2024-09-21T08:28:03.318 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:03.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:03.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:03.343 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:03.361 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:03.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:03.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:03.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:03.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:03.568 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:03.580 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:28:03.580 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:28:03.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:03.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:03.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:03.659 INFO:teuthology.orchestra.run.smithi137.stderr:marked in osd.1. 2024-09-21T08:28:03.663 INFO:tasks.rados.rados.0.smithi137.stdout:1084: done (4 left) 2024-09-21T08:28:03.664 INFO:tasks.rados.rados.0.smithi137.stdout:1085: done (3 left) 2024-09-21T08:28:03.664 INFO:tasks.rados.rados.0.smithi137.stdout:1086: done (2 left) 2024-09-21T08:28:03.664 INFO:tasks.rados.rados.0.smithi137.stdout:1087: done (1 left) 2024-09-21T08:28:03.664 INFO:tasks.rados.rados.0.smithi137.stdout:1088: done (0 left) 2024-09-21T08:28:03.664 INFO:tasks.rados.rados.0.smithi137.stdout:1089: read oid 28 snap -1 2024-09-21T08:28:03.664 INFO:tasks.rados.rados.0.smithi137.stdout:1089: expect (ObjNum 387 snap 94 seq_num 387) 2024-09-21T08:28:03.664 INFO:tasks.rados.rados.0.smithi137.stdout:1090: delete oid 35 current snap is 101 2024-09-21T08:28:03.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:03.722 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:03.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:03.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:03 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:03 smithi165 ceph-mon[32893]: pgmap v338: 168 pgs: 11 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 112 MiB data, 398 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:03 smithi165 ceph-mon[32893]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-21T08:28:03.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:03 smithi165 ceph-mon[32893]: osdmap e289: 8 total, 8 up, 4 in 2024-09-21T08:28:03.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:03 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:28:03.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:03 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:28:03.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:03 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/426135059' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-21T08:28:03.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:03 smithi165 ceph-mon[32893]: from='client.45329 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[29634]: pgmap v338: 168 pgs: 11 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 112 MiB data, 398 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[29634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[29634]: osdmap e289: 8 total, 8 up, 4 in 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/426135059' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[29634]: from='client.45329 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[41044]: pgmap v338: 168 pgs: 11 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 112 MiB data, 398 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[41044]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 4 pgs inactive, 5 pgs peering) 2024-09-21T08:28:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[41044]: osdmap e289: 8 total, 8 up, 4 in 2024-09-21T08:28:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:28:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:28:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/426135059' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-21T08:28:03.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:03 smithi137 ceph-mon[41044]: from='client.45329 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["1"]}]: dispatch 2024-09-21T08:28:03.824 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:03.847 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:03.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:03.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:03.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:03.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:03.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:03.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:04.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:04.093 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:04.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:04.094 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:04.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T08:28:04.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:04.216 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:04.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:04.217 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:04.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:04.339 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:04.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:04.340 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:04.438 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:04.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:04.442 INFO:tasks.thrashosds.thrasher:Added osd 1 2024-09-21T08:28:04.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:04.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:04.464 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:04.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:04.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:04.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:04.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:04.674 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:04 smithi137 ceph-mon[29634]: from='client.45329 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-21T08:28:04.674 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:04 smithi137 ceph-mon[29634]: osdmap e290: 8 total, 8 up, 5 in 2024-09-21T08:28:04.674 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:04 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1304231932' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:04.674 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:04 smithi137 ceph-mon[29634]: from='client.45335 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:04.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:04 smithi137 ceph-mon[41044]: from='client.45329 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-21T08:28:04.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:04 smithi137 ceph-mon[41044]: osdmap e290: 8 total, 8 up, 5 in 2024-09-21T08:28:04.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:04 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1304231932' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:04.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:04 smithi137 ceph-mon[41044]: from='client.45335 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:04.675 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:28:04.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:04.692 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:04 smithi165 ceph-mon[32893]: from='client.45329 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["1"]}]': finished 2024-09-21T08:28:04.692 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:04 smithi165 ceph-mon[32893]: osdmap e290: 8 total, 8 up, 5 in 2024-09-21T08:28:04.692 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:04 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1304231932' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:04.692 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:04 smithi165 ceph-mon[32893]: from='client.45335 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:04.729 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:04.729 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:04.730 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:04.830 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:04.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:04.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:04.852 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:04.938 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:28:04.938 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:28:04.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:04.974 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:04.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:04.975 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:05.076 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:05.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:05.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:05.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:05.198 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:05.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:05.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:05.221 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:05.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:28:05.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:05.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:05.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:05.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:05.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:05.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:05.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:05.466 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:05.487 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:05.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:05.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:05.588 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:05.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:05.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:05.711 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:05.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:05.712 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:05.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:05.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T08:28:05.833 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:05.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:05.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:05.923 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:05 smithi137 ceph-mon[29634]: pgmap v341: 168 pgs: 11 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 112 MiB data, 398 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:05.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:05 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:05.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:05 smithi137 ceph-mon[29634]: from='client.45335 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:05.924 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:05 smithi137 ceph-mon[29634]: osdmap e291: 8 total, 8 up, 5 in 2024-09-21T08:28:05.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:05 smithi137 ceph-mon[41044]: pgmap v341: 168 pgs: 11 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 112 MiB data, 398 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:05.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:05 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:05.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:05 smithi137 ceph-mon[41044]: from='client.45335 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:05.924 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:05 smithi137 ceph-mon[41044]: osdmap e291: 8 total, 8 up, 5 in 2024-09-21T08:28:05.935 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:05.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:05.957 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:05.957 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:05 smithi165 ceph-mon[32893]: pgmap v341: 168 pgs: 11 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 154 active+clean; 112 MiB data, 398 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:05 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:05 smithi165 ceph-mon[32893]: from='client.45335 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:05 smithi165 ceph-mon[32893]: osdmap e291: 8 total, 8 up, 5 in 2024-09-21T08:28:06.031 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:06.032 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:06.038 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:06.039 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:06.054 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:06.054 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:06.058 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:06.062 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:06.070 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:06.070 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:06.080 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:06.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:06.081 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:06.086 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:06.086 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:06.103 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:06.103 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:06.119 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:06.119 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:06.135 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:06.135 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:06.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:06.203 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:06.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:06.204 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:06.305 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:06.322 INFO:tasks.rados.rados.0.smithi137.stdout:1089: expect (ObjNum 387 snap 94 seq_num 387) 2024-09-21T08:28:06.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:06.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:06.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:06.422 INFO:tasks.rados.rados.0.smithi137.stdout:1089: done (1 left) 2024-09-21T08:28:06.422 INFO:tasks.rados.rados.0.smithi137.stdout:1090: done (0 left) 2024-09-21T08:28:06.423 INFO:tasks.rados.rados.0.smithi137.stdout:1091: rollback oid 32 current snap is 101 2024-09-21T08:28:06.423 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 32 to 99 2024-09-21T08:28:06.423 INFO:tasks.rados.rados.0.smithi137.stdout:1092: delete oid 39 current snap is 101 2024-09-21T08:28:06.424 INFO:tasks.rados.rados.0.smithi137.stdout:1092: done (1 left) 2024-09-21T08:28:06.424 INFO:tasks.rados.rados.0.smithi137.stdout:1093: write oid 42 current snap is 101 2024-09-21T08:28:06.424 INFO:tasks.rados.rados.0.smithi137.stdout:1093: seq_num 409 ranges {474063=543550,1650547=514133,2765651=108412} 2024-09-21T08:28:06.430 INFO:tasks.rados.rados.0.smithi137.stdout:1093: writing smithi13779221-42 from 474063 to 1017613 tid 1 2024-09-21T08:28:06.436 INFO:tasks.rados.rados.0.smithi137.stdout:1093: writing smithi13779221-42 from 1650547 to 2164680 tid 2 2024-09-21T08:28:06.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:06.442 INFO:tasks.rados.rados.0.smithi137.stdout:1093: writing smithi13779221-42 from 2765651 to 2874063 tid 3 2024-09-21T08:28:06.442 INFO:tasks.rados.rados.0.smithi137.stdout:1094: read oid 43 snap 99 2024-09-21T08:28:06.442 INFO:tasks.rados.rados.0.smithi137.stdout:1094: expect deleted 2024-09-21T08:28:06.442 INFO:tasks.rados.rados.0.smithi137.stdout:1095: copy_from oid 25 from oid 34 current snap is 101 2024-09-21T08:28:06.442 INFO:tasks.rados.rados.0.smithi137.stdout:1091: finishing rollback tid 0 to smithi13779221-32 2024-09-21T08:28:06.442 INFO:tasks.rados.rados.0.smithi137.stdout:1091: finishing rollback tid 1 to smithi13779221-32 2024-09-21T08:28:06.442 INFO:tasks.rados.rados.0.smithi137.stdout:1091: finishing rollback tid 2 to smithi13779221-32 2024-09-21T08:28:06.442 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 221 (ObjNum 378 snap 93 seq_num 378) dirty exists 2024-09-21T08:28:06.443 INFO:tasks.rados.rados.0.smithi137.stdout:1093: finishing write tid 1 to smithi13779221-42 2024-09-21T08:28:06.443 INFO:tasks.rados.rados.0.smithi137.stdout:1093: finishing write tid 2 to smithi13779221-42 2024-09-21T08:28:06.443 INFO:tasks.rados.rados.0.smithi137.stdout:1091: done (3 left) 2024-09-21T08:28:06.443 INFO:tasks.rados.rados.0.smithi137.stdout:1096: rollback oid 16 current snap is 101 2024-09-21T08:28:06.443 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 16 to 99 2024-09-21T08:28:06.443 INFO:tasks.rados.rados.0.smithi137.stdout:1097: snap_create 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:1093: finishing write tid 3 to smithi13779221-42 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:1096: finishing rollback tid 0 to smithi13779221-16 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:1093: finishing write tid 4 to smithi13779221-42 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:1096: finishing rollback tid 1 to smithi13779221-16 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:1093: finishing write tid 5 to smithi13779221-42 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:1093: oid 42 updating version 0 to 273 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:1093: oid 42 updating version 273 to 274 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:1093: oid 42 updating version 274 to 275 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:1093: oid 42 updating version 275 to 276 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 276 (ObjNum 409 snap 101 seq_num 409) dirty exists 2024-09-21T08:28:06.444 INFO:tasks.rados.rados.0.smithi137.stdout:1093: left oid 42 (ObjNum 409 snap 101 seq_num 409) 2024-09-21T08:28:06.445 INFO:tasks.rados.rados.0.smithi137.stdout:1096: finishing rollback tid 2 to smithi13779221-16 2024-09-21T08:28:06.445 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 321 (ObjNum 360 snap 85 seq_num 360) dirty exists 2024-09-21T08:28:06.450 INFO:tasks.rados.rados.0.smithi137.stdout:1095: finishing copy_from racing read to smithi13779221-25 2024-09-21T08:28:06.453 INFO:tasks.rados.rados.0.smithi137.stdout:1095: finishing copy_from to smithi13779221-25 2024-09-21T08:28:06.453 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 277 (ObjNum 266 snap 59 seq_num 266) dirty exists 2024-09-21T08:28:06.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:06.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:06.540 INFO:tasks.rados.rados.0.smithi137.stdout:1093: done (4 left) 2024-09-21T08:28:06.540 INFO:tasks.rados.rados.0.smithi137.stdout:1094: done (3 left) 2024-09-21T08:28:06.540 INFO:tasks.rados.rados.0.smithi137.stdout:1095: done (2 left) 2024-09-21T08:28:06.540 INFO:tasks.rados.rados.0.smithi137.stdout:1096: done (1 left) 2024-09-21T08:28:06.540 INFO:tasks.rados.rados.0.smithi137.stdout:1097: done (0 left) 2024-09-21T08:28:06.540 INFO:tasks.rados.rados.0.smithi137.stdout:1098: snap_create 2024-09-21T08:28:06.554 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:28:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:28:06.554 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:28:06.554 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:28:06.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:06.584 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:06.584 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:06.585 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:06.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:06.707 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:06.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:06.708 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:06.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:06 smithi165 ceph-mon[32893]: osdmap e292: 8 total, 8 up, 5 in 2024-09-21T08:28:06.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:06 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2981088314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:06.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:06 smithi165 ceph-mon[32893]: pgmap v344: 168 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 9 activating+undersized+degraded+remapped, 1 active+clean+snaptrim, 154 active+clean; 107 MiB data, 436 MiB used, 447 GiB / 447 GiB avail; 15 MiB/s rd, 5.8 MiB/s wr, 33 op/s; 149/747 objects degraded (19.946%); 66/747 objects misplaced (8.835%); 5.7 MiB/s, 29 objects/s recovering 2024-09-21T08:28:06.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:06 smithi165 ceph-mon[32893]: osdmap e293: 8 total, 8 up, 5 in 2024-09-21T08:28:06.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:06 smithi137 ceph-mon[41044]: osdmap e292: 8 total, 8 up, 5 in 2024-09-21T08:28:06.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:06 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2981088314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:06.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:06 smithi137 ceph-mon[41044]: pgmap v344: 168 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 9 activating+undersized+degraded+remapped, 1 active+clean+snaptrim, 154 active+clean; 107 MiB data, 436 MiB used, 447 GiB / 447 GiB avail; 15 MiB/s rd, 5.8 MiB/s wr, 33 op/s; 149/747 objects degraded (19.946%); 66/747 objects misplaced (8.835%); 5.7 MiB/s, 29 objects/s recovering 2024-09-21T08:28:06.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:06 smithi137 ceph-mon[41044]: osdmap e293: 8 total, 8 up, 5 in 2024-09-21T08:28:06.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:06 smithi137 ceph-mon[29634]: osdmap e292: 8 total, 8 up, 5 in 2024-09-21T08:28:06.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:06 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2981088314' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:06.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:06 smithi137 ceph-mon[29634]: pgmap v344: 168 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 9 activating+undersized+degraded+remapped, 1 active+clean+snaptrim, 154 active+clean; 107 MiB data, 436 MiB used, 447 GiB / 447 GiB avail; 15 MiB/s rd, 5.8 MiB/s wr, 33 op/s; 149/747 objects degraded (19.946%); 66/747 objects misplaced (8.835%); 5.7 MiB/s, 29 objects/s recovering 2024-09-21T08:28:06.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:06 smithi137 ceph-mon[29634]: osdmap e293: 8 total, 8 up, 5 in 2024-09-21T08:28:06.809 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:06.830 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:06.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:06.831 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:06.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:06.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:06.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:06.954 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:07.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:07.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:07.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:07.076 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:07.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:07.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:07.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:07.201 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:07.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:28:07.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:07.326 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:07.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:07.327 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:07.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T08:28:07.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:07.449 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:07.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:07.449 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:07.541 INFO:tasks.rados.rados.0.smithi137.stdout:1098: done (0 left) 2024-09-21T08:28:07.541 INFO:tasks.rados.rados.0.smithi137.stdout:1099: rollback oid 43 current snap is 103 2024-09-21T08:28:07.541 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 43 to 101 2024-09-21T08:28:07.541 INFO:tasks.rados.rados.0.smithi137.stdout:1100: delete oid 9 current snap is 103 2024-09-21T08:28:07.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:07.553 INFO:tasks.rados.rados.0.smithi137.stdout:1100: done (1 left) 2024-09-21T08:28:07.553 INFO:tasks.rados.rados.0.smithi137.stdout:1101: write (excl) oid 20 current snap is 103 2024-09-21T08:28:07.553 INFO:tasks.rados.rados.0.smithi137.stdout:1101: seq_num 411 ranges {507677=410401,1367511=517802,2420497=487180} 2024-09-21T08:28:07.554 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:07.558 INFO:tasks.rados.rados.0.smithi137.stdout:1101: writing smithi13779221-20 from 507677 to 918078 tid 1 2024-09-21T08:28:07.563 INFO:tasks.rados.rados.0.smithi137.stdout:1101: writing smithi13779221-20 from 1367511 to 1885313 tid 2 2024-09-21T08:28:07.570 INFO:tasks.rados.rados.0.smithi137.stdout:1101: writing smithi13779221-20 from 2420497 to 2907677 tid 3 2024-09-21T08:28:07.570 INFO:tasks.rados.rados.0.smithi137.stdout:1102: read oid 4 snap 102 2024-09-21T08:28:07.570 INFO:tasks.rados.rados.0.smithi137.stdout:1102: expect deleted 2024-09-21T08:28:07.570 INFO:tasks.rados.rados.0.smithi137.stdout:1103: read oid 3 snap 99 2024-09-21T08:28:07.570 INFO:tasks.rados.rados.0.smithi137.stdout:1103: expect deleted 2024-09-21T08:28:07.570 INFO:tasks.rados.rados.0.smithi137.stdout:1104: write oid 23 current snap is 103 2024-09-21T08:28:07.570 INFO:tasks.rados.rados.0.smithi137.stdout:1104: seq_num 412 ranges {524484=485650,1425993=661460,2847920=76564} 2024-09-21T08:28:07.575 INFO:tasks.rados.rados.0.smithi137.stdout:1104: writing smithi13779221-23 from 524484 to 1010134 tid 1 2024-09-21T08:28:07.578 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:07.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:07.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:07.581 INFO:tasks.rados.rados.0.smithi137.stdout:1104: writing smithi13779221-23 from 1425993 to 2087453 tid 2 2024-09-21T08:28:07.586 INFO:tasks.rados.rados.0.smithi137.stdout:1104: writing smithi13779221-23 from 2847920 to 2924484 tid 3 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1105: delete oid 37 current snap is 103 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1099: finishing rollback tid 1 to smithi13779221-43 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 0 (ObjNum 177 snap 0 seq_num 3761041584) dirty dne 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1101: finishing write tid 1 to smithi13779221-20 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1101: finishing write tid 2 to smithi13779221-20 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1101: finishing write tid 3 to smithi13779221-20 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1101: finishing write tid 4 to smithi13779221-20 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1101: finishing write tid 5 to smithi13779221-20 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1101: oid 20 updating version 0 to 364 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1101: oid 20 version 364 is already newer than 363 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1101: oid 20 version 364 is already newer than 362 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1101: oid 20 version 364 is already newer than 361 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 364 (ObjNum 411 snap 103 seq_num 411) dirty exists 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1101: left oid 20 (ObjNum 411 snap 103 seq_num 411) 2024-09-21T08:28:07.587 INFO:tasks.rados.rados.0.smithi137.stdout:1104: finishing write tid 1 to smithi13779221-23 2024-09-21T08:28:07.588 INFO:tasks.rados.rados.0.smithi137.stdout:1104: finishing write tid 2 to smithi13779221-23 2024-09-21T08:28:07.588 INFO:tasks.rados.rados.0.smithi137.stdout:1104: finishing write tid 3 to smithi13779221-23 2024-09-21T08:28:07.588 INFO:tasks.rados.rados.0.smithi137.stdout:1099: done (5 left) 2024-09-21T08:28:07.588 INFO:tasks.rados.rados.0.smithi137.stdout:1101: done (4 left) 2024-09-21T08:28:07.588 INFO:tasks.rados.rados.0.smithi137.stdout:1102: done (3 left) 2024-09-21T08:28:07.588 INFO:tasks.rados.rados.0.smithi137.stdout:1103: done (2 left) 2024-09-21T08:28:07.589 INFO:tasks.rados.rados.0.smithi137.stdout:1105: done (1 left) 2024-09-21T08:28:07.589 INFO:tasks.rados.rados.0.smithi137.stdout:1106: read oid 8 snap -1 2024-09-21T08:28:07.589 INFO:tasks.rados.rados.0.smithi137.stdout:1106: expect (ObjNum 404 snap 99 seq_num 404) 2024-09-21T08:28:07.589 INFO:tasks.rados.rados.0.smithi137.stdout:1107: write (excl) oid 24 current snap is 103 2024-09-21T08:28:07.589 INFO:tasks.rados.rados.0.smithi137.stdout:1107: seq_num 414 ranges {558098=636148,1826604=665129} 2024-09-21T08:28:07.595 INFO:tasks.rados.rados.0.smithi137.stdout:1107: writing smithi13779221-24 from 558098 to 1194246 tid 1 2024-09-21T08:28:07.602 INFO:tasks.rados.rados.0.smithi137.stdout:1107: writing smithi13779221-24 from 1826604 to 2491733 tid 2 2024-09-21T08:28:07.602 INFO:tasks.rados.rados.0.smithi137.stdout:1108: read oid 10 snap -1 2024-09-21T08:28:07.603 INFO:tasks.rados.rados.0.smithi137.stdout:1108: expect (ObjNum 384 snap 94 seq_num 384) 2024-09-21T08:28:07.603 INFO:tasks.rados.rados.0.smithi137.stdout:1109: write (excl) oid 17 current snap is 103 2024-09-21T08:28:07.603 INFO:tasks.rados.rados.0.smithi137.stdout:1104: finishing write tid 4 to smithi13779221-23 2024-09-21T08:28:07.603 INFO:tasks.rados.rados.0.smithi137.stdout:1109: seq_num 415 ranges {574905=711397,1885086=525140,2930189=44716} 2024-09-21T08:28:07.609 INFO:tasks.rados.rados.0.smithi137.stdout:1109: writing smithi13779221-17 from 574905 to 1286302 tid 1 2024-09-21T08:28:07.615 INFO:tasks.rados.rados.0.smithi137.stdout:1109: writing smithi13779221-17 from 1885086 to 2410226 tid 2 2024-09-21T08:28:07.619 INFO:tasks.rados.rados.0.smithi137.stdout:1109: writing smithi13779221-17 from 2930189 to 2974905 tid 3 2024-09-21T08:28:07.619 INFO:tasks.rados.rados.0.smithi137.stdout:1110: write (excl) oid 19 current snap is 103 2024-09-21T08:28:07.619 INFO:tasks.rados.rados.0.smithi137.stdout:1104: finishing write tid 5 to smithi13779221-23 2024-09-21T08:28:07.619 INFO:tasks.rados.rados.0.smithi137.stdout:1104: oid 23 updating version 0 to 181 2024-09-21T08:28:07.619 INFO:tasks.rados.rados.0.smithi137.stdout:1104: oid 23 updating version 181 to 182 2024-09-21T08:28:07.619 INFO:tasks.rados.rados.0.smithi137.stdout:1104: oid 23 updating version 182 to 184 2024-09-21T08:28:07.619 INFO:tasks.rados.rados.0.smithi137.stdout:1104: oid 23 version 184 is already newer than 183 2024-09-21T08:28:07.619 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 184 (ObjNum 412 snap 103 seq_num 412) dirty exists 2024-09-21T08:28:07.620 INFO:tasks.rados.rados.0.smithi137.stdout:1104: left oid 23 (ObjNum 412 snap 103 seq_num 412) 2024-09-21T08:28:07.620 INFO:tasks.rados.rados.0.smithi137.stdout:1106: expect (ObjNum 404 snap 99 seq_num 404) 2024-09-21T08:28:07.635 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:07.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:07.701 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:07.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:07.702 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:07.798 INFO:tasks.rados.rados.0.smithi137.stdout:1107: finishing write tid 1 to smithi13779221-24 2024-09-21T08:28:07.798 INFO:tasks.rados.rados.0.smithi137.stdout:1110: seq_num 416 ranges {591712=786646,1943568=668798} 2024-09-21T08:28:07.802 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:07.805 INFO:tasks.rados.rados.0.smithi137.stdout:1110: writing smithi13779221-19 from 591712 to 1378358 tid 1 2024-09-21T08:28:07.812 INFO:tasks.rados.rados.0.smithi137.stdout:1110: writing smithi13779221-19 from 1943568 to 2612366 tid 2 2024-09-21T08:28:07.812 INFO:tasks.rados.rados.0.smithi137.stdout:1104: done (5 left) 2024-09-21T08:28:07.812 INFO:tasks.rados.rados.0.smithi137.stdout:1106: done (4 left) 2024-09-21T08:28:07.812 INFO:tasks.rados.rados.0.smithi137.stdout:1111: write (excl) oid 43 current snap is 103 2024-09-21T08:28:07.812 INFO:tasks.rados.rados.0.smithi137.stdout:1107: finishing write tid 2 to smithi13779221-24 2024-09-21T08:28:07.812 INFO:tasks.rados.rados.0.smithi137.stdout:1107: finishing write tid 3 to smithi13779221-24 2024-09-21T08:28:07.812 INFO:tasks.rados.rados.0.smithi137.stdout:1111: seq_num 417 ranges {608519=461895,1602050=528809,2585035=423484} 2024-09-21T08:28:07.817 INFO:tasks.rados.rados.0.smithi137.stdout:1111: writing smithi13779221-43 from 608519 to 1070414 tid 1 2024-09-21T08:28:07.823 INFO:tasks.rados.rados.0.smithi137.stdout:1111: writing smithi13779221-43 from 1602050 to 2130859 tid 2 2024-09-21T08:28:07.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:07.824 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:07.825 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:07.829 INFO:tasks.rados.rados.0.smithi137.stdout:1111: writing smithi13779221-43 from 2585035 to 3008519 tid 3 2024-09-21T08:28:07.829 INFO:tasks.rados.rados.0.smithi137.stdout:1112: copy_from oid 16 from oid 48 current snap is 103 2024-09-21T08:28:07.829 INFO:tasks.rados.rados.0.smithi137.stdout:1107: finishing write tid 4 to smithi13779221-24 2024-09-21T08:28:07.829 INFO:tasks.rados.rados.0.smithi137.stdout:1107: oid 24 updating version 0 to 277 2024-09-21T08:28:07.829 INFO:tasks.rados.rados.0.smithi137.stdout:1107: oid 24 updating version 277 to 278 2024-09-21T08:28:07.829 INFO:tasks.rados.rados.0.smithi137.stdout:1107: oid 24 version 278 is already newer than 276 2024-09-21T08:28:07.829 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 278 (ObjNum 414 snap 103 seq_num 414) dirty exists 2024-09-21T08:28:07.829 INFO:tasks.rados.rados.0.smithi137.stdout:1107: left oid 24 (ObjNum 414 snap 103 seq_num 414) 2024-09-21T08:28:07.829 INFO:tasks.rados.rados.0.smithi137.stdout:1108: expect (ObjNum 384 snap 94 seq_num 384) 2024-09-21T08:28:07.917 INFO:tasks.rados.rados.0.smithi137.stdout:1109: finishing write tid 1 to smithi13779221-17 2024-09-21T08:28:07.917 INFO:tasks.rados.rados.0.smithi137.stdout:1109: finishing write tid 2 to smithi13779221-17 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1109: finishing write tid 3 to smithi13779221-17 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1109: finishing write tid 4 to smithi13779221-17 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1109: finishing write tid 5 to smithi13779221-17 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1109: oid 17 updating version 0 to 283 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1109: oid 17 version 283 is already newer than 281 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1109: oid 17 version 283 is already newer than 280 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1109: oid 17 version 283 is already newer than 282 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 283 (ObjNum 415 snap 103 seq_num 415) dirty exists 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1109: left oid 17 (ObjNum 415 snap 103 seq_num 415) 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1107: done (5 left) 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1108: done (4 left) 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1109: done (3 left) 2024-09-21T08:28:07.918 INFO:tasks.rados.rados.0.smithi137.stdout:1113: read oid 6 snap -1 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1113: expect (ObjNum 312 snap 70 seq_num 312) 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1114: copy_from oid 9 from oid 35 current snap is 103 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1110: finishing write tid 1 to smithi13779221-19 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1110: finishing write tid 2 to smithi13779221-19 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1110: finishing write tid 3 to smithi13779221-19 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1110: finishing write tid 4 to smithi13779221-19 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1110: oid 19 updating version 0 to 111 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1110: oid 19 version 111 is already newer than 109 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1110: oid 19 version 111 is already newer than 110 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 111 (ObjNum 416 snap 103 seq_num 416) dirty exists 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1110: left oid 19 (ObjNum 416 snap 103 seq_num 416) 2024-09-21T08:28:07.919 INFO:tasks.rados.rados.0.smithi137.stdout:1111: finishing write tid 1 to smithi13779221-43 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1110: done (4 left) 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1115: write oid 10 current snap is 103 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1111: finishing write tid 2 to smithi13779221-43 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1111: finishing write tid 3 to smithi13779221-43 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1111: finishing write tid 4 to smithi13779221-43 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1111: finishing write tid 5 to smithi13779221-43 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1111: oid 43 updating version 0 to 302 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1111: oid 43 version 302 is already newer than 300 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1111: oid 43 version 302 is already newer than 301 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1111: oid 43 updating version 302 to 303 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 303 (ObjNum 417 snap 103 seq_num 417) dirty exists 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1111: left oid 43 (ObjNum 417 snap 103 seq_num 417) 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1112: finishing copy_from racing read to smithi13779221-16 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:1112: finishing copy_from to smithi13779221-16 2024-09-21T08:28:07.920 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 298 (ObjNum 398 snap 97 seq_num 398) dirty exists 2024-09-21T08:28:07.921 INFO:tasks.rados.rados.0.smithi137.stdout:1115: seq_num 418 ranges {625326=537144,1944179=672467} 2024-09-21T08:28:07.923 INFO:tasks.rados.rados.0.smithi137.stdout:1115: writing smithi13779221-10 from 625326 to 1162470 tid 1 2024-09-21T08:28:07.926 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:07.931 INFO:tasks.rados.rados.0.smithi137.stdout:1115: writing smithi13779221-10 from 1944179 to 2616646 tid 2 2024-09-21T08:28:07.932 INFO:tasks.rados.rados.0.smithi137.stdout:1111: done (4 left) 2024-09-21T08:28:07.932 INFO:tasks.rados.rados.0.smithi137.stdout:1112: done (3 left) 2024-09-21T08:28:07.932 INFO:tasks.rados.rados.0.smithi137.stdout:1116: rollback oid 7 current snap is 103 2024-09-21T08:28:07.932 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 7 to 100 2024-09-21T08:28:07.932 INFO:tasks.rados.rados.0.smithi137.stdout:1114: finishing copy_from to smithi13779221-9 2024-09-21T08:28:07.932 INFO:tasks.rados.rados.0.smithi137.stdout:1114: got expected ENOENT (src dne) 2024-09-21T08:28:07.932 INFO:tasks.rados.rados.0.smithi137.stdout:1114: finishing copy_from racing read to smithi13779221-9 2024-09-21T08:28:07.932 INFO:tasks.rados.rados.0.smithi137.stdout:1115: finishing write tid 1 to smithi13779221-10 2024-09-21T08:28:07.932 INFO:tasks.rados.rados.0.smithi137.stdout:1113: expect (ObjNum 312 snap 70 seq_num 312) 2024-09-21T08:28:07.947 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:07.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:07.947 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:07 smithi165 ceph-mon[32893]: Health check failed: Degraded data redundancy: 149/747 objects degraded (19.946%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T08:28:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:07 smithi165 ceph-mon[32893]: osdmap e294: 8 total, 8 up, 5 in 2024-09-21T08:28:08.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:07 smithi137 ceph-mon[29634]: Health check failed: Degraded data redundancy: 149/747 objects degraded (19.946%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T08:28:08.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:07 smithi137 ceph-mon[29634]: osdmap e294: 8 total, 8 up, 5 in 2024-09-21T08:28:08.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:07 smithi137 ceph-mon[41044]: Health check failed: Degraded data redundancy: 149/747 objects degraded (19.946%), 9 pgs degraded (PG_DEGRADED) 2024-09-21T08:28:08.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:07 smithi137 ceph-mon[41044]: osdmap e294: 8 total, 8 up, 5 in 2024-09-21T08:28:08.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:08.069 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:08.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:08.070 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:08.120 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:28:08.120 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:28:08.120 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:28:08.120 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:28:08.120 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:28:08.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:08.180 INFO:tasks.rados.rados.0.smithi137.stdout:1116: finishing rollback tid 1 to smithi13779221-7 2024-09-21T08:28:08.180 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 0 (ObjNum 305 snap 0 seq_num 379559424) dirty dne 2024-09-21T08:28:08.181 INFO:tasks.rados.rados.0.smithi137.stdout:1115: finishing write tid 2 to smithi13779221-10 2024-09-21T08:28:08.181 INFO:tasks.rados.rados.0.smithi137.stdout:1115: finishing write tid 3 to smithi13779221-10 2024-09-21T08:28:08.181 INFO:tasks.rados.rados.0.smithi137.stdout:1113: done (3 left) 2024-09-21T08:28:08.181 INFO:tasks.rados.rados.0.smithi137.stdout:1114: done (2 left) 2024-09-21T08:28:08.181 INFO:tasks.rados.rados.0.smithi137.stdout:1116: done (1 left) 2024-09-21T08:28:08.181 INFO:tasks.rados.rados.0.smithi137.stdout:1117: write oid 5 current snap is 103 2024-09-21T08:28:08.181 INFO:tasks.rados.rados.0.smithi137.stdout:1115: finishing write tid 4 to smithi13779221-10 2024-09-21T08:28:08.181 INFO:tasks.rados.rados.0.smithi137.stdout:1115: oid 10 updating version 0 to 301 2024-09-21T08:28:08.181 INFO:tasks.rados.rados.0.smithi137.stdout:1115: oid 10 version 301 is already newer than 299 2024-09-21T08:28:08.181 INFO:tasks.rados.rados.0.smithi137.stdout:1115: oid 10 version 301 is already newer than 300 2024-09-21T08:28:08.182 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 301 (ObjNum 418 snap 103 seq_num 418) dirty exists 2024-09-21T08:28:08.182 INFO:tasks.rados.rados.0.smithi137.stdout:1115: left oid 10 (ObjNum 418 snap 103 seq_num 418) 2024-09-21T08:28:08.182 INFO:tasks.rados.rados.0.smithi137.stdout:1117: seq_num 419 ranges {642133=728746,2119014=532478} 2024-09-21T08:28:08.187 INFO:tasks.rados.rados.0.smithi137.stdout:1117: writing smithi13779221-5 from 642133 to 1370879 tid 1 2024-09-21T08:28:08.192 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:08.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:08.193 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:08.195 INFO:tasks.rados.rados.0.smithi137.stdout:1117: writing smithi13779221-5 from 2119014 to 2651492 tid 2 2024-09-21T08:28:08.195 INFO:tasks.rados.rados.0.smithi137.stdout:1115: done (1 left) 2024-09-21T08:28:08.195 INFO:tasks.rados.rados.0.smithi137.stdout:1118: copy_from oid 35 from oid 30 current snap is 103 2024-09-21T08:28:08.195 INFO:tasks.rados.rados.0.smithi137.stdout:1117: finishing write tid 1 to smithi13779221-5 2024-09-21T08:28:08.195 INFO:tasks.rados.rados.0.smithi137.stdout:1119: write oid 3 current snap is 103 2024-09-21T08:28:08.195 INFO:tasks.rados.rados.0.smithi137.stdout:1119: seq_num 420 ranges {658940=403995,1777496=676136,2950951=107989} 2024-09-21T08:28:08.201 INFO:tasks.rados.rados.0.smithi137.stdout:1119: writing smithi13779221-3 from 658940 to 1062935 tid 1 2024-09-21T08:28:08.209 INFO:tasks.rados.rados.0.smithi137.stdout:1119: writing smithi13779221-3 from 1777496 to 2453632 tid 2 2024-09-21T08:28:08.212 INFO:tasks.rados.rados.0.smithi137.stdout:1119: writing smithi13779221-3 from 2950951 to 3058940 tid 3 2024-09-21T08:28:08.212 INFO:tasks.rados.rados.0.smithi137.stdout:1120: write (excl) oid 4 current snap is 103 2024-09-21T08:28:08.212 INFO:tasks.rados.rados.0.smithi137.stdout:1117: finishing write tid 2 to smithi13779221-5 2024-09-21T08:28:08.212 INFO:tasks.rados.rados.0.smithi137.stdout:1120: seq_num 421 ranges {675747=479244,1835978=536147} 2024-09-21T08:28:08.218 INFO:tasks.rados.rados.0.smithi137.stdout:1120: writing smithi13779221-4 from 675747 to 1154991 tid 1 2024-09-21T08:28:08.226 INFO:tasks.rados.rados.0.smithi137.stdout:1120: writing smithi13779221-4 from 1835978 to 2372125 tid 2 2024-09-21T08:28:08.226 INFO:tasks.rados.rados.0.smithi137.stdout:1121: read oid 15 snap 102 2024-09-21T08:28:08.226 INFO:tasks.rados.rados.0.smithi137.stdout:1121: expect deleted 2024-09-21T08:28:08.226 INFO:tasks.rados.rados.0.smithi137.stdout:1122: copy_from oid 34 from oid 21 current snap is 103 2024-09-21T08:28:08.226 INFO:tasks.rados.rados.0.smithi137.stdout:1117: finishing write tid 3 to smithi13779221-5 2024-09-21T08:28:08.226 INFO:tasks.rados.rados.0.smithi137.stdout:1117: finishing write tid 4 to smithi13779221-5 2024-09-21T08:28:08.226 INFO:tasks.rados.rados.0.smithi137.stdout:1117: oid 5 updating version 0 to 328 2024-09-21T08:28:08.226 INFO:tasks.rados.rados.0.smithi137.stdout:1117: oid 5 version 328 is already newer than 327 2024-09-21T08:28:08.226 INFO:tasks.rados.rados.0.smithi137.stdout:1117: oid 5 version 328 is already newer than 326 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 328 (ObjNum 419 snap 103 seq_num 419) dirty exists 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1117: left oid 5 (ObjNum 419 snap 103 seq_num 419) 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1118: finishing copy_from racing read to smithi13779221-35 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1118: finishing copy_from to smithi13779221-35 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1119: finishing write tid 1 to smithi13779221-3 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1119: finishing write tid 2 to smithi13779221-3 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1117: done (5 left) 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1118: done (4 left) 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1123: snap_create 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1119: finishing write tid 3 to smithi13779221-3 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1119: finishing write tid 4 to smithi13779221-3 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1119: finishing write tid 5 to smithi13779221-3 2024-09-21T08:28:08.227 INFO:tasks.rados.rados.0.smithi137.stdout:1119: oid 3 updating version 0 to 182 2024-09-21T08:28:08.228 INFO:tasks.rados.rados.0.smithi137.stdout:1119: oid 3 updating version 182 to 183 2024-09-21T08:28:08.228 INFO:tasks.rados.rados.0.smithi137.stdout:1119: oid 3 version 183 is already newer than 181 2024-09-21T08:28:08.228 INFO:tasks.rados.rados.0.smithi137.stdout:1119: oid 3 version 183 is already newer than 180 2024-09-21T08:28:08.228 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 183 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:28:08.228 INFO:tasks.rados.rados.0.smithi137.stdout:1119: left oid 3 (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:08.228 INFO:tasks.rados.rados.0.smithi137.stdout:1120: finishing write tid 1 to smithi13779221-4 2024-09-21T08:28:08.230 INFO:tasks.rados.rados.0.smithi137.stdout:1120: finishing write tid 2 to smithi13779221-4 2024-09-21T08:28:08.230 INFO:tasks.rados.rados.0.smithi137.stdout:1120: finishing write tid 3 to smithi13779221-4 2024-09-21T08:28:08.230 INFO:tasks.rados.rados.0.smithi137.stdout:1120: finishing write tid 4 to smithi13779221-4 2024-09-21T08:28:08.230 INFO:tasks.rados.rados.0.smithi137.stdout:1120: oid 4 updating version 0 to 296 2024-09-21T08:28:08.231 INFO:tasks.rados.rados.0.smithi137.stdout:1120: oid 4 updating version 296 to 297 2024-09-21T08:28:08.231 INFO:tasks.rados.rados.0.smithi137.stdout:1120: oid 4 updating version 297 to 298 2024-09-21T08:28:08.231 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 298 (ObjNum 421 snap 103 seq_num 421) dirty exists 2024-09-21T08:28:08.231 INFO:tasks.rados.rados.0.smithi137.stdout:1120: left oid 4 (ObjNum 421 snap 103 seq_num 421) 2024-09-21T08:28:08.239 INFO:tasks.rados.rados.0.smithi137.stdout:1122: finishing copy_from racing read to smithi13779221-34 2024-09-21T08:28:08.248 INFO:tasks.rados.rados.0.smithi137.stdout:1122: finishing copy_from to smithi13779221-34 2024-09-21T08:28:08.248 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 108 (ObjNum 332 snap 79 seq_num 332) dirty exists 2024-09-21T08:28:08.293 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:08.315 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:08.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:08.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:08.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:08.437 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:08.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:08.438 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:08.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:08.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:08.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:08.562 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:08.662 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:08.677 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:28:08.684 INFO:tasks.rados.rados.0.smithi137.stdout:1119: done (4 left) 2024-09-21T08:28:08.684 INFO:tasks.rados.rados.0.smithi137.stdout:1120: done (3 left) 2024-09-21T08:28:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:1121: done (2 left) 2024-09-21T08:28:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:1122: done (1 left) 2024-09-21T08:28:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:1123: done (0 left) 2024-09-21T08:28:08.685 INFO:tasks.rados.rados.0.smithi137.stdout:1124: delete oid 49 current snap is 104 2024-09-21T08:28:08.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:08.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:08.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:08.691 INFO:tasks.rados.rados.0.smithi137.stdout:1124: done (0 left) 2024-09-21T08:28:08.691 INFO:tasks.rados.rados.0.smithi137.stdout:1125: snap_create 2024-09-21T08:28:08.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:08 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:28:08.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:08 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:28:08.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:08 smithi165 ceph-mon[32893]: pgmap v347: 168 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 9 activating+undersized+degraded+remapped, 1 active+clean+snaptrim, 154 active+clean; 107 MiB data, 436 MiB used, 447 GiB / 447 GiB avail; 15 MiB/s rd, 5.8 MiB/s wr, 33 op/s; 149/747 objects degraded (19.946%); 66/747 objects misplaced (8.835%); 5.7 MiB/s, 29 objects/s recovering 2024-09-21T08:28:08.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:08 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1663534989' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:08.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:08.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:28:08.809 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:08.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:08.810 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:08.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:08.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T08:28:08.932 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:08.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:08.933 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:08.995 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:08 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:28:08.995 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:08 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:28:08.995 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:08 smithi137 ceph-mon[29634]: pgmap v347: 168 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 9 activating+undersized+degraded+remapped, 1 active+clean+snaptrim, 154 active+clean; 107 MiB data, 436 MiB used, 447 GiB / 447 GiB avail; 15 MiB/s rd, 5.8 MiB/s wr, 33 op/s; 149/747 objects degraded (19.946%); 66/747 objects misplaced (8.835%); 5.7 MiB/s, 29 objects/s recovering 2024-09-21T08:28:08.995 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:08 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1663534989' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:08.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:08 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:28:08.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:08 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:28:08.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:08 smithi137 ceph-mon[41044]: pgmap v347: 168 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 9 activating+undersized+degraded+remapped, 1 active+clean+snaptrim, 154 active+clean; 107 MiB data, 436 MiB used, 447 GiB / 447 GiB avail; 15 MiB/s rd, 5.8 MiB/s wr, 33 op/s; 149/747 objects degraded (19.946%); 66/747 objects misplaced (8.835%); 5.7 MiB/s, 29 objects/s recovering 2024-09-21T08:28:08.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:08 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1663534989' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:09.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:09.054 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:09.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:09.055 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:09.156 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:09.169 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:09.177 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:09.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:09.178 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:09.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:09.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:09.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:09.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:09.403 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:09.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:09.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:09.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:09.443 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 1] out_osds: [4, 6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:28:09.443 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:28:09.443 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T08:28:09.443 INFO:tasks.thrashosds.thrasher:Setting osd 7 primary_affinity to 0.243311 2024-09-21T08:28:09.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 7 0.24331079899676422 2024-09-21T08:28:09.479 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:09.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:09.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:09.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:09.548 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:09.599 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1083 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94837804037760 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 211.43791192800001, 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.395092834, 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.600 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1083 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:28:09.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159586+0000", 2024-09-21T08:28:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.48e-06 2024-09-21T08:28:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159599+0000", 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3699e-05 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159630+0000", 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0629000000000001e-05 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159647+0000", 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7030000000000001e-05 2024-09-21T08:28:09.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159767+0000", 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119571 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159781+0000", 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3844e-05 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.549248+0000", 2024-09-21T08:28:09.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.38946733 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.549559+0000", 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000311057 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552031+0000", 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024720720000000001 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552119+0000", 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8261000000000001e-05 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.554673+0000", 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025538610000000002 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1031 7.0 7:00f4c0d4:::smithi13779221-18:head [copy-from ver 6] snapc 22=[22,21,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T08:28:09.606 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:40.814163+0000", 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 148.78332903399999, 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012690369, 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1031 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814163+0000", 2024-09-21T08:28:09.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814163+0000", 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814165+0000", 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8679999999999999e-06 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814173+0000", 2024-09-21T08:28:09.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7030000000000006e-06 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814184+0000", 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1423999999999999e-05 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814190+0000", 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4709999999999998e-06 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814237+0000", 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7290000000000003e-05 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.814301+0000", 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3720000000000007e-05 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.818021+0000", 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003720105 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.821257+0000", 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032359979999999999 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.826736+0000", 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054788179999999999 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.826774+0000", 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8127000000000003e-05 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.826811+0000", 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7144999999999998e-05 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:40.826854+0000", 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2700000000000001e-05 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1159 7.b 7:df2b42d9:::smithi13779221-36:head [write 642067~696188,stat] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 138.70737354799999, 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62427366799999995, 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.613 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1159 2024-09-21T08:28:09.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.614 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:28:09.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890120+0000", 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6950000000000001e-06 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890849+0000", 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072891299999999998 2024-09-21T08:28:09.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890861+0000", 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1899000000000001e-05 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890868+0000", 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2110000000000001e-06 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890947+0000", 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8598999999999997e-05 2024-09-21T08:28:09.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890954+0000", 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1180000000000003e-06 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508089+0000", 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61713446299999997 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508263+0000", 2024-09-21T08:28:09.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174671 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508366+0000", 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102786 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511111+0000", 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027452710000000001 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.513304+0000", 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021930560000000001 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514317+0000", 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010129959999999999 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514355+0000", 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7404000000000003e-05 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514392+0000", 2024-09-21T08:28:09.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7586000000000002e-05 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1160 7.b 7:df2b42d9:::smithi13779221-36:head [write 2025476~459397] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 138.70092565900001, 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61820804500000004, 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.620 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1160 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:28:09.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896568+0000", 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2610000000000001e-06 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896958+0000", 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039041199999999997 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896984+0000", 2024-09-21T08:28:09.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6115e-05 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896989+0000", 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0379999999999999e-06 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.897069+0000", 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9196000000000005e-05 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.897075+0000", 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.28e-06 2024-09-21T08:28:09.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509576+0000", 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61250132700000004 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509636+0000", 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9753999999999997e-05 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509725+0000", 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9524000000000005e-05 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511333+0000", 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001607619 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.513767+0000", 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024336819999999999 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514703+0000", 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093633599999999996 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514736+0000", 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2438999999999999e-05 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514775+0000", 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9062e-05 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1161 7.b 7:df2b42d9:::smithi13779221-36:head [write 2971091~70976] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:28:09.626 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 138.698046734, 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61638660599999995, 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1161 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.627 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899446+0000", 2024-09-21T08:28:09.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0800000000000004e-07 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899537+0000", 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0414999999999994e-05 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899544+0000", 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8580000000000002e-06 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899547+0000", 2024-09-21T08:28:09.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8970000000000001e-06 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899597+0000", 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9058999999999999e-05 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899605+0000", 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3990000000000004e-06 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510251+0000", 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61064600599999996 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510324+0000", 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2961999999999997e-05 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510438+0000", 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113793 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511337+0000", 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089906399999999998 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514615+0000", 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00327853 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515770+0000", 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001154715 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515802+0000", 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1956999999999999e-05 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515832+0000", 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0142999999999999e-05 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1162 7.b 7:df2b42d9:::smithi13779221-36:head [setxattr _header (57) in=7b,truncate 3042067] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:28:09.633 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 138.697937666, 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61641711799999999, 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1162 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:28:09.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9100000000000004e-07 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.635 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899557+0000", 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3989999999999998e-06 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899561+0000", 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9349999999999996e-06 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899564+0000", 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7700000000000002e-06 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.636 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899633+0000", 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9268000000000002e-05 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899637+0000", 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9650000000000002e-06 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510758+0000", 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61112027999999996 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.637 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510819+0000", 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0866999999999998e-05 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510929+0000", 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110914 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511340+0000", 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041042100000000001 2024-09-21T08:28:09.638 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514641+0000", 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003301511 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515865+0000", 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012240319999999999 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515898+0000", 2024-09-21T08:28:09.639 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3036e-05 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515972+0000", 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3229e-05 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1163 7.b 7:df2b42d9:::smithi13779221-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 138.69790625300001, 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61652938899999998, 2024-09-21T08:28:09.640 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1163 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.641 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8599999999999998e-07 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899589+0000", 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1299999999999999e-06 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.642 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899592+0000", 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9190000000000001e-06 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899595+0000", 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9639999999999999e-06 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899654+0000", 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8958e-05 2024-09-21T08:28:09.643 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899658+0000", 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2669999999999998e-06 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511140+0000", 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61148218399999998 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.644 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511235+0000", 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4927999999999996e-05 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516004+0000", 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047691640000000002 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516060+0000", 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5726999999999997e-05 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.645 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516115+0000", 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5661999999999997e-05 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1187 7.0 7:0979c720:::smithi13779221-16:head [copy-from ver 34] snapc 26=[26,25] ondisk+write+known_if_redirected+supports_pool_eio e158)", 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:51.549803+0000", 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 138.04768914600001, 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012728847, 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.646 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1187 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549803+0000", 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549803+0000", 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.647 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549805+0000", 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2019999999999998e-06 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549811+0000", 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3229999999999997e-06 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549822+0000", 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1093e-05 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.648 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549827+0000", 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3669999999999996e-06 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549875+0000", 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7865999999999999e-05 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.549935+0000", 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0117000000000001e-05 2024-09-21T08:28:09.649 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.554466+0000", 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045311630000000004 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.558093+0000", 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036264240000000001 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.562399+0000", 2024-09-21T08:28:09.650 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043062860000000003 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.562467+0000", 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7776999999999998e-05 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.562502+0000", 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5117999999999998e-05 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.562532+0000", 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0111e-05 2024-09-21T08:28:09.651 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1214 7.0 7:00f4c0d4:::smithi13779221-18:head [copy-from ver 91] snapc 27=[25] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:54.540239+0000", 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 135.057253624, 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016093084000000001, 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1214 2024-09-21T08:28:09.652 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540239+0000", 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540239+0000", 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540241+0000", 2024-09-21T08:28:09.653 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0499999999999999e-06 2024-09-21T08:28:09.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.654 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.654 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540249+0000", 2024-09-21T08:28:09.654 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8049999999999996e-06 2024-09-21T08:28:09.654 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.654 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:09.655 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.655 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540263+0000", 2024-09-21T08:28:09.655 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3329999999999999e-05 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540268+0000", 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6470000000000003e-06 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540325+0000", 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6918e-05 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.656 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540389+0000", 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3990000000000002e-05 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.547214+0000", 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068248240000000002 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.551118+0000", 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039037669999999998 2024-09-21T08:28:09.657 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556219+0000", 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051015089999999997 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556258+0000", 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8606999999999998e-05 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.658 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556298+0000", 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0096999999999999e-05 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556332+0000", 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3540000000000001e-05 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:1744 7.a e197/154 7:53537bd3:::smithi13779221-39:head v 197'56, mlcod=193'53)", 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:26:30.598253+0000", 2024-09-21T08:28:09.659 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 98.999239610000004, 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013121868, 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.598253+0000", 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.598253+0000", 2024-09-21T08:28:09.660 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.598255+0000", 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1849999999999998e-06 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605829+0000", 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075738150000000002 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605842+0000", 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3251999999999999e-05 2024-09-21T08:28:09.661 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605849+0000", 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0700000000000001e-06 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605900+0000", 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0698000000000001e-05 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.605916+0000", 2024-09-21T08:28:09.662 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6528000000000002e-05 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.611321+0000", 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005404598 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:26:30.611374+0000", 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3721999999999998e-05 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.663 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2116 7.b 7:df2b42d9:::smithi13779221-36:head [copy-from ver 318] snapc 4a=[4a,49,48,47,45,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 68.074267227000007, 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030972873000000001, 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2116 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.664 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523228+0000", 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6469999999999998e-06 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.665 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523237+0000", 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4590000000000004e-06 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523251+0000", 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3856000000000001e-05 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523260+0000", 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.666 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523351+0000", 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0787999999999999e-05 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523359+0000", 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5860000000000005e-06 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.526688+0000", 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033287339999999999 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.667 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.526800+0000", 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111566 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.531948+0000", 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051486199999999996 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.538302+0000", 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063539729999999997 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.553780+0000", 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015477451 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554118+0000", 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000338549 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554160+0000", 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2218000000000001e-05 2024-09-21T08:28:09.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554198+0000", 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7475999999999998e-05 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2117 7.b 7:df2b42d9:::smithi13779221-36:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e228)", 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:01.523289+0000", 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 68.074203385000004, 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014200134999999999, 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.670 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2117 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523289+0000", 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523289+0000", 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523290+0000", 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9900000000000009e-07 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523299+0000", 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4180000000000003e-06 2024-09-21T08:28:09.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523309+0000", 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6530000000000006e-06 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523316+0000", 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1389999999999998e-06 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523446+0000", 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000130204 2024-09-21T08:28:09.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523453+0000", 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6499999999999999e-06 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.527019+0000", 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035662649999999999 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.527078+0000", 2024-09-21T08:28:09.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8684000000000003e-05 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.537333+0000", 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010255597999999999 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.537408+0000", 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4251000000000004e-05 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.537489+0000", 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1273999999999995e-05 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:2259 7.18 e239/186 7:1a1aa5cd:::smithi13779221-31:head v 239'172, mlcod=187'168)", 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:12.807624+0000", 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.789867749000003, 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013355837000000001, 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:12.807624+0000", 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:12.807624+0000", 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:12.807625+0000", 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7100000000000001e-07 2024-09-21T08:28:09.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:12.813344+0000", 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057191760000000003 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:12.813370+0000", 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6180999999999999e-05 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:12.813385+0000", 2024-09-21T08:28:09.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4781e-05 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:12.813487+0000", 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000101859 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:12.813520+0000", 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2548000000000001e-05 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:12.820944+0000", 2024-09-21T08:28:09.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074240340000000004 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:12.820980+0000", 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6587000000000001e-05 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2314 7.b 7:df2b42d9:::smithi13779221-36:head [copy-from ver 279] snapc 4f=[4f,4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:13.531460+0000", 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.066032407999998, 2024-09-21T08:28:09.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013677514999999999, 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2314 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.531460+0000", 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.531460+0000", 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.531463+0000", 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3569999999999998e-06 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.531474+0000", 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0511e-05 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.531491+0000", 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6901999999999998e-05 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.531500+0000", 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5729999999999997e-06 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.531564+0000", 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3534000000000005e-05 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.531645+0000", 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1145999999999999e-05 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.535379+0000", 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003734561 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.539842+0000", 2024-09-21T08:28:09.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0044625039999999999 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.544963+0000", 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051210359999999998 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.545059+0000", 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6230999999999994e-05 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.545100+0000", 2024-09-21T08:28:09.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.066e-05 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.545137+0000", 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7499999999999997e-05 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.686 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2322 7.b 7:df2b42d9:::smithi13779221-36:head [append 0~0] snapc 4f=[4f,4e,4d,48,3d] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T08:28:09.687 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:13.572465+0000", 2024-09-21T08:28:09.687 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.025027424000001, 2024-09-21T08:28:09.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014526439, 2024-09-21T08:28:09.687 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.687 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.687 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.687 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2322 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572465+0000", 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572465+0000", 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572467+0000", 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8250000000000001e-06 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572472+0000", 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6370000000000004e-06 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572483+0000", 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0883000000000001e-05 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572491+0000", 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3520000000000007e-06 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572551+0000", 2024-09-21T08:28:09.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9799000000000001e-05 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572556+0000", 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9019999999999998e-06 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.574436+0000", 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018797410000000001 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.691 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.574553+0000", 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011684999999999999 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.574732+0000", 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000179548 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.576412+0000", 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016801629999999999 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.586450+0000", 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010037621 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.586917+0000", 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046688099999999999 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.586956+0000", 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8798999999999999e-05 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.586991+0000", 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5438000000000002e-05 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2323 7.b 7:df2b42d9:::smithi13779221-36:head [rollback 48] snapc 4f=[4f,4e,4d,48,3d] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:13.572505+0000", 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.024986947999999, 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014608846, 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.694 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2323 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572505+0000", 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572505+0000", 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572506+0000", 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.99e-07 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572528+0000", 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2022e-05 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572534+0000", 2024-09-21T08:28:09.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9669999999999998e-06 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572540+0000", 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7899999999999996e-06 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572599+0000", 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9120999999999999e-05 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572604+0000", 2024-09-21T08:28:09.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7359999999999998e-06 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.575509+0000", 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029042540000000002 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.575560+0000", 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0889e-05 2024-09-21T08:28:09.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.575675+0000", 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011489600000000001 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.576417+0000", 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074203300000000002 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.586511+0000", 2024-09-21T08:28:09.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010094792999999999 2024-09-21T08:28:09.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587046+0000", 2024-09-21T08:28:09.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053456399999999996 2024-09-21T08:28:09.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587082+0000", 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6531e-05 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587114+0000", 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1650999999999999e-05 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.701 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2324 7.b 7:df2b42d9:::smithi13779221-36:head [append 0~0] snapc 4f=[4f,4e,4d,48,3d] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:13.572548+0000", 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 56.024944300999998, 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014668708000000001, 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2324 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.702 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572548+0000", 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572548+0000", 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572549+0000", 2024-09-21T08:28:09.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9500000000000002e-07 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572551+0000", 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.779e-06 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572556+0000", 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1909999999999999e-06 2024-09-21T08:28:09.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572562+0000", 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3170000000000001e-06 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572620+0000", 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8405000000000001e-05 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572623+0000", 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2619999999999999e-06 2024-09-21T08:28:09.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.575994+0000", 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033708710000000001 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.576058+0000", 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3571000000000005e-05 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.576214+0000", 2024-09-21T08:28:09.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155771 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.577231+0000", 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010176829999999999 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.586539+0000", 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093072429999999998 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587152+0000", 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00061338899999999999 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587187+0000", 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4817999999999998e-05 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587217+0000", 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9813000000000001e-05 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.708 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2577 7.1c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 174] snapc 59=[59,56] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.034741029999999, 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0145323420000001, 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2577 2024-09-21T08:28:09.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562753+0000", 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.745e-06 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562759+0000", 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1330000000000004e-06 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562770+0000", 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0617e-05 2024-09-21T08:28:09.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562775+0000", 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4500000000000003e-06 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562831+0000", 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6233999999999997e-05 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562838+0000", 2024-09-21T08:28:09.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7379999999999997e-06 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552540+0000", 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98970149799999996 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552665+0000", 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012560800000000001 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:09.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.562825+0000", 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010159916 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.569448+0000", 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066225670000000002 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577182+0000", 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077346680000000001 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577203+0000", 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0619999999999999e-05 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577240+0000", 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6613999999999999e-05 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577284+0000", 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3934000000000003e-05 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.715 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2578 7.1c 7:3cae00e7:::smithi13779221-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.034695153000001, 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0041372150000001, 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:09.716 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2578 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562798+0000", 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5000000000000003e-07 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562800+0000", 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2460000000000002e-06 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562806+0000", 2024-09-21T08:28:09.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6620000000000002e-06 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562809+0000", 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8500000000000004e-06 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562869+0000", 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9892999999999997e-05 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:09.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562879+0000", 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0054999999999999e-05 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552788+0000", 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.989908287 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552833+0000", 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5427999999999999e-05 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566793+0000", 2024-09-21T08:28:09.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013959875 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566858+0000", 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4870999999999996e-05 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566934+0000", 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6497999999999998e-05 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:2828 7.d e287/259 7:b0b32695:::smithi13779221-19:head v 287'89, mlcod=262'86)", 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:00.649064+0000", 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 8.9484284699999996, 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014116995, 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:00.649064+0000", 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:00.649064+0000", 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:00.649064+0000", 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3999999999999998e-07 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:00.656754+0000", 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076902300000000002 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:00.656765+0000", 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0835999999999999e-05 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:00.656775+0000", 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7960000000000007e-06 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:00.657053+0000", 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027813499999999999 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:00.657076+0000", 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2538e-05 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:00.663138+0000", 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0060623209999999999 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:00.663181+0000", 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2898999999999997e-05 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.725 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.726 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.726 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:09.726 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:09.726 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:28:09.726 INFO:tasks.rados.rados.0.smithi137.stdout:1125: done (0 left) 2024-09-21T08:28:09.726 INFO:tasks.rados.rados.0.smithi137.stdout:1126: snap_create 2024-09-21T08:28:09.746 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:09.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:09.748 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:09.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:09.870 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:09.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:09.871 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:09.907 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1663534989' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[29634]: osdmap e295: 8 total, 8 up, 5 in 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[29634]: 7.1b scrub starts 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[29634]: 7.1b scrub ok 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[29634]: 7.1e scrub starts 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[29634]: 7.1e scrub ok 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1663534989' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[41044]: osdmap e295: 8 total, 8 up, 5 in 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[41044]: 7.1b scrub starts 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[41044]: 7.1b scrub ok 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[41044]: 7.1e scrub starts 2024-09-21T08:28:09.908 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:09 smithi137 ceph-mon[41044]: 7.1e scrub ok 2024-09-21T08:28:09.972 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:09.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:09.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:09.995 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:10.007 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:09 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1663534989' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:28:10.007 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:09 smithi165 ceph-mon[32893]: osdmap e295: 8 total, 8 up, 5 in 2024-09-21T08:28:10.007 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:09 smithi165 ceph-mon[32893]: 7.1b scrub starts 2024-09-21T08:28:10.007 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:09 smithi165 ceph-mon[32893]: 7.1b scrub ok 2024-09-21T08:28:10.007 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:09 smithi165 ceph-mon[32893]: 7.1e scrub starts 2024-09-21T08:28:10.007 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:09 smithi165 ceph-mon[32893]: 7.1e scrub ok 2024-09-21T08:28:10.007 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:10.007 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:10.025 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:10.025 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:10.034 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:10.041 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:10.042 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:10.058 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:10.059 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:10.076 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:10.076 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:10.094 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:10.094 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:10.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:10.110 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:10.110 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:10.117 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:10.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:10.117 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:10.127 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:10.127 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:10.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:10.240 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:10.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:10.240 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:10.341 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:10.362 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:10.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:10.363 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:10.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:10.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:10.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:10.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:10.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T08:28:10.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:10.633 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:10.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:10.634 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:10.692 INFO:tasks.rados.rados.0.smithi137.stdout:1126: done (0 left) 2024-09-21T08:28:10.692 INFO:tasks.rados.rados.0.smithi137.stdout:1127: read oid 40 snap -1 2024-09-21T08:28:10.692 INFO:tasks.rados.rados.0.smithi137.stdout:1127: expect (ObjNum 287 snap 66 seq_num 287) 2024-09-21T08:28:10.692 INFO:tasks.rados.rados.0.smithi137.stdout:1128: write (excl) oid 10 current snap is 106 2024-09-21T08:28:10.692 INFO:tasks.rados.rados.0.smithi137.stdout:1128: seq_num 423 ranges {709361=629742,1836589=539816,3033220=76141} 2024-09-21T08:28:10.699 INFO:tasks.rados.rados.0.smithi137.stdout:1128: writing smithi13779221-10 from 709361 to 1339103 tid 1 2024-09-21T08:28:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:1128: writing smithi13779221-10 from 1836589 to 2376405 tid 2 2024-09-21T08:28:10.711 INFO:tasks.rados.rados.0.smithi137.stdout:1128: writing smithi13779221-10 from 3033220 to 3109361 tid 3 2024-09-21T08:28:10.711 INFO:tasks.rados.rados.0.smithi137.stdout:1129: snap_create 2024-09-21T08:28:10.711 INFO:tasks.rados.rados.0.smithi137.stdout:1127: expect (ObjNum 287 snap 66 seq_num 287) 2024-09-21T08:28:10.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:10.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:10.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:10.756 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:10.764 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: 5.1c scrub starts 2024-09-21T08:28:10.764 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: 5.1c scrub ok 2024-09-21T08:28:10.764 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: osdmap e296: 8 total, 8 up, 5 in 2024-09-21T08:28:10.764 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:28:10.764 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2646335724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: pgmap v350: 168 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 9 activating+undersized+degraded+remapped, 1 active+clean+snaptrim, 154 active+clean; 107 MiB data, 436 MiB used, 447 GiB / 447 GiB avail; 149/747 objects degraded (19.946%); 66/747 objects misplaced (8.835%) 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: 7.f scrub starts 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: 7.f scrub ok 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: 7.23 scrub starts 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: 7.23 scrub ok 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: 6.19 scrub starts 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[29634]: 6.19 scrub ok 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: 5.1c scrub starts 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: 5.1c scrub ok 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: osdmap e296: 8 total, 8 up, 5 in 2024-09-21T08:28:10.765 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:28:10.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:28:10.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2646335724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:10.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: pgmap v350: 168 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 9 activating+undersized+degraded+remapped, 1 active+clean+snaptrim, 154 active+clean; 107 MiB data, 436 MiB used, 447 GiB / 447 GiB avail; 149/747 objects degraded (19.946%); 66/747 objects misplaced (8.835%) 2024-09-21T08:28:10.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: 7.f scrub starts 2024-09-21T08:28:10.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: 7.f scrub ok 2024-09-21T08:28:10.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: 7.23 scrub starts 2024-09-21T08:28:10.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: 7.23 scrub ok 2024-09-21T08:28:10.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: 6.19 scrub starts 2024-09-21T08:28:10.766 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-mon[41044]: 6.19 scrub ok 2024-09-21T08:28:10.766 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:28:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:28:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:28:10.825 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:10.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:10.877 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:10.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:10.878 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:10.911 INFO:tasks.rados.rados.0.smithi137.stdout:1128: finishing write tid 1 to smithi13779221-10 2024-09-21T08:28:10.911 INFO:tasks.rados.rados.0.smithi137.stdout:1128: finishing write tid 2 to smithi13779221-10 2024-09-21T08:28:10.911 INFO:tasks.rados.rados.0.smithi137.stdout:1128: finishing write tid 3 to smithi13779221-10 2024-09-21T08:28:10.911 INFO:tasks.rados.rados.0.smithi137.stdout:1128: finishing write tid 4 to smithi13779221-10 2024-09-21T08:28:10.911 INFO:tasks.rados.rados.0.smithi137.stdout:1128: finishing write tid 5 to smithi13779221-10 2024-09-21T08:28:10.911 INFO:tasks.rados.rados.0.smithi137.stdout:1128: oid 10 updating version 0 to 303 2024-09-21T08:28:10.911 INFO:tasks.rados.rados.0.smithi137.stdout:1128: oid 10 version 303 is already newer than 302 2024-09-21T08:28:10.911 INFO:tasks.rados.rados.0.smithi137.stdout:1128: oid 10 updating version 303 to 305 2024-09-21T08:28:10.912 INFO:tasks.rados.rados.0.smithi137.stdout:1128: oid 10 version 305 is already newer than 304 2024-09-21T08:28:10.912 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 305 (ObjNum 423 snap 106 seq_num 423) dirty exists 2024-09-21T08:28:10.912 INFO:tasks.rados.rados.0.smithi137.stdout:1128: left oid 10 (ObjNum 423 snap 106 seq_num 423) 2024-09-21T08:28:10.943 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: 5.1c scrub starts 2024-09-21T08:28:10.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: 5.1c scrub ok 2024-09-21T08:28:10.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: osdmap e296: 8 total, 8 up, 5 in 2024-09-21T08:28:10.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:28:10.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:28:10.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2646335724' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:10.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: pgmap v350: 168 pgs: 1 activating+undersized+remapped, 3 activating+remapped, 9 activating+undersized+degraded+remapped, 1 active+clean+snaptrim, 154 active+clean; 107 MiB data, 436 MiB used, 447 GiB / 447 GiB avail; 149/747 objects degraded (19.946%); 66/747 objects misplaced (8.835%) 2024-09-21T08:28:10.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: 7.f scrub starts 2024-09-21T08:28:10.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: 7.f scrub ok 2024-09-21T08:28:10.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: 7.23 scrub starts 2024-09-21T08:28:10.944 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: 7.23 scrub ok 2024-09-21T08:28:10.945 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: 6.19 scrub starts 2024-09-21T08:28:10.945 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:10 smithi165 ceph-mon[32893]: 6.19 scrub ok 2024-09-21T08:28:10.979 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:11.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:11.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:11.001 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:11.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:11.123 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:11.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:11.124 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:11.225 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:11.246 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:11.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:11.247 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:11.280 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:28:11.280 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:28:11.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:11.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:28:11.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:11.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:11.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:11.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:11.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:11.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:11.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:11.542 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.7 primary-affinity to 0.243311 (8159452) 2024-09-21T08:28:11.552 INFO:tasks.rados.rados.0.smithi137.stdout:1127: done (2 left) 2024-09-21T08:28:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:1128: done (1 left) 2024-09-21T08:28:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:1129: done (0 left) 2024-09-21T08:28:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:1130: write (excl) oid 48 current snap is 107 2024-09-21T08:28:11.553 INFO:tasks.rados.rados.0.smithi137.stdout:1130: seq_num 424 ranges {726168=704991,1895071=683474} 2024-09-21T08:28:11.560 INFO:tasks.rados.rados.0.smithi137.stdout:1130: writing smithi13779221-48 from 726168 to 1431159 tid 1 2024-09-21T08:28:11.568 INFO:tasks.rados.rados.0.smithi137.stdout:1130: writing smithi13779221-48 from 1895071 to 2578545 tid 2 2024-09-21T08:28:11.569 INFO:tasks.rados.rados.0.smithi137.stdout:1131: read oid 20 snap 100 2024-09-21T08:28:11.569 INFO:tasks.rados.rados.0.smithi137.stdout:1131: expect (ObjNum 388 snap 94 seq_num 388) 2024-09-21T08:28:11.569 INFO:tasks.rados.rados.0.smithi137.stdout:1130: finishing write tid 1 to smithi13779221-48 2024-09-21T08:28:11.569 INFO:tasks.rados.rados.0.smithi137.stdout:1132: read oid 41 snap -1 2024-09-21T08:28:11.569 INFO:tasks.rados.rados.0.smithi137.stdout:1132: expect (ObjNum 377 snap 93 seq_num 377) 2024-09-21T08:28:11.569 INFO:tasks.rados.rados.0.smithi137.stdout:1133: write (excl) oid 28 current snap is 107 2024-09-21T08:28:11.569 INFO:tasks.rados.rados.0.smithi137.stdout:1133: seq_num 425 ranges {742975=780240,1953553=427132,2971713=171262} 2024-09-21T08:28:11.579 INFO:tasks.rados.rados.0.smithi137.stdout:1133: writing smithi13779221-28 from 742975 to 1523215 tid 1 2024-09-21T08:28:11.584 INFO:tasks.rados.rados.0.smithi137.stdout:1133: writing smithi13779221-28 from 1953553 to 2380685 tid 2 2024-09-21T08:28:11.588 INFO:tasks.rados.rados.0.smithi137.stdout:1133: writing smithi13779221-28 from 2971713 to 3142975 tid 3 2024-09-21T08:28:11.588 INFO:tasks.rados.rados.0.smithi137.stdout:1130: finishing write tid 2 to smithi13779221-48 2024-09-21T08:28:11.588 INFO:tasks.rados.rados.0.smithi137.stdout:1130: finishing write tid 3 to smithi13779221-48 2024-09-21T08:28:11.589 INFO:tasks.rados.rados.0.smithi137.stdout:1130: finishing write tid 4 to smithi13779221-48 2024-09-21T08:28:11.589 INFO:tasks.rados.rados.0.smithi137.stdout:1130: oid 48 updating version 0 to 307 2024-09-21T08:28:11.589 INFO:tasks.rados.rados.0.smithi137.stdout:1130: oid 48 updating version 307 to 308 2024-09-21T08:28:11.589 INFO:tasks.rados.rados.0.smithi137.stdout:1130: oid 48 version 308 is already newer than 306 2024-09-21T08:28:11.589 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 308 (ObjNum 424 snap 107 seq_num 424) dirty exists 2024-09-21T08:28:11.589 INFO:tasks.rados.rados.0.smithi137.stdout:1130: left oid 48 (ObjNum 424 snap 107 seq_num 424) 2024-09-21T08:28:11.589 INFO:tasks.rados.rados.0.smithi137.stdout:1132: expect (ObjNum 377 snap 93 seq_num 377) 2024-09-21T08:28:11.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:11.617 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:11.617 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:11.618 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:11.718 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:11.740 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:11.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:11.741 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:11.791 INFO:tasks.rados.rados.0.smithi137.stdout:1133: finishing write tid 1 to smithi13779221-28 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1133: finishing write tid 2 to smithi13779221-28 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1133: finishing write tid 3 to smithi13779221-28 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1133: finishing write tid 4 to smithi13779221-28 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1133: finishing write tid 5 to smithi13779221-28 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1133: oid 28 updating version 0 to 260 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1133: oid 28 updating version 260 to 262 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1133: oid 28 version 262 is already newer than 261 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1133: oid 28 version 262 is already newer than 259 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 262 (ObjNum 425 snap 107 seq_num 425) dirty exists 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1133: left oid 28 (ObjNum 425 snap 107 seq_num 425) 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1130: done (3 left) 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1132: done (2 left) 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1133: done (1 left) 2024-09-21T08:28:11.792 INFO:tasks.rados.rados.0.smithi137.stdout:1134: write oid 9 current snap is 107 2024-09-21T08:28:11.793 INFO:tasks.rados.rados.0.smithi137.stdout:1134: seq_num 426 ranges {759782=571842,2012035=687143} 2024-09-21T08:28:11.798 INFO:tasks.rados.rados.0.smithi137.stdout:1134: writing smithi13779221-9 from 759782 to 1331624 tid 1 2024-09-21T08:28:11.806 INFO:tasks.rados.rados.0.smithi137.stdout:1134: writing smithi13779221-9 from 2012035 to 2699178 tid 2 2024-09-21T08:28:11.806 INFO:tasks.rados.rados.0.smithi137.stdout:1135: read oid 27 snap 104 2024-09-21T08:28:11.806 INFO:tasks.rados.rados.0.smithi137.stdout:1135: expect deleted 2024-09-21T08:28:11.806 INFO:tasks.rados.rados.0.smithi137.stdout:1134: finishing write tid 1 to smithi13779221-9 2024-09-21T08:28:11.806 INFO:tasks.rados.rados.0.smithi137.stdout:1136: read oid 29 snap -1 2024-09-21T08:28:11.806 INFO:tasks.rados.rados.0.smithi137.stdout:1136: expect (ObjNum 402 snap 99 seq_num 402) 2024-09-21T08:28:11.806 INFO:tasks.rados.rados.0.smithi137.stdout:1137: snap_remove snap 99 2024-09-21T08:28:11.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:11.863 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:11.863 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:11.864 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:11.891 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-mon[32893]: 6.14 scrub starts 2024-09-21T08:28:11.892 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-mon[32893]: 6.14 scrub ok 2024-09-21T08:28:11.892 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-mon[32893]: osdmap e297: 8 total, 8 up, 5 in 2024-09-21T08:28:11.892 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4010895922' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.24331079899676422}]: dispatch 2024-09-21T08:28:11.892 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-mon[32893]: from='client.45359 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.24331079899676422}]: dispatch 2024-09-21T08:28:11.892 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-mon[32893]: 7.d scrub starts 2024-09-21T08:28:11.892 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-mon[32893]: 7.d scrub ok 2024-09-21T08:28:11.892 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:11.892 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-mon[32893]: from='client.45359 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.24331079899676422}]': finished 2024-09-21T08:28:11.892 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-mon[32893]: osdmap e298: 8 total, 8 up, 5 in 2024-09-21T08:28:11.892 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:28:11 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:28:11.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T08:28:11.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:11.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:11.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:11.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:12.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[41044]: 6.14 scrub starts 2024-09-21T08:28:12.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[41044]: 6.14 scrub ok 2024-09-21T08:28:12.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[41044]: osdmap e297: 8 total, 8 up, 5 in 2024-09-21T08:28:12.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4010895922' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.24331079899676422}]: dispatch 2024-09-21T08:28:12.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[41044]: from='client.45359 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.24331079899676422}]: dispatch 2024-09-21T08:28:12.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[41044]: 7.d scrub starts 2024-09-21T08:28:12.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[41044]: 7.d scrub ok 2024-09-21T08:28:12.012 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[41044]: from='client.45359 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.24331079899676422}]': finished 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[41044]: osdmap e298: 8 total, 8 up, 5 in 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[29634]: 6.14 scrub starts 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[29634]: 6.14 scrub ok 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[29634]: osdmap e297: 8 total, 8 up, 5 in 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4010895922' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.24331079899676422}]: dispatch 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[29634]: from='client.45359 ' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.24331079899676422}]: dispatch 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[29634]: 7.d scrub starts 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[29634]: 7.d scrub ok 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[29634]: from='client.45359 ' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 7, "weight": 0.24331079899676422}]': finished 2024-09-21T08:28:12.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:11 smithi137 ceph-mon[29634]: osdmap e298: 8 total, 8 up, 5 in 2024-09-21T08:28:12.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:12.110 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:12.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:12.111 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:12.167 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:12.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:12.234 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:12.235 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:12.235 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:12.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:12.357 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:12.358 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:12.358 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:12.399 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:12.459 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:12.480 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:12.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:12.481 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:1137: done (4 left) 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:1138: delete oid 30 current snap is 107 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:1134: finishing write tid 2 to smithi13779221-9 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:1134: finishing write tid 3 to smithi13779221-9 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:1134: finishing write tid 4 to smithi13779221-9 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:1134: oid 9 updating version 0 to 329 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:1134: oid 9 updating version 329 to 330 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:1134: oid 9 updating version 330 to 331 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 9 v 331 (ObjNum 426 snap 107 seq_num 426) dirty exists 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:1134: left oid 9 (ObjNum 426 snap 107 seq_num 426) 2024-09-21T08:28:12.545 INFO:tasks.rados.rados.0.smithi137.stdout:1136: expect (ObjNum 402 snap 99 seq_num 402) 2024-09-21T08:28:12.582 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:12.603 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:12.603 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:12.604 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:12.641 INFO:tasks.rados.rados.0.smithi137.stdout:1131: expect (ObjNum 388 snap 94 seq_num 388) 2024-09-21T08:28:12.652 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:28:12.652 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:28:12.705 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:12.728 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:12.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:12.729 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:12.751 INFO:tasks.rados.rados.0.smithi137.stdout:1131: done (4 left) 2024-09-21T08:28:12.751 INFO:tasks.rados.rados.0.smithi137.stdout:1134: done (3 left) 2024-09-21T08:28:12.751 INFO:tasks.rados.rados.0.smithi137.stdout:1135: done (2 left) 2024-09-21T08:28:12.751 INFO:tasks.rados.rados.0.smithi137.stdout:1136: done (1 left) 2024-09-21T08:28:12.751 INFO:tasks.rados.rados.0.smithi137.stdout:1138: done (0 left) 2024-09-21T08:28:12.751 INFO:tasks.rados.rados.0.smithi137.stdout:1139: copy_from oid 22 from oid 10 current snap is 107 2024-09-21T08:28:12.751 INFO:tasks.rados.rados.0.smithi137.stdout:1140: read oid 18 snap 102 2024-09-21T08:28:12.751 INFO:tasks.rados.rados.0.smithi137.stdout:1140: expect (ObjNum 406 snap 99 seq_num 406) 2024-09-21T08:28:12.751 INFO:tasks.rados.rados.0.smithi137.stdout:1141: snap_create 2024-09-21T08:28:12.757 INFO:tasks.rados.rados.0.smithi137.stdout:1140: expect (ObjNum 406 snap 99 seq_num 406) 2024-09-21T08:28:12.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:28:12.830 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:12.848 INFO:tasks.rados.rados.0.smithi137.stdout:1139: finishing copy_from racing read to smithi13779221-22 2024-09-21T08:28:12.848 INFO:tasks.rados.rados.0.smithi137.stdout:1139: finishing copy_from to smithi13779221-22 2024-09-21T08:28:12.848 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 305 (ObjNum 423 snap 106 seq_num 423) dirty exists 2024-09-21T08:28:12.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:12.851 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:12.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:12.952 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:12.974 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:12.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:12.975 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:13.058 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:13.075 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:13.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:13.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:13.097 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:13.198 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:13.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:13.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:13.220 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:13.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:13.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:13.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:13.344 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:13.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T08:28:13.445 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:13.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:13.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:13.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:13.510 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:13.510 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:13.527 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:13.527 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:13.545 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:13.545 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:13.561 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:13.561 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:13.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:13.579 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:13.579 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:13.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:13.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:13.590 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:13.596 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:13.596 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:13.612 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:13.612 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:13.628 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:13.629 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:13.658 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:13.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:13.696 INFO:tasks.rados.rados.0.smithi137.stdout:1139: done (2 left) 2024-09-21T08:28:13.696 INFO:tasks.rados.rados.0.smithi137.stdout:1140: done (1 left) 2024-09-21T08:28:13.696 INFO:tasks.rados.rados.0.smithi137.stdout:1141: done (0 left) 2024-09-21T08:28:13.696 INFO:tasks.rados.rados.0.smithi137.stdout:1142: rollback oid 37 current snap is 108 2024-09-21T08:28:13.696 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 37 to 102 2024-09-21T08:28:13.696 INFO:tasks.rados.rados.0.smithi137.stdout:1143: snap_remove snap 107 2024-09-21T08:28:13.696 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:28:13.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:13.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:13.715 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:13.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: pgmap v353: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 143 MiB data, 510 MiB used, 447 GiB / 447 GiB avail; 2.0 MiB/s, 34 objects/s recovering 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: 3.1e scrub starts 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: 3.1e scrub ok 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: 7.1d scrub starts 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: 7.1d scrub ok 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: 3.12 scrub starts 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: 3.12 scrub ok 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: 6.1a scrub starts 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: 6.1a scrub ok 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: osdmap e299: 8 total, 8 up, 5 in 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 149/747 objects degraded (19.946%), 9 pgs degraded) 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/216910163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:13.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: from='client.45365 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:13.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:13 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3346476205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:13.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: pgmap v353: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 143 MiB data, 510 MiB used, 447 GiB / 447 GiB avail; 2.0 MiB/s, 34 objects/s recovering 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: 3.1e scrub starts 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: 3.1e scrub ok 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: 7.1d scrub starts 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: 7.1d scrub ok 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: 3.12 scrub starts 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: 3.12 scrub ok 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: 6.1a scrub starts 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: 6.1a scrub ok 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: osdmap e299: 8 total, 8 up, 5 in 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 149/747 objects degraded (19.946%), 9 pgs degraded) 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:28:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/216910163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: from='client.45365 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3346476205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: pgmap v353: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 143 MiB data, 510 MiB used, 447 GiB / 447 GiB avail; 2.0 MiB/s, 34 objects/s recovering 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: 3.1e scrub starts 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: 3.1e scrub ok 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: 7.1d scrub starts 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: 7.1d scrub ok 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: 3.12 scrub starts 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: 3.12 scrub ok 2024-09-21T08:28:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: 6.1a scrub starts 2024-09-21T08:28:13.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: 6.1a scrub ok 2024-09-21T08:28:13.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: osdmap e299: 8 total, 8 up, 5 in 2024-09-21T08:28:13.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 149/747 objects degraded (19.946%), 9 pgs degraded) 2024-09-21T08:28:13.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:28:13.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:28:13.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/216910163' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:13.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: from='client.45365 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:13.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:13 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3346476205' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:13.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:13.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:13.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:13.838 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:13.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:13.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:13.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:13.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:14.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:14.084 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:14.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:14.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:14.152 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:28:14.152 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:28:14.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.56106591899999, 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049090319999999998, 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.153 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567959+0000", 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.762e-06 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567965+0000", 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567980+0000", 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4742e-05 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567990+0000", 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8800000000000003e-06 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568114+0000", 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012444499999999999 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568250+0000", 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135205 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568390+0000", 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140335 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569250+0000", 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085971599999999995 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572132+0000", 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028820819999999998 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572759+0000", 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062750100000000004 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572810+0000", 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0488000000000002e-05 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572867+0000", 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6999000000000001e-05 2024-09-21T08:28:14.158 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.56059911, 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045637589999999997, 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.159 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568426+0000", 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6470000000000001e-06 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568432+0000", 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.395e-06 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568443+0000", 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1070000000000001e-05 2024-09-21T08:28:14.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568456+0000", 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3862e-05 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568688+0000", 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023171699999999999 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568773+0000", 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5307000000000002e-05 2024-09-21T08:28:14.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568890+0000", 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116082 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569279+0000", 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038986899999999999 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572238+0000", 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029581009999999999 2024-09-21T08:28:14.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572906+0000", 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000668945 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572955+0000", 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8455000000000003e-05 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572988+0000", 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3309000000000002e-05 2024-09-21T08:28:14.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:47 5.1f 5:f95f44c2:::notify.0:head [watch watch cookie 94074441540560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.55581803000001, 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026954729999999999, 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:28:14.165 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 47 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573208+0000", 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9970000000000001e-06 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573213+0000", 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5679999999999999e-06 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573224+0000", 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1270000000000001e-05 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573233+0000", 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8939999999999999e-06 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573770+0000", 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053668100000000001 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573841+0000", 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.135e-05 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.574869+0000", 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010281719999999999 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.574911+0000", 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1631000000000002e-05 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575071+0000", 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015977 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575463+0000", 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039199299999999999 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575737+0000", 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027415300000000002 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575785+0000", 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7970999999999999e-05 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575832+0000", 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7120999999999998e-05 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575901+0000", 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8901999999999999e-05 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:192 4.5 4:a2ace8b3:::data_log.0:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:28:14.171 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.51319734399999, 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031080779999999998, 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:28:14.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615827+0000", 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1629999999999999e-06 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615831+0000", 2024-09-21T08:28:14.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0790000000000002e-06 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615841+0000", 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8279999999999998e-06 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615847+0000", 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2660000000000001e-06 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615879+0000", 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2512999999999999e-05 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615924+0000", 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4709999999999997e-05 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.618934+0000", 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030105190000000001 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1084 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94837804042368 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 215.969405997, 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40449157200000002, 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1084 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159622+0000", 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1400000000000002e-06 2024-09-21T08:28:14.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159631+0000", 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9670000000000007e-06 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159655+0000", 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4037999999999999e-05 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159667+0000", 2024-09-21T08:28:14.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2427e-05 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159756+0000", 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9103000000000004e-05 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159772+0000", 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5877000000000001e-05 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.554979+0000", 2024-09-21T08:28:14.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39520667999999998 2024-09-21T08:28:14.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:14.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.555262+0000", 2024-09-21T08:28:14.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028320500000000002 2024-09-21T08:28:14.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.560260+0000", 2024-09-21T08:28:14.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049979630000000002 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.560331+0000", 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1048000000000001e-05 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.564109+0000", 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037781239999999999 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.182 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1085 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94837808833664 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 215.96926676499999, 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40437364799999997, 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1085 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.183 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159758+0000", 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.539e-06 2024-09-21T08:28:14.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159764+0000", 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1330000000000004e-06 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159773+0000", 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3250000000000008e-06 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159779+0000", 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5400000000000001e-06 2024-09-21T08:28:14.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159844+0000", 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4580999999999997e-05 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159854+0000", 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.893e-06 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552880+0000", 2024-09-21T08:28:14.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.393026184 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.553195+0000", 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031526299999999998 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559454+0000", 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062587809999999997 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559541+0000", 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7379999999999993e-05 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.564130+0000", 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045890289999999997 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:265 7.b e93/80 7:dd97af7e:::smithi13779221-30:head v 93'26, mlcod=86'23)", 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 207.56527171100001, 2024-09-21T08:28:14.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036488330000000002, 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563753+0000", 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4000000000000001e-07 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565430+0000", 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016774439999999999 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565446+0000", 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5540999999999999e-05 2024-09-21T08:28:14.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565455+0000", 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9580000000000002e-06 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565513+0000", 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7312000000000003e-05 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565542+0000", 2024-09-21T08:28:14.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8856999999999998e-05 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.567363+0000", 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018212300000000001 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.567401+0000", 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7851000000000001e-05 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:321 7.c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 6] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 207.31939634599999, 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013109218000000001, 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 321 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809632+0000", 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9559999999999996e-06 2024-09-21T08:28:14.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809648+0000", 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5814000000000001e-05 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809680+0000", 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1732e-05 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809696+0000", 2024-09-21T08:28:14.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6293000000000001e-05 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809815+0000", 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119442 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809903+0000", 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7033999999999994e-05 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:28:14.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.815434+0000", 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055318420000000004 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.820591+0000", 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051568229999999996 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.821125+0000", 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053413400000000002 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:14.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822656+0000", 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001531035 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822697+0000", 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0488999999999998e-05 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822736+0000", 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9623999999999999e-05 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.198 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:322 7.c 7:3cae00e7:::smithi13779221-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 207.319298094, 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093384190000000006, 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 322 2024-09-21T08:28:14.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809727+0000", 2024-09-21T08:28:14.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.13e-06 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809734+0000", 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9569999999999996e-06 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809744+0000", 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0633e-05 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809753+0000", 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7090000000000008e-06 2024-09-21T08:28:14.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.810046+0000", 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029301499999999999 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.810079+0000", 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3293000000000001e-05 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.818920+0000", 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088405299999999992 2024-09-21T08:28:14.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.818982+0000", 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999000000000001e-05 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.819064+0000", 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2152999999999998e-05 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:329 7.c 7:361ee62b:::smithi13779221-40:head [sparse-read 0~1272916,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 206.442845242, 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90863698400000004, 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 329 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686182+0000", 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0459999999999999e-06 2024-09-21T08:28:14.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686196+0000", 2024-09-21T08:28:14.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4097000000000001e-05 2024-09-21T08:28:14.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686225+0000", 2024-09-21T08:28:14.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8846000000000001e-05 2024-09-21T08:28:14.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686240+0000", 2024-09-21T08:28:14.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4256e-05 2024-09-21T08:28:14.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686362+0000", 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012248200000000001 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686390+0000", 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7546e-05 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.591277+0000", 2024-09-21T08:28:14.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90488744700000001 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.591416+0000", 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013935900000000001 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.594815+0000", 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033989049999999998 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:330 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2655473,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 206.44275630300001, 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91026091200000003, 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 330 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686269+0000", 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7060000000000001e-06 2024-09-21T08:28:14.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686277+0000", 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4470000000000006e-06 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686294+0000", 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6099999999999998e-05 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686308+0000", 2024-09-21T08:28:14.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4516e-05 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686428+0000", 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119834 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686436+0000", 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1450000000000008e-06 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.595025+0000", 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90858932599999997 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.595096+0000", 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0821999999999994e-05 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596528+0000", 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014320159999999999 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:331 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2655473,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 206.44269681099999, 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91109795900000001, 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 331 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.215 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686328+0000", 2024-09-21T08:28:14.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.387e-06 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686336+0000", 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4030000000000001e-06 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686350+0000", 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4713e-05 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686364+0000", 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3716999999999999e-05 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686455+0000", 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0737999999999998e-05 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686460+0000", 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1290000000000004e-06 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596614+0000", 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91015405299999996 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596687+0000", 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2700000000000005e-05 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.597425+0000", 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00073811899999999997 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.219 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:338 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 205.56119187100001, 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038704049999999999, 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 338 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.220 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567835+0000", 2024-09-21T08:28:14.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0019999999999998e-06 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567843+0000", 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0090000000000007e-06 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567861+0000", 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8748000000000001e-05 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567872+0000", 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0173000000000001e-05 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567975+0000", 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103398 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568085+0000", 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109691 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571702+0000", 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003617384 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:339 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 205.56111677199999, 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054334199999999996, 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.224 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 339 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567908+0000", 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-06 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567913+0000", 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.536e-06 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567922+0000", 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0730000000000005e-06 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567928+0000", 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4489999999999996e-06 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571741+0000", 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038125590000000001 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571859+0000", 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118249 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.573340+0000", 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001481472 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:340 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:28:14.228 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 205.56091824699999, 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082399729999999994, 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 340 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568107+0000", 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1850000000000001e-06 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568112+0000", 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8019999999999997e-06 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568122+0000", 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6970000000000005e-06 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568129+0000", 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2119999999999997e-06 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.231 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.575080+0000", 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069511850000000003 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.575168+0000", 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7613000000000003e-05 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.576345+0000", 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001177279 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.232 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.7 95/94 [PushReplyOp(7:e1536981:::smithi13779221-32:head)])", 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 205.472332752, 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030062309999999998, 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656692+0000", 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0690000000000001e-06 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656696+0000", 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3590000000000001e-06 2024-09-21T08:28:14.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656705+0000", 2024-09-21T08:28:14.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5639999999999997e-06 2024-09-21T08:28:14.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:14.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656710+0000", 2024-09-21T08:28:14.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-21T08:28:14.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656749+0000", 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9026999999999998e-05 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.659697+0000", 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029482829999999999 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.c 95/94 [PushReplyOp(7:3cae00e7:::smithi13779221-28:5)])", 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 205.45089872400001, 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028000960000000002, 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678126+0000", 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.327e-06 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678131+0000", 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4920000000000004e-06 2024-09-21T08:28:14.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678141+0000", 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9839999999999996e-06 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678146+0000", 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.446e-06 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678184+0000", 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8259000000000001e-05 2024-09-21T08:28:14.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.680925+0000", 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002740588 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.c 95/94 [PushReplyOp(7:3cae00e7:::smithi13779221-28:head)])", 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 205.44059567900001, 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027171439999999999, 2024-09-21T08:28:14.241 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.242 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688429+0000", 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688434+0000", 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999996e-06 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688443+0000", 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4769999999999992e-06 2024-09-21T08:28:14.243 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688448+0000", 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3499999999999996e-06 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688491+0000", 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2216999999999999e-05 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.691145+0000", 2024-09-21T08:28:14.244 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026543550000000002 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:353 7.c 7:361ee62b:::smithi13779221-40:head [copy-from ver 7] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 204.226732129, 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014212783, 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.245 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 353 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.246 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902296+0000", 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1509999999999997e-06 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902323+0000", 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7491000000000002e-05 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902342+0000", 2024-09-21T08:28:14.247 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8522999999999999e-05 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902350+0000", 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8119999999999997e-06 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902448+0000", 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7867000000000003e-05 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902544+0000", 2024-09-21T08:28:14.248 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5612999999999994e-05 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.907790+0000", 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052465360000000004 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.913037+0000", 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052469129999999998 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916372+0000", 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003334914 2024-09-21T08:28:14.249 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916404+0000", 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1327e-05 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916459+0000", 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4917000000000003e-05 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916504+0000", 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5719000000000001e-05 2024-09-21T08:28:14.250 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:354 7.c 7:361ee62b:::smithi13779221-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 204.22665118399999, 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089418770000000009, 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 354 2024-09-21T08:28:14.251 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902373+0000", 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6499999999999998e-07 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.252 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902376+0000", 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.616e-06 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902402+0000", 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5890000000000001e-05 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902407+0000", 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0019999999999997e-06 2024-09-21T08:28:14.253 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902704+0000", 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029720499999999998 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902740+0000", 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6383e-05 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911116+0000", 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.008375136 2024-09-21T08:28:14.254 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911208+0000", 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2791e-05 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911314+0000", 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105989 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:14.255 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:28:14.276 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:14.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:14.277 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:14.378 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:14.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:14.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:14.402 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:14.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:14.525 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:14.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:14.525 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:14.626 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:14.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:14.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:14.648 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:14.698 INFO:tasks.rados.rados.0.smithi137.stdout:1143: done (1 left) 2024-09-21T08:28:14.698 INFO:tasks.rados.rados.0.smithi137.stdout:1144: write (excl) oid 23 current snap is 108 2024-09-21T08:28:14.698 INFO:tasks.rados.rados.0.smithi137.stdout:1144: seq_num 428 ranges {793396=722340,2128999=690812} 2024-09-21T08:28:14.706 INFO:tasks.rados.rados.0.smithi137.stdout:1144: writing smithi13779221-23 from 793396 to 1515736 tid 1 2024-09-21T08:28:14.713 INFO:tasks.rados.rados.0.smithi137.stdout:1144: writing smithi13779221-23 from 2128999 to 2819811 tid 2 2024-09-21T08:28:14.713 INFO:tasks.rados.rados.0.smithi137.stdout:1145: snap_remove snap 101 2024-09-21T08:28:14.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:28:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:28:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[29634]: 7.17 scrub starts 2024-09-21T08:28:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[29634]: 7.17 scrub ok 2024-09-21T08:28:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:28:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:28:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[29634]: from='client.45365 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[29634]: osdmap e300: 8 total, 8 up, 5 in 2024-09-21T08:28:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:28:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:28:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[41044]: 7.17 scrub starts 2024-09-21T08:28:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[41044]: 7.17 scrub ok 2024-09-21T08:28:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:28:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:28:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[41044]: from='client.45365 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:14.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:14 smithi137 ceph-mon[41044]: osdmap e300: 8 total, 8 up, 5 in 2024-09-21T08:28:14.771 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:14.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:14.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:14.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:14.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:14.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:14.895 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:14.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:14 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:28:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:14 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:28:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:14 smithi165 ceph-mon[32893]: 7.17 scrub starts 2024-09-21T08:28:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:14 smithi165 ceph-mon[32893]: 7.17 scrub ok 2024-09-21T08:28:15.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:14 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:28:15.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:14 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:28:15.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:14 smithi165 ceph-mon[32893]: from='client.45365 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:15.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:14 smithi165 ceph-mon[32893]: osdmap e300: 8 total, 8 up, 5 in 2024-09-21T08:28:15.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:15.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:15.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:15.120 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:15.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:15.143 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:15.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:15.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T08:28:15.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:15.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:15.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:15.267 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:15.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:15.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:15.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:15.390 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:15.427 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:15.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:15.512 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:15.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:15.513 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:15.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:15.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:15.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:15.635 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:15.705 INFO:tasks.rados.rados.0.smithi137.stdout:1145: done (2 left) 2024-09-21T08:28:15.705 INFO:tasks.rados.rados.0.smithi137.stdout:1146: write oid 6 current snap is 108 2024-09-21T08:28:15.705 INFO:tasks.rados.rados.0.smithi137.stdout:1142: finishing rollback tid 1 to smithi13779221-37 2024-09-21T08:28:15.705 INFO:tasks.rados.rados.0.smithi137.stdout:1146: seq_num 429 ranges {410203=797589,1787481=434470,2681405=528798} 2024-09-21T08:28:15.711 INFO:tasks.rados.rados.0.smithi137.stdout:1146: writing smithi13779221-6 from 410203 to 1207792 tid 1 2024-09-21T08:28:15.717 INFO:tasks.rados.rados.0.smithi137.stdout:1146: writing smithi13779221-6 from 1787481 to 2221951 tid 2 2024-09-21T08:28:15.723 INFO:tasks.rados.rados.0.smithi137.stdout:1146: writing smithi13779221-6 from 2681405 to 3210203 tid 3 2024-09-21T08:28:15.723 INFO:tasks.rados.rados.0.smithi137.stdout:1147: snap_remove snap 105 2024-09-21T08:28:15.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:15.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:15.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:15.758 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:15.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: 4.5 scrub starts 2024-09-21T08:28:15.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: 4.5 scrub ok 2024-09-21T08:28:15.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: pgmap v356: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 143 MiB data, 510 MiB used, 447 GiB / 447 GiB avail; 2.0 MiB/s, 34 objects/s recovering 2024-09-21T08:28:15.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: 5.1f scrub starts 2024-09-21T08:28:15.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: 5.1f scrub ok 2024-09-21T08:28:15.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: 3.3 scrub starts 2024-09-21T08:28:15.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: 3.3 scrub ok 2024-09-21T08:28:15.793 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: 4.e scrub starts 2024-09-21T08:28:15.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: 4.e scrub ok 2024-09-21T08:28:15.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: 4.15 scrub starts 2024-09-21T08:28:15.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: 4.15 scrub ok 2024-09-21T08:28:15.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: osdmap e301: 8 total, 8 up, 5 in 2024-09-21T08:28:15.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:15.794 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:15 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:15.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:15.880 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:15.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:15.881 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:15.889 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:28:15.889 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:28:15.889 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:28:15.889 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:28:15.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:16.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:16.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:16.004 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: 4.5 scrub starts 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: 4.5 scrub ok 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: pgmap v356: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 143 MiB data, 510 MiB used, 447 GiB / 447 GiB avail; 2.0 MiB/s, 34 objects/s recovering 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: 5.1f scrub starts 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: 5.1f scrub ok 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: 3.3 scrub starts 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: 3.3 scrub ok 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: 4.e scrub starts 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: 4.e scrub ok 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: 4.15 scrub starts 2024-09-21T08:28:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: 4.15 scrub ok 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: osdmap e301: 8 total, 8 up, 5 in 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: 4.5 scrub starts 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: 4.5 scrub ok 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: pgmap v356: 168 pgs: 1 active+clean+snaptrim, 167 active+clean; 143 MiB data, 510 MiB used, 447 GiB / 447 GiB avail; 2.0 MiB/s, 34 objects/s recovering 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: 5.1f scrub starts 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: 5.1f scrub ok 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: 3.3 scrub starts 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: 3.3 scrub ok 2024-09-21T08:28:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: 4.e scrub starts 2024-09-21T08:28:16.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: 4.e scrub ok 2024-09-21T08:28:16.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: 4.15 scrub starts 2024-09-21T08:28:16.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: 4.15 scrub ok 2024-09-21T08:28:16.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: osdmap e301: 8 total, 8 up, 5 in 2024-09-21T08:28:16.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:16.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:15 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:16.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:16.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:16.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:16.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:16.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:16.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:16.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:16.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:16.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:16.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:16.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:16.374 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:16.430 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:28:16.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:16.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:16.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:16.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:16.598 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:16.619 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:16.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:16.620 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:16.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T08:28:16.682 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:16.717 INFO:tasks.rados.rados.0.smithi137.stdout:1147: done (3 left) 2024-09-21T08:28:16.717 INFO:tasks.rados.rados.0.smithi137.stdout:1148: copy_from oid 29 from oid 14 current snap is 108 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1142: finishing rollback tid 2 to smithi13779221-37 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 503 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1144: finishing write tid 1 to smithi13779221-23 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1144: finishing write tid 2 to smithi13779221-23 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1144: finishing write tid 3 to smithi13779221-23 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1144: finishing write tid 4 to smithi13779221-23 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1144: oid 23 updating version 0 to 186 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1144: oid 23 version 186 is already newer than 185 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1144: oid 23 updating version 186 to 187 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 187 (ObjNum 428 snap 108 seq_num 428) dirty exists 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1144: left oid 23 (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1146: finishing write tid 1 to smithi13779221-6 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1146: finishing write tid 2 to smithi13779221-6 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1146: finishing write tid 3 to smithi13779221-6 2024-09-21T08:28:16.718 INFO:tasks.rados.rados.0.smithi137.stdout:1146: finishing write tid 4 to smithi13779221-6 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:1146: finishing write tid 5 to smithi13779221-6 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:1146: oid 6 updating version 0 to 255 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:1146: oid 6 version 255 is already newer than 254 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:1146: oid 6 version 255 is already newer than 253 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:1146: oid 6 updating version 255 to 256 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 256 (ObjNum 429 snap 108 seq_num 429) dirty exists 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:1146: left oid 6 (ObjNum 429 snap 108 seq_num 429) 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:1142: done (3 left) 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:1144: done (2 left) 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:1146: done (1 left) 2024-09-21T08:28:16.719 INFO:tasks.rados.rados.0.smithi137.stdout:1149: delete oid 15 current snap is 108 2024-09-21T08:28:16.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:16.722 INFO:tasks.rados.rados.0.smithi137.stdout:1149: done (1 left) 2024-09-21T08:28:16.722 INFO:tasks.rados.rados.0.smithi137.stdout:1150: snap_create 2024-09-21T08:28:16.728 INFO:tasks.rados.rados.0.smithi137.stdout:1148: finishing copy_from racing read to smithi13779221-29 2024-09-21T08:28:16.732 INFO:tasks.rados.rados.0.smithi137.stdout:1148: finishing copy_from to smithi13779221-29 2024-09-21T08:28:16.732 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 252 (ObjNum 395 snap 97 seq_num 395) dirty exists 2024-09-21T08:28:16.744 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:16.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:16.745 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:16.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:16 smithi137 ceph-mon[29634]: 6.10 scrub starts 2024-09-21T08:28:16.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:16 smithi137 ceph-mon[29634]: 6.10 scrub ok 2024-09-21T08:28:16.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:16 smithi137 ceph-mon[29634]: osdmap e302: 8 total, 8 up, 5 in 2024-09-21T08:28:16.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:16 smithi137 ceph-mon[41044]: 6.10 scrub starts 2024-09-21T08:28:16.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:16 smithi137 ceph-mon[41044]: 6.10 scrub ok 2024-09-21T08:28:16.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:16 smithi137 ceph-mon[41044]: osdmap e302: 8 total, 8 up, 5 in 2024-09-21T08:28:16.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:16.856 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:28:16.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:16.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:16.868 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:16.877 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:16.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:16.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:16 smithi165 ceph-mon[32893]: 6.10 scrub starts 2024-09-21T08:28:16.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:16 smithi165 ceph-mon[32893]: 6.10 scrub ok 2024-09-21T08:28:16.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:16 smithi165 ceph-mon[32893]: osdmap e302: 8 total, 8 up, 5 in 2024-09-21T08:28:16.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:16.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:16.991 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:17.092 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:17.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:17.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:17.115 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:17.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:17.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:17.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:17.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:17.297 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 1] out_osds: [4, 6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:28:17.297 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:28:17.297 INFO:tasks.thrashosds.thrasher:inject_pause on osd.3 2024-09-21T08:28:17.297 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T08:28:17.297 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T08:28:17.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 config set filestore_inject_stall 3 2024-09-21T08:28:17.305 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:17.318 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:28:17.318 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:28:17.318 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:28:17.318 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:28:17.319 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:28:17.339 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:17.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:17.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:17.361 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:17.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:17.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:17.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:17.485 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:17.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:17.609 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:17.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:17.610 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:17.680 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:17 smithi165 ceph-mon[32893]: pgmap v359: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 159 MiB data, 556 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:28:17.680 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:17 smithi165 ceph-mon[32893]: 6.11 scrub starts 2024-09-21T08:28:17.680 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:17 smithi165 ceph-mon[32893]: 6.11 scrub ok 2024-09-21T08:28:17.680 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:17 smithi165 ceph-mon[32893]: 4.7 scrub starts 2024-09-21T08:28:17.680 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:17 smithi165 ceph-mon[32893]: 4.7 scrub ok 2024-09-21T08:28:17.680 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:17 smithi165 ceph-mon[32893]: osdmap e303: 8 total, 8 up, 5 in 2024-09-21T08:28:17.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:17 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1265900146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:28:17.681 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:17 smithi165 ceph-mon[32893]: from='client.45377 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:28:17.695 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[29634]: pgmap v359: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 159 MiB data, 556 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:28:17.695 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[29634]: 6.11 scrub starts 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[29634]: 6.11 scrub ok 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[29634]: 4.7 scrub starts 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[29634]: 4.7 scrub ok 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[29634]: osdmap e303: 8 total, 8 up, 5 in 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1265900146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[29634]: from='client.45377 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[41044]: pgmap v359: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 159 MiB data, 556 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[41044]: 6.11 scrub starts 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[41044]: 6.11 scrub ok 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[41044]: 4.7 scrub starts 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[41044]: 4.7 scrub ok 2024-09-21T08:28:17.696 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[41044]: osdmap e303: 8 total, 8 up, 5 in 2024-09-21T08:28:17.697 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1265900146' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:28:17.697 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:17 smithi137 ceph-mon[41044]: from='client.45377 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:28:17.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:17.714 INFO:tasks.rados.rados.0.smithi137.stdout:1148: done (1 left) 2024-09-21T08:28:17.714 INFO:tasks.rados.rados.0.smithi137.stdout:1150: done (0 left) 2024-09-21T08:28:17.714 INFO:tasks.rados.rados.0.smithi137.stdout:1151: delete oid 37 current snap is 109 2024-09-21T08:28:17.716 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:28:17.720 INFO:tasks.rados.rados.0.smithi137.stdout:1151: done (0 left) 2024-09-21T08:28:17.720 INFO:tasks.rados.rados.0.smithi137.stdout:1152: read oid 28 snap 102 2024-09-21T08:28:17.720 INFO:tasks.rados.rados.0.smithi137.stdout:1152: expect (ObjNum 387 snap 94 seq_num 387) 2024-09-21T08:28:17.720 INFO:tasks.rados.rados.0.smithi137.stdout:1153: snap_create 2024-09-21T08:28:17.727 INFO:tasks.rados.rados.0.smithi137.stdout:1152: expect (ObjNum 387 snap 94 seq_num 387) 2024-09-21T08:28:17.737 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:17.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:17.738 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:17.804 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:17.804 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:17.822 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:17.822 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:17.838 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:17.838 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:17.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:17.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:17.854 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:17.860 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:17.861 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:17.870 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:17.870 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:17.886 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:17.886 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:17.902 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:17.903 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:17.919 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:17.919 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:17.962 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:17.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:17.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:17.984 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:18.085 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:18.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:18.107 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:18.107 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:18.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T08:28:18.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:18.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:18.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:18.247 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:18.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:18.369 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:18.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:18.370 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:18.427 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:18.470 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:18.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:18.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:18.496 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:18.597 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:18.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:18.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:18.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:18.718 INFO:tasks.rados.rados.0.smithi137.stdout:1152: done (1 left) 2024-09-21T08:28:18.718 INFO:tasks.rados.rados.0.smithi137.stdout:1153: done (0 left) 2024-09-21T08:28:18.718 INFO:tasks.rados.rados.0.smithi137.stdout:1154: delete oid 44 current snap is 110 2024-09-21T08:28:18.720 INFO:tasks.rados.rados.0.smithi137.stdout:1154: done (0 left) 2024-09-21T08:28:18.720 INFO:tasks.rados.rados.0.smithi137.stdout:1155: rollback oid 32 current snap is 110 2024-09-21T08:28:18.720 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 32 to 109 2024-09-21T08:28:18.720 INFO:tasks.rados.rados.0.smithi137.stdout:1156: write (excl) oid 24 current snap is 110 2024-09-21T08:28:18.720 INFO:tasks.rados.rados.0.smithi137.stdout:1156: seq_num 433 ranges {477431=698585,1905056=441808,3074744=202687} 2024-09-21T08:28:18.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:18.727 INFO:tasks.rados.rados.0.smithi137.stdout:1156: writing smithi13779221-24 from 477431 to 1176016 tid 1 2024-09-21T08:28:18.733 INFO:tasks.rados.rados.0.smithi137.stdout:1156: writing smithi13779221-24 from 1905056 to 2346864 tid 2 2024-09-21T08:28:18.739 INFO:tasks.rados.rados.0.smithi137.stdout:1156: writing smithi13779221-24 from 3074744 to 3277431 tid 3 2024-09-21T08:28:18.739 INFO:tasks.rados.rados.0.smithi137.stdout:1157: write (excl) oid 12 current snap is 110 2024-09-21T08:28:18.739 INFO:tasks.rados.rados.0.smithi137.stdout:1155: finishing rollback tid 0 to smithi13779221-32 2024-09-21T08:28:18.739 INFO:tasks.rados.rados.0.smithi137.stdout:1157: seq_num 434 ranges {494238=490187,1563538=701819,2818520=475718} 2024-09-21T08:28:18.744 INFO:tasks.rados.rados.0.smithi137.stdout:1157: writing smithi13779221-12 from 494238 to 984425 tid 1 2024-09-21T08:28:18.750 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:18.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:18.751 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:18.751 INFO:tasks.rados.rados.0.smithi137.stdout:1157: writing smithi13779221-12 from 1563538 to 2265357 tid 2 2024-09-21T08:28:18.757 INFO:tasks.rados.rados.0.smithi137.stdout:1157: writing smithi13779221-12 from 2818520 to 3294238 tid 3 2024-09-21T08:28:18.757 INFO:tasks.rados.rados.0.smithi137.stdout:1158: copy_from oid 37 from oid 3 current snap is 110 2024-09-21T08:28:18.757 INFO:tasks.rados.rados.0.smithi137.stdout:1155: finishing rollback tid 1 to smithi13779221-32 2024-09-21T08:28:18.757 INFO:tasks.rados.rados.0.smithi137.stdout:1155: finishing rollback tid 2 to smithi13779221-32 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 224 (ObjNum 378 snap 93 seq_num 378) dirty exists 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1156: finishing write tid 1 to smithi13779221-24 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1156: finishing write tid 2 to smithi13779221-24 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1156: finishing write tid 3 to smithi13779221-24 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1156: finishing write tid 4 to smithi13779221-24 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1156: finishing write tid 5 to smithi13779221-24 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1156: oid 24 updating version 0 to 293 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1156: oid 24 updating version 293 to 295 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1156: oid 24 version 295 is already newer than 292 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1156: oid 24 version 295 is already newer than 294 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 295 (ObjNum 433 snap 110 seq_num 433) dirty exists 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1156: left oid 24 (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1157: finishing write tid 1 to smithi13779221-12 2024-09-21T08:28:18.758 INFO:tasks.rados.rados.0.smithi137.stdout:1157: finishing write tid 2 to smithi13779221-12 2024-09-21T08:28:18.759 INFO:tasks.rados.rados.0.smithi137.stdout:1155: done (3 left) 2024-09-21T08:28:18.759 INFO:tasks.rados.rados.0.smithi137.stdout:1156: done (2 left) 2024-09-21T08:28:18.759 INFO:tasks.rados.rados.0.smithi137.stdout:1159: rollback oid 20 current snap is 110 2024-09-21T08:28:18.759 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 20 to 102 2024-09-21T08:28:18.759 INFO:tasks.rados.rados.0.smithi137.stdout:1160: write oid 29 current snap is 110 2024-09-21T08:28:18.759 INFO:tasks.rados.rados.0.smithi137.stdout:1160: seq_num 435 ranges {511045=565436,1622020=445477,2729590=556855} 2024-09-21T08:28:18.763 INFO:tasks.rados.rados.0.smithi137.stdout:1160: writing smithi13779221-29 from 511045 to 1076481 tid 1 2024-09-21T08:28:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[29634]: 3.13 scrub starts 2024-09-21T08:28:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[29634]: 3.13 scrub ok 2024-09-21T08:28:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[29634]: 6.4 scrub starts 2024-09-21T08:28:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[29634]: 6.4 scrub ok 2024-09-21T08:28:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[29634]: from='client.45377 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:28:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[29634]: osdmap e304: 8 total, 8 up, 5 in 2024-09-21T08:28:18.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4183477243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:18.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[41044]: 3.13 scrub starts 2024-09-21T08:28:18.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[41044]: 3.13 scrub ok 2024-09-21T08:28:18.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[41044]: 6.4 scrub starts 2024-09-21T08:28:18.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[41044]: 6.4 scrub ok 2024-09-21T08:28:18.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:18.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[41044]: from='client.45377 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:28:18.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[41044]: osdmap e304: 8 total, 8 up, 5 in 2024-09-21T08:28:18.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:18 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4183477243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:18.769 INFO:tasks.rados.rados.0.smithi137.stdout:1160: writing smithi13779221-29 from 1622020 to 2067497 tid 2 2024-09-21T08:28:18.776 INFO:tasks.rados.rados.0.smithi137.stdout:1160: writing smithi13779221-29 from 2729590 to 3286445 tid 3 2024-09-21T08:28:18.776 INFO:tasks.rados.rados.0.smithi137.stdout:1161: write oid 16 current snap is 110 2024-09-21T08:28:18.776 INFO:tasks.rados.rados.0.smithi137.stdout:1157: finishing write tid 3 to smithi13779221-12 2024-09-21T08:28:18.776 INFO:tasks.rados.rados.0.smithi137.stdout:1161: seq_num 436 ranges {527852=640685,1680502=589135,2757013=570839} 2024-09-21T08:28:18.783 INFO:tasks.rados.rados.0.smithi137.stdout:1161: writing smithi13779221-16 from 527852 to 1168537 tid 1 2024-09-21T08:28:18.788 INFO:tasks.rados.rados.0.smithi137.stdout:1161: writing smithi13779221-16 from 1680502 to 2269637 tid 2 2024-09-21T08:28:18.795 INFO:tasks.rados.rados.0.smithi137.stdout:1161: writing smithi13779221-16 from 2757013 to 3327852 tid 3 2024-09-21T08:28:18.795 INFO:tasks.rados.rados.0.smithi137.stdout:1162: snap_create 2024-09-21T08:28:18.795 INFO:tasks.rados.rados.0.smithi137.stdout:1157: finishing write tid 4 to smithi13779221-12 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1157: finishing write tid 5 to smithi13779221-12 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1157: oid 12 updating version 0 to 333 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1157: oid 12 updating version 333 to 335 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1157: oid 12 version 335 is already newer than 332 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1157: oid 12 version 335 is already newer than 334 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 335 (ObjNum 434 snap 110 seq_num 434) dirty exists 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1157: left oid 12 (ObjNum 434 snap 110 seq_num 434) 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1159: finishing rollback tid 0 to smithi13779221-20 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1159: finishing rollback tid 1 to smithi13779221-20 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1159: finishing rollback tid 2 to smithi13779221-20 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 378 (ObjNum 388 snap 94 seq_num 388) dirty exists 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1160: finishing write tid 1 to smithi13779221-29 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1158: finishing copy_from racing read to smithi13779221-37 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:1158: finishing copy_from to smithi13779221-37 2024-09-21T08:28:18.796 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 183 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:28:18.797 INFO:tasks.rados.rados.0.smithi137.stdout:1160: finishing write tid 2 to smithi13779221-29 2024-09-21T08:28:18.797 INFO:tasks.rados.rados.0.smithi137.stdout:1160: finishing write tid 3 to smithi13779221-29 2024-09-21T08:28:18.797 INFO:tasks.rados.rados.0.smithi137.stdout:1160: finishing write tid 4 to smithi13779221-29 2024-09-21T08:28:18.797 INFO:tasks.rados.rados.0.smithi137.stdout:1160: finishing write tid 5 to smithi13779221-29 2024-09-21T08:28:18.797 INFO:tasks.rados.rados.0.smithi137.stdout:1160: oid 29 updating version 0 to 502 2024-09-21T08:28:18.797 INFO:tasks.rados.rados.0.smithi137.stdout:1160: oid 29 version 502 is already newer than 501 2024-09-21T08:28:18.797 INFO:tasks.rados.rados.0.smithi137.stdout:1160: oid 29 version 502 is already newer than 500 2024-09-21T08:28:18.797 INFO:tasks.rados.rados.0.smithi137.stdout:1160: oid 29 updating version 502 to 503 2024-09-21T08:28:18.798 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 503 (ObjNum 435 snap 110 seq_num 435) dirty exists 2024-09-21T08:28:18.798 INFO:tasks.rados.rados.0.smithi137.stdout:1160: left oid 29 (ObjNum 435 snap 110 seq_num 435) 2024-09-21T08:28:18.798 INFO:tasks.rados.rados.0.smithi137.stdout:1161: finishing write tid 1 to smithi13779221-16 2024-09-21T08:28:18.798 INFO:tasks.rados.rados.0.smithi137.stdout:1161: finishing write tid 2 to smithi13779221-16 2024-09-21T08:28:18.799 INFO:tasks.rados.rados.0.smithi137.stdout:1161: finishing write tid 3 to smithi13779221-16 2024-09-21T08:28:18.799 INFO:tasks.rados.rados.0.smithi137.stdout:1161: finishing write tid 4 to smithi13779221-16 2024-09-21T08:28:18.799 INFO:tasks.rados.rados.0.smithi137.stdout:1161: finishing write tid 5 to smithi13779221-16 2024-09-21T08:28:18.799 INFO:tasks.rados.rados.0.smithi137.stdout:1161: oid 16 updating version 0 to 323 2024-09-21T08:28:18.799 INFO:tasks.rados.rados.0.smithi137.stdout:1161: oid 16 updating version 323 to 325 2024-09-21T08:28:18.799 INFO:tasks.rados.rados.0.smithi137.stdout:1161: oid 16 version 325 is already newer than 322 2024-09-21T08:28:18.799 INFO:tasks.rados.rados.0.smithi137.stdout:1161: oid 16 version 325 is already newer than 324 2024-09-21T08:28:18.799 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 325 (ObjNum 436 snap 110 seq_num 436) dirty exists 2024-09-21T08:28:18.799 INFO:tasks.rados.rados.0.smithi137.stdout:1161: left oid 16 (ObjNum 436 snap 110 seq_num 436) 2024-09-21T08:28:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:18 smithi165 ceph-mon[32893]: 3.13 scrub starts 2024-09-21T08:28:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:18 smithi165 ceph-mon[32893]: 3.13 scrub ok 2024-09-21T08:28:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:18 smithi165 ceph-mon[32893]: 6.4 scrub starts 2024-09-21T08:28:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:18 smithi165 ceph-mon[32893]: 6.4 scrub ok 2024-09-21T08:28:18.802 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:18 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:18.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:18 smithi165 ceph-mon[32893]: from='client.45377 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:28:18.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:18 smithi165 ceph-mon[32893]: osdmap e304: 8 total, 8 up, 5 in 2024-09-21T08:28:18.803 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:18 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4183477243' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:18.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:18.863 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:18.874 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:18.874 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:18.875 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:18.909 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:28:18.910 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:28:18.975 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:18.997 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:18.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:18.998 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:19.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:19.120 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:19.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:19.121 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:19.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:19.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:19.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:19.244 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:19.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:19.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:19.367 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:19.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:19.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:19.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:19.490 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:19.491 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:19.591 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:19.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:19.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:19.614 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:19.634 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:19 smithi137 ceph-mon[29634]: pgmap v362: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 159 MiB data, 556 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:28:19.634 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:19 smithi137 ceph-mon[29634]: osdmap e305: 8 total, 8 up, 5 in 2024-09-21T08:28:19.634 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:19 smithi137 ceph-mon[41044]: pgmap v362: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 159 MiB data, 556 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:28:19.634 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:19 smithi137 ceph-mon[41044]: osdmap e305: 8 total, 8 up, 5 in 2024-09-21T08:28:19.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T08:28:19.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:19.721 INFO:tasks.rados.rados.0.smithi137.stdout:1157: done (5 left) 2024-09-21T08:28:19.721 INFO:tasks.rados.rados.0.smithi137.stdout:1158: done (4 left) 2024-09-21T08:28:19.722 INFO:tasks.rados.rados.0.smithi137.stdout:1159: done (3 left) 2024-09-21T08:28:19.722 INFO:tasks.rados.rados.0.smithi137.stdout:1160: done (2 left) 2024-09-21T08:28:19.722 INFO:tasks.rados.rados.0.smithi137.stdout:1161: done (1 left) 2024-09-21T08:28:19.722 INFO:tasks.rados.rados.0.smithi137.stdout:1162: done (0 left) 2024-09-21T08:28:19.722 INFO:tasks.rados.rados.0.smithi137.stdout:1163: copy_from oid 6 from oid 33 current snap is 111 2024-09-21T08:28:19.722 INFO:tasks.rados.rados.0.smithi137.stdout:1164: snap_remove snap 102 2024-09-21T08:28:19.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:19.742 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:19.742 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:19.843 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:19.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:19.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:19.865 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:19.886 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:28:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "" 2024-09-21T08:28:19.887 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:28:19.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:19.988 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:19.988 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:19.988 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:19 smithi165 ceph-mon[32893]: pgmap v362: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 159 MiB data, 556 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:28:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:19 smithi165 ceph-mon[32893]: osdmap e305: 8 total, 8 up, 5 in 2024-09-21T08:28:20.090 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:20.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:20.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:20.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:20.213 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:20.235 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:20.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:20.236 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:20.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:20.358 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:20.359 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:20.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:20.483 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:20.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:20.483 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:20.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:20.608 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:20.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:20.609 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:20.639 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:20.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:20.732 INFO:tasks.rados.rados.0.smithi137.stdout:1164: done (1 left) 2024-09-21T08:28:20.732 INFO:tasks.rados.rados.0.smithi137.stdout:1165: snap_remove snap 104 2024-09-21T08:28:20.736 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:20.737 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:20.737 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:20.760 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:20 smithi137 ceph-mon[29634]: osdmap e306: 8 total, 8 up, 5 in 2024-09-21T08:28:20.760 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:20 smithi137 ceph-mon[29634]: pgmap v365: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 159 MiB data, 556 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:28:20.761 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:20 smithi137 ceph-mon[41044]: osdmap e306: 8 total, 8 up, 5 in 2024-09-21T08:28:20.761 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:20 smithi137 ceph-mon[41044]: pgmap v365: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 159 MiB data, 556 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:28:20.761 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:28:20 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:28:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:28:20.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:28:20.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:20.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:20.860 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:20.860 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:20.961 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:20.982 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:20.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:20.983 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:20 smithi165 ceph-mon[32893]: osdmap e306: 8 total, 8 up, 5 in 2024-09-21T08:28:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:20 smithi165 ceph-mon[32893]: pgmap v365: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 159 MiB data, 556 MiB used, 446 GiB / 447 GiB avail 2024-09-21T08:28:21.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:21.095 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:28:21.096 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:28:21.109 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:21.109 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:21.110 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:21.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:21.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:21.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:21.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:21.334 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:21.355 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:21.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:21.356 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:21.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:28:21.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:21.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:21.479 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:21.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:21.546 INFO:tasks.rados.rados.0.smithi137.stdout:1165: done (1 left) 2024-09-21T08:28:21.546 INFO:tasks.rados.rados.0.smithi137.stdout:1166: read oid 20 snap -1 2024-09-21T08:28:21.546 INFO:tasks.rados.rados.0.smithi137.stdout:1166: expect (ObjNum 388 snap 94 seq_num 388) 2024-09-21T08:28:21.546 INFO:tasks.rados.rados.0.smithi137.stdout:1167: snap_remove snap 109 2024-09-21T08:28:21.581 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:21.601 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:21.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:21.602 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:21.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:21.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:21.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:21.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:21.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T08:28:21.814 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:21.826 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:21.847 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:21.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:21.848 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:21.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:21.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:21.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:21.971 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:22.009 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:21 smithi137 ceph-mon[29634]: osdmap e307: 8 total, 8 up, 5 in 2024-09-21T08:28:22.009 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:21 smithi137 ceph-mon[29634]: osdmap e308: 8 total, 8 up, 5 in 2024-09-21T08:28:22.009 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:21 smithi137 ceph-mon[41044]: osdmap e307: 8 total, 8 up, 5 in 2024-09-21T08:28:22.009 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:21 smithi137 ceph-mon[41044]: osdmap e308: 8 total, 8 up, 5 in 2024-09-21T08:28:22.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:21 smithi165 ceph-mon[32893]: osdmap e307: 8 total, 8 up, 5 in 2024-09-21T08:28:22.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:21 smithi165 ceph-mon[32893]: osdmap e308: 8 total, 8 up, 5 in 2024-09-21T08:28:22.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:28:21 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:28:22.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:22.093 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:22.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:22.094 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:22.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:22.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:22.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:22.219 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:22.320 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:22.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:22.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:22.343 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:22.435 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:22.435 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:22.435 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:22.435 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:22.435 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:22.435 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:22.436 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:22.436 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:22.436 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:22.436 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:22.436 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:22.436 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:22.436 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:22.436 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:22.437 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:22.437 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:22.443 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:22.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:22.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:22.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:22.546 INFO:tasks.rados.rados.0.smithi137.stdout:1167: done (2 left) 2024-09-21T08:28:22.546 INFO:tasks.rados.rados.0.smithi137.stdout:1168: read oid 37 snap -1 2024-09-21T08:28:22.547 INFO:tasks.rados.rados.0.smithi137.stdout:1168: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:22.547 INFO:tasks.rados.rados.0.smithi137.stdout:1169: delete oid 19 current snap is 111 2024-09-21T08:28:22.547 INFO:tasks.rados.rados.0.smithi137.stdout:1163: finishing copy_from racing read to smithi13779221-6 2024-09-21T08:28:22.547 INFO:tasks.rados.rados.0.smithi137.stdout:1163: finishing copy_from to smithi13779221-6 2024-09-21T08:28:22.547 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 126 (ObjNum 302 snap 70 seq_num 302) dirty exists 2024-09-21T08:28:22.547 INFO:tasks.rados.rados.0.smithi137.stdout:1166: expect (ObjNum 388 snap 94 seq_num 388) 2024-09-21T08:28:22.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:22.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:22.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:22.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:22.653 INFO:tasks.rados.rados.0.smithi137.stdout:1163: done (3 left) 2024-09-21T08:28:22.653 INFO:tasks.rados.rados.0.smithi137.stdout:1166: done (2 left) 2024-09-21T08:28:22.653 INFO:tasks.rados.rados.0.smithi137.stdout:1169: done (1 left) 2024-09-21T08:28:22.654 INFO:tasks.rados.rados.0.smithi137.stdout:1170: snap_remove snap 106 2024-09-21T08:28:22.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:22.714 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:22.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:22.715 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:22.720 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:22.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:22.837 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:22.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:22.838 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:22.939 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:22.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:22.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:22.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:22 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:22 smithi165 ceph-mon[32893]: pgmap v368: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 176 MiB data, 609 MiB used, 446 GiB / 447 GiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 12 op/s 2024-09-21T08:28:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:22 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3935682540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:22 smithi165 ceph-mon[32893]: osdmap e309: 8 total, 8 up, 5 in 2024-09-21T08:28:23.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:22 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:23.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:22 smithi137 ceph-mon[29634]: pgmap v368: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 176 MiB data, 609 MiB used, 446 GiB / 447 GiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 12 op/s 2024-09-21T08:28:23.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:22 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3935682540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:23.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:22 smithi137 ceph-mon[29634]: osdmap e309: 8 total, 8 up, 5 in 2024-09-21T08:28:23.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:22 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:23.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:22 smithi137 ceph-mon[41044]: pgmap v368: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 176 MiB data, 609 MiB used, 446 GiB / 447 GiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 12 op/s 2024-09-21T08:28:23.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:22 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3935682540' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:23.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:22 smithi137 ceph-mon[41044]: osdmap e309: 8 total, 8 up, 5 in 2024-09-21T08:28:23.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:23.084 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:23.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:23.085 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:23.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:23.207 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:23.207 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:23.208 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:23.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:23.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:23.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:23.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:23.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:23.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:23.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:23.455 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:23.546 INFO:tasks.rados.rados.0.smithi137.stdout:1170: done (1 left) 2024-09-21T08:28:23.546 INFO:tasks.rados.rados.0.smithi137.stdout:1171: snap_remove snap 100 2024-09-21T08:28:23.557 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:23.578 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:23.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:23.579 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:23.623 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:28:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:28:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:28:23.623 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:28:23.623 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:28:23.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:23.702 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:23.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:23.703 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:23.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:23.823 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:23.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:23.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:23.827 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:23.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:23.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:23.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:23.952 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:24.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:24.073 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:24.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:24.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:24.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:24.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:24.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:24.200 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:24.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:24.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:24.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:24.324 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:24.348 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T08:28:24.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:24.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:24.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:24.453 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:24.553 INFO:tasks.rados.rados.0.smithi137.stdout:1171: done (1 left) 2024-09-21T08:28:24.553 INFO:tasks.rados.rados.0.smithi137.stdout:1172: read oid 15 snap -1 2024-09-21T08:28:24.553 INFO:tasks.rados.rados.0.smithi137.stdout:1172: expect deleted 2024-09-21T08:28:24.553 INFO:tasks.rados.rados.0.smithi137.stdout:1173: copy_from oid 25 from oid 9 current snap is 111 2024-09-21T08:28:24.553 INFO:tasks.rados.rados.0.smithi137.stdout:1174: read oid 32 snap -1 2024-09-21T08:28:24.553 INFO:tasks.rados.rados.0.smithi137.stdout:1174: expect (ObjNum 378 snap 93 seq_num 378) 2024-09-21T08:28:24.554 INFO:tasks.rados.rados.0.smithi137.stdout:1175: snap_create 2024-09-21T08:28:24.554 INFO:tasks.rados.rados.0.smithi137.stdout:1168: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:24.554 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:24.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:24.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:24.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:24 smithi137 ceph-mon[29634]: osdmap e310: 8 total, 8 up, 5 in 2024-09-21T08:28:24.605 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:24 smithi137 ceph-mon[41044]: osdmap e310: 8 total, 8 up, 5 in 2024-09-21T08:28:24.670 INFO:tasks.rados.rados.0.smithi137.stdout:1174: expect (ObjNum 378 snap 93 seq_num 378) 2024-09-21T08:28:24.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:24.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:24.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:24.716 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:24.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:24.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:24.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:24.841 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:24.843 INFO:tasks.rados.rados.0.smithi137.stdout:1173: finishing copy_from racing read to smithi13779221-25 2024-09-21T08:28:24.843 INFO:tasks.rados.rados.0.smithi137.stdout:1173: finishing copy_from to smithi13779221-25 2024-09-21T08:28:24.844 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 331 (ObjNum 426 snap 107 seq_num 426) dirty exists 2024-09-21T08:28:24.856 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:24.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:24.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:24.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:24.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:24 smithi165 ceph-mon[32893]: osdmap e310: 8 total, 8 up, 5 in 2024-09-21T08:28:25.068 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:25.090 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:25.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:25.091 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:25.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:28:25.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:25.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:25.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:25.215 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:25.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:25.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:25.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:25.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:25.376 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:28:25.376 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:28:25.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:28:25.376 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:28:25.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.376 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:336 7.9 7:9d4d36a0:::smithi13779221-27:head [copy-from ver 7] snapc 6=[6] ondisk+write+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:28:25.376 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:28:25.376 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 216.80338877200001, 2024-09-21T08:28:25.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.050439811000000001, 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 336 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569726+0000", 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569727+0000", 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4899999999999999e-06 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569733+0000", 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9120000000000003e-06 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569741+0000", 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4999999999999999e-06 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.569749+0000", 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2099999999999996e-06 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.583356+0000", 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013607731 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584178+0000", 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082153700000000001 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,7", 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.610330+0000", 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026152526999999998 2024-09-21T08:28:25.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.617047+0000", 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067167470000000003 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.619683+0000", 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026361039999999998 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620040+0000", 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00035616499999999998 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620096+0000", 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6922000000000002e-05 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.620165+0000", 2024-09-21T08:28:25.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8966000000000003e-05 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:337 7.9 7:9d4d36a0:::smithi13779221-27:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 216.79544894, 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.038882593999999999, 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.383 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 337 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577665+0000", 2024-09-21T08:28:25.384 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577668+0000", 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2960000000000001e-06 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577674+0000", 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5479999999999998e-06 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.385 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577687+0000", 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2829e-05 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.577698+0000", 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0832999999999999e-05 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584334+0000", 2024-09-21T08:28:25.386 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0066355900000000002 2024-09-21T08:28:25.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:25.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.584379+0000", 2024-09-21T08:28:25.387 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5575999999999999e-05 2024-09-21T08:28:25.387 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.387 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.387 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.387 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616393+0000", 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032013592 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616468+0000", 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5150999999999999e-05 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.616548+0000", 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0179000000000006e-05 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.388 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1594 7.16 7:6a4067f0:::smithi13779221-23:head [write 433680~466703] snapc 32=[32] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.326885+0000", 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.046229764, 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.62075412799999996, 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1594 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.389 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.326885+0000", 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.326885+0000", 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.326887+0000", 2024-09-21T08:28:25.390 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9489999999999999e-06 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327091+0000", 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020473400000000001 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327103+0000", 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.153e-05 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.391 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327108+0000", 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4489999999999998e-06 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327157+0000", 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8429e-05 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.327164+0000", 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6899999999999992e-06 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.392 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.925242+0000", 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59807755799999995 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928049+0000", 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028070149999999999 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.944633+0000", 2024-09-21T08:28:25.393 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016583665000000001 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.944714+0000", 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1816000000000004e-05 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.944826+0000", 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111823 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.394 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946814+0000", 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001987815 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946867+0000", 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3052e-05 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947560+0000", 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069266300000000005 2024-09-21T08:28:25.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947595+0000", 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5394e-05 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947639+0000", 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3545999999999999e-05 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.396 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1595 7.16 7:6a4067f0:::smithi13779221-23:head [write 1577796~455884] snapc 32=[32] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.332973+0000", 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.04014136699999, 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61484140200000004, 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1595 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.397 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.332973+0000", 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.332973+0000", 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.332975+0000", 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5680000000000001e-06 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333220+0000", 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024565600000000002 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333230+0000", 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0223e-05 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.399 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333236+0000", 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5230000000000003e-06 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333274+0000", 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8232999999999998e-05 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333280+0000", 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2040000000000002e-06 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.400 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928405+0000", 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59512471499999997 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928464+0000", 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9123000000000003e-05 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.945609+0000", 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017145059000000001 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.401 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.945660+0000", 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0562999999999997e-05 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.945757+0000", 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6949999999999998e-05 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947123+0000", 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001366173 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947230+0000", 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010691700000000001 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947760+0000", 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053026199999999999 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947791+0000", 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0923000000000002e-05 2024-09-21T08:28:25.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947814+0000", 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3309999999999999e-05 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1596 7.16 7:6a4067f0:::smithi13779221-23:head [setxattr _header (57) in=7b,truncate 2033680] snapc 32=[32] ondisk+write+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:28:25.404 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.333246+0000", 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.03986828999999, 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61472346600000005, 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1596 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.406 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333246+0000", 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333246+0000", 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.407 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333247+0000", 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1299999999999998e-07 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333251+0000", 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1699999999999999e-06 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333256+0000", 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3340000000000001e-06 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.408 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333262+0000", 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2780000000000001e-06 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333310+0000", 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8572000000000002e-05 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333320+0000", 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8099999999999992e-06 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.409 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928508+0000", 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59518784499999999 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928553+0000", 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4545000000000001e-05 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946225+0000", 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017672832999999999 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946285+0000", 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9466999999999998e-05 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946398+0000", 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011340000000000001 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.411 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947240+0000", 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00084159800000000004 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947726+0000", 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048660799999999997 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947904+0000", 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017706899999999999 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947931+0000", 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7742e-05 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.947970+0000", 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8281999999999997e-05 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.413 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1597 7.16 7:6a4067f0:::smithi13779221-23:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e183)", 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:17.333275+0000", 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 128.03983898600001, 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.61483585500000004, 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1597 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.414 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333275+0000", 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333275+0000", 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333276+0000", 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2099999999999996e-07 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333278+0000", 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4229999999999999e-06 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333285+0000", 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0820000000000001e-06 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333291+0000", 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2089999999999996e-06 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333334+0000", 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3482999999999997e-05 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.333338+0000", 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0169999999999998e-06 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928654+0000", 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.59531576200000003 2024-09-21T08:28:25.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.928713+0000", 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8971999999999997e-05 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946717+0000", 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018003894999999999 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.946799+0000", 2024-09-21T08:28:25.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2521000000000005e-05 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.948007+0000", 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012079180000000001 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.948053+0000", 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5661000000000002e-05 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:17.948111+0000", 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8291000000000001e-05 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1742 7.d 7:b31d99ce:::smithi13779221-21:head [copy-from ver 99] snapc 39=[39] ondisk+write+known_if_redirected+supports_pool_eio e197)", 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:30.594911+0000", 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 114.778203045, 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021953376, 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.420 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1742 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594911+0000", 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594911+0000", 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594913+0000", 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2190000000000001e-06 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594919+0000", 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2049999999999999e-06 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.594930+0000", 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1416e-05 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595001+0000", 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0678000000000003e-05 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595054+0000", 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2966000000000001e-05 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.595167+0000", 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113167 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.605259+0000", 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010091829 2024-09-21T08:28:25.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.610873+0000", 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056143 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616680+0000", 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058072159999999996 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616794+0000", 2024-09-21T08:28:25.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113318 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616826+0000", 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2264999999999997e-05 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:30.616865+0000", 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8797000000000001e-05 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1910 7.f 7:f1c8dcbe:::smithi13779221-1:head [write 423609~701524] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.544612+0000", 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 96.828502263000004, 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0451588970000001, 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1910 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.544612+0000", 2024-09-21T08:28:25.428 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.544612+0000", 2024-09-21T08:28:25.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.544614+0000", 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7349999999999999e-06 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545147+0000", 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053351400000000002 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545158+0000", 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0241000000000001e-05 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545168+0000", 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0047000000000001e-05 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545223+0000", 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5858e-05 2024-09-21T08:28:25.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545235+0000", 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1525000000000001e-05 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.568108+0000", 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.022872655 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.568937+0000", 2024-09-21T08:28:25.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00082958499999999998 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585576+0000", 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016639025000000002 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585673+0000", 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6496999999999994e-05 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585866+0000", 2024-09-21T08:28:25.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000193019 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589564+0000", 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036982180000000001 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589584+0000", 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0097000000000001e-05 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589637+0000", 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2794000000000003e-05 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589704+0000", 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7438e-05 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589771+0000", 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6649000000000005e-05 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.435 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1911 7.f 7:f1c8dcbe:::smithi13779221-1:head [write 1628332~472089] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.545246+0000", 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 96.827868335000005, 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0460972529999999, 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.436 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1911 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545246+0000", 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545246+0000", 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545247+0000", 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5799999999999998e-07 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545571+0000", 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000323703 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545581+0000", 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0515e-05 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545588+0000", 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8689999999999998e-06 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545632+0000", 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4149e-05 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.545641+0000", 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4509999999999992e-06 2024-09-21T08:28:25.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569011+0000", 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0233707439999999 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569071+0000", 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9499000000000001e-05 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587283+0000", 2024-09-21T08:28:25.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018211871000000001 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587400+0000", 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117216 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587560+0000", 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016029999999999999 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589591+0000", 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002030853 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589853+0000", 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00026147699999999998 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591174+0000", 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001321046 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591220+0000", 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6112999999999997e-05 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591343+0000", 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012348900000000001 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1912 7.f 7:f1c8dcbe:::smithi13779221-1:head [write 2648987~174622] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:28:25.443 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.549709+0000", 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 96.823405554000004, 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.041840002, 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1912 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549709+0000", 2024-09-21T08:28:25.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549709+0000", 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549712+0000", 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4740000000000001e-06 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549841+0000", 2024-09-21T08:28:25.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012874699999999999 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549873+0000", 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2208000000000001e-05 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549885+0000", 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1878e-05 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549977+0000", 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2044000000000002e-05 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549988+0000", 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0723999999999999e-05 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569102+0000", 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.019114064 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:25.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569150+0000", 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7576e-05 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.588268+0000", 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019118133999999998 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.588389+0000", 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012087999999999999 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.588544+0000", 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015529300000000001 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589784+0000", 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001239696 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589910+0000", 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000126747 2024-09-21T08:28:25.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591404+0000", 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014933500000000001 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591452+0000", 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8714000000000001e-05 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591549+0000", 2024-09-21T08:28:25.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6472999999999996e-05 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1913 7.f 7:f1c8dcbe:::smithi13779221-1:head [setxattr _header (56) in=7b,truncate 2823609] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.550029+0000", 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 96.823085293999995, 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0418481909999999, 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1913 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550029+0000", 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550029+0000", 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550030+0000", 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1100000000000004e-07 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550036+0000", 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5999999999999997e-06 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550044+0000", 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6090000000000001e-06 2024-09-21T08:28:25.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550051+0000", 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5150000000000003e-06 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550105+0000", 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4258999999999997e-05 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550113+0000", 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7549999999999994e-06 2024-09-21T08:28:25.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569285+0000", 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0191723500000001 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569337+0000", 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2369e-05 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589107+0000", 2024-09-21T08:28:25.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019769201 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589204+0000", 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7133999999999995e-05 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589314+0000", 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110308 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.589972+0000", 2024-09-21T08:28:25.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000658079 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.590152+0000", 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017941499999999999 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591636+0000", 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0014841979999999999 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591704+0000", 2024-09-21T08:28:25.457 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8051000000000003e-05 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591877+0000", 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000173437 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1914 7.f 7:f1c8dcbe:::smithi13779221-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.550141+0000", 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 96.822973567000005, 2024-09-21T08:28:25.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.042047948, 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1914 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550141+0000", 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550141+0000", 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550142+0000", 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1099999999999995e-07 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550145+0000", 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2550000000000002e-06 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550153+0000", 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0439999999999996e-06 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550158+0000", 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2290000000000002e-06 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550217+0000", 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9219999999999999e-05 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550224+0000", 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.199e-06 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569368+0000", 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.019143197 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.569416+0000", 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8168999999999999e-05 2024-09-21T08:28:25.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.590013+0000", 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020597133 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.590102+0000", 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9364999999999996e-05 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.591921+0000", 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0018190389999999999 2024-09-21T08:28:25.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.592009+0000", 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8096000000000005e-05 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.592189+0000", 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017929099999999999 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2114 7.4 7:26d6e516:::smithi13779221-33:head [copy-from ver 279] snapc 4a=[4a,49,48,47,45,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.523396+0000", 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 83.849718108000005, 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030629719, 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2114 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523396+0000", 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523396+0000", 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523397+0000", 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4199999999999996e-07 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523401+0000", 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5970000000000002e-06 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523406+0000", 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0159999999999999e-06 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523411+0000", 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4070000000000003e-06 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523463+0000", 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1997999999999999e-05 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523469+0000", 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6810000000000001e-06 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.528702+0000", 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0052329919999999997 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.528799+0000", 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6323000000000006e-05 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,5", 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540822+0000", 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012022812000000001 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.547580+0000", 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0067587519999999998 2024-09-21T08:28:25.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553592+0000", 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0060113479999999997 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553788+0000", 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00019646900000000001 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553858+0000", 2024-09-21T08:28:25.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9536999999999995e-05 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.554026+0000", 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000168245 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2115 7.4 7:26d6e516:::smithi13779221-33:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e228)", 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.523423+0000", 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 83.849691668999995, 2024-09-21T08:28:25.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020869774000000001, 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2115 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523423+0000", 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.472 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523423+0000", 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523423+0000", 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4999999999999998e-07 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523426+0000", 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.633e-06 2024-09-21T08:28:25.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523472+0000", 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5775999999999997e-05 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.523476+0000", 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6460000000000001e-06 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.529003+0000", 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055263200000000004 2024-09-21T08:28:25.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:25.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.529053+0000", 2024-09-21T08:28:25.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0130000000000003e-05 2024-09-21T08:28:25.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:25.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.544083+0000", 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015030646 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.544186+0000", 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102518 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.544292+0000", 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000106655 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.477 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2215 7.12 7:4ccee2c4:::smithi13779221-50:head [copy-from ver 311] snapc 4e=[4e,4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544476+0000", 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 74.828638322000003, 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026654285999999999, 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2215 2024-09-21T08:28:25.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544476+0000", 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544476+0000", 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544477+0000", 2024-09-21T08:28:25.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6400000000000002e-07 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544480+0000", 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2179999999999999e-06 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544485+0000", 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7620000000000003e-06 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544489+0000", 2024-09-21T08:28:25.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7180000000000004e-06 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544550+0000", 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0439000000000002e-05 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544557+0000", 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1069999999999999e-06 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.547211+0000", 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026542789999999998 2024-09-21T08:28:25.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.547316+0000", 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104938 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.562697+0000", 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015381183 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.568461+0000", 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057634590000000003 2024-09-21T08:28:25.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.569259+0000", 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00079830400000000001 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.571009+0000", 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0017497599999999999 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.571062+0000", 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2991999999999997e-05 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.571130+0000", 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8563000000000001e-05 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2216 7.12 7:4ccee2c4:::smithi13779221-50:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544502+0000", 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 74.828612512999996, 2024-09-21T08:28:25.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022611549000000002, 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2216 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544502+0000", 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544502+0000", 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544502+0000", 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5199999999999996e-07 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544505+0000", 2024-09-21T08:28:25.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6230000000000001e-06 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544511+0000", 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8579999999999996e-06 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544516+0000", 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1649999999999997e-06 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544596+0000", 2024-09-21T08:28:25.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9684000000000005e-05 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544602+0000", 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2249999999999997e-06 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.547457+0000", 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028550049999999999 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.547501+0000", 2024-09-21T08:28:25.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3945e-05 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.566950+0000", 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019448564000000002 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.567030+0000", 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0321999999999994e-05 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.567113+0000", 2024-09-21T08:28:25.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3505999999999996e-05 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2222 7.12 7:4d793d31:::smithi13779221-9:head [append 0~0] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.527713+0000", 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 73.845401065000004, 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022209310999999999, 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.490 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2222 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527713+0000", 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527713+0000", 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527716+0000", 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6129999999999998e-06 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527723+0000", 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9059999999999996e-06 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527739+0000", 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6589e-05 2024-09-21T08:28:25.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527748+0000", 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3119999999999994e-06 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527857+0000", 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010948800000000001 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527864+0000", 2024-09-21T08:28:25.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2579999999999998e-06 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.533682+0000", 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058173829999999998 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.533784+0000", 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102514 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:25.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.533993+0000", 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020875399999999999 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.541706+0000", 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0077127029999999996 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.543228+0000", 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001522203 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.549783+0000", 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065552730000000003 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.549860+0000", 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6810000000000005e-05 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.549923+0000", 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2504999999999999e-05 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.496 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2223 7.12 7:4d793d31:::smithi13779221-9:head [rollback 4d] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.527776+0000", 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 73.845338256000005, 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027805044000000001, 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2223 2024-09-21T08:28:25.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527776+0000", 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527776+0000", 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527777+0000", 2024-09-21T08:28:25.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5000000000000001e-07 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527780+0000", 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4580000000000002e-06 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527789+0000", 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1780000000000003e-06 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.499 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527794+0000", 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6319999999999996e-06 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527912+0000", 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000117972 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527918+0000", 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0680000000000002e-06 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.539337+0000", 2024-09-21T08:28:25.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011418991999999999 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.539460+0000", 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012270400000000001 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.539688+0000", 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022830499999999999 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.541712+0000", 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020237190000000002 2024-09-21T08:28:25.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.544647+0000", 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002934688 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555467+0000", 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010820299 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555525+0000", 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8349e-05 2024-09-21T08:28:25.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555581+0000", 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5829999999999999e-05 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2224 7.12 7:4d793d31:::smithi13779221-9:head [append 0~0] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.527802+0000", 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 73.845312278999998, 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027898684999999999, 2024-09-21T08:28:25.503 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2224 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527802+0000", 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527802+0000", 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527803+0000", 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6400000000000001e-07 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527806+0000", 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.056e-06 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527811+0000", 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5879999999999997e-06 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527816+0000", 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9930000000000003e-06 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527939+0000", 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012283699999999999 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.527945+0000", 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2900000000000002e-06 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.540300+0000", 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012355457 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.540404+0000", 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000104356 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.540576+0000", 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017131899999999999 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.541717+0000", 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001140907 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.544877+0000", 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0031601649999999999 2024-09-21T08:28:25.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555618+0000", 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010741550000000001 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555665+0000", 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6968000000000002e-05 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555701+0000", 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5435000000000002e-05 2024-09-21T08:28:25.509 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2828 7.d 7:b0b32695:::smithi13779221-19:head [copy-from ver 108] snapc 64=[64] ondisk+write+known_if_redirected+supports_pool_eio e287)", 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:00.639263+0000", 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 24.733851173000001, 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024131550000000002, 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:25.510 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2828 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.639263+0000", 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.639263+0000", 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.639266+0000", 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3009999999999998e-06 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.639278+0000", 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.166e-05 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.639297+0000", 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8464000000000001e-05 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.512 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.639307+0000", 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0552e-05 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.639382+0000", 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5240000000000005e-05 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.639388+0000", 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5500000000000002e-06 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.643788+0000", 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044001550000000002 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.643878+0000", 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9497e-05 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.648659+0000", 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004781614 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.655855+0000", 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0071953490000000002 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.663188+0000", 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073332550000000003 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.663287+0000", 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8913000000000007e-05 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.663348+0000", 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1237999999999999e-05 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:00.663395+0000", 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6761999999999997e-05 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:25.516 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:28:25.525 INFO:tasks.thrashosds.thrasher:in_osds: [0, 2, 5, 7, 1] out_osds: [4, 6, 3] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:28:25.525 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:28:25.525 INFO:tasks.thrashosds.thrasher:Removing osd 0, in_osds are: [0, 2, 5, 7, 1] 2024-09-21T08:28:25.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd out 0 2024-09-21T08:28:25.537 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:25.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:25.538 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:25.551 INFO:tasks.rados.rados.0.smithi137.stdout:1168: done (4 left) 2024-09-21T08:28:25.551 INFO:tasks.rados.rados.0.smithi137.stdout:1172: done (3 left) 2024-09-21T08:28:25.551 INFO:tasks.rados.rados.0.smithi137.stdout:1173: done (2 left) 2024-09-21T08:28:25.552 INFO:tasks.rados.rados.0.smithi137.stdout:1174: done (1 left) 2024-09-21T08:28:25.552 INFO:tasks.rados.rados.0.smithi137.stdout:1175: done (0 left) 2024-09-21T08:28:25.552 INFO:tasks.rados.rados.0.smithi137.stdout:1176: snap_create 2024-09-21T08:28:25.552 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:28:25.639 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:25.665 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:25.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:25.666 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:25.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:25.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:25.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:25.793 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:25.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:25.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:25.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:25.921 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:25 smithi165 ceph-mon[32893]: pgmap v371: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 176 MiB data, 609 MiB used, 446 GiB / 447 GiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 12 op/s 2024-09-21T08:28:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:25 smithi165 ceph-mon[32893]: osdmap e311: 8 total, 8 up, 5 in 2024-09-21T08:28:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:25 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3326626894' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:25 smithi165 ceph-mon[32893]: from='client.45395 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:26.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:25 smithi137 ceph-mon[29634]: pgmap v371: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 176 MiB data, 609 MiB used, 446 GiB / 447 GiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 12 op/s 2024-09-21T08:28:26.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:25 smithi137 ceph-mon[29634]: osdmap e311: 8 total, 8 up, 5 in 2024-09-21T08:28:26.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:25 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3326626894' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:26.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:25 smithi137 ceph-mon[29634]: from='client.45395 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:26.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:25 smithi137 ceph-mon[41044]: pgmap v371: 168 pgs: 4 active+clean+snaptrim, 18 active+clean+snaptrim_wait, 146 active+clean; 176 MiB data, 609 MiB used, 446 GiB / 447 GiB avail; 1.7 MiB/s rd, 3.6 MiB/s wr, 12 op/s 2024-09-21T08:28:26.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:25 smithi137 ceph-mon[41044]: osdmap e311: 8 total, 8 up, 5 in 2024-09-21T08:28:26.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:25 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3326626894' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:26.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:25 smithi137 ceph-mon[41044]: from='client.45395 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:26.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:26.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:26.049 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:26.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:26.171 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:26.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:26.273 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:26.294 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:26.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:26.295 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:26.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:26.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:26.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:26.417 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:26.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:26.539 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:26.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:26.540 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:26.554 INFO:tasks.rados.rados.0.smithi137.stdout:1176: done (0 left) 2024-09-21T08:28:26.554 INFO:tasks.rados.rados.0.smithi137.stdout:1177: rollback oid 32 current snap is 113 2024-09-21T08:28:26.554 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 32 to 111 2024-09-21T08:28:26.554 INFO:tasks.rados.rados.0.smithi137.stdout:1178: read oid 19 snap -1 2024-09-21T08:28:26.554 INFO:tasks.rados.rados.0.smithi137.stdout:1178: expect deleted 2024-09-21T08:28:26.554 INFO:tasks.rados.rados.0.smithi137.stdout:1179: copy_from oid 10 from oid 11 current snap is 113 2024-09-21T08:28:26.554 INFO:tasks.rados.rados.0.smithi137.stdout:1180: snap_remove snap 110 2024-09-21T08:28:26.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:26.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:26.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:26.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:26.729 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:26.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:26.764 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:26.785 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:26.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:26.787 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:26.852 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:26 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:26.852 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:26 smithi137 ceph-mon[29634]: from='client.45395 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:26.852 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:26 smithi137 ceph-mon[29634]: osdmap e312: 8 total, 8 up, 5 in 2024-09-21T08:28:26.852 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:26 smithi137 ceph-mon[29634]: osdmap e313: 8 total, 8 up, 5 in 2024-09-21T08:28:26.852 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:26 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:26.853 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:26 smithi137 ceph-mon[41044]: from='client.45395 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:26.853 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:26 smithi137 ceph-mon[41044]: osdmap e312: 8 total, 8 up, 5 in 2024-09-21T08:28:26.853 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:26 smithi137 ceph-mon[41044]: osdmap e313: 8 total, 8 up, 5 in 2024-09-21T08:28:26.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:26.908 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:26.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:26.909 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:27.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:26 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:26 smithi165 ceph-mon[32893]: from='client.45395 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:26 smithi165 ceph-mon[32893]: osdmap e312: 8 total, 8 up, 5 in 2024-09-21T08:28:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:26 smithi165 ceph-mon[32893]: osdmap e313: 8 total, 8 up, 5 in 2024-09-21T08:28:27.033 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:27.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:27.034 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:27.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:27.157 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:27.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:27.158 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:27.258 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:27.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:27.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:27.283 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:27.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:27.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:27.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:27.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:27.451 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:27.451 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:27.468 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:27.468 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:27.484 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:27.485 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:27.500 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:27.501 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:27.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:27.518 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:27.518 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:27.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:27.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:27.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:27.534 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:27.535 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:27.551 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:27.551 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:27.555 INFO:tasks.rados.rados.0.smithi137.stdout:1180: done (3 left) 2024-09-21T08:28:27.555 INFO:tasks.rados.rados.0.smithi137.stdout:1181: write oid 12 current snap is 113 2024-09-21T08:28:27.555 INFO:tasks.rados.rados.0.smithi137.stdout:1181: seq_num 438 ranges {561466=791183,2081113=592804,3095506=265960} 2024-09-21T08:28:27.562 INFO:tasks.rados.rados.0.smithi137.stdout:1181: writing smithi13779221-12 from 561466 to 1352649 tid 1 2024-09-21T08:28:27.570 INFO:tasks.rados.rados.0.smithi137.stdout:1181: writing smithi13779221-12 from 2081113 to 2673917 tid 2 2024-09-21T08:28:27.574 INFO:tasks.rados.rados.0.smithi137.stdout:1181: writing smithi13779221-12 from 3095506 to 3361466 tid 3 2024-09-21T08:28:27.575 INFO:tasks.rados.rados.0.smithi137.stdout:1182: read oid 33 snap 108 2024-09-21T08:28:27.575 INFO:tasks.rados.rados.0.smithi137.stdout:1182: expect (ObjNum 302 snap 70 seq_num 302) 2024-09-21T08:28:27.575 INFO:tasks.rados.rados.0.smithi137.stdout:1183: write oid 26 current snap is 113 2024-09-21T08:28:27.575 INFO:tasks.rados.rados.0.smithi137.stdout:1183: seq_num 439 ranges {578273=466432,1739595=452815,2722929=655344} 2024-09-21T08:28:27.575 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:27.575 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:27.580 INFO:tasks.rados.rados.0.smithi137.stdout:1183: writing smithi13779221-26 from 578273 to 1044705 tid 1 2024-09-21T08:28:27.587 INFO:tasks.rados.rados.0.smithi137.stdout:1183: writing smithi13779221-26 from 1739595 to 2192410 tid 2 2024-09-21T08:28:27.595 INFO:tasks.rados.rados.0.smithi137.stdout:1183: writing smithi13779221-26 from 2722929 to 3378273 tid 3 2024-09-21T08:28:27.595 INFO:tasks.rados.rados.0.smithi137.stdout:1184: copy_from oid 30 from oid 23 current snap is 113 2024-09-21T08:28:27.595 INFO:tasks.rados.rados.0.smithi137.stdout:1177: finishing rollback tid 0 to smithi13779221-32 2024-09-21T08:28:27.595 INFO:tasks.rados.rados.0.smithi137.stdout:1177: finishing rollback tid 1 to smithi13779221-32 2024-09-21T08:28:27.595 INFO:tasks.rados.rados.0.smithi137.stdout:1177: finishing rollback tid 2 to smithi13779221-32 2024-09-21T08:28:27.595 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 227 (ObjNum 378 snap 93 seq_num 378) dirty exists 2024-09-21T08:28:27.595 INFO:tasks.rados.rados.0.smithi137.stdout:1179: finishing copy_from racing read to smithi13779221-10 2024-09-21T08:28:27.595 INFO:tasks.rados.rados.0.smithi137.stdout:1179: finishing copy_from to smithi13779221-10 2024-09-21T08:28:27.596 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 218 (ObjNum 385 snap 94 seq_num 385) dirty exists 2024-09-21T08:28:27.596 INFO:tasks.rados.rados.0.smithi137.stdout:1181: finishing write tid 1 to smithi13779221-12 2024-09-21T08:28:27.596 INFO:tasks.rados.rados.0.smithi137.stdout:1181: finishing write tid 2 to smithi13779221-12 2024-09-21T08:28:27.596 INFO:tasks.rados.rados.0.smithi137.stdout:1177: done (6 left) 2024-09-21T08:28:27.596 INFO:tasks.rados.rados.0.smithi137.stdout:1178: done (5 left) 2024-09-21T08:28:27.596 INFO:tasks.rados.rados.0.smithi137.stdout:1179: done (4 left) 2024-09-21T08:28:27.596 INFO:tasks.rados.rados.0.smithi137.stdout:1185: delete oid 1 current snap is 113 2024-09-21T08:28:27.596 INFO:tasks.rados.rados.0.smithi137.stdout:1182: expect (ObjNum 302 snap 70 seq_num 302) 2024-09-21T08:28:27.629 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:27.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:27.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:27.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:27.752 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[29634]: pgmap v374: 168 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 141 active+clean; 167 MiB data, 598 MiB used, 446 GiB / 447 GiB avail; 6.5 MiB/s rd, 771 KiB/s wr, 7 op/s 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/577574469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[29634]: osdmap e314: 8 total, 8 up, 5 in 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[41044]: pgmap v374: 168 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 141 active+clean; 167 MiB data, 598 MiB used, 446 GiB / 447 GiB avail; 6.5 MiB/s rd, 771 KiB/s wr, 7 op/s 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:28:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:28:27.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:27.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/577574469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:27.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-mon[41044]: osdmap e314: 8 total, 8 up, 5 in 2024-09-21T08:28:27.769 INFO:tasks.rados.rados.0.smithi137.stdout:1181: finishing write tid 3 to smithi13779221-12 2024-09-21T08:28:27.770 INFO:tasks.rados.rados.0.smithi137.stdout:1181: finishing write tid 4 to smithi13779221-12 2024-09-21T08:28:27.770 INFO:tasks.rados.rados.0.smithi137.stdout:1181: finishing write tid 5 to smithi13779221-12 2024-09-21T08:28:27.770 INFO:tasks.rados.rados.0.smithi137.stdout:1181: oid 12 updating version 0 to 339 2024-09-21T08:28:27.770 INFO:tasks.rados.rados.0.smithi137.stdout:1181: oid 12 version 339 is already newer than 337 2024-09-21T08:28:27.770 INFO:tasks.rados.rados.0.smithi137.stdout:1181: oid 12 version 339 is already newer than 336 2024-09-21T08:28:27.770 INFO:tasks.rados.rados.0.smithi137.stdout:1181: oid 12 version 339 is already newer than 338 2024-09-21T08:28:27.770 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 339 (ObjNum 438 snap 113 seq_num 438) dirty exists 2024-09-21T08:28:27.770 INFO:tasks.rados.rados.0.smithi137.stdout:1181: left oid 12 (ObjNum 438 snap 113 seq_num 438) 2024-09-21T08:28:27.770 INFO:tasks.rados.rados.0.smithi137.stdout:1183: finishing write tid 1 to smithi13779221-26 2024-09-21T08:28:27.770 INFO:tasks.rados.rados.0.smithi137.stdout:1183: finishing write tid 2 to smithi13779221-26 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:1183: finishing write tid 3 to smithi13779221-26 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:1183: finishing write tid 4 to smithi13779221-26 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:1183: finishing write tid 5 to smithi13779221-26 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:1183: oid 26 updating version 0 to 306 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:1183: oid 26 updating version 306 to 307 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:1183: oid 26 version 307 is already newer than 304 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:1183: oid 26 version 307 is already newer than 305 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 307 (ObjNum 439 snap 113 seq_num 439) dirty exists 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:1183: left oid 26 (ObjNum 439 snap 113 seq_num 439) 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:1184: finishing copy_from racing read to smithi13779221-30 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:1184: finishing copy_from to smithi13779221-30 2024-09-21T08:28:27.771 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 187 (ObjNum 428 snap 108 seq_num 428) dirty exists 2024-09-21T08:28:27.772 INFO:tasks.rados.rados.0.smithi137.stdout:1181: done (4 left) 2024-09-21T08:28:27.772 INFO:tasks.rados.rados.0.smithi137.stdout:1182: done (3 left) 2024-09-21T08:28:27.772 INFO:tasks.rados.rados.0.smithi137.stdout:1183: done (2 left) 2024-09-21T08:28:27.772 INFO:tasks.rados.rados.0.smithi137.stdout:1184: done (1 left) 2024-09-21T08:28:27.772 INFO:tasks.rados.rados.0.smithi137.stdout:1185: done (0 left) 2024-09-21T08:28:27.772 INFO:tasks.rados.rados.0.smithi137.stdout:1186: write oid 41 current snap is 113 2024-09-21T08:28:27.772 INFO:tasks.rados.rados.0.smithi137.stdout:1186: seq_num 441 ranges {611887=733283,1972912=456484,2894128=517759} 2024-09-21T08:28:27.774 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:27.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:27.775 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:27.778 INFO:tasks.rados.rados.0.smithi137.stdout:1186: writing smithi13779221-41 from 611887 to 1345170 tid 1 2024-09-21T08:28:27.784 INFO:tasks.rados.rados.0.smithi137.stdout:1186: writing smithi13779221-41 from 1972912 to 2429396 tid 2 2024-09-21T08:28:27.789 INFO:tasks.rados.rados.0.smithi137.stdout:1186: writing smithi13779221-41 from 2894128 to 3411887 tid 3 2024-09-21T08:28:27.790 INFO:tasks.rados.rados.0.smithi137.stdout:1187: snap_create 2024-09-21T08:28:27.790 INFO:tasks.rados.rados.0.smithi137.stdout:1186: finishing write tid 1 to smithi13779221-41 2024-09-21T08:28:27.790 INFO:tasks.rados.rados.0.smithi137.stdout:1186: finishing write tid 2 to smithi13779221-41 2024-09-21T08:28:27.792 INFO:tasks.rados.rados.0.smithi137.stdout:1186: finishing write tid 3 to smithi13779221-41 2024-09-21T08:28:27.792 INFO:tasks.rados.rados.0.smithi137.stdout:1186: finishing write tid 4 to smithi13779221-41 2024-09-21T08:28:27.792 INFO:tasks.rados.rados.0.smithi137.stdout:1186: finishing write tid 5 to smithi13779221-41 2024-09-21T08:28:27.792 INFO:tasks.rados.rados.0.smithi137.stdout:1186: oid 41 updating version 0 to 309 2024-09-21T08:28:27.792 INFO:tasks.rados.rados.0.smithi137.stdout:1186: oid 41 updating version 309 to 310 2024-09-21T08:28:27.792 INFO:tasks.rados.rados.0.smithi137.stdout:1186: oid 41 version 310 is already newer than 308 2024-09-21T08:28:27.792 INFO:tasks.rados.rados.0.smithi137.stdout:1186: oid 41 updating version 310 to 311 2024-09-21T08:28:27.792 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 311 (ObjNum 441 snap 113 seq_num 441) dirty exists 2024-09-21T08:28:27.792 INFO:tasks.rados.rados.0.smithi137.stdout:1186: left oid 41 (ObjNum 441 snap 113 seq_num 441) 2024-09-21T08:28:27.876 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:27.898 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:27.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:27.899 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:28.000 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:27 smithi165 ceph-mon[32893]: pgmap v374: 168 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 141 active+clean; 167 MiB data, 598 MiB used, 446 GiB / 447 GiB avail; 6.5 MiB/s rd, 771 KiB/s wr, 7 op/s 2024-09-21T08:28:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:27 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:28:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:27 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:28:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:27 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:28:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:27 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:27 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/577574469' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:27 smithi165 ceph-mon[32893]: osdmap e314: 8 total, 8 up, 5 in 2024-09-21T08:28:28.022 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:28.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:28.023 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:28.124 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:28.145 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:28.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:28.146 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:28.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:28.268 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:27 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29611]: 2024-09-21T08:28:27.871+0000 7f1058bb7640 -1 mon.a@0(leader).osd e314 definitely_dead 0 2024-09-21T08:28:28.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:28.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:28.270 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:28.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:28.392 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:28.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:28.393 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:28.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T08:28:28.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:28.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:28.516 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:28.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:28.582 INFO:tasks.rados.rados.0.smithi137.stdout:1186: done (1 left) 2024-09-21T08:28:28.583 INFO:tasks.rados.rados.0.smithi137.stdout:1187: done (0 left) 2024-09-21T08:28:28.583 INFO:tasks.rados.rados.0.smithi137.stdout:1188: read oid 37 snap -1 2024-09-21T08:28:28.583 INFO:tasks.rados.rados.0.smithi137.stdout:1188: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:28.583 INFO:tasks.rados.rados.0.smithi137.stdout:1189: read oid 48 snap 111 2024-09-21T08:28:28.583 INFO:tasks.rados.rados.0.smithi137.stdout:1189: expect (ObjNum 424 snap 107 seq_num 424) 2024-09-21T08:28:28.583 INFO:tasks.rados.rados.0.smithi137.stdout:1190: delete oid 44 current snap is 114 2024-09-21T08:28:28.583 INFO:teuthology.orchestra.run.smithi137.stderr:marked out osd.0. 2024-09-21T08:28:28.617 INFO:tasks.rados.rados.0.smithi137.stdout:1189: expect (ObjNum 424 snap 107 seq_num 424) 2024-09-21T08:28:28.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:28.642 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:28.642 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:28.643 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:28.719 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:28:28.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:28 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/455697062' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-21T08:28:28.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:28 smithi137 ceph-mon[29634]: from='client.45407 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-21T08:28:28.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:28 smithi137 ceph-mon[29634]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-21T08:28:28.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:28 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/455697062' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-21T08:28:28.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:28 smithi137 ceph-mon[41044]: from='client.45407 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-21T08:28:28.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:28 smithi137 ceph-mon[41044]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-21T08:28:28.743 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:28.765 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:28.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:28.766 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:28.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:28.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:28.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:28.889 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:28.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:28 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/455697062' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-21T08:28:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:28 smithi165 ceph-mon[32893]: from='client.45407 ' entity='client.admin' cmd=[{"prefix": "osd out", "ids": ["0"]}]: dispatch 2024-09-21T08:28:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:28 smithi165 ceph-mon[32893]: Client client.admin marked osd.0 out, while it was still marked up 2024-09-21T08:28:29.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:29.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:29.014 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:29.115 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:29.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:29.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:29.139 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:29.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:29.262 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:29.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:29.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:29.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:29.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:29.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:29.390 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:29.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:29.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:29.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:29.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:29.604 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:29.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:29.640 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:29.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:29.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:29.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:29.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:29.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:29.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:29.787 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:29.888 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:29.893 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:29 smithi137 ceph-mon[29634]: pgmap v377: 168 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 141 active+clean; 167 MiB data, 598 MiB used, 446 GiB / 447 GiB avail; 6.5 MiB/s rd, 771 KiB/s wr, 7 op/s 2024-09-21T08:28:29.893 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:29 smithi137 ceph-mon[29634]: from='client.45407 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-21T08:28:29.893 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:29 smithi137 ceph-mon[29634]: osdmap e315: 8 total, 8 up, 4 in 2024-09-21T08:28:29.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:29 smithi137 ceph-mon[41044]: pgmap v377: 168 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 141 active+clean; 167 MiB data, 598 MiB used, 446 GiB / 447 GiB avail; 6.5 MiB/s rd, 771 KiB/s wr, 7 op/s 2024-09-21T08:28:29.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:29 smithi137 ceph-mon[41044]: from='client.45407 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-21T08:28:29.894 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:29 smithi137 ceph-mon[41044]: osdmap e315: 8 total, 8 up, 4 in 2024-09-21T08:28:29.911 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:29.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:29.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:30.013 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:30.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:29 smithi165 ceph-mon[32893]: pgmap v377: 168 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 141 active+clean; 167 MiB data, 598 MiB used, 446 GiB / 447 GiB avail; 6.5 MiB/s rd, 771 KiB/s wr, 7 op/s 2024-09-21T08:28:30.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:29 smithi165 ceph-mon[32893]: from='client.45407 ' entity='client.admin' cmd='[{"prefix": "osd out", "ids": ["0"]}]': finished 2024-09-21T08:28:30.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:29 smithi165 ceph-mon[32893]: osdmap e315: 8 total, 8 up, 4 in 2024-09-21T08:28:30.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:30.035 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:30.136 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:30.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:30.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:30.160 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:30.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:30.284 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:30.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:30.385 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:30.409 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:30.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:30.410 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:30.473 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:28:30.473 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:28:30.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:30.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:30.534 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:30.618 INFO:tasks.rados.rados.0.smithi137.stdout:1189: done (2 left) 2024-09-21T08:28:30.618 INFO:tasks.rados.rados.0.smithi137.stdout:1190: done (1 left) 2024-09-21T08:28:30.618 INFO:tasks.rados.rados.0.smithi137.stdout:1191: read oid 8 snap -1 2024-09-21T08:28:30.618 INFO:tasks.rados.rados.0.smithi137.stdout:1191: expect (ObjNum 404 snap 99 seq_num 404) 2024-09-21T08:28:30.618 INFO:tasks.rados.rados.0.smithi137.stdout:1192: write oid 7 current snap is 114 2024-09-21T08:28:30.618 INFO:tasks.rados.rados.0.smithi137.stdout:1192: seq_num 443 ranges {645501=483781,1573523=460153,2548974=763384} 2024-09-21T08:28:30.624 INFO:tasks.rados.rados.0.smithi137.stdout:1192: writing smithi13779221-7 from 645501 to 1129282 tid 1 2024-09-21T08:28:30.629 INFO:tasks.rados.rados.0.smithi137.stdout:1192: writing smithi13779221-7 from 1573523 to 2033676 tid 2 2024-09-21T08:28:30.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:30.637 INFO:tasks.rados.rados.0.smithi137.stdout:1192: writing smithi13779221-7 from 2548974 to 3312358 tid 3 2024-09-21T08:28:30.637 INFO:tasks.rados.rados.0.smithi137.stdout:1193: copy_from oid 40 from oid 41 current snap is 114 2024-09-21T08:28:30.637 INFO:tasks.rados.rados.0.smithi137.stdout:1191: expect (ObjNum 404 snap 99 seq_num 404) 2024-09-21T08:28:30.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:30.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:30.664 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:30.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:30.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:30 smithi137 ceph-mon[29634]: osdmap e316: 8 total, 8 up, 4 in 2024-09-21T08:28:30.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:30.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:30.768 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:28:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:28:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:28:30.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:30 smithi137 ceph-mon[41044]: osdmap e316: 8 total, 8 up, 4 in 2024-09-21T08:28:30.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:30.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:30.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:30.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:30.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:30.837 INFO:tasks.rados.rados.0.smithi137.stdout:1188: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:30.880 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:28:30.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:30.940 INFO:tasks.rados.rados.0.smithi137.stdout:1192: finishing write tid 1 to smithi13779221-7 2024-09-21T08:28:30.940 INFO:tasks.rados.rados.0.smithi137.stdout:1192: finishing write tid 2 to smithi13779221-7 2024-09-21T08:28:30.940 INFO:tasks.rados.rados.0.smithi137.stdout:1188: done (3 left) 2024-09-21T08:28:30.940 INFO:tasks.rados.rados.0.smithi137.stdout:1191: done (2 left) 2024-09-21T08:28:30.940 INFO:tasks.rados.rados.0.smithi137.stdout:1194: copy_from oid 31 from oid 34 current snap is 114 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1192: finishing write tid 3 to smithi13779221-7 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1192: finishing write tid 4 to smithi13779221-7 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1192: finishing write tid 5 to smithi13779221-7 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1192: oid 7 updating version 0 to 506 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1192: oid 7 updating version 506 to 508 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1192: oid 7 updating version 508 to 509 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1192: oid 7 version 509 is already newer than 507 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 509 (ObjNum 443 snap 114 seq_num 443) dirty exists 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1192: left oid 7 (ObjNum 443 snap 114 seq_num 443) 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1193: finishing copy_from racing read to smithi13779221-40 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1193: finishing copy_from to smithi13779221-40 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 311 (ObjNum 441 snap 113 seq_num 441) dirty exists 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1192: done (2 left) 2024-09-21T08:28:30.941 INFO:tasks.rados.rados.0.smithi137.stdout:1193: done (1 left) 2024-09-21T08:28:30.942 INFO:tasks.rados.rados.0.smithi137.stdout:1195: snap_create 2024-09-21T08:28:30.946 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:30.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:30.947 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:30.952 INFO:tasks.rados.rados.0.smithi137.stdout:1194: finishing copy_from racing read to smithi13779221-31 2024-09-21T08:28:30.956 INFO:tasks.rados.rados.0.smithi137.stdout:1194: finishing copy_from to smithi13779221-31 2024-09-21T08:28:30.957 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 108 (ObjNum 332 snap 79 seq_num 332) dirty exists 2024-09-21T08:28:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:30 smithi165 ceph-mon[32893]: osdmap e316: 8 total, 8 up, 4 in 2024-09-21T08:28:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:28:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:31.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:31.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:31.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:31.072 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:31.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T08:28:31.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:31.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:31.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:31.201 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:31.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:31.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:31.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:31.325 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:31.425 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:31.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:31.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:31.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:31.448 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:31.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:31.558 INFO:tasks.rados.rados.0.smithi137.stdout:1194: done (1 left) 2024-09-21T08:28:31.558 INFO:tasks.rados.rados.0.smithi137.stdout:1195: done (0 left) 2024-09-21T08:28:31.559 INFO:tasks.rados.rados.0.smithi137.stdout:1196: write (excl) oid 21 current snap is 115 2024-09-21T08:28:31.559 INFO:tasks.rados.rados.0.smithi137.stdout:1196: seq_num 444 ranges {662308=559030,1632005=603811,2860044=602264} 2024-09-21T08:28:31.563 INFO:tasks.rados.rados.0.smithi137.stdout:1196: writing smithi13779221-21 from 662308 to 1221338 tid 1 2024-09-21T08:28:31.570 INFO:tasks.rados.rados.0.smithi137.stdout:1196: writing smithi13779221-21 from 1632005 to 2235816 tid 2 2024-09-21T08:28:31.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:31.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:31.573 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:31.579 INFO:tasks.rados.rados.0.smithi137.stdout:1196: writing smithi13779221-21 from 2860044 to 3462308 tid 3 2024-09-21T08:28:31.579 INFO:tasks.rados.rados.0.smithi137.stdout:1197: write (excl) oid 42 current snap is 115 2024-09-21T08:28:31.579 INFO:tasks.rados.rados.0.smithi137.stdout:1197: seq_num 445 ranges {679115=634279,2090487=463822,3003820=475295} 2024-09-21T08:28:31.587 INFO:tasks.rados.rados.0.smithi137.stdout:1197: writing smithi13779221-42 from 679115 to 1313394 tid 1 2024-09-21T08:28:31.595 INFO:tasks.rados.rados.0.smithi137.stdout:1197: writing smithi13779221-42 from 2090487 to 2554309 tid 2 2024-09-21T08:28:31.600 INFO:tasks.rados.rados.0.smithi137.stdout:1197: writing smithi13779221-42 from 3003820 to 3479115 tid 3 2024-09-21T08:28:31.601 INFO:tasks.rados.rados.0.smithi137.stdout:1198: write oid 18 current snap is 115 2024-09-21T08:28:31.601 INFO:tasks.rados.rados.0.smithi137.stdout:1198: seq_num 446 ranges {695922=709528,2032616=607480,3198537=297385} 2024-09-21T08:28:31.608 INFO:tasks.rados.rados.0.smithi137.stdout:1198: writing smithi13779221-18 from 695922 to 1405450 tid 1 2024-09-21T08:28:31.615 INFO:tasks.rados.rados.0.smithi137.stdout:1198: writing smithi13779221-18 from 2032616 to 2640096 tid 2 2024-09-21T08:28:31.620 INFO:tasks.rados.rados.0.smithi137.stdout:1198: writing smithi13779221-18 from 3198537 to 3495922 tid 3 2024-09-21T08:28:31.621 INFO:tasks.rados.rados.0.smithi137.stdout:1199: read oid 27 snap -1 2024-09-21T08:28:31.621 INFO:tasks.rados.rados.0.smithi137.stdout:1199: expect deleted 2024-09-21T08:28:31.621 INFO:tasks.rados.rados.0.smithi137.stdout:1200: snap_create 2024-09-21T08:28:31.621 INFO:tasks.rados.rados.0.smithi137.stdout:1198: finishing write tid 1 to smithi13779221-18 2024-09-21T08:28:31.621 INFO:tasks.rados.rados.0.smithi137.stdout:1198: finishing write tid 2 to smithi13779221-18 2024-09-21T08:28:31.623 INFO:tasks.rados.rados.0.smithi137.stdout:1198: finishing write tid 3 to smithi13779221-18 2024-09-21T08:28:31.623 INFO:tasks.rados.rados.0.smithi137.stdout:1198: finishing write tid 4 to smithi13779221-18 2024-09-21T08:28:31.623 INFO:tasks.rados.rados.0.smithi137.stdout:1198: finishing write tid 5 to smithi13779221-18 2024-09-21T08:28:31.623 INFO:tasks.rados.rados.0.smithi137.stdout:1198: oid 18 updating version 0 to 140 2024-09-21T08:28:31.623 INFO:tasks.rados.rados.0.smithi137.stdout:1198: oid 18 updating version 140 to 141 2024-09-21T08:28:31.623 INFO:tasks.rados.rados.0.smithi137.stdout:1198: oid 18 updating version 141 to 142 2024-09-21T08:28:31.623 INFO:tasks.rados.rados.0.smithi137.stdout:1198: oid 18 updating version 142 to 143 2024-09-21T08:28:31.623 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 143 (ObjNum 446 snap 115 seq_num 446) dirty exists 2024-09-21T08:28:31.623 INFO:tasks.rados.rados.0.smithi137.stdout:1198: left oid 18 (ObjNum 446 snap 115 seq_num 446) 2024-09-21T08:28:31.675 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:31.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:31.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:31.697 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:31.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:31 smithi137 ceph-mon[29634]: pgmap v380: 168 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 141 active+clean; 167 MiB data, 485 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:31.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:31 smithi137 ceph-mon[29634]: osdmap e317: 8 total, 8 up, 4 in 2024-09-21T08:28:31.705 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:31 smithi137 ceph-mon[29634]: osdmap e318: 8 total, 8 up, 4 in 2024-09-21T08:28:31.705 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:31 smithi137 ceph-mon[41044]: pgmap v380: 168 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 141 active+clean; 167 MiB data, 485 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:31.705 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:31 smithi137 ceph-mon[41044]: osdmap e317: 8 total, 8 up, 4 in 2024-09-21T08:28:31.705 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:31 smithi137 ceph-mon[41044]: osdmap e318: 8 total, 8 up, 4 in 2024-09-21T08:28:31.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:31.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:31.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:31.820 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:31.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:31.925 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:31.935 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:31.936 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:31.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:31.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:31.945 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:31.954 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:31.954 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:31.973 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:31.973 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:31.989 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:31.989 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:32.006 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:32.006 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:31 smithi165 ceph-mon[32893]: pgmap v380: 168 pgs: 7 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 141 active+clean; 167 MiB data, 485 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:31 smithi165 ceph-mon[32893]: osdmap e317: 8 total, 8 up, 4 in 2024-09-21T08:28:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:31 smithi165 ceph-mon[32893]: osdmap e318: 8 total, 8 up, 4 in 2024-09-21T08:28:32.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:28:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:28:32.022 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:32.023 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:32.039 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:32.039 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:32.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:32.055 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:32.056 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:32.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:32.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:32.068 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:32.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:32.190 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:32.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:32.191 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:32.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:32.314 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:32.315 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:32.315 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:32.416 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:32.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:32.439 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:32.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:32.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:32.554 INFO:tasks.rados.rados.0.smithi137.stdout:1198: done (4 left) 2024-09-21T08:28:32.554 INFO:tasks.rados.rados.0.smithi137.stdout:1199: done (3 left) 2024-09-21T08:28:32.554 INFO:tasks.rados.rados.0.smithi137.stdout:1200: done (2 left) 2024-09-21T08:28:32.555 INFO:tasks.rados.rados.0.smithi137.stdout:1201: write oid 40 current snap is 116 2024-09-21T08:28:32.555 INFO:tasks.rados.rados.0.smithi137.stdout:1201: seq_num 447 ranges {712729=784777,2091098=467491,3342313=170416} 2024-09-21T08:28:32.555 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:28:32.559 INFO:tasks.rados.rados.0.smithi137.stdout:1201: writing smithi13779221-40 from 712729 to 1497506 tid 1 2024-09-21T08:28:32.566 INFO:tasks.rados.rados.0.smithi137.stdout:1201: writing smithi13779221-40 from 2091098 to 2558589 tid 2 2024-09-21T08:28:32.569 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:32.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:32.570 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:32.572 INFO:tasks.rados.rados.0.smithi137.stdout:1201: writing smithi13779221-40 from 3342313 to 3512729 tid 3 2024-09-21T08:28:32.572 INFO:tasks.rados.rados.0.smithi137.stdout:1202: read oid 19 snap -1 2024-09-21T08:28:32.572 INFO:tasks.rados.rados.0.smithi137.stdout:1202: expect deleted 2024-09-21T08:28:32.572 INFO:tasks.rados.rados.0.smithi137.stdout:1203: copy_from oid 17 from oid 37 current snap is 116 2024-09-21T08:28:32.572 INFO:tasks.rados.rados.0.smithi137.stdout:1196: finishing write tid 1 to smithi13779221-21 2024-09-21T08:28:32.572 INFO:tasks.rados.rados.0.smithi137.stdout:1204: copy_from oid 50 from oid 47 current snap is 116 2024-09-21T08:28:32.572 INFO:tasks.rados.rados.0.smithi137.stdout:1205: copy_from oid 9 from oid 43 current snap is 116 2024-09-21T08:28:32.572 INFO:tasks.rados.rados.0.smithi137.stdout:1206: snap_create 2024-09-21T08:28:32.573 INFO:tasks.rados.rados.0.smithi137.stdout:1196: finishing write tid 2 to smithi13779221-21 2024-09-21T08:28:32.573 INFO:tasks.rados.rados.0.smithi137.stdout:1196: finishing write tid 3 to smithi13779221-21 2024-09-21T08:28:32.573 INFO:tasks.rados.rados.0.smithi137.stdout:1196: finishing write tid 4 to smithi13779221-21 2024-09-21T08:28:32.573 INFO:tasks.rados.rados.0.smithi137.stdout:1196: finishing write tid 5 to smithi13779221-21 2024-09-21T08:28:32.573 INFO:tasks.rados.rados.0.smithi137.stdout:1196: oid 21 updating version 0 to 116 2024-09-21T08:28:32.573 INFO:tasks.rados.rados.0.smithi137.stdout:1196: oid 21 version 116 is already newer than 113 2024-09-21T08:28:32.573 INFO:tasks.rados.rados.0.smithi137.stdout:1196: oid 21 version 116 is already newer than 115 2024-09-21T08:28:32.573 INFO:tasks.rados.rados.0.smithi137.stdout:1196: oid 21 version 116 is already newer than 114 2024-09-21T08:28:32.574 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 116 (ObjNum 444 snap 115 seq_num 444) dirty exists 2024-09-21T08:28:32.574 INFO:tasks.rados.rados.0.smithi137.stdout:1196: left oid 21 (ObjNum 444 snap 115 seq_num 444) 2024-09-21T08:28:32.576 INFO:tasks.rados.rados.0.smithi137.stdout:1201: finishing write tid 1 to smithi13779221-40 2024-09-21T08:28:32.576 INFO:tasks.rados.rados.0.smithi137.stdout:1201: finishing write tid 2 to smithi13779221-40 2024-09-21T08:28:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:1201: finishing write tid 3 to smithi13779221-40 2024-09-21T08:28:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:1201: finishing write tid 4 to smithi13779221-40 2024-09-21T08:28:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:1201: finishing write tid 5 to smithi13779221-40 2024-09-21T08:28:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:1201: oid 40 updating version 0 to 313 2024-09-21T08:28:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:1201: oid 40 updating version 313 to 314 2024-09-21T08:28:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:1201: oid 40 updating version 314 to 315 2024-09-21T08:28:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:1201: oid 40 version 315 is already newer than 312 2024-09-21T08:28:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 315 (ObjNum 447 snap 116 seq_num 447) dirty exists 2024-09-21T08:28:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:1201: left oid 40 (ObjNum 447 snap 116 seq_num 447) 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:1197: finishing write tid 1 to smithi13779221-42 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:1197: finishing write tid 2 to smithi13779221-42 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:1197: finishing write tid 3 to smithi13779221-42 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:1197: finishing write tid 4 to smithi13779221-42 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:1197: finishing write tid 5 to smithi13779221-42 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:1197: oid 42 updating version 0 to 284 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:1197: oid 42 version 284 is already newer than 283 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:1197: oid 42 version 284 is already newer than 282 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:1197: oid 42 updating version 284 to 285 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 285 (ObjNum 445 snap 115 seq_num 445) dirty exists 2024-09-21T08:28:32.583 INFO:tasks.rados.rados.0.smithi137.stdout:1197: left oid 42 (ObjNum 445 snap 115 seq_num 445) 2024-09-21T08:28:32.590 INFO:tasks.rados.rados.0.smithi137.stdout:1203: finishing copy_from racing read to smithi13779221-17 2024-09-21T08:28:32.591 INFO:tasks.rados.rados.0.smithi137.stdout:1204: finishing copy_from racing read to smithi13779221-50 2024-09-21T08:28:32.591 INFO:tasks.rados.rados.0.smithi137.stdout:1205: finishing copy_from racing read to smithi13779221-9 2024-09-21T08:28:32.593 INFO:tasks.rados.rados.0.smithi137.stdout:1205: finishing copy_from to smithi13779221-9 2024-09-21T08:28:32.593 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 9 v 303 (ObjNum 417 snap 103 seq_num 417) dirty exists 2024-09-21T08:28:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1203: finishing copy_from to smithi13779221-17 2024-09-21T08:28:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 183 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:28:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1204: finishing copy_from to smithi13779221-50 2024-09-21T08:28:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:28:32.636 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:28:32.636 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:28:32.636 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:28:32.636 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:28:32.636 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:28:32.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:32.692 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:32.693 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:32.693 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/892425238' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[29634]: from='client.45413 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/630122070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[29634]: from='client.45413 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:28:32.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[29634]: osdmap e319: 8 total, 8 up, 4 in 2024-09-21T08:28:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/892425238' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:32.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[41044]: from='client.45413 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:32.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/630122070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:32.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:32.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[41044]: from='client.45413 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:28:32.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:32 smithi137 ceph-mon[41044]: osdmap e319: 8 total, 8 up, 4 in 2024-09-21T08:28:32.794 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:32.815 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:32.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:32.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:32.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:32.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:32.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:32.940 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:32 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/892425238' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:32 smithi165 ceph-mon[32893]: from='client.45413 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:32 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/630122070' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:32 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:33.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:32 smithi165 ceph-mon[32893]: from='client.45413 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:28:33.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:32 smithi165 ceph-mon[32893]: osdmap e319: 8 total, 8 up, 4 in 2024-09-21T08:28:33.041 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:33.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:33.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:33.064 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:33.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:33.187 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:33.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:33.188 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:33.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:33.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:33.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:33.312 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:33.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:33.434 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:33.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:33.435 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:33.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:33.547 INFO:tasks.rados.rados.0.smithi137.stdout:1196: done (7 left) 2024-09-21T08:28:33.547 INFO:tasks.rados.rados.0.smithi137.stdout:1197: done (6 left) 2024-09-21T08:28:33.547 INFO:tasks.rados.rados.0.smithi137.stdout:1201: done (5 left) 2024-09-21T08:28:33.547 INFO:tasks.rados.rados.0.smithi137.stdout:1202: done (4 left) 2024-09-21T08:28:33.547 INFO:tasks.rados.rados.0.smithi137.stdout:1203: done (3 left) 2024-09-21T08:28:33.547 INFO:tasks.rados.rados.0.smithi137.stdout:1204: done (2 left) 2024-09-21T08:28:33.547 INFO:tasks.rados.rados.0.smithi137.stdout:1205: done (1 left) 2024-09-21T08:28:33.548 INFO:tasks.rados.rados.0.smithi137.stdout:1206: done (0 left) 2024-09-21T08:28:33.548 INFO:tasks.rados.rados.0.smithi137.stdout:1207: snap_remove snap 115 2024-09-21T08:28:33.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:33.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:33.560 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:33.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:33.681 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:33.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:33.682 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:33.759 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:33.760 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[29634]: pgmap v383: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 7 active+undersized+remapped, 3 active+clean+snaptrim, 1 active+remapped, 9 active+clean+snaptrim_wait, 143 active+clean; 178 MiB data, 504 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 3.7 MiB/s wr, 16 op/s; 22/822 objects degraded (2.676%); 16/822 objects misplaced (1.946%); 11 MiB/s, 74 objects/s recovering 2024-09-21T08:28:33.760 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[29634]: Health check failed: Degraded data redundancy: 22/822 objects degraded (2.676%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T08:28:33.760 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:28:33.760 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:28:33.760 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[29634]: osdmap e320: 8 total, 8 up, 4 in 2024-09-21T08:28:33.760 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:33.760 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[41044]: pgmap v383: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 7 active+undersized+remapped, 3 active+clean+snaptrim, 1 active+remapped, 9 active+clean+snaptrim_wait, 143 active+clean; 178 MiB data, 504 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 3.7 MiB/s wr, 16 op/s; 22/822 objects degraded (2.676%); 16/822 objects misplaced (1.946%); 11 MiB/s, 74 objects/s recovering 2024-09-21T08:28:33.760 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[41044]: Health check failed: Degraded data redundancy: 22/822 objects degraded (2.676%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T08:28:33.760 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:28:33.760 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:28:33.761 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:33 smithi137 ceph-mon[41044]: osdmap e320: 8 total, 8 up, 4 in 2024-09-21T08:28:33.783 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:33.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T08:28:33.804 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:33.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:33.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:33.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:33.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:33.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:33.929 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:34.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:33 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:34.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:33 smithi165 ceph-mon[32893]: pgmap v383: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 7 active+undersized+remapped, 3 active+clean+snaptrim, 1 active+remapped, 9 active+clean+snaptrim_wait, 143 active+clean; 178 MiB data, 504 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 3.7 MiB/s wr, 16 op/s; 22/822 objects degraded (2.676%); 16/822 objects misplaced (1.946%); 11 MiB/s, 74 objects/s recovering 2024-09-21T08:28:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:33 smithi165 ceph-mon[32893]: Health check failed: Degraded data redundancy: 22/822 objects degraded (2.676%), 2 pgs degraded (PG_DEGRADED) 2024-09-21T08:28:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:33 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:28:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:33 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:28:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:33 smithi165 ceph-mon[32893]: osdmap e320: 8 total, 8 up, 4 in 2024-09-21T08:28:34.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:34.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:34.053 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:34.053 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:34.154 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:34.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:34.262 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:34.262 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:34.263 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:34.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:34.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:34.387 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:34.487 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:34.509 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:34.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:34.510 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:34.550 INFO:tasks.rados.rados.0.smithi137.stdout:1207: done (0 left) 2024-09-21T08:28:34.550 INFO:tasks.rados.rados.0.smithi137.stdout:1208: read oid 34 snap -1 2024-09-21T08:28:34.550 INFO:tasks.rados.rados.0.smithi137.stdout:1208: expect (ObjNum 332 snap 79 seq_num 332) 2024-09-21T08:28:34.550 INFO:tasks.rados.rados.0.smithi137.stdout:1209: rollback oid 28 current snap is 117 2024-09-21T08:28:34.551 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 28 to 116 2024-09-21T08:28:34.551 INFO:tasks.rados.rados.0.smithi137.stdout:1210: read oid 23 snap -1 2024-09-21T08:28:34.551 INFO:tasks.rados.rados.0.smithi137.stdout:1210: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:28:34.551 INFO:tasks.rados.rados.0.smithi137.stdout:1211: write oid 30 current snap is 117 2024-09-21T08:28:34.551 INFO:tasks.rados.rados.0.smithi137.stdout:1211: seq_num 448 ranges {729536=460026,1749580=611149,2853383=452450} 2024-09-21T08:28:34.556 INFO:tasks.rados.rados.0.smithi137.stdout:1211: writing smithi13779221-30 from 729536 to 1189562 tid 1 2024-09-21T08:28:34.563 INFO:tasks.rados.rados.0.smithi137.stdout:1211: writing smithi13779221-30 from 1749580 to 2360729 tid 2 2024-09-21T08:28:34.569 INFO:tasks.rados.rados.0.smithi137.stdout:1211: writing smithi13779221-30 from 2853383 to 3305833 tid 3 2024-09-21T08:28:34.569 INFO:tasks.rados.rados.0.smithi137.stdout:1212: snap_create 2024-09-21T08:28:34.569 INFO:tasks.rados.rados.0.smithi137.stdout:1209: finishing rollback tid 0 to smithi13779221-28 2024-09-21T08:28:34.569 INFO:tasks.rados.rados.0.smithi137.stdout:1208: expect (ObjNum 332 snap 79 seq_num 332) 2024-09-21T08:28:34.588 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1] out_osds: [4, 6, 3, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:28:34.588 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:28:34.589 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T08:28:34.589 INFO:tasks.thrashosds.thrasher:Setting osd 2 primary_affinity to 0.305256 2024-09-21T08:28:34.589 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 2 0.3052560012589116 2024-09-21T08:28:34.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:34.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:34.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:34.663 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:34.734 INFO:tasks.rados.rados.0.smithi137.stdout:1209: finishing rollback tid 1 to smithi13779221-28 2024-09-21T08:28:34.735 INFO:tasks.rados.rados.0.smithi137.stdout:1209: finishing rollback tid 2 to smithi13779221-28 2024-09-21T08:28:34.735 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 265 (ObjNum 425 snap 107 seq_num 425) dirty exists 2024-09-21T08:28:34.735 INFO:tasks.rados.rados.0.smithi137.stdout:1211: finishing write tid 1 to smithi13779221-30 2024-09-21T08:28:34.735 INFO:tasks.rados.rados.0.smithi137.stdout:1211: finishing write tid 2 to smithi13779221-30 2024-09-21T08:28:34.735 INFO:tasks.rados.rados.0.smithi137.stdout:1210: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:28:34.742 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:28:34.742 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 438.26332318099998, 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27973036899999998, 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.743 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.744 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473701+0000", 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1320000000000001e-06 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473716+0000", 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5622999999999999e-05 2024-09-21T08:28:34.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473730+0000", 2024-09-21T08:28:34.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3259e-05 2024-09-21T08:28:34.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473740+0000", 2024-09-21T08:28:34.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0746e-05 2024-09-21T08:28:34.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473802+0000", 2024-09-21T08:28:34.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2043000000000002e-05 2024-09-21T08:28:34.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:34.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473810+0000", 2024-09-21T08:28:34.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1500000000000002e-06 2024-09-21T08:28:34.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752339+0000", 2024-09-21T08:28:34.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27852951199999998 2024-09-21T08:28:34.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752485+0000", 2024-09-21T08:28:34.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014628699999999999 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752609+0000", 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012376499999999999 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753145+0000", 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053571000000000001 2024-09-21T08:28:34.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753326+0000", 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018102799999999999 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753367+0000", 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1253e-05 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753400+0000", 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3065999999999998e-05 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753429+0000", 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8795000000000001e-05 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:422 7.d 7:b31d99ce:::smithi13779221-21:head [write 443718~615603] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:28:34.751 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:28:34.752 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 218.232046199, 2024-09-21T08:28:34.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034885076000000001, 2024-09-21T08:28:34.752 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.752 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.752 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.752 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.752 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.752 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.752 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 422 2024-09-21T08:28:34.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504978+0000", 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0090000000000002e-06 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506268+0000", 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001290444 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506279+0000", 2024-09-21T08:28:34.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0526999999999999e-05 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506288+0000", 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5470000000000002e-06 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506343+0000", 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5287000000000001e-05 2024-09-21T08:28:34.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506354+0000", 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0798999999999999e-05 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.525308+0000", 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018953616 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:34.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529390+0000", 2024-09-21T08:28:34.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040819029999999996 2024-09-21T08:28:34.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533318+0000", 2024-09-21T08:28:34.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039282190000000002 2024-09-21T08:28:34.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533380+0000", 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1890000000000005e-05 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533709+0000", 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032948400000000001 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536916+0000", 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032063399999999998 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539553+0000", 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026370909999999998 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539658+0000", 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105585 2024-09-21T08:28:34.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539723+0000", 2024-09-21T08:28:34.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4343000000000004e-05 2024-09-21T08:28:34.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539861+0000", 2024-09-21T08:28:34.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137992 2024-09-21T08:28:34.761 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.761 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.761 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.761 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:423 7.d 7:b31d99ce:::smithi13779221-21:head [write 1564138~470341] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:28:34.761 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:28:34.761 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 218.23065653099999, 2024-09-21T08:28:34.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033782404000000002, 2024-09-21T08:28:34.761 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.762 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.762 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.762 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.762 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.762 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.762 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 423 2024-09-21T08:28:34.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.762 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.762 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506366+0000", 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0449999999999999e-06 2024-09-21T08:28:34.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507012+0000", 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064568899999999996 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507021+0000", 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2550000000000003e-06 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507029+0000", 2024-09-21T08:28:34.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2559999999999996e-06 2024-09-21T08:28:34.765 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.765 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.765 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507079+0000", 2024-09-21T08:28:34.765 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0957e-05 2024-09-21T08:28:34.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:34.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507088+0000", 2024-09-21T08:28:34.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3650000000000004e-06 2024-09-21T08:28:34.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529457+0000", 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022369185999999999 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529526+0000", 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8683999999999995e-05 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535157+0000", 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056315849999999997 2024-09-21T08:28:34.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535244+0000", 2024-09-21T08:28:34.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6840000000000002e-05 2024-09-21T08:28:34.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:28:34.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535398+0000", 2024-09-21T08:28:34.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015404399999999999 2024-09-21T08:28:34.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539569+0000", 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004170361 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539613+0000", 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4101999999999999e-05 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539985+0000", 2024-09-21T08:28:34.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037279300000000001 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540037+0000", 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1345999999999999e-05 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540148+0000", 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110896 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.770 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.771 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:424 7.d 7:b31d99ce:::smithi13779221-21:head [write 2633066~610652] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:28:34.771 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:28:34.771 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 218.223667517, 2024-09-21T08:28:34.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027015847999999999, 2024-09-21T08:28:34.771 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.771 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.771 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.772 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.772 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.772 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.772 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 424 2024-09-21T08:28:34.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.772 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:28:34.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:28:34.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513356+0000", 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8789999999999999e-06 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513753+0000", 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039670999999999999 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513764+0000", 2024-09-21T08:28:34.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.154e-05 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513772+0000", 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9149999999999996e-06 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513833+0000", 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1072000000000001e-05 2024-09-21T08:28:34.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513843+0000", 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8349999999999999e-06 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529644+0000", 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015801084 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:34.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529688+0000", 2024-09-21T08:28:34.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3568e-05 2024-09-21T08:28:34.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536031+0000", 2024-09-21T08:28:34.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063434659999999999 2024-09-21T08:28:34.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536135+0000", 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103599 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536291+0000", 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155668 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539575+0000", 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032840450000000002 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539925+0000", 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034988899999999998 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540200+0000", 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027585000000000001 2024-09-21T08:28:34.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540250+0000", 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9413000000000003e-05 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540370+0000", 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120315 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.780 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:425 7.d 7:b31d99ce:::smithi13779221-21:head [setxattr _header (56) in=7b,truncate 3243718] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:28:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:28:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 218.22316487699999, 2024-09-21T08:28:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030130239, 2024-09-21T08:28:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.781 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 425 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513858+0000", 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9199999999999999e-07 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513863+0000", 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7650000000000001e-06 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513869+0000", 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9880000000000001e-06 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513874+0000", 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.643e-06 2024-09-21T08:28:34.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515915+0000", 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020407279999999999 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515925+0000", 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0051000000000001e-05 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529722+0000", 2024-09-21T08:28:34.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013796780999999999 2024-09-21T08:28:34.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:34.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529804+0000", 2024-09-21T08:28:34.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1663999999999996e-05 2024-09-21T08:28:34.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540423+0000", 2024-09-21T08:28:34.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010619635000000001 2024-09-21T08:28:34.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540671+0000", 2024-09-21T08:28:34.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024766600000000001 2024-09-21T08:28:34.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:28:34.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540795+0000", 2024-09-21T08:28:34.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012383899999999999 2024-09-21T08:28:34.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541324+0000", 2024-09-21T08:28:34.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052910200000000004 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.542654+0000", 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013304370000000001 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543785+0000", 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001130603 2024-09-21T08:28:34.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543841+0000", 2024-09-21T08:28:34.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5939000000000001e-05 2024-09-21T08:28:34.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543987+0000", 2024-09-21T08:28:34.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014640599999999999 2024-09-21T08:28:34.791 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.791 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.791 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.791 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:426 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:28:34.791 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 218.221205932, 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028482668999999999, 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 426 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.792 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:28:34.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.793 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:28:34.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515818+0000", 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7999999999999999e-06 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515825+0000", 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0790000000000003e-06 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.795 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515840+0000", 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4765e-05 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515857+0000", 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7414999999999998e-05 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515968+0000", 2024-09-21T08:28:34.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111161 2024-09-21T08:28:34.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:34.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515975+0000", 2024-09-21T08:28:34.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7499999999999997e-06 2024-09-21T08:28:34.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529841+0000", 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013866387000000001 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529951+0000", 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010953100000000001 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541110+0000", 2024-09-21T08:28:34.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011159591 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541190+0000", 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9243e-05 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544032+0000", 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028424050000000001 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544118+0000", 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5798999999999996e-05 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544299+0000", 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180743 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.800 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:430 7.d 7:b0b32695:::smithi13779221-19:head [delete] snapc a=[9] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 218.22103525399999, 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028508957000000001, 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 430 2024-09-21T08:28:34.801 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.802 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1900000000000001e-07 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515991+0000", 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6899999999999998e-06 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.803 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515997+0000", 2024-09-21T08:28:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2999999999999998e-06 2024-09-21T08:28:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516003+0000", 2024-09-21T08:28:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0220000000000001e-06 2024-09-21T08:28:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.804 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516045+0000", 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1069000000000003e-05 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516052+0000", 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9890000000000003e-06 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.805 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529980+0000", 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013927999999999999 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.532305+0000", 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023250580000000001 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541672+0000", 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093675080000000001 2024-09-21T08:28:34.806 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541749+0000", 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6829000000000006e-05 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541934+0000", 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018531 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.807 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543560+0000", 2024-09-21T08:28:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00162525 2024-09-21T08:28:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543661+0000", 2024-09-21T08:28:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101604 2024-09-21T08:28:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.808 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544339+0000", 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067745299999999995 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544394+0000", 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5515e-05 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544496+0000", 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010144100000000001 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.809 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:552 7.d 7:b0b32695:::smithi13779221-19:head [copy-from ver 34] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:09.528772+0000", 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 205.20825022299999, 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022405184000000002, 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.810 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 552 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528772+0000", 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528772+0000", 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.811 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.812 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.812 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528776+0000", 2024-09-21T08:28:34.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:34 smithi137 ceph-mon[29634]: 7.18 scrub starts 2024-09-21T08:28:34.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:34 smithi137 ceph-mon[29634]: 7.18 scrub ok 2024-09-21T08:28:34.812 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:34 smithi137 ceph-mon[29634]: osdmap e321: 8 total, 8 up, 4 in 2024-09-21T08:28:34.812 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:34 smithi137 ceph-mon[41044]: 7.18 scrub starts 2024-09-21T08:28:34.813 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:34 smithi137 ceph-mon[41044]: 7.18 scrub ok 2024-09-21T08:28:34.813 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:34 smithi137 ceph-mon[41044]: osdmap e321: 8 total, 8 up, 4 in 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3510000000000004e-06 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528788+0000", 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2542e-05 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528810+0000", 2024-09-21T08:28:34.814 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1467e-05 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528822+0000", 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1973e-05 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528913+0000", 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1458000000000004e-05 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.815 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528922+0000", 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0780000000000002e-06 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.531438+0000", 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002515541 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.531531+0000", 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3175000000000002e-05 2024-09-21T08:28:34.816 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.539639+0000", 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081080649999999994 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.547323+0000", 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076836079999999998 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.817 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.550791+0000", 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034681719999999998 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.551056+0000", 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000264966 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.551116+0000", 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9905000000000003e-05 2024-09-21T08:28:34.818 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.551177+0000", 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0883e-05 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:553 7.d 7:b0b32695:::smithi13779221-19:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e116)", 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:09.528845+0000", 2024-09-21T08:28:34.819 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 205.20817711800001, 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015306544, 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 553 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.820 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528845+0000", 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528845+0000", 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.821 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528846+0000", 2024-09-21T08:28:34.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9899999999999999e-07 2024-09-21T08:28:34.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.822 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.822 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.822 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528857+0000", 2024-09-21T08:28:34.822 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1755e-05 2024-09-21T08:28:34.822 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528864+0000", 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0389999999999999e-06 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528871+0000", 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1229999999999997e-06 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.823 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528987+0000", 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116597 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528992+0000", 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1250000000000001e-06 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.531699+0000", 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027068890000000001 2024-09-21T08:28:34.824 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.532428+0000", 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00072873100000000004 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.543914+0000", 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011486322 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.825 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.544028+0000", 2024-09-21T08:28:34.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000114063 2024-09-21T08:28:34.826 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.826 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.826 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.826 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.544151+0000", 2024-09-21T08:28:34.826 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000123001 2024-09-21T08:28:34.826 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.826 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.827 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.827 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.827 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:571 7.1 7:88bf4ad2:::smithi13779221-6:head [copy-from ver 13] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:28:34.827 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:28:34.827 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 204.18996859999999, 2024-09-21T08:28:34.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037746406000000003, 2024-09-21T08:28:34.827 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.827 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.827 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 571 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547056+0000", 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.146e-06 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547067+0000", 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0609e-05 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547090+0000", 2024-09-21T08:28:34.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2714999999999999e-05 2024-09-21T08:28:34.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547100+0000", 2024-09-21T08:28:34.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9920000000000002e-06 2024-09-21T08:28:34.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547203+0000", 2024-09-21T08:28:34.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102828 2024-09-21T08:28:34.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:34.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547210+0000", 2024-09-21T08:28:34.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3629999999999996e-06 2024-09-21T08:28:34.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553577+0000", 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063672829999999996 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553669+0000", 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1856999999999998e-05 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573941+0000", 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020272168 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.583019+0000", 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090777630000000008 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584643+0000", 2024-09-21T08:28:34.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016245319999999999 2024-09-21T08:28:34.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584686+0000", 2024-09-21T08:28:34.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2259999999999999e-05 2024-09-21T08:28:34.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584748+0000", 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2049000000000002e-05 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584800+0000", 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1841000000000001e-05 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:572 7.1 7:88bf4ad2:::smithi13779221-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 204.18989957100001, 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032210599999999999, 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.836 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 572 2024-09-21T08:28:34.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.837 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:28:34.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547123+0000", 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5300000000000003e-07 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547138+0000", 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4718999999999999e-05 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547145+0000", 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2520000000000002e-06 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547151+0000", 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7649999999999999e-06 2024-09-21T08:28:34.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547263+0000", 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112001 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547269+0000", 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8919999999999997e-06 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553824+0000", 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065557499999999999 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553873+0000", 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8727999999999998e-05 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579133+0000", 2024-09-21T08:28:34.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025259964999999999 2024-09-21T08:28:34.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579227+0000", 2024-09-21T08:28:34.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3517999999999995e-05 2024-09-21T08:28:34.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579333+0000", 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010615699999999999 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:576 7.2 7:4031afd1:::smithi13779221-48:head [append 0~0] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547160+0000", 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 204.18986186000001, 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026627913, 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.843 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.844 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.844 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.844 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.844 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 576 2024-09-21T08:28:34.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.844 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547160+0000", 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547160+0000", 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547161+0000", 2024-09-21T08:28:34.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.44e-07 2024-09-21T08:28:34.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547163+0000", 2024-09-21T08:28:34.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5790000000000002e-06 2024-09-21T08:28:34.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547169+0000", 2024-09-21T08:28:34.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7610000000000004e-06 2024-09-21T08:28:34.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547174+0000", 2024-09-21T08:28:34.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7849999999999999e-06 2024-09-21T08:28:34.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547266+0000", 2024-09-21T08:28:34.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1978999999999995e-05 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547272+0000", 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3629999999999999e-06 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.562646+0000", 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015374095000000001 2024-09-21T08:28:34.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.562751+0000", 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010446 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.562953+0000", 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000202151 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564215+0000", 2024-09-21T08:28:34.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012625589999999999 2024-09-21T08:28:34.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567784+0000", 2024-09-21T08:28:34.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035684559999999998 2024-09-21T08:28:34.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573649+0000", 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058654110000000001 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573729+0000", 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9555999999999995e-05 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573788+0000", 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9113999999999997e-05 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:577 7.2 7:4031afd1:::smithi13779221-48:head [rollback 14] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 204.18983520800001, 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026796044000000002, 2024-09-21T08:28:34.852 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.853 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.853 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.853 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.853 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.853 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.853 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 577 2024-09-21T08:28:34.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.853 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:28:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:28:34.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:28:34.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3e-07 2024-09-21T08:28:34.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547190+0000", 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3980000000000002e-06 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547196+0000", 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5239999999999997e-06 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547201+0000", 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5839999999999996e-06 2024-09-21T08:28:34.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547293+0000", 2024-09-21T08:28:34.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2613000000000003e-05 2024-09-21T08:28:34.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:34.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547298+0000", 2024-09-21T08:28:34.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6500000000000004e-06 2024-09-21T08:28:34.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564248+0000", 2024-09-21T08:28:34.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01694959 2024-09-21T08:28:34.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564348+0000", 2024-09-21T08:28:34.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100893 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564547+0000", 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000198278 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565560+0000", 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001013391 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567845+0000", 2024-09-21T08:28:34.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022853090000000001 2024-09-21T08:28:34.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573898+0000", 2024-09-21T08:28:34.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.006052094 2024-09-21T08:28:34.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573944+0000", 2024-09-21T08:28:34.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6928000000000002e-05 2024-09-21T08:28:34.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573983+0000", 2024-09-21T08:28:34.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8161999999999998e-05 2024-09-21T08:28:34.863 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.863 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.863 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.864 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:578 7.2 7:4031afd1:::smithi13779221-48:head [append 0~0] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:28:34.864 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:28:34.864 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 204.189815947, 2024-09-21T08:28:34.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026976508, 2024-09-21T08:28:34.864 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.864 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.864 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.864 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 578 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:28:34.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:28:34.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8100000000000003e-07 2024-09-21T08:28:34.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547209+0000", 2024-09-21T08:28:34.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4549999999999998e-06 2024-09-21T08:28:34.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547214+0000", 2024-09-21T08:28:34.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2889999999999997e-06 2024-09-21T08:28:34.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547219+0000", 2024-09-21T08:28:34.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7419999999999997e-06 2024-09-21T08:28:34.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547317+0000", 2024-09-21T08:28:34.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8029000000000006e-05 2024-09-21T08:28:34.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:34.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547321+0000", 2024-09-21T08:28:34.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2880000000000003e-06 2024-09-21T08:28:34.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566078+0000", 2024-09-21T08:28:34.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018756951000000001 2024-09-21T08:28:34.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566167+0000", 2024-09-21T08:28:34.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9068999999999997e-05 2024-09-21T08:28:34.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:34.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566318+0000", 2024-09-21T08:28:34.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015045999999999999 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567184+0000", 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00086585299999999998 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.568942+0000", 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001758103 2024-09-21T08:28:34.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574047+0000", 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051054719999999998 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574143+0000", 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5616000000000001e-05 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574182+0000", 2024-09-21T08:28:34.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.96e-05 2024-09-21T08:28:34.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.873 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.873 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:830 7.b 7:d8f73c7b:::smithi13779221-38:head [sparse-read 0~2050454,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:28:34.873 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.705969+0000", 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 185.03105307300001, 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017749276000000001, 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 830 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.874 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705969+0000", 2024-09-21T08:28:34.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705969+0000", 2024-09-21T08:28:34.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705973+0000", 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7360000000000001e-06 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.705988+0000", 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.413e-05 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706011+0000", 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3484000000000001e-05 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706023+0000", 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1899000000000001e-05 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706110+0000", 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7439000000000001e-05 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706117+0000", 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2190000000000001e-06 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.719472+0000", 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013354927000000001 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.719550+0000", 2024-09-21T08:28:34.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8486999999999996e-05 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.723718+0000", 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041679550000000001 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:831 7.b 7:d8f73c7b:::smithi13779221-38:head [read 0~2050454,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:28:34.879 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 185.030969957, 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018156002000000001, 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 831 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706052+0000", 2024-09-21T08:28:34.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706053+0000", 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0299999999999997e-07 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706058+0000", 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0490000000000003e-06 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706071+0000", 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3542e-05 2024-09-21T08:28:34.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706077+0000", 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0730000000000001e-06 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706181+0000", 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103132 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:34.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706185+0000", 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.831e-06 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.723755+0000", 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017570065999999999 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.723835+0000", 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9320000000000006e-05 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.724208+0000", 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037308600000000001 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:832 7.b 7:d8f73c7b:::smithi13779221-38:head [sparse-read 0~2050454] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e136)", 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:29.706101+0000", 2024-09-21T08:28:34.885 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 185.03092052400001, 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018300291, 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 832 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706101+0000", 2024-09-21T08:28:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706101+0000", 2024-09-21T08:28:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706103+0000", 2024-09-21T08:28:34.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.508e-06 2024-09-21T08:28:34.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706108+0000", 2024-09-21T08:28:34.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2979999999999999e-06 2024-09-21T08:28:34.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706118+0000", 2024-09-21T08:28:34.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0033e-05 2024-09-21T08:28:34.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706128+0000", 2024-09-21T08:28:34.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4360000000000008e-06 2024-09-21T08:28:34.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706208+0000", 2024-09-21T08:28:34.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9973000000000002e-05 2024-09-21T08:28:34.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:34.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.706212+0000", 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9280000000000001e-06 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.724233+0000", 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018020673000000001 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.724281+0000", 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8254000000000003e-05 2024-09-21T08:28:34.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:29.724402+0000", 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120188 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1014 7.b 7:d6c5a045:::smithi13779221-46:head [copy-from ver 47] snapc 22=[22,21,1e,1d,1c] ondisk+write+known_if_redirected+supports_pool_eio e147)", 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:40.781122+0000", 2024-09-21T08:28:34.892 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 173.95589998700001, 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018020263000000002, 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1014 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781122+0000", 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781122+0000", 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781124+0000", 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5119999999999998e-06 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781133+0000", 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5169999999999999e-06 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781148+0000", 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5481e-05 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781158+0000", 2024-09-21T08:28:34.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2869999999999993e-06 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781222+0000", 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4428e-05 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.781227+0000", 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0690000000000001e-06 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.784918+0000", 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0036913620000000001 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.785019+0000", 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100597 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.790943+0000", 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059236510000000003 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.797751+0000", 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068083559999999998 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.798733+0000", 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00098198100000000004 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.799028+0000", 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029540499999999999 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.799088+0000", 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9206999999999997e-05 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:40.799142+0000", 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4410000000000003e-05 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3155 7.9 7:9247b5b1:::smithi13779221-17:head [copy-from ver 183] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:28:34.899 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.1645257569999998, 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029797267999999998, 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3155 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572497+0000", 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4899999999999995e-07 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572500+0000", 2024-09-21T08:28:34.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.05e-06 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572505+0000", 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2820000000000004e-06 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572510+0000", 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8969999999999999e-06 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573522+0000", 2024-09-21T08:28:34.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001012481 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573635+0000", 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112253 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.583897+0000", 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010262288 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594011+0000", 2024-09-21T08:28:34.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010113641 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594202+0000", 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000191418 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602174+0000", 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079717079999999992 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602233+0000", 2024-09-21T08:28:34.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9565999999999999e-05 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602293+0000", 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0134999999999999e-05 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3156 7.9 7:9247b5b1:::smithi13779221-17:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.572523+0000", 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.1644985459999999, 2024-09-21T08:28:34.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017642620000000001, 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3156 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572523+0000", 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572523+0000", 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572524+0000", 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9999999999999997e-07 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572526+0000", 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.295e-06 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572532+0000", 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.755e-06 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572537+0000", 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2349999999999999e-06 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573992+0000", 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001455214 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.574031+0000", 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8529999999999999e-05 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.589924+0000", 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015893280999999999 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.590040+0000", 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116118 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:34.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.590166+0000", 2024-09-21T08:28:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000125592 2024-09-21T08:28:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:34.910 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:34.910 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:28:34.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:34.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:34.932 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:34.982 INFO:tasks.rados.rados.0.smithi137.stdout:1211: finishing write tid 3 to smithi13779221-30 2024-09-21T08:28:34.982 INFO:tasks.rados.rados.0.smithi137.stdout:1211: finishing write tid 4 to smithi13779221-30 2024-09-21T08:28:34.982 INFO:tasks.rados.rados.0.smithi137.stdout:1211: finishing write tid 5 to smithi13779221-30 2024-09-21T08:28:34.982 INFO:tasks.rados.rados.0.smithi137.stdout:1211: oid 30 updating version 0 to 520 2024-09-21T08:28:34.983 INFO:tasks.rados.rados.0.smithi137.stdout:1211: oid 30 updating version 520 to 521 2024-09-21T08:28:34.983 INFO:tasks.rados.rados.0.smithi137.stdout:1211: oid 30 version 521 is already newer than 519 2024-09-21T08:28:34.983 INFO:tasks.rados.rados.0.smithi137.stdout:1211: oid 30 updating version 521 to 522 2024-09-21T08:28:34.983 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 522 (ObjNum 448 snap 117 seq_num 448) dirty exists 2024-09-21T08:28:34.983 INFO:tasks.rados.rados.0.smithi137.stdout:1211: left oid 30 (ObjNum 448 snap 117 seq_num 448) 2024-09-21T08:28:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:34 smithi165 ceph-mon[32893]: 7.18 scrub starts 2024-09-21T08:28:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:34 smithi165 ceph-mon[32893]: 7.18 scrub ok 2024-09-21T08:28:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:34 smithi165 ceph-mon[32893]: osdmap e321: 8 total, 8 up, 4 in 2024-09-21T08:28:35.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:35.055 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:35.056 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:35.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:35.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:35.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:35.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:35.180 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:35.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:35.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:35.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:35.302 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:35.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:28:35.404 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:35.426 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:35.427 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:35.427 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:35.528 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:35.551 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:35.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:35.552 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:35.553 INFO:tasks.rados.rados.0.smithi137.stdout:1208: done (4 left) 2024-09-21T08:28:35.554 INFO:tasks.rados.rados.0.smithi137.stdout:1209: done (3 left) 2024-09-21T08:28:35.554 INFO:tasks.rados.rados.0.smithi137.stdout:1210: done (2 left) 2024-09-21T08:28:35.554 INFO:tasks.rados.rados.0.smithi137.stdout:1211: done (1 left) 2024-09-21T08:28:35.554 INFO:tasks.rados.rados.0.smithi137.stdout:1212: done (0 left) 2024-09-21T08:28:35.554 INFO:tasks.rados.rados.0.smithi137.stdout:1213: write oid 14 current snap is 118 2024-09-21T08:28:35.554 INFO:tasks.rados.rados.0.smithi137.stdout:1213: seq_num 449 ranges {746343=651628,1924415=754807,3397159=149184} 2024-09-21T08:28:35.561 INFO:tasks.rados.rados.0.smithi137.stdout:1213: writing smithi13779221-14 from 746343 to 1397971 tid 1 2024-09-21T08:28:35.570 INFO:tasks.rados.rados.0.smithi137.stdout:1213: writing smithi13779221-14 from 1924415 to 2679222 tid 2 2024-09-21T08:28:35.575 INFO:tasks.rados.rados.0.smithi137.stdout:1213: writing smithi13779221-14 from 3397159 to 3546343 tid 3 2024-09-21T08:28:35.576 INFO:tasks.rados.rados.0.smithi137.stdout:1214: copy_from oid 6 from oid 24 current snap is 118 2024-09-21T08:28:35.576 INFO:tasks.rados.rados.0.smithi137.stdout:1213: finishing write tid 1 to smithi13779221-14 2024-09-21T08:28:35.576 INFO:tasks.rados.rados.0.smithi137.stdout:1213: finishing write tid 2 to smithi13779221-14 2024-09-21T08:28:35.576 INFO:tasks.rados.rados.0.smithi137.stdout:1215: read oid 12 snap -1 2024-09-21T08:28:35.576 INFO:tasks.rados.rados.0.smithi137.stdout:1215: expect (ObjNum 438 snap 113 seq_num 438) 2024-09-21T08:28:35.576 INFO:tasks.rados.rados.0.smithi137.stdout:1216: read oid 39 snap -1 2024-09-21T08:28:35.576 INFO:tasks.rados.rados.0.smithi137.stdout:1216: expect deleted 2024-09-21T08:28:35.576 INFO:tasks.rados.rados.0.smithi137.stdout:1217: write oid 20 current snap is 118 2024-09-21T08:28:35.576 INFO:tasks.rados.rados.0.smithi137.stdout:1217: seq_num 450 ranges {763150=726877,1982897=614818,3024582=474994} 2024-09-21T08:28:35.585 INFO:tasks.rados.rados.0.smithi137.stdout:1217: writing smithi13779221-20 from 763150 to 1490027 tid 1 2024-09-21T08:28:35.592 INFO:tasks.rados.rados.0.smithi137.stdout:1217: writing smithi13779221-20 from 1982897 to 2597715 tid 2 2024-09-21T08:28:35.596 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T08:28:35.597 INFO:tasks.rados.rados.0.smithi137.stdout:1217: writing smithi13779221-20 from 3024582 to 3499576 tid 3 2024-09-21T08:28:35.597 INFO:tasks.rados.rados.0.smithi137.stdout:1218: snap_remove snap 114 2024-09-21T08:28:35.619 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:35.653 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:35.658 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:35.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:35.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:35.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:35.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:35.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:35.804 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:35.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[41044]: pgmap v386: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 7 active+undersized+remapped, 3 active+clean+snaptrim, 1 active+remapped, 9 active+clean+snaptrim_wait, 143 active+clean; 178 MiB data, 504 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 3.7 MiB/s wr, 16 op/s; 22/822 objects degraded (2.676%); 16/822 objects misplaced (1.946%); 11 MiB/s, 74 objects/s recovering 2024-09-21T08:28:35.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[41044]: 6.0 scrub starts 2024-09-21T08:28:35.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[41044]: 6.0 scrub ok 2024-09-21T08:28:35.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[41044]: 5.19 scrub starts 2024-09-21T08:28:35.887 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[41044]: 5.19 scrub ok 2024-09-21T08:28:35.887 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[41044]: osdmap e322: 8 total, 8 up, 4 in 2024-09-21T08:28:35.887 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[29634]: pgmap v386: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 7 active+undersized+remapped, 3 active+clean+snaptrim, 1 active+remapped, 9 active+clean+snaptrim_wait, 143 active+clean; 178 MiB data, 504 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 3.7 MiB/s wr, 16 op/s; 22/822 objects degraded (2.676%); 16/822 objects misplaced (1.946%); 11 MiB/s, 74 objects/s recovering 2024-09-21T08:28:35.887 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[29634]: 6.0 scrub starts 2024-09-21T08:28:35.887 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[29634]: 6.0 scrub ok 2024-09-21T08:28:35.887 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[29634]: 5.19 scrub starts 2024-09-21T08:28:35.887 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[29634]: 5.19 scrub ok 2024-09-21T08:28:35.887 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:35 smithi137 ceph-mon[29634]: osdmap e322: 8 total, 8 up, 4 in 2024-09-21T08:28:35.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:35.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:35.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:35.930 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:36.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:35 smithi165 ceph-mon[32893]: pgmap v386: 168 pgs: 1 active+recovering+undersized+degraded+remapped, 1 active+recovery_wait+undersized+degraded+remapped, 3 active+recovering+undersized+remapped, 7 active+undersized+remapped, 3 active+clean+snaptrim, 1 active+remapped, 9 active+clean+snaptrim_wait, 143 active+clean; 178 MiB data, 504 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 3.7 MiB/s wr, 16 op/s; 22/822 objects degraded (2.676%); 16/822 objects misplaced (1.946%); 11 MiB/s, 74 objects/s recovering 2024-09-21T08:28:36.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:35 smithi165 ceph-mon[32893]: 6.0 scrub starts 2024-09-21T08:28:36.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:35 smithi165 ceph-mon[32893]: 6.0 scrub ok 2024-09-21T08:28:36.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:35 smithi165 ceph-mon[32893]: 5.19 scrub starts 2024-09-21T08:28:36.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:35 smithi165 ceph-mon[32893]: 5.19 scrub ok 2024-09-21T08:28:36.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:35 smithi165 ceph-mon[32893]: osdmap e322: 8 total, 8 up, 4 in 2024-09-21T08:28:36.030 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:36.056 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:36.057 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:36.057 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:36.158 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:36.183 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:36.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:36.184 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:36.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:28:36.284 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:36.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:36.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:36.330 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:36.431 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:36.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:36.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:36.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:36.549 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.2 primary-affinity to 0.305256 (8200052) 2024-09-21T08:28:36.553 INFO:tasks.rados.rados.0.smithi137.stdout:1218: done (5 left) 2024-09-21T08:28:36.553 INFO:tasks.rados.rados.0.smithi137.stdout:1219: delete oid 18 current snap is 118 2024-09-21T08:28:36.553 INFO:tasks.rados.rados.0.smithi137.stdout:1213: finishing write tid 3 to smithi13779221-14 2024-09-21T08:28:36.553 INFO:tasks.rados.rados.0.smithi137.stdout:1213: finishing write tid 4 to smithi13779221-14 2024-09-21T08:28:36.553 INFO:tasks.rados.rados.0.smithi137.stdout:1213: finishing write tid 5 to smithi13779221-14 2024-09-21T08:28:36.553 INFO:tasks.rados.rados.0.smithi137.stdout:1213: oid 14 updating version 0 to 258 2024-09-21T08:28:36.553 INFO:tasks.rados.rados.0.smithi137.stdout:1213: oid 14 updating version 258 to 260 2024-09-21T08:28:36.553 INFO:tasks.rados.rados.0.smithi137.stdout:1213: oid 14 version 260 is already newer than 259 2024-09-21T08:28:36.554 INFO:tasks.rados.rados.0.smithi137.stdout:1213: oid 14 version 260 is already newer than 257 2024-09-21T08:28:36.554 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 260 (ObjNum 449 snap 118 seq_num 449) dirty exists 2024-09-21T08:28:36.554 INFO:tasks.rados.rados.0.smithi137.stdout:1213: left oid 14 (ObjNum 449 snap 118 seq_num 449) 2024-09-21T08:28:36.554 INFO:tasks.rados.rados.0.smithi137.stdout:1215: expect (ObjNum 438 snap 113 seq_num 438) 2024-09-21T08:28:36.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:36.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:36.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:36.588 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:36.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:36.710 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:36.711 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:36.711 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:36.713 INFO:tasks.rados.rados.0.smithi137.stdout:1213: done (5 left) 2024-09-21T08:28:36.713 INFO:tasks.rados.rados.0.smithi137.stdout:1215: done (4 left) 2024-09-21T08:28:36.713 INFO:tasks.rados.rados.0.smithi137.stdout:1216: done (3 left) 2024-09-21T08:28:36.713 INFO:tasks.rados.rados.0.smithi137.stdout:1219: done (2 left) 2024-09-21T08:28:36.713 INFO:tasks.rados.rados.0.smithi137.stdout:1220: rollback oid 8 current snap is 118 2024-09-21T08:28:36.713 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 8 to 103 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1214: finishing copy_from racing read to smithi13779221-6 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1214: finishing copy_from to smithi13779221-6 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 295 (ObjNum 433 snap 110 seq_num 433) dirty exists 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1217: finishing write tid 1 to smithi13779221-20 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1217: finishing write tid 2 to smithi13779221-20 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1217: finishing write tid 3 to smithi13779221-20 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1217: finishing write tid 4 to smithi13779221-20 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1217: finishing write tid 5 to smithi13779221-20 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1217: oid 20 updating version 0 to 391 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1217: oid 20 version 391 is already newer than 388 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1217: oid 20 version 391 is already newer than 389 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:1217: oid 20 version 391 is already newer than 390 2024-09-21T08:28:36.714 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 391 (ObjNum 450 snap 118 seq_num 450) dirty exists 2024-09-21T08:28:36.715 INFO:tasks.rados.rados.0.smithi137.stdout:1217: left oid 20 (ObjNum 450 snap 118 seq_num 450) 2024-09-21T08:28:36.715 INFO:tasks.rados.rados.0.smithi137.stdout:1214: done (2 left) 2024-09-21T08:28:36.715 INFO:tasks.rados.rados.0.smithi137.stdout:1217: done (1 left) 2024-09-21T08:28:36.715 INFO:tasks.rados.rados.0.smithi137.stdout:1221: snap_create 2024-09-21T08:28:36.812 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:36.834 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:36.835 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:36.835 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:36.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:36 smithi137 ceph-mon[29634]: 6.6 scrub starts 2024-09-21T08:28:36.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:36 smithi137 ceph-mon[29634]: 6.6 scrub ok 2024-09-21T08:28:36.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:36 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3084582064' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3052560012589116}]: dispatch 2024-09-21T08:28:36.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:36 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3084582064' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3052560012589116}]': finished 2024-09-21T08:28:36.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:36 smithi137 ceph-mon[29634]: osdmap e323: 8 total, 8 up, 4 in 2024-09-21T08:28:36.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:36 smithi137 ceph-mon[41044]: 6.6 scrub starts 2024-09-21T08:28:36.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:36 smithi137 ceph-mon[41044]: 6.6 scrub ok 2024-09-21T08:28:36.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:36 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3084582064' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3052560012589116}]: dispatch 2024-09-21T08:28:36.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:36 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3084582064' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3052560012589116}]': finished 2024-09-21T08:28:36.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:36 smithi137 ceph-mon[41044]: osdmap e323: 8 total, 8 up, 4 in 2024-09-21T08:28:36.935 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:36.961 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:36.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:36.962 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:36 smithi165 ceph-mon[32893]: 6.6 scrub starts 2024-09-21T08:28:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:36 smithi165 ceph-mon[32893]: 6.6 scrub ok 2024-09-21T08:28:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:36 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3084582064' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3052560012589116}]: dispatch 2024-09-21T08:28:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:36 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3084582064' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 2, "weight": 0.3052560012589116}]': finished 2024-09-21T08:28:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:36 smithi165 ceph-mon[32893]: osdmap e323: 8 total, 8 up, 4 in 2024-09-21T08:28:37.064 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:37.089 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:37.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:37.090 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:37.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:37.214 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:37.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:37.215 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:37.250 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:37.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:37.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:37.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:37.339 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:37.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:37.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:37.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:37.461 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:37.555 INFO:tasks.rados.rados.0.smithi137.stdout:1221: done (1 left) 2024-09-21T08:28:37.555 INFO:tasks.rados.rados.0.smithi137.stdout:1222: read oid 27 snap 117 2024-09-21T08:28:37.555 INFO:tasks.rados.rados.0.smithi137.stdout:1222: expect deleted 2024-09-21T08:28:37.555 INFO:tasks.rados.rados.0.smithi137.stdout:1223: write (excl) oid 2 current snap is 119 2024-09-21T08:28:37.555 INFO:tasks.rados.rados.0.smithi137.stdout:1223: seq_num 452 ranges {796764=477375,1983508=618487,3363075=233689} 2024-09-21T08:28:37.556 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:28:37.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:37.563 INFO:tasks.rados.rados.0.smithi137.stdout:1223: writing smithi13779221-2 from 796764 to 1274139 tid 1 2024-09-21T08:28:37.571 INFO:tasks.rados.rados.0.smithi137.stdout:1223: writing smithi13779221-2 from 1983508 to 2601995 tid 2 2024-09-21T08:28:37.576 INFO:tasks.rados.rados.0.smithi137.stdout:1223: writing smithi13779221-2 from 3363075 to 3596764 tid 3 2024-09-21T08:28:37.576 INFO:tasks.rados.rados.0.smithi137.stdout:1224: read oid 4 snap 111 2024-09-21T08:28:37.577 INFO:tasks.rados.rados.0.smithi137.stdout:1224: expect (ObjNum 421 snap 103 seq_num 421) 2024-09-21T08:28:37.577 INFO:tasks.rados.rados.0.smithi137.stdout:1225: write (excl) oid 48 current snap is 119 2024-09-21T08:28:37.577 INFO:tasks.rados.rados.0.smithi137.stdout:1225: seq_num 453 ranges {413571=552624,1641990=762145,2990498=425163} 2024-09-21T08:28:37.582 INFO:tasks.rados.rados.0.smithi137.stdout:1225: writing smithi13779221-48 from 413571 to 966195 tid 1 2024-09-21T08:28:37.590 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:37.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:37.591 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:37.591 INFO:tasks.rados.rados.0.smithi137.stdout:1225: writing smithi13779221-48 from 1641990 to 2404135 tid 2 2024-09-21T08:28:37.597 INFO:tasks.rados.rados.0.smithi137.stdout:1225: writing smithi13779221-48 from 2990498 to 3415661 tid 3 2024-09-21T08:28:37.597 INFO:tasks.rados.rados.0.smithi137.stdout:1226: read oid 23 snap -1 2024-09-21T08:28:37.597 INFO:tasks.rados.rados.0.smithi137.stdout:1226: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:28:37.597 INFO:tasks.rados.rados.0.smithi137.stdout:1227: read oid 37 snap -1 2024-09-21T08:28:37.597 INFO:tasks.rados.rados.0.smithi137.stdout:1227: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:37.598 INFO:tasks.rados.rados.0.smithi137.stdout:1228: snap_create 2024-09-21T08:28:37.598 INFO:tasks.rados.rados.0.smithi137.stdout:1223: finishing write tid 1 to smithi13779221-2 2024-09-21T08:28:37.598 INFO:tasks.rados.rados.0.smithi137.stdout:1220: finishing rollback tid 0 to smithi13779221-8 2024-09-21T08:28:37.598 INFO:tasks.rados.rados.0.smithi137.stdout:1220: finishing rollback tid 1 to smithi13779221-8 2024-09-21T08:28:37.598 INFO:tasks.rados.rados.0.smithi137.stdout:1220: finishing rollback tid 2 to smithi13779221-8 2024-09-21T08:28:37.598 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 301 (ObjNum 404 snap 99 seq_num 404) dirty exists 2024-09-21T08:28:37.598 INFO:tasks.rados.rados.0.smithi137.stdout:1223: finishing write tid 2 to smithi13779221-2 2024-09-21T08:28:37.598 INFO:tasks.rados.rados.0.smithi137.stdout:1223: finishing write tid 3 to smithi13779221-2 2024-09-21T08:28:37.598 INFO:tasks.rados.rados.0.smithi137.stdout:1223: finishing write tid 4 to smithi13779221-2 2024-09-21T08:28:37.598 INFO:tasks.rados.rados.0.smithi137.stdout:1223: finishing write tid 5 to smithi13779221-2 2024-09-21T08:28:37.599 INFO:tasks.rados.rados.0.smithi137.stdout:1223: oid 2 updating version 0 to 144 2024-09-21T08:28:37.599 INFO:tasks.rados.rados.0.smithi137.stdout:1223: oid 2 version 144 is already newer than 143 2024-09-21T08:28:37.599 INFO:tasks.rados.rados.0.smithi137.stdout:1223: oid 2 version 144 is already newer than 142 2024-09-21T08:28:37.599 INFO:tasks.rados.rados.0.smithi137.stdout:1223: oid 2 updating version 144 to 145 2024-09-21T08:28:37.599 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 145 (ObjNum 452 snap 119 seq_num 452) dirty exists 2024-09-21T08:28:37.599 INFO:tasks.rados.rados.0.smithi137.stdout:1223: left oid 2 (ObjNum 452 snap 119 seq_num 452) 2024-09-21T08:28:37.599 INFO:tasks.rados.rados.0.smithi137.stdout:1225: finishing write tid 1 to smithi13779221-48 2024-09-21T08:28:37.599 INFO:tasks.rados.rados.0.smithi137.stdout:1224: expect (ObjNum 421 snap 103 seq_num 421) 2024-09-21T08:28:37.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:37.705 INFO:tasks.rados.rados.0.smithi137.stdout:1225: finishing write tid 2 to smithi13779221-48 2024-09-21T08:28:37.705 INFO:tasks.rados.rados.0.smithi137.stdout:1225: finishing write tid 3 to smithi13779221-48 2024-09-21T08:28:37.705 INFO:tasks.rados.rados.0.smithi137.stdout:1225: finishing write tid 4 to smithi13779221-48 2024-09-21T08:28:37.705 INFO:tasks.rados.rados.0.smithi137.stdout:1225: finishing write tid 5 to smithi13779221-48 2024-09-21T08:28:37.705 INFO:tasks.rados.rados.0.smithi137.stdout:1225: oid 48 updating version 0 to 316 2024-09-21T08:28:37.705 INFO:tasks.rados.rados.0.smithi137.stdout:1225: oid 48 version 316 is already newer than 315 2024-09-21T08:28:37.706 INFO:tasks.rados.rados.0.smithi137.stdout:1225: oid 48 updating version 316 to 317 2024-09-21T08:28:37.706 INFO:tasks.rados.rados.0.smithi137.stdout:1225: oid 48 version 317 is already newer than 314 2024-09-21T08:28:37.706 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 317 (ObjNum 453 snap 119 seq_num 453) dirty exists 2024-09-21T08:28:37.706 INFO:tasks.rados.rados.0.smithi137.stdout:1225: left oid 48 (ObjNum 453 snap 119 seq_num 453) 2024-09-21T08:28:37.706 INFO:tasks.rados.rados.0.smithi137.stdout:1226: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:28:37.713 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:37.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:37.714 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[29634]: pgmap v389: 168 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 151 active+clean; 199 MiB data, 650 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 6.5 MiB/s wr, 20 op/s; 11 MiB/s, 19 objects/s recovering 2024-09-21T08:28:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[29634]: 6.9 scrub starts 2024-09-21T08:28:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[29634]: 6.9 scrub ok 2024-09-21T08:28:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[29634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/822 objects degraded (2.676%), 2 pgs degraded) 2024-09-21T08:28:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2562331320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[29634]: from='client.45431 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:28:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:28:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[29634]: from='client.45431 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[29634]: osdmap e324: 8 total, 8 up, 4 in 2024-09-21T08:28:37.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[41044]: pgmap v389: 168 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 151 active+clean; 199 MiB data, 650 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 6.5 MiB/s wr, 20 op/s; 11 MiB/s, 19 objects/s recovering 2024-09-21T08:28:37.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[41044]: 6.9 scrub starts 2024-09-21T08:28:37.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[41044]: 6.9 scrub ok 2024-09-21T08:28:37.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[41044]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/822 objects degraded (2.676%), 2 pgs degraded) 2024-09-21T08:28:37.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2562331320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:37.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[41044]: from='client.45431 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:37.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:28:37.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:28:37.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[41044]: from='client.45431 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:37.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:37 smithi137 ceph-mon[41044]: osdmap e324: 8 total, 8 up, 4 in 2024-09-21T08:28:37.815 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:37.837 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:37.837 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:37.838 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:37.907 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:28:37.907 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:28:37.907 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:28:37.907 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:28:37.932 INFO:tasks.rados.rados.0.smithi137.stdout:1227: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:37.938 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:37.960 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:37.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:37.961 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:37 smithi165 ceph-mon[32893]: pgmap v389: 168 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 151 active+clean; 199 MiB data, 650 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 6.5 MiB/s wr, 20 op/s; 11 MiB/s, 19 objects/s recovering 2024-09-21T08:28:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:37 smithi165 ceph-mon[32893]: 6.9 scrub starts 2024-09-21T08:28:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:37 smithi165 ceph-mon[32893]: 6.9 scrub ok 2024-09-21T08:28:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:37 smithi165 ceph-mon[32893]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 22/822 objects degraded (2.676%), 2 pgs degraded) 2024-09-21T08:28:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2562331320' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:37 smithi165 ceph-mon[32893]: from='client.45431 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:37 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:28:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:37 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:28:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:37 smithi165 ceph-mon[32893]: from='client.45431 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:38.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:37 smithi165 ceph-mon[32893]: osdmap e324: 8 total, 8 up, 4 in 2024-09-21T08:28:38.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:38.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:38.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:38.086 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:38.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:38.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:38.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:38.211 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:38.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:38.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:38.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:38.336 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:38.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:38.458 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:38.459 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:38.459 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:38.556 INFO:tasks.rados.rados.0.smithi137.stdout:1220: done (7 left) 2024-09-21T08:28:38.556 INFO:tasks.rados.rados.0.smithi137.stdout:1222: done (6 left) 2024-09-21T08:28:38.556 INFO:tasks.rados.rados.0.smithi137.stdout:1223: done (5 left) 2024-09-21T08:28:38.556 INFO:tasks.rados.rados.0.smithi137.stdout:1224: done (4 left) 2024-09-21T08:28:38.556 INFO:tasks.rados.rados.0.smithi137.stdout:1225: done (3 left) 2024-09-21T08:28:38.556 INFO:tasks.rados.rados.0.smithi137.stdout:1226: done (2 left) 2024-09-21T08:28:38.556 INFO:tasks.rados.rados.0.smithi137.stdout:1227: done (1 left) 2024-09-21T08:28:38.556 INFO:tasks.rados.rados.0.smithi137.stdout:1228: done (0 left) 2024-09-21T08:28:38.556 INFO:tasks.rados.rados.0.smithi137.stdout:1229: read oid 2 snap -1 2024-09-21T08:28:38.556 INFO:tasks.rados.rados.0.smithi137.stdout:1229: expect (ObjNum 452 snap 119 seq_num 452) 2024-09-21T08:28:38.557 INFO:tasks.rados.rados.0.smithi137.stdout:1230: rollback oid 33 current snap is 120 2024-09-21T08:28:38.557 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 33 to 108 2024-09-21T08:28:38.557 INFO:tasks.rados.rados.0.smithi137.stdout:1231: read oid 10 snap -1 2024-09-21T08:28:38.557 INFO:tasks.rados.rados.0.smithi137.stdout:1231: expect (ObjNum 385 snap 94 seq_num 385) 2024-09-21T08:28:38.557 INFO:tasks.rados.rados.0.smithi137.stdout:1232: snap_create 2024-09-21T08:28:38.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:38.562 INFO:tasks.rados.rados.0.smithi137.stdout:1230: finishing rollback tid 0 to smithi13779221-33 2024-09-21T08:28:38.562 INFO:tasks.rados.rados.0.smithi137.stdout:1230: finishing rollback tid 1 to smithi13779221-33 2024-09-21T08:28:38.564 INFO:tasks.rados.rados.0.smithi137.stdout:1229: expect (ObjNum 452 snap 119 seq_num 452) 2024-09-21T08:28:38.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:38.588 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:38.589 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:38.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:38.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:38.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:38.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:38.761 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:38.799 INFO:tasks.rados.rados.0.smithi137.stdout:1231: expect (ObjNum 385 snap 94 seq_num 385) 2024-09-21T08:28:38.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:38.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:38.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:38.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:38.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:38 smithi137 ceph-mon[29634]: 6.3 scrub starts 2024-09-21T08:28:38.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:38 smithi137 ceph-mon[29634]: 6.3 scrub ok 2024-09-21T08:28:38.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:38 smithi137 ceph-mon[29634]: 3.17 scrub starts 2024-09-21T08:28:38.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:38 smithi137 ceph-mon[29634]: 3.17 scrub ok 2024-09-21T08:28:38.864 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:38 smithi137 ceph-mon[29634]: osdmap e325: 8 total, 8 up, 4 in 2024-09-21T08:28:38.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:38 smithi137 ceph-mon[41044]: 6.3 scrub starts 2024-09-21T08:28:38.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:38 smithi137 ceph-mon[41044]: 6.3 scrub ok 2024-09-21T08:28:38.864 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:38 smithi137 ceph-mon[41044]: 3.17 scrub starts 2024-09-21T08:28:38.865 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:38 smithi137 ceph-mon[41044]: 3.17 scrub ok 2024-09-21T08:28:38.865 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:38 smithi137 ceph-mon[41044]: osdmap e325: 8 total, 8 up, 4 in 2024-09-21T08:28:38.933 INFO:tasks.rados.rados.0.smithi137.stdout:1230: finishing rollback tid 2 to smithi13779221-33 2024-09-21T08:28:38.933 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 282 (ObjNum 302 snap 70 seq_num 302) dirty exists 2024-09-21T08:28:38.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:38.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:38.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:38.959 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:38 smithi165 ceph-mon[32893]: 6.3 scrub starts 2024-09-21T08:28:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:38 smithi165 ceph-mon[32893]: 6.3 scrub ok 2024-09-21T08:28:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:38 smithi165 ceph-mon[32893]: 3.17 scrub starts 2024-09-21T08:28:39.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:38 smithi165 ceph-mon[32893]: 3.17 scrub ok 2024-09-21T08:28:39.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:38 smithi165 ceph-mon[32893]: osdmap e325: 8 total, 8 up, 4 in 2024-09-21T08:28:39.060 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:39.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:39.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:39.082 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:39.183 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:39.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:39.205 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:39.205 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:39.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:39.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:39.328 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:39.328 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:39.429 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:39.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:39.455 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:39.456 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:39.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T08:28:39.555 INFO:tasks.rados.rados.0.smithi137.stdout:1229: done (3 left) 2024-09-21T08:28:39.555 INFO:tasks.rados.rados.0.smithi137.stdout:1230: done (2 left) 2024-09-21T08:28:39.555 INFO:tasks.rados.rados.0.smithi137.stdout:1231: done (1 left) 2024-09-21T08:28:39.555 INFO:tasks.rados.rados.0.smithi137.stdout:1232: done (0 left) 2024-09-21T08:28:39.555 INFO:tasks.rados.rados.0.smithi137.stdout:1233: write (excl) oid 16 current snap is 121 2024-09-21T08:28:39.555 INFO:tasks.rados.rados.0.smithi137.stdout:1233: seq_num 454 ranges {430378=627873,1700472=622156,3017921=612457} 2024-09-21T08:28:39.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:39.561 INFO:tasks.rados.rados.0.smithi137.stdout:1233: writing smithi13779221-16 from 430378 to 1058251 tid 1 2024-09-21T08:28:39.568 INFO:tasks.rados.rados.0.smithi137.stdout:1233: writing smithi13779221-16 from 1700472 to 2322628 tid 2 2024-09-21T08:28:39.576 INFO:tasks.rados.rados.0.smithi137.stdout:1233: writing smithi13779221-16 from 3017921 to 3630378 tid 3 2024-09-21T08:28:39.576 INFO:tasks.rados.rados.0.smithi137.stdout:1234: rollback oid 5 current snap is 121 2024-09-21T08:28:39.576 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 5 to 103 2024-09-21T08:28:39.576 INFO:tasks.rados.rados.0.smithi137.stdout:1233: finishing write tid 1 to smithi13779221-16 2024-09-21T08:28:39.577 INFO:tasks.rados.rados.0.smithi137.stdout:1233: finishing write tid 2 to smithi13779221-16 2024-09-21T08:28:39.577 INFO:tasks.rados.rados.0.smithi137.stdout:1235: write (excl) oid 25 current snap is 121 2024-09-21T08:28:39.577 INFO:tasks.rados.rados.0.smithi137.stdout:1235: seq_num 455 ranges {447185=703122,1642601=765814,2928991=447707} 2024-09-21T08:28:39.582 INFO:tasks.rados.rados.0.smithi137.stdout:1235: writing smithi13779221-25 from 447185 to 1150307 tid 1 2024-09-21T08:28:39.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:39.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:39.585 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:39.589 INFO:tasks.rados.rados.0.smithi137.stdout:1235: writing smithi13779221-25 from 1642601 to 2408415 tid 2 2024-09-21T08:28:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:1235: writing smithi13779221-25 from 2928991 to 3376698 tid 3 2024-09-21T08:28:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:1236: snap_create 2024-09-21T08:28:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:1233: finishing write tid 3 to smithi13779221-16 2024-09-21T08:28:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:1233: finishing write tid 4 to smithi13779221-16 2024-09-21T08:28:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:1233: finishing write tid 5 to smithi13779221-16 2024-09-21T08:28:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:1233: oid 16 updating version 0 to 326 2024-09-21T08:28:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:1233: oid 16 updating version 326 to 329 2024-09-21T08:28:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:1233: oid 16 version 329 is already newer than 327 2024-09-21T08:28:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:1233: oid 16 version 329 is already newer than 328 2024-09-21T08:28:39.595 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 329 (ObjNum 454 snap 121 seq_num 454) dirty exists 2024-09-21T08:28:39.596 INFO:tasks.rados.rados.0.smithi137.stdout:1233: left oid 16 (ObjNum 454 snap 121 seq_num 454) 2024-09-21T08:28:39.596 INFO:tasks.rados.rados.0.smithi137.stdout:1234: finishing rollback tid 0 to smithi13779221-5 2024-09-21T08:28:39.596 INFO:tasks.rados.rados.0.smithi137.stdout:1234: finishing rollback tid 1 to smithi13779221-5 2024-09-21T08:28:39.596 INFO:tasks.rados.rados.0.smithi137.stdout:1234: finishing rollback tid 2 to smithi13779221-5 2024-09-21T08:28:39.596 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 345 (ObjNum 419 snap 103 seq_num 419) dirty exists 2024-09-21T08:28:39.596 INFO:tasks.rados.rados.0.smithi137.stdout:1235: finishing write tid 1 to smithi13779221-25 2024-09-21T08:28:39.596 INFO:tasks.rados.rados.0.smithi137.stdout:1235: finishing write tid 2 to smithi13779221-25 2024-09-21T08:28:39.597 INFO:tasks.rados.rados.0.smithi137.stdout:1235: finishing write tid 3 to smithi13779221-25 2024-09-21T08:28:39.597 INFO:tasks.rados.rados.0.smithi137.stdout:1235: finishing write tid 4 to smithi13779221-25 2024-09-21T08:28:39.599 INFO:tasks.rados.rados.0.smithi137.stdout:1235: finishing write tid 5 to smithi13779221-25 2024-09-21T08:28:39.600 INFO:tasks.rados.rados.0.smithi137.stdout:1235: oid 25 updating version 0 to 334 2024-09-21T08:28:39.600 INFO:tasks.rados.rados.0.smithi137.stdout:1235: oid 25 version 334 is already newer than 333 2024-09-21T08:28:39.600 INFO:tasks.rados.rados.0.smithi137.stdout:1235: oid 25 updating version 334 to 335 2024-09-21T08:28:39.600 INFO:tasks.rados.rados.0.smithi137.stdout:1235: oid 25 version 335 is already newer than 332 2024-09-21T08:28:39.600 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 335 (ObjNum 455 snap 121 seq_num 455) dirty exists 2024-09-21T08:28:39.600 INFO:tasks.rados.rados.0.smithi137.stdout:1235: left oid 25 (ObjNum 455 snap 121 seq_num 455) 2024-09-21T08:28:39.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:39.710 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:39.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:39.711 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:39.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[29634]: 5.10 scrub starts 2024-09-21T08:28:39.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[29634]: 5.10 scrub ok 2024-09-21T08:28:39.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[29634]: pgmap v392: 168 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 151 active+clean; 199 MiB data, 650 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 6.5 MiB/s wr, 20 op/s; 11 MiB/s, 19 objects/s recovering 2024-09-21T08:28:39.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[29634]: 5.6 scrub starts 2024-09-21T08:28:39.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[29634]: 5.6 scrub ok 2024-09-21T08:28:39.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[29634]: 4.1a scrub starts 2024-09-21T08:28:39.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[29634]: 4.1a scrub ok 2024-09-21T08:28:39.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[29634]: osdmap e326: 8 total, 8 up, 4 in 2024-09-21T08:28:39.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[41044]: 5.10 scrub starts 2024-09-21T08:28:39.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[41044]: 5.10 scrub ok 2024-09-21T08:28:39.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[41044]: pgmap v392: 168 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 151 active+clean; 199 MiB data, 650 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 6.5 MiB/s wr, 20 op/s; 11 MiB/s, 19 objects/s recovering 2024-09-21T08:28:39.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[41044]: 5.6 scrub starts 2024-09-21T08:28:39.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[41044]: 5.6 scrub ok 2024-09-21T08:28:39.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[41044]: 4.1a scrub starts 2024-09-21T08:28:39.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[41044]: 4.1a scrub ok 2024-09-21T08:28:39.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:39 smithi137 ceph-mon[41044]: osdmap e326: 8 total, 8 up, 4 in 2024-09-21T08:28:39.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:39.834 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:39.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:39.836 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:39.936 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:39.959 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:39.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:39.961 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:39 smithi165 ceph-mon[32893]: 5.10 scrub starts 2024-09-21T08:28:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:39 smithi165 ceph-mon[32893]: 5.10 scrub ok 2024-09-21T08:28:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:39 smithi165 ceph-mon[32893]: pgmap v392: 168 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 151 active+clean; 199 MiB data, 650 MiB used, 357 GiB / 358 GiB avail; 9.1 MiB/s rd, 6.5 MiB/s wr, 20 op/s; 11 MiB/s, 19 objects/s recovering 2024-09-21T08:28:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:39 smithi165 ceph-mon[32893]: 5.6 scrub starts 2024-09-21T08:28:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:39 smithi165 ceph-mon[32893]: 5.6 scrub ok 2024-09-21T08:28:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:39 smithi165 ceph-mon[32893]: 4.1a scrub starts 2024-09-21T08:28:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:39 smithi165 ceph-mon[32893]: 4.1a scrub ok 2024-09-21T08:28:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:39 smithi165 ceph-mon[32893]: osdmap e326: 8 total, 8 up, 4 in 2024-09-21T08:28:40.030 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:40.030 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:40.031 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:40.048 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:40.048 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:40.062 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:40.070 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:40.071 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:40.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:40.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:40.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:40.087 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:40.087 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:40.104 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:40.104 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:40.121 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:40.121 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:40.137 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:40.138 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:40.154 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:40.154 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:40.185 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:40.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:40.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:40.209 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:40.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:40.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:40.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:40.333 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:40.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:40.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:40.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:40.457 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:40.557 INFO:tasks.rados.rados.0.smithi137.stdout:1233: done (3 left) 2024-09-21T08:28:40.558 INFO:tasks.rados.rados.0.smithi137.stdout:1234: done (2 left) 2024-09-21T08:28:40.558 INFO:tasks.rados.rados.0.smithi137.stdout:1235: done (1 left) 2024-09-21T08:28:40.558 INFO:tasks.rados.rados.0.smithi137.stdout:1236: done (0 left) 2024-09-21T08:28:40.558 INFO:tasks.rados.rados.0.smithi137.stdout:1237: snap_create 2024-09-21T08:28:40.558 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:40.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:28:40.581 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:40.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:40.582 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:40.667 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:28:40.667 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:28:40.683 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:40.704 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:40.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:40.705 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:40.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:40.828 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:40.829 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:40.829 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:40.930 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:40.951 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:40.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:40.952 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:40.977 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[29634]: 5.1e scrub starts 2024-09-21T08:28:40.977 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[29634]: 5.1e scrub ok 2024-09-21T08:28:40.977 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[29634]: 6.17 scrub starts 2024-09-21T08:28:40.977 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[29634]: 6.17 scrub ok 2024-09-21T08:28:40.977 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1784389615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:40.977 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[29634]: osdmap e327: 8 total, 8 up, 4 in 2024-09-21T08:28:40.978 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:28:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:28:40.978 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[41044]: 5.1e scrub starts 2024-09-21T08:28:40.978 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[41044]: 5.1e scrub ok 2024-09-21T08:28:40.978 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[41044]: 6.17 scrub starts 2024-09-21T08:28:40.978 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[41044]: 6.17 scrub ok 2024-09-21T08:28:40.978 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1784389615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:40.978 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:40 smithi137 ceph-mon[41044]: osdmap e327: 8 total, 8 up, 4 in 2024-09-21T08:28:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:40 smithi165 ceph-mon[32893]: 5.1e scrub starts 2024-09-21T08:28:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:40 smithi165 ceph-mon[32893]: 5.1e scrub ok 2024-09-21T08:28:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:40 smithi165 ceph-mon[32893]: 6.17 scrub starts 2024-09-21T08:28:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:40 smithi165 ceph-mon[32893]: 6.17 scrub ok 2024-09-21T08:28:41.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:40 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1784389615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:41.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:40 smithi165 ceph-mon[32893]: osdmap e327: 8 total, 8 up, 4 in 2024-09-21T08:28:41.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:41.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:41.076 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:41.076 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:41.177 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:41.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:41.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:41.200 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:41.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T08:28:41.301 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:41.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:41.324 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:41.324 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:41.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:41.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:41.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:41.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:41.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:41.553 INFO:tasks.rados.rados.0.smithi137.stdout:1237: done (0 left) 2024-09-21T08:28:41.553 INFO:tasks.rados.rados.0.smithi137.stdout:1238: copy_from oid 40 from oid 35 current snap is 123 2024-09-21T08:28:41.554 INFO:tasks.rados.rados.0.smithi137.stdout:1239: rollback oid 13 current snap is 123 2024-09-21T08:28:41.554 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 13 to 116 2024-09-21T08:28:41.554 INFO:tasks.rados.rados.0.smithi137.stdout:1240: copy_from oid 14 from oid 8 current snap is 123 2024-09-21T08:28:41.554 INFO:tasks.rados.rados.0.smithi137.stdout:1241: snap_remove snap 108 2024-09-21T08:28:41.578 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:41.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:41.579 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:41.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:41.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:41.703 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:41.703 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:41.721 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:41.804 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:41.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:41.826 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:41.826 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:41.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:41.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:41.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:41.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:41.976 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:41 smithi137 ceph-mon[29634]: 5.1a scrub starts 2024-09-21T08:28:41.976 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:41 smithi137 ceph-mon[29634]: 5.1a scrub ok 2024-09-21T08:28:41.976 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:41 smithi137 ceph-mon[29634]: pgmap v395: 168 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 151 active+clean; 199 MiB data, 650 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:41.976 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:41 smithi137 ceph-mon[29634]: osdmap e328: 8 total, 8 up, 4 in 2024-09-21T08:28:41.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:41 smithi137 ceph-mon[41044]: 5.1a scrub starts 2024-09-21T08:28:41.976 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:41 smithi137 ceph-mon[41044]: 5.1a scrub ok 2024-09-21T08:28:41.977 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:41 smithi137 ceph-mon[41044]: pgmap v395: 168 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 151 active+clean; 199 MiB data, 650 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:41.977 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:41 smithi137 ceph-mon[41044]: osdmap e328: 8 total, 8 up, 4 in 2024-09-21T08:28:41.988 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:41 smithi165 ceph-mon[32893]: 5.1a scrub starts 2024-09-21T08:28:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:41 smithi165 ceph-mon[32893]: 5.1a scrub ok 2024-09-21T08:28:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:41 smithi165 ceph-mon[32893]: pgmap v395: 168 pgs: 4 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 151 active+clean; 199 MiB data, 650 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:42.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:41 smithi165 ceph-mon[32893]: osdmap e328: 8 total, 8 up, 4 in 2024-09-21T08:28:42.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:28:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:28:42.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:42.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:42.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:42.075 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:42.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:42.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:42.203 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:42.203 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:42.238 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1] out_osds: [4, 6, 3, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:28:42.238 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:28:42.239 INFO:tasks.thrashosds.thrasher:primary_affinity 2024-09-21T08:28:42.239 INFO:tasks.thrashosds.thrasher:Setting osd 1 primary_affinity to 1.000000 2024-09-21T08:28:42.239 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd primary-affinity 1 1 2024-09-21T08:28:42.304 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:42.330 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:42.330 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:42.331 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:42.431 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:42.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:42.457 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:42.457 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:42.556 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:28:42.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:42.562 INFO:tasks.rados.rados.0.smithi137.stdout:1241: done (3 left) 2024-09-21T08:28:42.562 INFO:tasks.rados.rados.0.smithi137.stdout:1242: copy_from oid 27 from oid 45 current snap is 123 2024-09-21T08:28:42.562 INFO:tasks.rados.rados.0.smithi137.stdout:1239: finishing rollback tid 0 to smithi13779221-13 2024-09-21T08:28:42.562 INFO:tasks.rados.rados.0.smithi137.stdout:1239: finishing rollback tid 1 to smithi13779221-13 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1239: finishing rollback tid 2 to smithi13779221-13 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 515 (ObjNum 340 snap 80 seq_num 340) dirty exists 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1240: finishing copy_from racing read to smithi13779221-14 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1238: finishing copy_from racing read to smithi13779221-40 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1240: finishing copy_from to smithi13779221-14 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 301 (ObjNum 404 snap 99 seq_num 404) dirty exists 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1238: finishing copy_from to smithi13779221-40 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1238: done (3 left) 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1239: done (2 left) 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1240: done (1 left) 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1243: copy_from oid 18 from oid 47 current snap is 123 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1244: read oid 50 snap -1 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1244: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:28:42.563 INFO:tasks.rados.rados.0.smithi137.stdout:1245: read oid 48 snap 111 2024-09-21T08:28:42.564 INFO:tasks.rados.rados.0.smithi137.stdout:1245: expect (ObjNum 424 snap 107 seq_num 424) 2024-09-21T08:28:42.564 INFO:tasks.rados.rados.0.smithi137.stdout:1246: read oid 25 snap 112 2024-09-21T08:28:42.564 INFO:tasks.rados.rados.0.smithi137.stdout:1246: expect (ObjNum 426 snap 107 seq_num 426) 2024-09-21T08:28:42.564 INFO:tasks.rados.rados.0.smithi137.stdout:1247: snap_remove snap 113 2024-09-21T08:28:42.589 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:42.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:42.590 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:42.667 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:28:42.668 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:28:42.691 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:42.713 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:42.713 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:42.714 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:42.815 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:42.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:42.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:42.839 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:42.939 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:42.961 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:42.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:42.961 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:43.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:42 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/515950689' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:28:43.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:42 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:43.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:42 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/515950689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:28:43.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:42 smithi137 ceph-mon[29634]: osdmap e329: 8 total, 8 up, 4 in 2024-09-21T08:28:43.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:42 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/515950689' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:28:43.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:42 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:43.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:42 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/515950689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:28:43.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:42 smithi137 ceph-mon[41044]: osdmap e329: 8 total, 8 up, 4 in 2024-09-21T08:28:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:42 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/515950689' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:28:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:42 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:42 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/515950689' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:28:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:42 smithi165 ceph-mon[32893]: osdmap e329: 8 total, 8 up, 4 in 2024-09-21T08:28:43.063 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:43.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:43.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:43.085 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:43.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:43.207 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:43.208 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:43.208 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:43.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:43.332 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:43.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:43.333 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:43.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:43.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:43.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:43.458 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:43.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:43.560 INFO:tasks.rados.rados.0.smithi137.stdout:1247: done (5 left) 2024-09-21T08:28:43.560 INFO:tasks.rados.rados.0.smithi137.stdout:1248: write (excl) oid 14 current snap is 123 2024-09-21T08:28:43.561 INFO:tasks.rados.rados.0.smithi137.stdout:1248: seq_num 456 ranges {463992=778371,1701083=625825,2956414=658979} 2024-09-21T08:28:43.566 INFO:tasks.rados.rados.0.smithi137.stdout:1248: writing smithi13779221-14 from 463992 to 1242363 tid 1 2024-09-21T08:28:43.573 INFO:tasks.rados.rados.0.smithi137.stdout:1248: writing smithi13779221-14 from 1701083 to 2326908 tid 2 2024-09-21T08:28:43.581 INFO:tasks.rados.rados.0.smithi137.stdout:1248: writing smithi13779221-14 from 2956414 to 3615393 tid 3 2024-09-21T08:28:43.581 INFO:tasks.rados.rados.0.smithi137.stdout:1249: delete oid 36 current snap is 123 2024-09-21T08:28:43.581 INFO:tasks.rados.rados.0.smithi137.stdout:1245: expect (ObjNum 424 snap 107 seq_num 424) 2024-09-21T08:28:43.587 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:43.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:43.588 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:43.688 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:43.709 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:43.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:43.710 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:43.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:28:43.811 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:43.813 INFO:tasks.rados.rados.0.smithi137.stdout:1244: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:28:43.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:43.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:43.832 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:43.933 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:43.957 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:43.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:43.959 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:43.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:43 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:43.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:43 smithi137 ceph-mon[29634]: 7.6 scrub starts 2024-09-21T08:28:43.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:43 smithi137 ceph-mon[29634]: 7.6 scrub ok 2024-09-21T08:28:43.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:43 smithi137 ceph-mon[29634]: pgmap v398: 168 pgs: 3 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 152 active+clean; 205 MiB data, 689 MiB used, 357 GiB / 358 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-21T08:28:43.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:43 smithi137 ceph-mon[29634]: osdmap e330: 8 total, 8 up, 4 in 2024-09-21T08:28:43.989 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:43 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:43.989 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:43 smithi137 ceph-mon[41044]: 7.6 scrub starts 2024-09-21T08:28:43.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:43 smithi137 ceph-mon[41044]: 7.6 scrub ok 2024-09-21T08:28:43.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:43 smithi137 ceph-mon[41044]: pgmap v398: 168 pgs: 3 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 152 active+clean; 205 MiB data, 689 MiB used, 357 GiB / 358 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-21T08:28:43.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:43 smithi137 ceph-mon[41044]: osdmap e330: 8 total, 8 up, 4 in 2024-09-21T08:28:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:43 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:43 smithi165 ceph-mon[32893]: 7.6 scrub starts 2024-09-21T08:28:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:43 smithi165 ceph-mon[32893]: 7.6 scrub ok 2024-09-21T08:28:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:43 smithi165 ceph-mon[32893]: pgmap v398: 168 pgs: 3 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 152 active+clean; 205 MiB data, 689 MiB used, 357 GiB / 358 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-21T08:28:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:43 smithi165 ceph-mon[32893]: osdmap e330: 8 total, 8 up, 4 in 2024-09-21T08:28:44.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T08:28:44.034 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:44.036 INFO:tasks.rados.rados.0.smithi137.stdout:1246: expect (ObjNum 426 snap 107 seq_num 426) 2024-09-21T08:28:44.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:44.081 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:44.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:44.084 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:44.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:44.146 INFO:tasks.rados.rados.0.smithi137.stdout:1243: finishing copy_from racing read to smithi13779221-18 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1242: finishing copy_from racing read to smithi13779221-27 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1242: finishing copy_from to smithi13779221-27 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 295 (ObjNum 384 snap 94 seq_num 384) dirty exists 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1243: finishing copy_from to smithi13779221-18 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1242: done (6 left) 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1243: done (5 left) 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1244: done (4 left) 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1245: done (3 left) 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1246: done (2 left) 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1249: done (1 left) 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1250: copy_from oid 11 from oid 22 current snap is 123 2024-09-21T08:28:44.147 INFO:tasks.rados.rados.0.smithi137.stdout:1248: finishing write tid 1 to smithi13779221-14 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1248: finishing write tid 2 to smithi13779221-14 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1248: finishing write tid 3 to smithi13779221-14 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1248: finishing write tid 4 to smithi13779221-14 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1248: finishing write tid 5 to smithi13779221-14 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1248: oid 14 updating version 0 to 305 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1248: oid 14 version 305 is already newer than 304 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1248: oid 14 version 305 is already newer than 302 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1248: oid 14 version 305 is already newer than 303 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 305 (ObjNum 456 snap 123 seq_num 456) dirty exists 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1248: left oid 14 (ObjNum 456 snap 123 seq_num 456) 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1248: done (1 left) 2024-09-21T08:28:44.148 INFO:tasks.rados.rados.0.smithi137.stdout:1251: snap_create 2024-09-21T08:28:44.157 INFO:tasks.rados.rados.0.smithi137.stdout:1250: finishing copy_from racing read to smithi13779221-11 2024-09-21T08:28:44.160 INFO:tasks.rados.rados.0.smithi137.stdout:1250: finishing copy_from to smithi13779221-11 2024-09-21T08:28:44.160 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 305 (ObjNum 423 snap 106 seq_num 423) dirty exists 2024-09-21T08:28:44.186 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:44.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:44.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:44.212 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:44.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:44.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:44.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:44.338 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:44.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:44.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:44.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:44.466 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:44.567 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:44.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:44.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:44.594 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:44.659 INFO:tasks.rados.rados.0.smithi137.stdout:1250: done (1 left) 2024-09-21T08:28:44.659 INFO:tasks.rados.rados.0.smithi137.stdout:1251: done (0 left) 2024-09-21T08:28:44.660 INFO:tasks.rados.rados.0.smithi137.stdout:1252: write (excl) oid 32 current snap is 124 2024-09-21T08:28:44.660 INFO:tasks.rados.rados.0.smithi137.stdout:1252: seq_num 458 ranges {497606=645222,1934400=629494,3243966=453640} 2024-09-21T08:28:44.665 INFO:tasks.rados.rados.0.smithi137.stdout:1252: writing smithi13779221-32 from 497606 to 1142828 tid 1 2024-09-21T08:28:44.674 INFO:tasks.rados.rados.0.smithi137.stdout:1252: writing smithi13779221-32 from 1934400 to 2563894 tid 2 2024-09-21T08:28:44.680 INFO:tasks.rados.rados.0.smithi137.stdout:1252: writing smithi13779221-32 from 3243966 to 3697606 tid 3 2024-09-21T08:28:44.680 INFO:tasks.rados.rados.0.smithi137.stdout:1253: read oid 34 snap -1 2024-09-21T08:28:44.680 INFO:tasks.rados.rados.0.smithi137.stdout:1253: expect (ObjNum 332 snap 79 seq_num 332) 2024-09-21T08:28:44.680 INFO:tasks.rados.rados.0.smithi137.stdout:1254: write (excl) oid 25 current snap is 124 2024-09-21T08:28:44.681 INFO:tasks.rados.rados.0.smithi137.stdout:1252: finishing write tid 1 to smithi13779221-32 2024-09-21T08:28:44.681 INFO:tasks.rados.rados.0.smithi137.stdout:1254: seq_num 459 ranges {514413=720471,1876529=773152,3438683=275730} 2024-09-21T08:28:44.687 INFO:tasks.rados.rados.0.smithi137.stdout:1254: writing smithi13779221-25 from 514413 to 1234884 tid 1 2024-09-21T08:28:44.695 INFO:tasks.rados.rados.0.smithi137.stdout:1254: writing smithi13779221-25 from 1876529 to 2649681 tid 2 2024-09-21T08:28:44.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:44.701 INFO:tasks.rados.rados.0.smithi137.stdout:1254: writing smithi13779221-25 from 3438683 to 3714413 tid 3 2024-09-21T08:28:44.701 INFO:tasks.rados.rados.0.smithi137.stdout:1255: read oid 15 snap -1 2024-09-21T08:28:44.701 INFO:tasks.rados.rados.0.smithi137.stdout:1255: expect deleted 2024-09-21T08:28:44.701 INFO:tasks.rados.rados.0.smithi137.stdout:1256: read oid 26 snap -1 2024-09-21T08:28:44.702 INFO:tasks.rados.rados.0.smithi137.stdout:1256: expect (ObjNum 439 snap 113 seq_num 439) 2024-09-21T08:28:44.702 INFO:tasks.rados.rados.0.smithi137.stdout:1257: delete oid 44 current snap is 124 2024-09-21T08:28:44.702 INFO:tasks.rados.rados.0.smithi137.stdout:1252: finishing write tid 2 to smithi13779221-32 2024-09-21T08:28:44.702 INFO:tasks.rados.rados.0.smithi137.stdout:1253: expect (ObjNum 332 snap 79 seq_num 332) 2024-09-21T08:28:44.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:44.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:44.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:44.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:44.842 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:44.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:44.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:1252: finishing write tid 3 to smithi13779221-32 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:1252: finishing write tid 4 to smithi13779221-32 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:1252: finishing write tid 5 to smithi13779221-32 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:1252: oid 32 updating version 0 to 309 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:1252: oid 32 version 309 is already newer than 307 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:1252: oid 32 version 309 is already newer than 308 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:1252: oid 32 version 309 is already newer than 306 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 309 (ObjNum 458 snap 124 seq_num 458) dirty exists 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:1252: left oid 32 (ObjNum 458 snap 124 seq_num 458) 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:1254: finishing write tid 1 to smithi13779221-25 2024-09-21T08:28:44.869 INFO:tasks.rados.rados.0.smithi137.stdout:1252: done (5 left) 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1253: done (4 left) 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1257: done (3 left) 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1258: copy_from oid 1 from oid 7 current snap is 124 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1254: finishing write tid 2 to smithi13779221-25 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1254: finishing write tid 3 to smithi13779221-25 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1254: finishing write tid 4 to smithi13779221-25 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1254: finishing write tid 5 to smithi13779221-25 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1254: oid 25 updating version 0 to 339 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1254: oid 25 version 339 is already newer than 338 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1254: oid 25 version 339 is already newer than 337 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1254: oid 25 version 339 is already newer than 336 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 339 (ObjNum 459 snap 124 seq_num 459) dirty exists 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1254: left oid 25 (ObjNum 459 snap 124 seq_num 459) 2024-09-21T08:28:44.870 INFO:tasks.rados.rados.0.smithi137.stdout:1256: expect (ObjNum 439 snap 113 seq_num 439) 2024-09-21T08:28:44.944 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:44.965 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:44.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:44.966 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:44.983 INFO:tasks.rados.rados.0.smithi137.stdout:1258: finishing copy_from racing read to smithi13779221-1 2024-09-21T08:28:44.983 INFO:tasks.rados.rados.0.smithi137.stdout:1258: finishing copy_from to smithi13779221-1 2024-09-21T08:28:44.983 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 509 (ObjNum 443 snap 114 seq_num 443) dirty exists 2024-09-21T08:28:44.983 INFO:tasks.rados.rados.0.smithi137.stdout:1254: done (3 left) 2024-09-21T08:28:44.983 INFO:tasks.rados.rados.0.smithi137.stdout:1255: done (2 left) 2024-09-21T08:28:44.983 INFO:tasks.rados.rados.0.smithi137.stdout:1256: done (1 left) 2024-09-21T08:28:44.983 INFO:tasks.rados.rados.0.smithi137.stdout:1258: done (0 left) 2024-09-21T08:28:44.983 INFO:tasks.rados.rados.0.smithi137.stdout:1259: write (excl) oid 38 current snap is 124 2024-09-21T08:28:44.983 INFO:tasks.rados.rados.0.smithi137.stdout:1259: seq_num 461 ranges {548027=470969,1593493=776821,3093529=631692} 2024-09-21T08:28:44.989 INFO:tasks.rados.rados.0.smithi137.stdout:1259: writing smithi13779221-38 from 548027 to 1018996 tid 1 2024-09-21T08:28:44.995 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:44.995 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:44.997 INFO:tasks.rados.rados.0.smithi137.stdout:1259: writing smithi13779221-38 from 1593493 to 2370314 tid 2 2024-09-21T08:28:45.006 INFO:tasks.rados.rados.0.smithi137.stdout:1259: writing smithi13779221-38 from 3093529 to 3725221 tid 3 2024-09-21T08:28:45.006 INFO:tasks.rados.rados.0.smithi137.stdout:1260: read oid 46 snap 112 2024-09-21T08:28:45.007 INFO:tasks.rados.rados.0.smithi137.stdout:1260: expect (ObjNum 403 snap 99 seq_num 403) 2024-09-21T08:28:45.007 INFO:tasks.rados.rados.0.smithi137.stdout:1261: copy_from oid 28 from oid 47 current snap is 124 2024-09-21T08:28:45.007 INFO:tasks.rados.rados.0.smithi137.stdout:1259: finishing write tid 1 to smithi13779221-38 2024-09-21T08:28:45.007 INFO:tasks.rados.rados.0.smithi137.stdout:1262: copy_from oid 11 from oid 13 current snap is 124 2024-09-21T08:28:45.007 INFO:tasks.rados.rados.0.smithi137.stdout:1259: finishing write tid 2 to smithi13779221-38 2024-09-21T08:28:45.007 INFO:tasks.rados.rados.0.smithi137.stdout:1263: snap_remove snap 121 2024-09-21T08:28:45.013 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:45.020 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:45.020 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:45.037 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:45.038 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:45.054 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:45.055 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:45.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:45.071 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:45.071 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:45.087 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:45.088 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:45.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:45.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:45.089 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:45.104 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:45.104 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:45.120 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:45.121 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:45.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:45.211 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:45.211 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:45.212 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:45.312 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:28:45.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:45.334 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:45.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:45.335 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:45.436 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:45.457 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:45.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:45.458 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:45.559 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:45.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:45.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:45.581 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:45.658 INFO:teuthology.orchestra.run.smithi137.stderr:set osd.1 primary-affinity to 1 (8655362) 2024-09-21T08:28:45.660 INFO:tasks.rados.rados.0.smithi137.stdout:1263: done (4 left) 2024-09-21T08:28:45.660 INFO:tasks.rados.rados.0.smithi137.stdout:1264: write (excl) oid 4 current snap is 124 2024-09-21T08:28:45.660 INFO:tasks.rados.rados.0.smithi137.stdout:1264: seq_num 462 ranges {564834=546218,1651975=520479,2720952=559317} 2024-09-21T08:28:45.667 INFO:tasks.rados.rados.0.smithi137.stdout:1264: writing smithi13779221-4 from 564834 to 1111052 tid 1 2024-09-21T08:28:45.667 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:28:45.667 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:28:45.667 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:28:45.667 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:28:45.673 INFO:tasks.rados.rados.0.smithi137.stdout:1264: writing smithi13779221-4 from 1651975 to 2172454 tid 2 2024-09-21T08:28:45.679 INFO:tasks.rados.rados.0.smithi137.stdout:1264: writing smithi13779221-4 from 2720952 to 3280269 tid 3 2024-09-21T08:28:45.680 INFO:tasks.rados.rados.0.smithi137.stdout:1265: write oid 50 current snap is 124 2024-09-21T08:28:45.680 INFO:tasks.rados.rados.0.smithi137.stdout:1265: seq_num 463 ranges {581641=621467,1994104=780490,3432022=349619} 2024-09-21T08:28:45.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:45.686 INFO:tasks.rados.rados.0.smithi137.stdout:1265: writing smithi13779221-50 from 581641 to 1203108 tid 1 2024-09-21T08:28:45.695 INFO:tasks.rados.rados.0.smithi137.stdout:1265: writing smithi13779221-50 from 1994104 to 2774594 tid 2 2024-09-21T08:28:45.700 INFO:tasks.rados.rados.0.smithi137.stdout:1265: writing smithi13779221-50 from 3432022 to 3781641 tid 3 2024-09-21T08:28:45.700 INFO:tasks.rados.rados.0.smithi137.stdout:1266: snap_create 2024-09-21T08:28:45.701 INFO:tasks.rados.rados.0.smithi137.stdout:1260: expect (ObjNum 403 snap 99 seq_num 403) 2024-09-21T08:28:45.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:45.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:45.705 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:45.795 INFO:tasks.rados.rados.0.smithi137.stdout:1259: finishing write tid 3 to smithi13779221-38 2024-09-21T08:28:45.795 INFO:tasks.rados.rados.0.smithi137.stdout:1259: finishing write tid 4 to smithi13779221-38 2024-09-21T08:28:45.795 INFO:tasks.rados.rados.0.smithi137.stdout:1259: finishing write tid 5 to smithi13779221-38 2024-09-21T08:28:45.795 INFO:tasks.rados.rados.0.smithi137.stdout:1259: oid 38 updating version 0 to 529 2024-09-21T08:28:45.795 INFO:tasks.rados.rados.0.smithi137.stdout:1259: oid 38 updating version 529 to 531 2024-09-21T08:28:45.795 INFO:tasks.rados.rados.0.smithi137.stdout:1259: oid 38 version 531 is already newer than 528 2024-09-21T08:28:45.795 INFO:tasks.rados.rados.0.smithi137.stdout:1259: oid 38 version 531 is already newer than 530 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 531 (ObjNum 461 snap 124 seq_num 461) dirty exists 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:1259: left oid 38 (ObjNum 461 snap 124 seq_num 461) 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:1261: finishing copy_from racing read to smithi13779221-28 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:1261: finishing copy_from to smithi13779221-28 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:1262: finishing copy_from racing read to smithi13779221-11 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:1262: finishing copy_from to smithi13779221-11 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 515 (ObjNum 340 snap 80 seq_num 340) dirty exists 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:1264: finishing write tid 1 to smithi13779221-4 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:1264: finishing write tid 2 to smithi13779221-4 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:1264: finishing write tid 3 to smithi13779221-4 2024-09-21T08:28:45.796 INFO:tasks.rados.rados.0.smithi137.stdout:1264: finishing write tid 4 to smithi13779221-4 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1264: finishing write tid 5 to smithi13779221-4 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1264: oid 4 updating version 0 to 304 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1264: oid 4 version 304 is already newer than 302 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1264: oid 4 version 304 is already newer than 303 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1264: oid 4 updating version 304 to 305 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 305 (ObjNum 462 snap 124 seq_num 462) dirty exists 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1264: left oid 4 (ObjNum 462 snap 124 seq_num 462) 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1265: finishing write tid 1 to smithi13779221-50 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1265: finishing write tid 2 to smithi13779221-50 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1265: finishing write tid 3 to smithi13779221-50 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1265: finishing write tid 4 to smithi13779221-50 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1265: finishing write tid 5 to smithi13779221-50 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1265: oid 50 updating version 0 to 354 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1265: oid 50 version 354 is already newer than 353 2024-09-21T08:28:45.797 INFO:tasks.rados.rados.0.smithi137.stdout:1265: oid 50 updating version 354 to 355 2024-09-21T08:28:45.798 INFO:tasks.rados.rados.0.smithi137.stdout:1265: oid 50 updating version 355 to 356 2024-09-21T08:28:45.798 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 356 (ObjNum 463 snap 124 seq_num 463) dirty exists 2024-09-21T08:28:45.798 INFO:tasks.rados.rados.0.smithi137.stdout:1265: left oid 50 (ObjNum 463 snap 124 seq_num 463) 2024-09-21T08:28:45.806 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:45.827 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:45.828 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:45.828 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:45.857 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:45 smithi137 ceph-mon[29634]: pgmap v401: 168 pgs: 3 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 152 active+clean; 205 MiB data, 689 MiB used, 357 GiB / 358 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-21T08:28:45.857 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:45 smithi137 ceph-mon[29634]: osdmap e331: 8 total, 8 up, 4 in 2024-09-21T08:28:45.858 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:45 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3239417216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:45.858 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:45 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2384791057' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-21T08:28:45.858 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:45 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:45.858 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:45 smithi137 ceph-mon[41044]: pgmap v401: 168 pgs: 3 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 152 active+clean; 205 MiB data, 689 MiB used, 357 GiB / 358 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-21T08:28:45.858 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:45 smithi137 ceph-mon[41044]: osdmap e331: 8 total, 8 up, 4 in 2024-09-21T08:28:45.858 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:45 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3239417216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:45.858 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:45 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2384791057' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-21T08:28:45.858 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:45 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:45.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:45.949 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:45.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:45.950 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:45 smithi165 ceph-mon[32893]: pgmap v401: 168 pgs: 3 active+clean+snaptrim, 13 active+clean+snaptrim_wait, 152 active+clean; 205 MiB data, 689 MiB used, 357 GiB / 358 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-21T08:28:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:45 smithi165 ceph-mon[32893]: osdmap e331: 8 total, 8 up, 4 in 2024-09-21T08:28:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:45 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3239417216' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:45 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2384791057' entity='client.admin' cmd=[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]: dispatch 2024-09-21T08:28:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:45 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:28:46.052 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:46.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:46.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:46.073 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:46.174 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:46.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:46.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:46.198 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:46.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:46.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:46.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:46.321 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:46.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:46.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:46.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:46.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:46.544 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:46.548 INFO:tasks.rados.rados.0.smithi137.stdout:1259: done (6 left) 2024-09-21T08:28:46.548 INFO:tasks.rados.rados.0.smithi137.stdout:1260: done (5 left) 2024-09-21T08:28:46.548 INFO:tasks.rados.rados.0.smithi137.stdout:1261: done (4 left) 2024-09-21T08:28:46.548 INFO:tasks.rados.rados.0.smithi137.stdout:1262: done (3 left) 2024-09-21T08:28:46.548 INFO:tasks.rados.rados.0.smithi137.stdout:1264: done (2 left) 2024-09-21T08:28:46.548 INFO:tasks.rados.rados.0.smithi137.stdout:1265: done (1 left) 2024-09-21T08:28:46.548 INFO:tasks.rados.rados.0.smithi137.stdout:1266: done (0 left) 2024-09-21T08:28:46.548 INFO:tasks.rados.rados.0.smithi137.stdout:1267: write (excl) oid 28 current snap is 125 2024-09-21T08:28:46.548 INFO:tasks.rados.rados.0.smithi137.stdout:1267: seq_num 464 ranges {598448=413069,1768939=524148,2775798=581861,3781488=16960} 2024-09-21T08:28:46.553 INFO:tasks.rados.rados.0.smithi137.stdout:1267: writing smithi13779221-28 from 598448 to 1011517 tid 1 2024-09-21T08:28:46.561 INFO:tasks.rados.rados.0.smithi137.stdout:1267: writing smithi13779221-28 from 1768939 to 2293087 tid 2 2024-09-21T08:28:46.567 INFO:tasks.rados.rados.0.smithi137.stdout:1267: writing smithi13779221-28 from 2775798 to 3357659 tid 3 2024-09-21T08:28:46.570 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:1267: writing smithi13779221-28 from 3781488 to 3798448 tid 4 2024-09-21T08:28:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:1268: rollback oid 20 current snap is 125 2024-09-21T08:28:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 20 to 123 2024-09-21T08:28:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:1267: finishing write tid 1 to smithi13779221-28 2024-09-21T08:28:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:1267: finishing write tid 2 to smithi13779221-28 2024-09-21T08:28:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:1267: finishing write tid 3 to smithi13779221-28 2024-09-21T08:28:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:1269: write (excl) oid 1 current snap is 125 2024-09-21T08:28:46.571 INFO:tasks.rados.rados.0.smithi137.stdout:1269: seq_num 465 ranges {615255=488318,1827421=784159,3203221=612034} 2024-09-21T08:28:46.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:46.572 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:46.576 INFO:tasks.rados.rados.0.smithi137.stdout:1269: writing smithi13779221-1 from 615255 to 1103573 tid 1 2024-09-21T08:28:46.585 INFO:tasks.rados.rados.0.smithi137.stdout:1269: writing smithi13779221-1 from 1827421 to 2611580 tid 2 2024-09-21T08:28:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1269: writing smithi13779221-1 from 3203221 to 3815255 tid 3 2024-09-21T08:28:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1270: rollback oid 30 current snap is 125 2024-09-21T08:28:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 30 to 111 2024-09-21T08:28:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1267: finishing write tid 4 to smithi13779221-28 2024-09-21T08:28:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1267: finishing write tid 5 to smithi13779221-28 2024-09-21T08:28:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1267: finishing write tid 6 to smithi13779221-28 2024-09-21T08:28:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1267: oid 28 updating version 0 to 267 2024-09-21T08:28:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1267: oid 28 updating version 267 to 270 2024-09-21T08:28:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1267: oid 28 version 270 is already newer than 269 2024-09-21T08:28:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1267: oid 28 version 270 is already newer than 268 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:1267: oid 28 version 270 is already newer than 266 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 270 (ObjNum 464 snap 125 seq_num 464) dirty exists 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:1267: left oid 28 (ObjNum 464 snap 125 seq_num 464) 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:1268: finishing rollback tid 0 to smithi13779221-20 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:1268: finishing rollback tid 1 to smithi13779221-20 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:1267: done (3 left) 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:1271: delete oid 22 current snap is 125 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:1268: finishing rollback tid 2 to smithi13779221-20 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 395 (ObjNum 450 snap 118 seq_num 450) dirty exists 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:1269: finishing write tid 1 to smithi13779221-1 2024-09-21T08:28:46.593 INFO:tasks.rados.rados.0.smithi137.stdout:1269: finishing write tid 2 to smithi13779221-1 2024-09-21T08:28:46.594 INFO:tasks.rados.rados.0.smithi137.stdout:1268: done (3 left) 2024-09-21T08:28:46.594 INFO:tasks.rados.rados.0.smithi137.stdout:1271: done (2 left) 2024-09-21T08:28:46.594 INFO:tasks.rados.rados.0.smithi137.stdout:1272: snap_create 2024-09-21T08:28:46.594 INFO:tasks.rados.rados.0.smithi137.stdout:1269: finishing write tid 3 to smithi13779221-1 2024-09-21T08:28:46.594 INFO:tasks.rados.rados.0.smithi137.stdout:1269: finishing write tid 4 to smithi13779221-1 2024-09-21T08:28:46.595 INFO:tasks.rados.rados.0.smithi137.stdout:1270: finishing rollback tid 0 to smithi13779221-30 2024-09-21T08:28:46.595 INFO:tasks.rados.rados.0.smithi137.stdout:1269: finishing write tid 5 to smithi13779221-1 2024-09-21T08:28:46.595 INFO:tasks.rados.rados.0.smithi137.stdout:1269: oid 1 updating version 0 to 513 2024-09-21T08:28:46.595 INFO:tasks.rados.rados.0.smithi137.stdout:1269: oid 1 version 513 is already newer than 511 2024-09-21T08:28:46.595 INFO:tasks.rados.rados.0.smithi137.stdout:1269: oid 1 version 513 is already newer than 510 2024-09-21T08:28:46.595 INFO:tasks.rados.rados.0.smithi137.stdout:1269: oid 1 version 513 is already newer than 512 2024-09-21T08:28:46.595 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 513 (ObjNum 465 snap 125 seq_num 465) dirty exists 2024-09-21T08:28:46.595 INFO:tasks.rados.rados.0.smithi137.stdout:1269: left oid 1 (ObjNum 465 snap 125 seq_num 465) 2024-09-21T08:28:46.595 INFO:tasks.rados.rados.0.smithi137.stdout:1270: finishing rollback tid 1 to smithi13779221-30 2024-09-21T08:28:46.595 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 534 (ObjNum 33 snap 0 seq_num 2161485939) dirty dne 2024-09-21T08:28:46.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:46.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:46.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:46.695 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:46.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:46 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2384791057' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T08:28:46.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:46 smithi137 ceph-mon[29634]: osdmap e332: 8 total, 8 up, 4 in 2024-09-21T08:28:46.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:46 smithi137 ceph-mon[29634]: osdmap e333: 8 total, 8 up, 4 in 2024-09-21T08:28:46.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:46 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2384791057' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T08:28:46.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:46 smithi137 ceph-mon[41044]: osdmap e332: 8 total, 8 up, 4 in 2024-09-21T08:28:46.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:46 smithi137 ceph-mon[41044]: osdmap e333: 8 total, 8 up, 4 in 2024-09-21T08:28:46.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:46.817 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:46.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:46.818 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:46.918 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:46.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:46.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:46.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:46.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T08:28:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:46 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2384791057' entity='client.admin' cmd='[{"prefix": "osd primary-affinity", "id": 1, "weight": 1.0}]': finished 2024-09-21T08:28:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:46 smithi165 ceph-mon[32893]: osdmap e332: 8 total, 8 up, 4 in 2024-09-21T08:28:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:46 smithi165 ceph-mon[32893]: osdmap e333: 8 total, 8 up, 4 in 2024-09-21T08:28:47.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:47.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:47.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:47.064 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:47.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:47.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:47.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:47.187 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:47.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:47.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:47.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:47.311 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:47.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:47.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:47.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:47.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:47.470 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:47.474 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:47.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:47.551 INFO:tasks.rados.rados.0.smithi137.stdout:1269: done (2 left) 2024-09-21T08:28:47.551 INFO:tasks.rados.rados.0.smithi137.stdout:1270: done (1 left) 2024-09-21T08:28:47.551 INFO:tasks.rados.rados.0.smithi137.stdout:1272: done (0 left) 2024-09-21T08:28:47.552 INFO:tasks.rados.rados.0.smithi137.stdout:1273: rollback oid 48 current snap is 126 2024-09-21T08:28:47.552 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 48 to 124 2024-09-21T08:28:47.552 INFO:tasks.rados.rados.0.smithi137.stdout:1274: rollback oid 2 current snap is 126 2024-09-21T08:28:47.552 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 2 to 124 2024-09-21T08:28:47.552 INFO:tasks.rados.rados.0.smithi137.stdout:1275: write (excl) oid 45 current snap is 126 2024-09-21T08:28:47.552 INFO:tasks.rados.rados.0.smithi137.stdout:1275: seq_num 467 ranges {648869=638816,1828032=787828,3141714=532030} 2024-09-21T08:28:47.559 INFO:tasks.rados.rados.0.smithi137.stdout:1275: writing smithi13779221-45 from 648869 to 1287685 tid 1 2024-09-21T08:28:47.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:47.563 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:47.563 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:47.567 INFO:tasks.rados.rados.0.smithi137.stdout:1275: writing smithi13779221-45 from 1828032 to 2615860 tid 2 2024-09-21T08:28:47.573 INFO:tasks.rados.rados.0.smithi137.stdout:1275: writing smithi13779221-45 from 3141714 to 3673744 tid 3 2024-09-21T08:28:47.573 INFO:tasks.rados.rados.0.smithi137.stdout:1276: write oid 16 current snap is 126 2024-09-21T08:28:47.573 INFO:tasks.rados.rados.0.smithi137.stdout:1273: finishing rollback tid 0 to smithi13779221-48 2024-09-21T08:28:47.573 INFO:tasks.rados.rados.0.smithi137.stdout:1276: seq_num 468 ranges {665676=714065,1886514=531486,3169137=696539} 2024-09-21T08:28:47.580 INFO:tasks.rados.rados.0.smithi137.stdout:1276: writing smithi13779221-16 from 665676 to 1379741 tid 1 2024-09-21T08:28:47.586 INFO:tasks.rados.rados.0.smithi137.stdout:1276: writing smithi13779221-16 from 1886514 to 2418000 tid 2 2024-09-21T08:28:47.595 INFO:tasks.rados.rados.0.smithi137.stdout:1276: writing smithi13779221-16 from 3169137 to 3865676 tid 3 2024-09-21T08:28:47.595 INFO:tasks.rados.rados.0.smithi137.stdout:1277: read oid 3 snap -1 2024-09-21T08:28:47.595 INFO:tasks.rados.rados.0.smithi137.stdout:1277: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:47.595 INFO:tasks.rados.rados.0.smithi137.stdout:1278: rollback oid 42 current snap is 126 2024-09-21T08:28:47.595 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 42 to 120 2024-09-21T08:28:47.595 INFO:tasks.rados.rados.0.smithi137.stdout:1274: finishing rollback tid 0 to smithi13779221-2 2024-09-21T08:28:47.595 INFO:tasks.rados.rados.0.smithi137.stdout:1273: finishing rollback tid 1 to smithi13779221-48 2024-09-21T08:28:47.595 INFO:tasks.rados.rados.0.smithi137.stdout:1273: finishing rollback tid 2 to smithi13779221-48 2024-09-21T08:28:47.595 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 325 (ObjNum 453 snap 119 seq_num 453) dirty exists 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1274: finishing rollback tid 1 to smithi13779221-2 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1275: finishing write tid 1 to smithi13779221-45 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1274: finishing rollback tid 2 to smithi13779221-2 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 151 (ObjNum 452 snap 119 seq_num 452) dirty exists 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1275: finishing write tid 2 to smithi13779221-45 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1273: done (5 left) 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1274: done (4 left) 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1279: snap_create 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1275: finishing write tid 3 to smithi13779221-45 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1275: finishing write tid 4 to smithi13779221-45 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1275: finishing write tid 5 to smithi13779221-45 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1275: oid 45 updating version 0 to 306 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1275: oid 45 updating version 306 to 309 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1275: oid 45 version 309 is already newer than 308 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:1275: oid 45 version 309 is already newer than 307 2024-09-21T08:28:47.596 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 309 (ObjNum 467 snap 126 seq_num 467) dirty exists 2024-09-21T08:28:47.597 INFO:tasks.rados.rados.0.smithi137.stdout:1275: left oid 45 (ObjNum 467 snap 126 seq_num 467) 2024-09-21T08:28:47.597 INFO:tasks.rados.rados.0.smithi137.stdout:1276: finishing write tid 1 to smithi13779221-16 2024-09-21T08:28:47.597 INFO:tasks.rados.rados.0.smithi137.stdout:1276: finishing write tid 2 to smithi13779221-16 2024-09-21T08:28:47.597 INFO:tasks.rados.rados.0.smithi137.stdout:1278: finishing rollback tid 0 to smithi13779221-42 2024-09-21T08:28:47.597 INFO:tasks.rados.rados.0.smithi137.stdout:1276: finishing write tid 3 to smithi13779221-16 2024-09-21T08:28:47.600 INFO:tasks.rados.rados.0.smithi137.stdout:1276: finishing write tid 4 to smithi13779221-16 2024-09-21T08:28:47.601 INFO:tasks.rados.rados.0.smithi137.stdout:1276: finishing write tid 5 to smithi13779221-16 2024-09-21T08:28:47.601 INFO:tasks.rados.rados.0.smithi137.stdout:1276: oid 16 updating version 0 to 330 2024-09-21T08:28:47.601 INFO:tasks.rados.rados.0.smithi137.stdout:1276: oid 16 updating version 330 to 332 2024-09-21T08:28:47.601 INFO:tasks.rados.rados.0.smithi137.stdout:1276: oid 16 updating version 332 to 333 2024-09-21T08:28:47.601 INFO:tasks.rados.rados.0.smithi137.stdout:1276: oid 16 version 333 is already newer than 331 2024-09-21T08:28:47.601 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 333 (ObjNum 468 snap 126 seq_num 468) dirty exists 2024-09-21T08:28:47.601 INFO:tasks.rados.rados.0.smithi137.stdout:1276: left oid 16 (ObjNum 468 snap 126 seq_num 468) 2024-09-21T08:28:47.602 INFO:tasks.rados.rados.0.smithi137.stdout:1278: finishing rollback tid 1 to smithi13779221-42 2024-09-21T08:28:47.602 INFO:tasks.rados.rados.0.smithi137.stdout:1278: finishing rollback tid 2 to smithi13779221-42 2024-09-21T08:28:47.602 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 291 (ObjNum 445 snap 115 seq_num 445) dirty exists 2024-09-21T08:28:47.602 INFO:tasks.rados.rados.0.smithi137.stdout:1277: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:47.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:47.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:47.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:47.686 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:47.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:47.807 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:47.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:47.808 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:47.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:47 smithi137 ceph-mon[29634]: pgmap v404: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 239 MiB data, 775 MiB used, 357 GiB / 358 GiB avail; 21 MiB/s rd, 8.5 MiB/s wr, 29 op/s 2024-09-21T08:28:47.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:47 smithi137 ceph-mon[29634]: osdmap e334: 8 total, 8 up, 4 in 2024-09-21T08:28:47.901 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:47 smithi137 ceph-mon[41044]: pgmap v404: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 239 MiB data, 775 MiB used, 357 GiB / 358 GiB avail; 21 MiB/s rd, 8.5 MiB/s wr, 29 op/s 2024-09-21T08:28:47.901 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:47 smithi137 ceph-mon[41044]: osdmap e334: 8 total, 8 up, 4 in 2024-09-21T08:28:47.910 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:47.932 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:47.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:47.933 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:47 smithi165 ceph-mon[32893]: pgmap v404: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 239 MiB data, 775 MiB used, 357 GiB / 358 GiB avail; 21 MiB/s rd, 8.5 MiB/s wr, 29 op/s 2024-09-21T08:28:48.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:47 smithi165 ceph-mon[32893]: osdmap e334: 8 total, 8 up, 4 in 2024-09-21T08:28:48.034 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:48.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:48.056 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:48.056 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:48.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:48.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:48.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:48.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:48.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:48.302 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:48.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:48.303 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:48.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:48.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:48.426 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:48.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:48.454 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:28:48.454 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:28:48.454 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:28:48.454 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:28:48.454 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:28:48.527 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:48.553 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:48.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:48.554 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:48.555 INFO:tasks.rados.rados.0.smithi137.stdout:1275: done (4 left) 2024-09-21T08:28:48.555 INFO:tasks.rados.rados.0.smithi137.stdout:1276: done (3 left) 2024-09-21T08:28:48.555 INFO:tasks.rados.rados.0.smithi137.stdout:1277: done (2 left) 2024-09-21T08:28:48.555 INFO:tasks.rados.rados.0.smithi137.stdout:1278: done (1 left) 2024-09-21T08:28:48.555 INFO:tasks.rados.rados.0.smithi137.stdout:1279: done (0 left) 2024-09-21T08:28:48.555 INFO:tasks.rados.rados.0.smithi137.stdout:1280: write oid 22 current snap is 127 2024-09-21T08:28:48.555 INFO:tasks.rados.rados.0.smithi137.stdout:1280: seq_num 469 ranges {682483=789314,1944996=791497,3196560=554574} 2024-09-21T08:28:48.556 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:28:48.563 INFO:tasks.rados.rados.0.smithi137.stdout:1280: writing smithi13779221-22 from 682483 to 1471797 tid 1 2024-09-21T08:28:48.570 INFO:tasks.rados.rados.0.smithi137.stdout:1280: writing smithi13779221-22 from 1944996 to 2736493 tid 2 2024-09-21T08:28:48.576 INFO:tasks.rados.rados.0.smithi137.stdout:1280: writing smithi13779221-22 from 3196560 to 3751134 tid 3 2024-09-21T08:28:48.576 INFO:tasks.rados.rados.0.smithi137.stdout:1281: read oid 1 snap -1 2024-09-21T08:28:48.577 INFO:tasks.rados.rados.0.smithi137.stdout:1281: expect (ObjNum 465 snap 125 seq_num 465) 2024-09-21T08:28:48.577 INFO:tasks.rados.rados.0.smithi137.stdout:1282: snap_create 2024-09-21T08:28:48.577 INFO:tasks.rados.rados.0.smithi137.stdout:1280: finishing write tid 1 to smithi13779221-22 2024-09-21T08:28:48.577 INFO:tasks.rados.rados.0.smithi137.stdout:1280: finishing write tid 2 to smithi13779221-22 2024-09-21T08:28:48.580 INFO:tasks.rados.rados.0.smithi137.stdout:1280: finishing write tid 3 to smithi13779221-22 2024-09-21T08:28:48.580 INFO:tasks.rados.rados.0.smithi137.stdout:1280: finishing write tid 4 to smithi13779221-22 2024-09-21T08:28:48.580 INFO:tasks.rados.rados.0.smithi137.stdout:1280: finishing write tid 5 to smithi13779221-22 2024-09-21T08:28:48.580 INFO:tasks.rados.rados.0.smithi137.stdout:1280: oid 22 updating version 0 to 332 2024-09-21T08:28:48.580 INFO:tasks.rados.rados.0.smithi137.stdout:1280: oid 22 version 332 is already newer than 330 2024-09-21T08:28:48.580 INFO:tasks.rados.rados.0.smithi137.stdout:1280: oid 22 version 332 is already newer than 331 2024-09-21T08:28:48.581 INFO:tasks.rados.rados.0.smithi137.stdout:1280: oid 22 version 332 is already newer than 329 2024-09-21T08:28:48.581 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 332 (ObjNum 469 snap 127 seq_num 469) dirty exists 2024-09-21T08:28:48.581 INFO:tasks.rados.rados.0.smithi137.stdout:1280: left oid 22 (ObjNum 469 snap 127 seq_num 469) 2024-09-21T08:28:48.592 INFO:tasks.rados.rados.0.smithi137.stdout:1281: expect (ObjNum 465 snap 125 seq_num 465) 2024-09-21T08:28:48.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:48.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:48.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:48.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:48.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:48.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:48.805 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:48.805 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:48.906 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:48.927 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:48.928 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:48.928 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:48 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3722096883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:48 smithi165 ceph-mon[32893]: from='client.45461 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:48 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:48 smithi165 ceph-mon[32893]: from='client.45461 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:48 smithi165 ceph-mon[32893]: osdmap e335: 8 total, 8 up, 4 in 2024-09-21T08:28:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:48 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3722096883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:48 smithi137 ceph-mon[29634]: from='client.45461 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:48 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:48 smithi137 ceph-mon[29634]: from='client.45461 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:49.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:48 smithi137 ceph-mon[29634]: osdmap e335: 8 total, 8 up, 4 in 2024-09-21T08:28:49.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:48 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3722096883' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:49.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:48 smithi137 ceph-mon[41044]: from='client.45461 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:49.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:48 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:49.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:48 smithi137 ceph-mon[41044]: from='client.45461 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:49.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:48 smithi137 ceph-mon[41044]: osdmap e335: 8 total, 8 up, 4 in 2024-09-21T08:28:49.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:49.050 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:49.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:49.051 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:49.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:49.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:49.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:49.180 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:49.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:49.302 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:49.303 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:49.303 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:49.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:28:49.404 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:49.464 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:49.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:49.465 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:49.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T08:28:49.553 INFO:tasks.rados.rados.0.smithi137.stdout:1280: done (2 left) 2024-09-21T08:28:49.553 INFO:tasks.rados.rados.0.smithi137.stdout:1281: done (1 left) 2024-09-21T08:28:49.553 INFO:tasks.rados.rados.0.smithi137.stdout:1282: done (0 left) 2024-09-21T08:28:49.553 INFO:tasks.rados.rados.0.smithi137.stdout:1283: delete oid 2 current snap is 128 2024-09-21T08:28:49.561 INFO:tasks.rados.rados.0.smithi137.stdout:1283: done (0 left) 2024-09-21T08:28:49.561 INFO:tasks.rados.rados.0.smithi137.stdout:1284: copy_from oid 27 from oid 19 current snap is 128 2024-09-21T08:28:49.561 INFO:tasks.rados.rados.0.smithi137.stdout:1285: write (excl) oid 44 current snap is 128 2024-09-21T08:28:49.561 INFO:tasks.rados.rados.0.smithi137.stdout:1285: seq_num 471 ranges {716097=539812,1945607=795166,3535053=381044} 2024-09-21T08:28:49.566 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:49.569 INFO:tasks.rados.rados.0.smithi137.stdout:1285: writing smithi13779221-44 from 716097 to 1255909 tid 1 2024-09-21T08:28:49.578 INFO:tasks.rados.rados.0.smithi137.stdout:1285: writing smithi13779221-44 from 1945607 to 2740773 tid 2 2024-09-21T08:28:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1285: writing smithi13779221-44 from 3535053 to 3916097 tid 3 2024-09-21T08:28:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1286: read oid 37 snap -1 2024-09-21T08:28:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1286: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1287: read oid 42 snap 125 2024-09-21T08:28:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1287: expect (ObjNum 445 snap 115 seq_num 445) 2024-09-21T08:28:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1288: copy_from oid 12 from oid 31 current snap is 128 2024-09-21T08:28:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1284: finishing copy_from to smithi13779221-27 2024-09-21T08:28:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1284: got expected ENOENT (src dne) 2024-09-21T08:28:49.586 INFO:tasks.rados.rados.0.smithi137.stdout:1284: finishing copy_from racing read to smithi13779221-27 2024-09-21T08:28:49.587 INFO:tasks.rados.rados.0.smithi137.stdout:1285: finishing write tid 1 to smithi13779221-44 2024-09-21T08:28:49.587 INFO:tasks.rados.rados.0.smithi137.stdout:1285: finishing write tid 2 to smithi13779221-44 2024-09-21T08:28:49.587 INFO:tasks.rados.rados.0.smithi137.stdout:1284: done (4 left) 2024-09-21T08:28:49.587 INFO:tasks.rados.rados.0.smithi137.stdout:1289: read oid 25 snap 127 2024-09-21T08:28:49.587 INFO:tasks.rados.rados.0.smithi137.stdout:1289: expect (ObjNum 459 snap 124 seq_num 459) 2024-09-21T08:28:49.587 INFO:tasks.rados.rados.0.smithi137.stdout:1290: snap_remove snap 120 2024-09-21T08:28:49.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:49.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:49.593 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:49.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:49.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:49.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:49.718 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:49.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:49.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:49.856 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:49.856 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:49.910 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:49.911 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:49.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:49.980 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:49.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:49.981 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:49 smithi165 ceph-mon[32893]: pgmap v407: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 239 MiB data, 775 MiB used, 357 GiB / 358 GiB avail; 21 MiB/s rd, 8.5 MiB/s wr, 29 op/s 2024-09-21T08:28:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:49 smithi165 ceph-mon[32893]: osdmap e336: 8 total, 8 up, 4 in 2024-09-21T08:28:50.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:49 smithi137 ceph-mon[29634]: pgmap v407: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 239 MiB data, 775 MiB used, 357 GiB / 358 GiB avail; 21 MiB/s rd, 8.5 MiB/s wr, 29 op/s 2024-09-21T08:28:50.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:49 smithi137 ceph-mon[29634]: osdmap e336: 8 total, 8 up, 4 in 2024-09-21T08:28:50.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:49 smithi137 ceph-mon[41044]: pgmap v407: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 239 MiB data, 775 MiB used, 357 GiB / 358 GiB avail; 21 MiB/s rd, 8.5 MiB/s wr, 29 op/s 2024-09-21T08:28:50.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:49 smithi137 ceph-mon[41044]: osdmap e336: 8 total, 8 up, 4 in 2024-09-21T08:28:50.082 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:50.105 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:50.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:50.106 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:50.207 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:50.228 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:50.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:50.229 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:50.330 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:50.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:50.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:50.356 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:50.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:50.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:50.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:50.484 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:50.555 INFO:tasks.rados.rados.0.smithi137.stdout:1290: done (5 left) 2024-09-21T08:28:50.555 INFO:tasks.rados.rados.0.smithi137.stdout:1291: rollback oid 3 current snap is 128 2024-09-21T08:28:50.555 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 3 to 125 2024-09-21T08:28:50.555 INFO:tasks.rados.rados.0.smithi137.stdout:1287: expect (ObjNum 445 snap 115 seq_num 445) 2024-09-21T08:28:50.585 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:50.592 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:1082 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94837804043520 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 252.43001558, 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.39751146399999998, 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.593 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1082 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159528+0000", 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2749999999999998e-06 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159533+0000", 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2480000000000004e-06 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159547+0000", 2024-09-21T08:28:50.595 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4516e-05 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159556+0000", 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9460000000000004e-06 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159617+0000", 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0999000000000003e-05 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159625+0000", 2024-09-21T08:28:50.596 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240000000000007e-06 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544062+0000", 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.38443652099999998 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544359+0000", 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000297378 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548773+0000", 2024-09-21T08:28:50.597 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044136440000000004 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548858+0000", 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5638999999999998e-05 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.557037+0000", 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081783740000000004 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:255 7.2 7:4031afd1:::smithi13779221-48:head [copy-from ver 3] snapc 4=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 245.08170518599999, 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015259864, 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 255 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.599 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507839+0000", 2024-09-21T08:28:50.600 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0089999999999999e-06 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507852+0000", 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3617e-05 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507872+0000", 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9482000000000001e-05 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507883+0000", 2024-09-21T08:28:50.601 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1036000000000001e-05 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507941+0000", 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7819999999999999e-05 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507946+0000", 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9080000000000003e-06 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.602 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510282+0000", 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023368170000000001 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510376+0000", 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3846999999999998e-05 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.516270+0000", 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005893934 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:50.603 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522776+0000", 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065057179999999997 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522902+0000", 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012641 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522990+0000", 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7491000000000006e-05 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.604 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523050+0000", 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0075999999999999e-05 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523096+0000", 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5698999999999997e-05 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:256 7.2 7:4031afd1:::smithi13779221-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 245.08163788799999, 2024-09-21T08:28:50.605 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013379356, 2024-09-21T08:28:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.606 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 256 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:28:50.607 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507904+0000", 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8000000000000005e-07 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507907+0000", 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.258e-06 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.608 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507915+0000", 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.571e-06 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507920+0000", 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1189999999999996e-06 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507998+0000", 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8181000000000002e-05 2024-09-21T08:28:50.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.508004+0000", 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3960000000000003e-06 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510529+0000", 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025244989999999999 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511053+0000", 2024-09-21T08:28:50.610 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052380199999999997 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521102+0000", 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010049361999999999 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521183+0000", 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0495999999999996e-05 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521282+0000", 2024-09-21T08:28:50.611 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9891999999999998e-05 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:260 7.8 e93/84 7:1fd29771:::smithi13779221-22:head v 93'20, mlcod=86'17)", 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 244.02059155500001, 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078947310000000003, 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.612 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568951+0000", 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1510000000000001e-06 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.613 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571221+0000", 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022708279999999999 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571244+0000", 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2759999999999999e-05 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571257+0000", 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2669e-05 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571601+0000", 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034448900000000001 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571645+0000", 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3498999999999998e-05 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576787+0000", 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051417010000000003 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.615 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576844+0000", 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7633999999999997e-05 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:294 7.2 7:4d793d31:::smithi13779221-9:head [copy-from ver 3] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 243.82687475899999, 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017337353, 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.616 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 294 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.617 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762668+0000", 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.353e-06 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762673+0000", 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1569999999999999e-06 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.618 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762683+0000", 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0331e-05 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762688+0000", 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1410000000000004e-06 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766904+0000", 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042154899999999997 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.619 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767004+0000", 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100135 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.771465+0000", 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044613170000000002 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779379+0000", 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079137649999999997 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.620 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779883+0000", 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050370000000000005 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779896+0000", 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3458999999999999e-05 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779950+0000", 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3851000000000003e-05 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.780004+0000", 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3653999999999998e-05 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:295 7.2 7:4d793d31:::smithi13779221-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 243.82684404099999, 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012332286, 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.622 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 295 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.623 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4400000000000001e-07 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762699+0000", 2024-09-21T08:28:50.624 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0099999999999998e-06 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762705+0000", 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0279999999999997e-06 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762710+0000", 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8210000000000001e-06 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.625 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767340+0000", 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046298770000000001 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.768012+0000", 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067169199999999999 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774900+0000", 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068879680000000004 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.626 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774961+0000", 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0980999999999998e-05 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.775029+0000", 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8465000000000003e-05 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1908 7.a 7:53537bd3:::smithi13779221-39:head [append 0~0] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:28:50.627 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 122.03965408099999, 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.037489017, 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1908 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:28:50.628 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549889+0000", 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.199e-06 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549894+0000", 2024-09-21T08:28:50.629 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9450000000000001e-06 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549902+0000", 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9459999999999998e-06 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549910+0000", 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6510000000000001e-06 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.630 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549962+0000", 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2741999999999997e-05 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549970+0000", 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0949999999999996e-06 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.556983+0000", 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.007012673 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.631 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557124+0000", 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014058199999999999 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583019+0000", 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025894830000000001 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583102+0000", 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3332999999999996e-05 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.632 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583767+0000", 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00066536999999999998 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583809+0000", 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1919999999999998e-05 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583935+0000", 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012586800000000001 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.633 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584880+0000", 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00094458400000000005 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585414+0000", 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000533892 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.586605+0000", 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011918479999999999 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.634 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587260+0000", 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065433099999999997 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587376+0000", 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116208 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.635 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1909 7.a 7:53537bd3:::smithi13779221-39:head [rollback 39] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 122.039558117, 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0375938790000001, 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1909 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.636 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549984+0000", 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5799999999999998e-07 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549987+0000", 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5099999999999999e-06 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549993+0000", 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6099999999999997e-06 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549998+0000", 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4890000000000003e-06 2024-09-21T08:28:50.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550039+0000", 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1155000000000002e-05 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550045+0000", 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7130000000000002e-06 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557614+0000", 2024-09-21T08:28:50.639 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.007568971 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557661+0000", 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6726e-05 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583163+0000", 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025501894000000001 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583309+0000", 2024-09-21T08:28:50.640 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014584299999999999 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584385+0000", 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010765250000000001 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584430+0000", 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5151000000000002e-05 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584564+0000", 2024-09-21T08:28:50.641 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133652 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585131+0000", 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00056728200000000001 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585577+0000", 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044534199999999999 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.642 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587426+0000", 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001849402 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587490+0000", 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4126000000000002e-05 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587577+0000", 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6630000000000002e-05 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.643 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:1376 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94837804043520 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 119.960655249, 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0087414380000000007, 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1376 2024-09-21T08:28:50.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628889+0000", 2024-09-21T08:28:50.645 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9299999999999999e-06 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628895+0000", 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6189999999999996e-06 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628909+0000", 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3963e-05 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628916+0000", 2024-09-21T08:28:50.646 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.131e-06 2024-09-21T08:28:50.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.647 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.647 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628998+0000", 2024-09-21T08:28:50.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1219999999999998e-05 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.629004+0000", 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7510000000000002e-06 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.634800+0000", 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005795311 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.634882+0000", 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2025000000000002e-05 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637627+0000", 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027454879999999999 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.649 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2114 7.4 e228/215 7:26d6e516:::smithi13779221-33:head v 228'62, mlcod=211'59)", 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 109.04860072699999, 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012725531, 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:28:50.650 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540942+0000", 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3549999999999999e-06 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543680+0000", 2024-09-21T08:28:50.651 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002738379 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543701+0000", 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1072000000000001e-05 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543712+0000", 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0691e-05 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.652 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543786+0000", 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4721999999999994e-05 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543809+0000", 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2731e-05 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553613+0000", 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098038980000000001 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553666+0000", 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2682999999999997e-05 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2124 7.7 e229/215 7:e55f2143:::smithi13779221-11:head v 229'87, mlcod=224'83)", 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 108.05678025, 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008804704, 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.654 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.655 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1900000000000003e-07 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535432+0000", 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026709979999999999 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535449+0000", 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6888000000000002e-05 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.656 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535460+0000", 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0947e-05 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535516+0000", 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5551999999999999e-05 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535538+0000", 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2135000000000001e-05 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.657 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.541520+0000", 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059823330000000003 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.541565+0000", 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5331999999999999e-05 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2177 7.2 7:43ab705f:::smithi13779221-10:head [rollback 44] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:28:50.658 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 101.05538400099999, 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0080117160000000003, 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2177 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534158+0000", 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0580000000000001e-06 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534200+0000", 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1875999999999997e-05 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534209+0000", 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0899999999999994e-06 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534215+0000", 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9499999999999998e-06 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.661 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:50.662 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:50.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534288+0000", 2024-09-21T08:28:50.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3223e-05 2024-09-21T08:28:50.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.663 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:50.663 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534294+0000", 2024-09-21T08:28:50.663 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.519e-06 2024-09-21T08:28:50.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.539853+0000", 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055597980000000003 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.539923+0000", 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9115000000000005e-05 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540058+0000", 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013511999999999999 2024-09-21T08:28:50.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540864+0000", 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00080602199999999999 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.541123+0000", 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025911100000000002 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542066+0000", 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000943541 2024-09-21T08:28:50.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542123+0000", 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7108999999999998e-05 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542169+0000", 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5184e-05 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2178 7.2 7:43ab705f:::smithi13779221-10:head [append 0~0] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 101.05521249900001, 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081322689999999993, 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2178 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.667 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534329+0000", 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8299999999999995e-07 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.668 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534351+0000", 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1662999999999999e-05 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534359+0000", 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9480000000000008e-06 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534364+0000", 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1329999999999998e-06 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.669 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534415+0000", 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1199999999999998e-05 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534421+0000", 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1200000000000001e-06 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540407+0000", 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059863010000000003 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.670 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540492+0000", 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4715000000000005e-05 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540616+0000", 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012489700000000001 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540903+0000", 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028699500000000001 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.671 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.541328+0000", 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042463800000000002 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542392+0000", 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010644199999999999 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542427+0000", 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4338999999999998e-05 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.672 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542461+0000", 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3917000000000001e-05 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2184 7.c 7:305aef79:::smithi13779221-15:head [copy-from ver 143] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 101.035955587, 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022239622000000001, 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.673 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2184 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.674 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553588+0000", 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.021e-06 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553596+0000", 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1400000000000002e-06 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553611+0000", 2024-09-21T08:28:50.675 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5478e-05 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553619+0000", 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9019999999999999e-06 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553685+0000", 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5977999999999997e-05 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.676 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553801+0000", 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116506 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.565629+0000", 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011827127999999999 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.574989+0000", 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093609639999999994 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.677 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575477+0000", 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048743400000000002 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575700+0000", 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022307499999999999 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575756+0000", 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6125999999999998e-05 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.678 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575825+0000", 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8869999999999996e-05 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2185 7.c 7:305aef79:::smithi13779221-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 101.035901972, 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016182583, 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.679 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2185 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:28:50.680 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553640+0000", 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.61e-07 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553643+0000", 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1020000000000001e-06 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553653+0000", 2024-09-21T08:28:50.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0249e-05 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553659+0000", 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3319999999999997e-06 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553963+0000", 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000303617 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.554000+0000", 2024-09-21T08:28:50.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6976e-05 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569623+0000", 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015623076 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569721+0000", 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7398000000000005e-05 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569822+0000", 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101072 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2219 7.c 7:361ee62b:::smithi13779221-40:head [sparse-read 0~3130511] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 100.044968025, 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095045249999999998, 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.684 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2219 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544574+0000", 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.297e-06 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544579+0000", 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2830000000000003e-06 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544588+0000", 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4849999999999998e-06 2024-09-21T08:28:50.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544596+0000", 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6930000000000007e-06 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544738+0000", 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014263700000000001 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544744+0000", 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6030000000000004e-06 2024-09-21T08:28:50.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.552342+0000", 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075976259999999997 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.552431+0000", 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9571e-05 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554077+0000", 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016463300000000001 2024-09-21T08:28:50.688 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2220 7.c 7:361ee62b:::smithi13779221-40:head [read 0~3546277,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 100.044936049, 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011021838000000001, 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2220 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5499999999999998e-07 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544610+0000", 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4349999999999999e-06 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544617+0000", 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2030000000000003e-06 2024-09-21T08:28:50.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544623+0000", 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0260000000000004e-06 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544805+0000", 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000181635 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544809+0000", 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5950000000000001e-06 2024-09-21T08:28:50.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554139+0000", 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.009329225 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554247+0000", 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010890700000000001 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.555627+0000", 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013792570000000001 2024-09-21T08:28:50.693 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2233 7.8 e238/218 7:17170b1f:::smithi13779221-4:head v 238'212, mlcod=238'208)", 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 99.033626412000004, 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018069034000000001, 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555917+0000", 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3089999999999998e-06 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556417+0000", 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050039099999999999 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556437+0000", 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9746e-05 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556446+0000", 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4590000000000004e-06 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.696 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556581+0000", 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013473700000000001 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556614+0000", 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3287000000000002e-05 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573937+0000", 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017322422000000001 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573984+0000", 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6683e-05 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2234 7.8 e238/218 7:17170b1f:::smithi13779221-4:head v 238'213, mlcod=238'208)", 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 99.033070748, 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017540864, 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.698 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556471+0000", 2024-09-21T08:28:50.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3099999999999997e-07 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556477+0000", 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9680000000000003e-06 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556486+0000", 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5719999999999992e-06 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556492+0000", 2024-09-21T08:28:50.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6149999999999996e-06 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.557226+0000", 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073403900000000002 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.557411+0000", 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018489400000000001 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573988+0000", 2024-09-21T08:28:50.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016576941000000001 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574011+0000", 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3104000000000001e-05 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2285 7.c 7:361ee62b:::smithi13779221-40:head [copy-from ver 252] snapc 4f=[4f,4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 97.728024259999998, 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010853527, 2024-09-21T08:28:50.702 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2285 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5999999999999999e-07 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861521+0000", 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7299999999999999e-06 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861524+0000", 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9510000000000002e-06 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861526+0000", 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5739999999999998e-06 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861577+0000", 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0764999999999999e-05 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861629+0000", 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1937000000000001e-05 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.863963+0000", 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002334175 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.869705+0000", 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057416610000000003 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872022+0000", 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023169670000000001 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872295+0000", 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027316900000000003 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872331+0000", 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6276999999999998e-05 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:28:50.707 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:28:50.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872370+0000", 2024-09-21T08:28:50.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8961000000000002e-05 2024-09-21T08:28:50.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.708 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.708 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:28:50.708 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:28:50.708 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:28:50.755 INFO:tasks.rados.rados.0.smithi137.stdout:1285: finishing write tid 3 to smithi13779221-44 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:1285: finishing write tid 4 to smithi13779221-44 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:1287: done (5 left) 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:1292: write (excl) oid 33 current snap is 128 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:1285: finishing write tid 5 to smithi13779221-44 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:1285: oid 44 updating version 0 to 316 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:1285: oid 44 version 316 is already newer than 315 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:1285: oid 44 version 316 is already newer than 314 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:1285: oid 44 updating version 316 to 317 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 317 (ObjNum 471 snap 128 seq_num 471) dirty exists 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:1285: left oid 44 (ObjNum 471 snap 128 seq_num 471) 2024-09-21T08:28:50.756 INFO:tasks.rados.rados.0.smithi137.stdout:1286: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:50.764 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:50.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:50.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:50.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:50.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:50.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:50.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:50.854 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:50.855 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:50.855 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:50.855 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:50.855 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:50.856 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:50.856 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:50.864 INFO:tasks.rados.rados.0.smithi137.stdout:1289: expect (ObjNum 459 snap 124 seq_num 459) 2024-09-21T08:28:50.873 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:50.873 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:50.889 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:50.889 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:50.890 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:28:50.906 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:28:50.906 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:28:50.913 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:50.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:50.914 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:28:51.014 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:51.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:28:50 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:28:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:28:51.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:51.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:51.039 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:51.082 INFO:tasks.rados.rados.0.smithi137.stdout:1288: finishing copy_from racing read to smithi13779221-12 2024-09-21T08:28:51.082 INFO:tasks.rados.rados.0.smithi137.stdout:1288: finishing copy_from to smithi13779221-12 2024-09-21T08:28:51.082 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 108 (ObjNum 332 snap 79 seq_num 332) dirty exists 2024-09-21T08:28:51.082 INFO:tasks.rados.rados.0.smithi137.stdout:1292: seq_num 472 ranges {732904=731414,2120442=538824,3278829=504743} 2024-09-21T08:28:51.089 INFO:tasks.rados.rados.0.smithi137.stdout:1292: writing smithi13779221-33 from 732904 to 1464318 tid 1 2024-09-21T08:28:51.096 INFO:tasks.rados.rados.0.smithi137.stdout:1292: writing smithi13779221-33 from 2120442 to 2659266 tid 2 2024-09-21T08:28:51.102 INFO:tasks.rados.rados.0.smithi137.stdout:1292: writing smithi13779221-33 from 3278829 to 3783572 tid 3 2024-09-21T08:28:51.103 INFO:tasks.rados.rados.0.smithi137.stdout:1285: done (5 left) 2024-09-21T08:28:51.103 INFO:tasks.rados.rados.0.smithi137.stdout:1286: done (4 left) 2024-09-21T08:28:51.103 INFO:tasks.rados.rados.0.smithi137.stdout:1288: done (3 left) 2024-09-21T08:28:51.103 INFO:tasks.rados.rados.0.smithi137.stdout:1289: done (2 left) 2024-09-21T08:28:51.103 INFO:tasks.rados.rados.0.smithi137.stdout:1293: snap_remove snap 119 2024-09-21T08:28:51.139 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:51.164 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:51.165 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:51.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:51.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:51.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:51.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:51.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:51.414 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:51.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:51.415 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:51.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:51.538 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:51.538 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:51.539 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:51.543 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1] out_osds: [4, 6, 3, 0] dead_osds: [] live_osds: [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:28:51.543 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:28:51.543 INFO:tasks.thrashosds.thrasher:Killing osd 7, live_osds are [0, 1, 2, 3, 4, 5, 6, 7] 2024-09-21T08:28:51.543 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-09-21T08:28:51.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl stop ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.7 2024-09-21T08:28:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:1293: done (2 left) 2024-09-21T08:28:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:1294: write oid 8 current snap is 128 2024-09-21T08:28:51.550 INFO:tasks.rados.rados.0.smithi137.stdout:1291: finishing rollback tid 0 to smithi13779221-3 2024-09-21T08:28:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:1291: finishing rollback tid 1 to smithi13779221-3 2024-09-21T08:28:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:1291: finishing rollback tid 2 to smithi13779221-3 2024-09-21T08:28:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 186 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:28:51.551 INFO:tasks.rados.rados.0.smithi137.stdout:1294: seq_num 473 ranges {749711=406663,1778924=682482,3189899=716015} 2024-09-21T08:28:51.557 INFO:tasks.rados.rados.0.smithi137.stdout:1294: writing smithi13779221-8 from 749711 to 1156374 tid 1 2024-09-21T08:28:51.564 INFO:tasks.rados.rados.0.smithi137.stdout:1294: writing smithi13779221-8 from 1778924 to 2461406 tid 2 2024-09-21T08:28:51.574 INFO:tasks.rados.rados.0.smithi137.stdout:1294: writing smithi13779221-8 from 3189899 to 3905914 tid 3 2024-09-21T08:28:51.574 INFO:tasks.rados.rados.0.smithi137.stdout:1291: done (2 left) 2024-09-21T08:28:51.574 INFO:tasks.rados.rados.0.smithi137.stdout:1295: copy_from oid 29 from oid 10 current snap is 128 2024-09-21T08:28:51.574 INFO:tasks.rados.rados.0.smithi137.stdout:1292: finishing write tid 1 to smithi13779221-33 2024-09-21T08:28:51.574 INFO:tasks.rados.rados.0.smithi137.stdout:1292: finishing write tid 2 to smithi13779221-33 2024-09-21T08:28:51.575 INFO:tasks.rados.rados.0.smithi137.stdout:1292: finishing write tid 3 to smithi13779221-33 2024-09-21T08:28:51.575 INFO:tasks.rados.rados.0.smithi137.stdout:1292: finishing write tid 4 to smithi13779221-33 2024-09-21T08:28:51.575 INFO:tasks.rados.rados.0.smithi137.stdout:1292: finishing write tid 5 to smithi13779221-33 2024-09-21T08:28:51.575 INFO:tasks.rados.rados.0.smithi137.stdout:1292: oid 33 updating version 0 to 285 2024-09-21T08:28:51.575 INFO:tasks.rados.rados.0.smithi137.stdout:1292: oid 33 version 285 is already newer than 283 2024-09-21T08:28:51.575 INFO:tasks.rados.rados.0.smithi137.stdout:1292: oid 33 updating version 285 to 286 2024-09-21T08:28:51.575 INFO:tasks.rados.rados.0.smithi137.stdout:1292: oid 33 version 286 is already newer than 284 2024-09-21T08:28:51.575 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 286 (ObjNum 472 snap 128 seq_num 472) dirty exists 2024-09-21T08:28:51.576 INFO:tasks.rados.rados.0.smithi137.stdout:1292: left oid 33 (ObjNum 472 snap 128 seq_num 472) 2024-09-21T08:28:51.576 INFO:tasks.rados.rados.0.smithi137.stdout:1294: finishing write tid 1 to smithi13779221-8 2024-09-21T08:28:51.576 INFO:tasks.rados.rados.0.smithi137.stdout:1292: done (2 left) 2024-09-21T08:28:51.576 INFO:tasks.rados.rados.0.smithi137.stdout:1296: read oid 13 snap -1 2024-09-21T08:28:51.576 INFO:tasks.rados.rados.0.smithi137.stdout:1296: expect (ObjNum 340 snap 80 seq_num 340) 2024-09-21T08:28:51.576 INFO:tasks.rados.rados.0.smithi137.stdout:1297: copy_from oid 41 from oid 50 current snap is 128 2024-09-21T08:28:51.576 INFO:tasks.rados.rados.0.smithi137.stdout:1294: finishing write tid 2 to smithi13779221-8 2024-09-21T08:28:51.577 INFO:tasks.rados.rados.0.smithi137.stdout:1298: snap_create 2024-09-21T08:28:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:1294: finishing write tid 3 to smithi13779221-8 2024-09-21T08:28:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:1294: finishing write tid 4 to smithi13779221-8 2024-09-21T08:28:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:1294: finishing write tid 5 to smithi13779221-8 2024-09-21T08:28:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:1294: oid 8 updating version 0 to 312 2024-09-21T08:28:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:1294: oid 8 version 312 is already newer than 310 2024-09-21T08:28:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:1294: oid 8 updating version 312 to 313 2024-09-21T08:28:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:1294: oid 8 version 313 is already newer than 311 2024-09-21T08:28:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 313 (ObjNum 473 snap 128 seq_num 473) dirty exists 2024-09-21T08:28:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:1294: left oid 8 (ObjNum 473 snap 128 seq_num 473) 2024-09-21T08:28:51.592 INFO:tasks.rados.rados.0.smithi137.stdout:1296: expect (ObjNum 340 snap 80 seq_num 340) 2024-09-21T08:28:51.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:51.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:51.662 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:51.662 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:51.763 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:51.784 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:51.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:51.785 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:51.815 INFO:tasks.rados.rados.0.smithi137.stdout:1295: finishing copy_from racing read to smithi13779221-29 2024-09-21T08:28:51.815 INFO:tasks.rados.rados.0.smithi137.stdout:1295: finishing copy_from to smithi13779221-29 2024-09-21T08:28:51.816 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 218 (ObjNum 385 snap 94 seq_num 385) dirty exists 2024-09-21T08:28:51.816 INFO:tasks.rados.rados.0.smithi137.stdout:1297: finishing copy_from racing read to smithi13779221-41 2024-09-21T08:28:51.816 INFO:tasks.rados.rados.0.smithi137.stdout:1297: finishing copy_from to smithi13779221-41 2024-09-21T08:28:51.816 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 356 (ObjNum 463 snap 124 seq_num 463) dirty exists 2024-09-21T08:28:51.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:51 smithi165 ceph-mon[32893]: pgmap v410: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 239 MiB data, 775 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:51.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:51 smithi165 ceph-mon[32893]: osdmap e337: 8 total, 8 up, 4 in 2024-09-21T08:28:51.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:51 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2783283218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:51.819 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:51 smithi165 ceph-mon[32893]: osdmap e338: 8 total, 8 up, 4 in 2024-09-21T08:28:51.819 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:51 smithi165 systemd[1]: Stopping Ceph osd.7 for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:28:51.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:28:51.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:51.894 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:51 smithi137 ceph-mon[29634]: pgmap v410: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 239 MiB data, 775 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:51.894 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:51 smithi137 ceph-mon[29634]: osdmap e337: 8 total, 8 up, 4 in 2024-09-21T08:28:51.894 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:51 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2783283218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:51.894 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:51 smithi137 ceph-mon[29634]: osdmap e338: 8 total, 8 up, 4 in 2024-09-21T08:28:51.895 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:51 smithi137 ceph-mon[41044]: pgmap v410: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 239 MiB data, 775 MiB used, 357 GiB / 358 GiB avail 2024-09-21T08:28:51.895 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:51 smithi137 ceph-mon[41044]: osdmap e337: 8 total, 8 up, 4 in 2024-09-21T08:28:51.895 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:51 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2783283218' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:51.895 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:51 smithi137 ceph-mon[41044]: osdmap e338: 8 total, 8 up, 4 in 2024-09-21T08:28:51.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:51.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:51.916 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:52.016 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:52.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:52.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:52.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:52.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:52.161 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:52.162 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:52.162 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:52.263 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:52.264 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:28:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:28:52.264 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7[56695]: 2024-09-21T08:28:51.867+0000 7f09e942b640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T08:28:52.264 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7[56695]: 2024-09-21T08:28:51.867+0000 7f09e942b640 -1 osd.7 338 *** Got signal Terminated *** 2024-09-21T08:28:52.264 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7[56695]: 2024-09-21T08:28:51.867+0000 7f09e942b640 -1 osd.7 338 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T08:28:52.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:52.286 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:52.286 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:52.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T08:28:52.387 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:52.409 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:52.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:52.410 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:52.457 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:52.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:52.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:52.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:52.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:52.573 INFO:tasks.rados.rados.0.smithi137.stdout:1294: done (4 left) 2024-09-21T08:28:52.573 INFO:tasks.rados.rados.0.smithi137.stdout:1295: done (3 left) 2024-09-21T08:28:52.573 INFO:tasks.rados.rados.0.smithi137.stdout:1296: done (2 left) 2024-09-21T08:28:52.573 INFO:tasks.rados.rados.0.smithi137.stdout:1297: done (1 left) 2024-09-21T08:28:52.573 INFO:tasks.rados.rados.0.smithi137.stdout:1298: done (0 left) 2024-09-21T08:28:52.574 INFO:tasks.rados.rados.0.smithi137.stdout:1299: snap_remove snap 126 2024-09-21T08:28:52.634 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:52.644 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:52.656 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:52.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:52.657 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:52.709 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:52 smithi137 ceph-mon[29634]: osd.7 marked itself down and dead 2024-09-21T08:28:52.709 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:52 smithi137 ceph-mon[41044]: osd.7 marked itself down and dead 2024-09-21T08:28:52.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:52.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:52.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:52.781 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:52.798 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:52 smithi165 ceph-mon[32893]: osd.7 marked itself down and dead 2024-09-21T08:28:52.882 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:52.903 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:52.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:52.904 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:53.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:53.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:53.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:53.045 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:53.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:53.149 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:28:53.149 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:28:53.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:53.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:53.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:53.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:53.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:53.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:53.297 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:53.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:53.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:53.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:53.420 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:53.420 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:53 smithi165 podman[90839]: 2024-09-21 08:28:53.146594732 +0000 UTC m=+1.398809097 container died 13f9507a957d7504ee74b5d4949287d7d38bdda2908621a38c192a72bee90da8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-09-21T08:28:53.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:53.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:53.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:53.544 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:53.573 INFO:tasks.rados.rados.0.smithi137.stdout:1299: done (0 left) 2024-09-21T08:28:53.573 INFO:tasks.rados.rados.0.smithi137.stdout:1300: snap_remove snap 116 2024-09-21T08:28:53.573 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:28:53.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:53.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:53.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:53.667 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:53.763 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:53 smithi165 podman[90839]: 2024-09-21 08:28:53.577612951 +0000 UTC m=+1.829827309 container cleanup 13f9507a957d7504ee74b5d4949287d7d38bdda2908621a38c192a72bee90da8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0) 2024-09-21T08:28:53.763 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:53 smithi165 bash[90839]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7 2024-09-21T08:28:53.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:53 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:53.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:53 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:28:53.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:53 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:28:53.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:53 smithi165 ceph-mon[32893]: pgmap v413: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 253 MiB data, 826 MiB used, 357 GiB / 358 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 25 op/s 2024-09-21T08:28:53.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:53 smithi165 ceph-mon[32893]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T08:28:53.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:53 smithi165 ceph-mon[32893]: osdmap e339: 8 total, 7 up, 4 in 2024-09-21T08:28:53.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:53 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3725262213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:53.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:53 smithi165 ceph-mon[32893]: from='client.45473 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:53.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:53.789 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:53.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:53.790 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:53.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:53.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:28:53.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:28:53.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[29634]: pgmap v413: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 253 MiB data, 826 MiB used, 357 GiB / 358 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 25 op/s 2024-09-21T08:28:53.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[29634]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T08:28:53.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[29634]: osdmap e339: 8 total, 7 up, 4 in 2024-09-21T08:28:53.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3725262213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:53.891 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[29634]: from='client.45473 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:53.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:28:53.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:28:53.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:28:53.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[41044]: pgmap v413: 168 pgs: 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 137 active+clean; 253 MiB data, 826 MiB used, 357 GiB / 358 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 25 op/s 2024-09-21T08:28:53.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[41044]: Health check failed: 1 osds down (OSD_DOWN) 2024-09-21T08:28:53.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[41044]: osdmap e339: 8 total, 7 up, 4 in 2024-09-21T08:28:53.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3725262213' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:53.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:53 smithi137 ceph-mon[41044]: from='client.45473 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:28:53.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:53.914 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:53.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:53.916 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:53.976 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:28:54.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:54.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:54.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:54.040 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:54.134 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:53 smithi165 podman[90997]: 2024-09-21 08:28:53.806337804 +0000 UTC m=+0.788362801 container remove 13f9507a957d7504ee74b5d4949287d7d38bdda2908621a38c192a72bee90da8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0) 2024-09-21T08:28:54.141 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:54.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:54.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:54.164 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:54.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:54.287 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:54.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:54.288 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:54.350 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:54.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:54.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T08:28:54.409 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:54.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:54.410 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:54.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:28:54.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:54.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:54.534 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:28:54.583 INFO:tasks.rados.rados.0.smithi137.stdout:1300: done (0 left) 2024-09-21T08:28:54.583 INFO:tasks.rados.rados.0.smithi137.stdout:1301: write oid 1 current snap is 129 2024-09-21T08:28:54.584 INFO:tasks.rados.rados.0.smithi137.stdout:1301: seq_num 474 ranges {766518=481912,1837406=542493,2933675=527287,3876584=89934} 2024-09-21T08:28:54.591 INFO:tasks.rados.rados.0.smithi137.stdout:1301: writing smithi13779221-1 from 766518 to 1248430 tid 1 2024-09-21T08:28:54.597 INFO:tasks.rados.rados.0.smithi137.stdout:1301: writing smithi13779221-1 from 1837406 to 2379899 tid 2 2024-09-21T08:28:54.604 INFO:tasks.rados.rados.0.smithi137.stdout:1301: writing smithi13779221-1 from 2933675 to 3460962 tid 3 2024-09-21T08:28:54.607 INFO:tasks.rados.rados.0.smithi137.stdout:1301: writing smithi13779221-1 from 3876584 to 3966518 tid 4 2024-09-21T08:28:54.607 INFO:tasks.rados.rados.0.smithi137.stdout:1302: snap_create 2024-09-21T08:28:54.608 INFO:tasks.rados.rados.0.smithi137.stdout:1301: finishing write tid 1 to smithi13779221-1 2024-09-21T08:28:54.608 INFO:tasks.rados.rados.0.smithi137.stdout:1301: finishing write tid 2 to smithi13779221-1 2024-09-21T08:28:54.608 INFO:tasks.rados.rados.0.smithi137.stdout:1301: finishing write tid 3 to smithi13779221-1 2024-09-21T08:28:54.609 INFO:tasks.rados.rados.0.smithi137.stdout:1301: finishing write tid 4 to smithi13779221-1 2024-09-21T08:28:54.609 INFO:tasks.rados.rados.0.smithi137.stdout:1301: finishing write tid 5 to smithi13779221-1 2024-09-21T08:28:54.610 INFO:tasks.rados.rados.0.smithi137.stdout:1301: finishing write tid 6 to smithi13779221-1 2024-09-21T08:28:54.610 INFO:tasks.rados.rados.0.smithi137.stdout:1301: oid 1 updating version 0 to 517 2024-09-21T08:28:54.610 INFO:tasks.rados.rados.0.smithi137.stdout:1301: oid 1 version 517 is already newer than 515 2024-09-21T08:28:54.610 INFO:tasks.rados.rados.0.smithi137.stdout:1301: oid 1 updating version 517 to 518 2024-09-21T08:28:54.610 INFO:tasks.rados.rados.0.smithi137.stdout:1301: oid 1 version 518 is already newer than 516 2024-09-21T08:28:54.610 INFO:tasks.rados.rados.0.smithi137.stdout:1301: oid 1 version 518 is already newer than 514 2024-09-21T08:28:54.610 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 518 (ObjNum 474 snap 129 seq_num 474) dirty exists 2024-09-21T08:28:54.610 INFO:tasks.rados.rados.0.smithi137.stdout:1301: left oid 1 (ObjNum 474 snap 129 seq_num 474) 2024-09-21T08:28:54.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:54.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:54.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:54.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:54.678 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:54.758 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:54 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:54 smithi165 ceph-mon[32893]: from='client.45473 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:28:54.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:54 smithi165 ceph-mon[32893]: osdmap e340: 8 total, 7 up, 4 in 2024-09-21T08:28:54.780 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:54.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:54.781 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:54.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:54 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:54.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:54 smithi137 ceph-mon[29634]: from='client.45473 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:28:54.855 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:54 smithi137 ceph-mon[29634]: osdmap e340: 8 total, 7 up, 4 in 2024-09-21T08:28:54.855 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:54 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:28:54.855 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:54 smithi137 ceph-mon[41044]: from='client.45473 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:28:54.856 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:54 smithi137 ceph-mon[41044]: osdmap e340: 8 total, 7 up, 4 in 2024-09-21T08:28:54.862 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:54.862 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:54.872 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:54.872 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:54.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:54.887 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:54.887 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:54.903 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:54.903 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:54.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:54.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:54.905 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:54.920 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:54.921 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:54.936 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:54.937 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:54.953 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:54.953 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:54.953 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:28:55.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:55.027 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:55.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:55.028 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:55.129 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:55.150 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:55.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:55.151 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:55.160 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:54 smithi165 podman[91280]: 2024-09-21 08:28:54.703204071 +0000 UTC m=+0.017685812 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T08:28:55.160 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:55 smithi165 podman[91280]: 2024-09-21 08:28:55.043979699 +0000 UTC m=+0.358461431 container create 48173940266ff4962ee231dfa6894ab74bb1ad3d688053b13368a8e3092a3789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T08:28:55.252 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:55.273 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:55.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:55.274 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:55.375 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:55.399 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:55.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:55.400 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:55.464 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:55 smithi165 podman[91280]: 2024-09-21 08:28:55.338381308 +0000 UTC m=+0.652863039 container init 48173940266ff4962ee231dfa6894ab74bb1ad3d688053b13368a8e3092a3789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-09-21T08:28:55.464 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:55 smithi165 podman[91280]: 2024-09-21 08:28:55.344882571 +0000 UTC m=+0.659364308 container start 48173940266ff4962ee231dfa6894ab74bb1ad3d688053b13368a8e3092a3789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-deactivate, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731) 2024-09-21T08:28:55.464 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:55 smithi165 podman[91280]: 2024-09-21 08:28:55.46339071 +0000 UTC m=+0.777872461 container attach 48173940266ff4962ee231dfa6894ab74bb1ad3d688053b13368a8e3092a3789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True) 2024-09-21T08:28:55.500 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:55.522 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:55.522 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:55.523 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:55.581 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:28:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:28:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:28:55.581 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:28:55.581 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:28:55.593 INFO:tasks.rados.rados.0.smithi137.stdout:1301: done (1 left) 2024-09-21T08:28:55.593 INFO:tasks.rados.rados.0.smithi137.stdout:1302: done (0 left) 2024-09-21T08:28:55.593 INFO:tasks.rados.rados.0.smithi137.stdout:1303: read oid 7 snap -1 2024-09-21T08:28:55.593 INFO:tasks.rados.rados.0.smithi137.stdout:1303: expect (ObjNum 443 snap 114 seq_num 443) 2024-09-21T08:28:55.593 INFO:tasks.rados.rados.0.smithi137.stdout:1304: read oid 14 snap -1 2024-09-21T08:28:55.593 INFO:tasks.rados.rados.0.smithi137.stdout:1304: expect (ObjNum 456 snap 123 seq_num 456) 2024-09-21T08:28:55.593 INFO:tasks.rados.rados.0.smithi137.stdout:1305: read oid 3 snap -1 2024-09-21T08:28:55.594 INFO:tasks.rados.rados.0.smithi137.stdout:1305: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:55.594 INFO:tasks.rados.rados.0.smithi137.stdout:1306: rollback oid 8 current snap is 130 2024-09-21T08:28:55.594 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 8 to 123 2024-09-21T08:28:55.594 INFO:tasks.rados.rados.0.smithi137.stdout:1307: snap_create 2024-09-21T08:28:55.600 INFO:tasks.rados.rados.0.smithi137.stdout:1306: finishing rollback tid 0 to smithi13779221-8 2024-09-21T08:28:55.601 INFO:tasks.rados.rados.0.smithi137.stdout:1306: finishing rollback tid 1 to smithi13779221-8 2024-09-21T08:28:55.601 INFO:tasks.rados.rados.0.smithi137.stdout:1306: finishing rollback tid 2 to smithi13779221-8 2024-09-21T08:28:55.602 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 316 (ObjNum 404 snap 99 seq_num 404) dirty exists 2024-09-21T08:28:55.603 INFO:tasks.rados.rados.0.smithi137.stdout:1303: expect (ObjNum 443 snap 114 seq_num 443) 2024-09-21T08:28:55.624 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:55.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:55.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:55.646 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:55.654 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:28:55.655 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:28:55.715 INFO:tasks.rados.rados.0.smithi137.stdout:1304: expect (ObjNum 456 snap 123 seq_num 456) 2024-09-21T08:28:55.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:55.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:55 smithi165 ceph-mon[32893]: pgmap v416: 168 pgs: 12 stale+active+clean, 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 125 active+clean; 253 MiB data, 826 MiB used, 357 GiB / 358 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 25 op/s 2024-09-21T08:28:55.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:55 smithi165 ceph-mon[32893]: osdmap e341: 8 total, 7 up, 4 in 2024-09-21T08:28:55.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:55 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4138755304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:55.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:55.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:55.769 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:55.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:55.892 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:55.893 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:55.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:55.959 INFO:tasks.rados.rados.0.smithi137.stdout:1305: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:28:55.994 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:56.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:56.016 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:56.016 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:56.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:55 smithi137 ceph-mon[29634]: pgmap v416: 168 pgs: 12 stale+active+clean, 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 125 active+clean; 253 MiB data, 826 MiB used, 357 GiB / 358 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 25 op/s 2024-09-21T08:28:56.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:55 smithi137 ceph-mon[29634]: osdmap e341: 8 total, 7 up, 4 in 2024-09-21T08:28:56.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:55 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4138755304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:55 smithi137 ceph-mon[41044]: pgmap v416: 168 pgs: 12 stale+active+clean, 4 active+clean+snaptrim, 27 active+clean+snaptrim_wait, 125 active+clean; 253 MiB data, 826 MiB used, 357 GiB / 358 GiB avail; 15 MiB/s rd, 3.5 MiB/s wr, 25 op/s 2024-09-21T08:28:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:55 smithi137 ceph-mon[41044]: osdmap e341: 8 total, 7 up, 4 in 2024-09-21T08:28:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:55 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4138755304' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:56.086 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:28:55 smithi165 podman[91280]: 2024-09-21 08:28:55.768236559 +0000 UTC m=+1.082718293 container died 48173940266ff4962ee231dfa6894ab74bb1ad3d688053b13368a8e3092a3789 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731) 2024-09-21T08:28:56.117 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:56.138 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:56.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:56.139 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:56.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:56.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:56.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:56.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:56.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:28:56.360 DEBUG:teuthology.orchestra.run.smithi165:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.7.service' 2024-09-21T08:28:56.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:56.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:56.391 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:56.391 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:56.411 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T08:28:56.411 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-09-21T08:28:56.412 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.7 2024-09-21T08:28:56.412 INFO:tasks.thrashosds.thrasher:remote for osd 7 is ubuntu@smithi165.front.sepia.ceph.com 2024-09-21T08:28:56.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.7 --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-21T08:28:56.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:56.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:56.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:56.515 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:56.541 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:56.555 INFO:tasks.rados.rados.0.smithi137.stdout:1303: done (4 left) 2024-09-21T08:28:56.555 INFO:tasks.rados.rados.0.smithi137.stdout:1304: done (3 left) 2024-09-21T08:28:56.555 INFO:tasks.rados.rados.0.smithi137.stdout:1305: done (2 left) 2024-09-21T08:28:56.555 INFO:tasks.rados.rados.0.smithi137.stdout:1306: done (1 left) 2024-09-21T08:28:56.555 INFO:tasks.rados.rados.0.smithi137.stdout:1307: done (0 left) 2024-09-21T08:28:56.555 INFO:tasks.rados.rados.0.smithi137.stdout:1308: write (excl) oid 47 current snap is 131 2024-09-21T08:28:56.555 INFO:tasks.rados.rados.0.smithi137.stdout:1308: seq_num 475 ranges {783325=557161,1779535=686151,3244745=454912} 2024-09-21T08:28:56.563 INFO:tasks.rados.rados.0.smithi137.stdout:1308: writing smithi13779221-47 from 783325 to 1340486 tid 1 2024-09-21T08:28:56.574 INFO:tasks.rados.rados.0.smithi137.stdout:1308: writing smithi13779221-47 from 1779535 to 2465686 tid 2 2024-09-21T08:28:56.581 INFO:tasks.rados.rados.0.smithi137.stdout:1308: writing smithi13779221-47 from 3244745 to 3699657 tid 3 2024-09-21T08:28:56.581 INFO:tasks.rados.rados.0.smithi137.stdout:1309: delete oid 38 current snap is 131 2024-09-21T08:28:56.593 INFO:tasks.rados.rados.0.smithi137.stdout:1308: finishing write tid 1 to smithi13779221-47 2024-09-21T08:28:56.593 INFO:tasks.rados.rados.0.smithi137.stdout:1308: finishing write tid 2 to smithi13779221-47 2024-09-21T08:28:56.593 INFO:tasks.rados.rados.0.smithi137.stdout:1308: finishing write tid 3 to smithi13779221-47 2024-09-21T08:28:56.593 INFO:tasks.rados.rados.0.smithi137.stdout:1308: finishing write tid 4 to smithi13779221-47 2024-09-21T08:28:56.593 INFO:tasks.rados.rados.0.smithi137.stdout:1308: finishing write tid 5 to smithi13779221-47 2024-09-21T08:28:56.593 INFO:tasks.rados.rados.0.smithi137.stdout:1308: oid 47 updating version 0 to 402 2024-09-21T08:28:56.593 INFO:tasks.rados.rados.0.smithi137.stdout:1308: oid 47 version 402 is already newer than 400 2024-09-21T08:28:56.594 INFO:tasks.rados.rados.0.smithi137.stdout:1308: oid 47 version 402 is already newer than 401 2024-09-21T08:28:56.594 INFO:tasks.rados.rados.0.smithi137.stdout:1308: oid 47 version 402 is already newer than 399 2024-09-21T08:28:56.594 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 402 (ObjNum 475 snap 131 seq_num 475) dirty exists 2024-09-21T08:28:56.594 INFO:tasks.rados.rados.0.smithi137.stdout:1308: left oid 47 (ObjNum 475 snap 131 seq_num 475) 2024-09-21T08:28:56.594 INFO:tasks.rados.rados.0.smithi137.stdout:1308: done (1 left) 2024-09-21T08:28:56.595 INFO:tasks.rados.rados.0.smithi137.stdout:1309: done (0 left) 2024-09-21T08:28:56.595 INFO:tasks.rados.rados.0.smithi137.stdout:1310: rollback oid 12 current snap is 131 2024-09-21T08:28:56.595 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 12 to 112 2024-09-21T08:28:56.595 INFO:tasks.rados.rados.0.smithi137.stdout:1311: write (excl) oid 39 current snap is 131 2024-09-21T08:28:56.595 INFO:tasks.rados.rados.0.smithi137.stdout:1311: seq_num 477 ranges {416939=707659,1896499=120440} 2024-09-21T08:28:56.600 INFO:tasks.rados.rados.0.smithi137.stdout:1311: writing smithi13779221-39 from 416939 to 1124598 tid 1 2024-09-21T08:28:56.606 INFO:tasks.rados.rados.0.smithi137.stdout:1311: writing smithi13779221-39 from 1896499 to 2016939 tid 2 2024-09-21T08:28:56.606 INFO:tasks.rados.rados.0.smithi137.stdout:1312: read oid 13 snap 125 2024-09-21T08:28:56.606 INFO:tasks.rados.rados.0.smithi137.stdout:1312: expect (ObjNum 340 snap 80 seq_num 340) 2024-09-21T08:28:56.606 INFO:tasks.rados.rados.0.smithi137.stdout:1313: write (excl) oid 40 current snap is 131 2024-09-21T08:28:56.606 INFO:tasks.rados.rados.0.smithi137.stdout:1310: finishing rollback tid 0 to smithi13779221-12 2024-09-21T08:28:56.606 INFO:tasks.rados.rados.0.smithi137.stdout:1313: seq_num 478 ranges {433746=782908,1954981=78765} 2024-09-21T08:28:56.612 INFO:tasks.rados.rados.0.smithi137.stdout:1313: writing smithi13779221-40 from 433746 to 1216654 tid 1 2024-09-21T08:28:56.617 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:56.618 INFO:tasks.rados.rados.0.smithi137.stdout:1313: writing smithi13779221-40 from 1954981 to 2033746 tid 2 2024-09-21T08:28:56.618 INFO:tasks.rados.rados.0.smithi137.stdout:1314: snap_create 2024-09-21T08:28:56.618 INFO:tasks.rados.rados.0.smithi137.stdout:1310: finishing rollback tid 1 to smithi13779221-12 2024-09-21T08:28:56.618 INFO:tasks.rados.rados.0.smithi137.stdout:1310: finishing rollback tid 2 to smithi13779221-12 2024-09-21T08:28:56.618 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 366 (ObjNum 434 snap 110 seq_num 434) dirty exists 2024-09-21T08:28:56.618 INFO:tasks.rados.rados.0.smithi137.stdout:1311: finishing write tid 1 to smithi13779221-39 2024-09-21T08:28:56.619 INFO:tasks.rados.rados.0.smithi137.stdout:1311: finishing write tid 2 to smithi13779221-39 2024-09-21T08:28:56.619 INFO:tasks.rados.rados.0.smithi137.stdout:1311: finishing write tid 3 to smithi13779221-39 2024-09-21T08:28:56.619 INFO:tasks.rados.rados.0.smithi137.stdout:1311: finishing write tid 4 to smithi13779221-39 2024-09-21T08:28:56.619 INFO:tasks.rados.rados.0.smithi137.stdout:1311: oid 39 updating version 0 to 226 2024-09-21T08:28:56.619 INFO:tasks.rados.rados.0.smithi137.stdout:1311: oid 39 updating version 226 to 227 2024-09-21T08:28:56.619 INFO:tasks.rados.rados.0.smithi137.stdout:1311: oid 39 version 227 is already newer than 225 2024-09-21T08:28:56.619 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 227 (ObjNum 477 snap 131 seq_num 477) dirty exists 2024-09-21T08:28:56.619 INFO:tasks.rados.rados.0.smithi137.stdout:1311: left oid 39 (ObjNum 477 snap 131 seq_num 477) 2024-09-21T08:28:56.619 INFO:tasks.rados.rados.0.smithi137.stdout:1312: expect (ObjNum 340 snap 80 seq_num 340) 2024-09-21T08:28:56.640 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:56.641 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:56.641 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:56.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:56.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:56.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:56.764 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:56.840 INFO:tasks.rados.rados.0.smithi137.stdout:1313: finishing write tid 1 to smithi13779221-40 2024-09-21T08:28:56.840 INFO:tasks.rados.rados.0.smithi137.stdout:1313: finishing write tid 2 to smithi13779221-40 2024-09-21T08:28:56.840 INFO:tasks.rados.rados.0.smithi137.stdout:1313: finishing write tid 3 to smithi13779221-40 2024-09-21T08:28:56.840 INFO:tasks.rados.rados.0.smithi137.stdout:1313: finishing write tid 4 to smithi13779221-40 2024-09-21T08:28:56.840 INFO:tasks.rados.rados.0.smithi137.stdout:1313: oid 40 updating version 0 to 317 2024-09-21T08:28:56.841 INFO:tasks.rados.rados.0.smithi137.stdout:1313: oid 40 updating version 317 to 318 2024-09-21T08:28:56.841 INFO:tasks.rados.rados.0.smithi137.stdout:1313: oid 40 version 318 is already newer than 316 2024-09-21T08:28:56.841 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 318 (ObjNum 478 snap 131 seq_num 478) dirty exists 2024-09-21T08:28:56.841 INFO:tasks.rados.rados.0.smithi137.stdout:1313: left oid 40 (ObjNum 478 snap 131 seq_num 478) 2024-09-21T08:28:56.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:56.872 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:56 smithi165 ceph-mon[32893]: 6.a scrub starts 2024-09-21T08:28:56.872 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:56 smithi165 ceph-mon[32893]: 6.a scrub ok 2024-09-21T08:28:56.872 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:56 smithi165 ceph-mon[32893]: osdmap e342: 8 total, 7 up, 4 in 2024-09-21T08:28:56.872 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:56 smithi165 ceph-mon[32893]: osdmap e343: 8 total, 7 up, 4 in 2024-09-21T08:28:56.886 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:56.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:56.887 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:56.899 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T08:28:56.944 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T08:28:56.963 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:56 smithi137 ceph-mon[29634]: 6.a scrub starts 2024-09-21T08:28:56.963 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:56 smithi137 ceph-mon[29634]: 6.a scrub ok 2024-09-21T08:28:56.963 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:56 smithi137 ceph-mon[29634]: osdmap e342: 8 total, 7 up, 4 in 2024-09-21T08:28:56.963 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:56 smithi137 ceph-mon[29634]: osdmap e343: 8 total, 7 up, 4 in 2024-09-21T08:28:56.963 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:56 smithi137 ceph-mon[41044]: 6.a scrub starts 2024-09-21T08:28:56.963 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:56 smithi137 ceph-mon[41044]: 6.a scrub ok 2024-09-21T08:28:56.963 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:56 smithi137 ceph-mon[41044]: osdmap e342: 8 total, 7 up, 4 in 2024-09-21T08:28:56.963 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:56 smithi137 ceph-mon[41044]: osdmap e343: 8 total, 7 up, 4 in 2024-09-21T08:28:56.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:57.010 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:57.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:57.011 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:57.112 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:57.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:57.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:57.135 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:57.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:57.256 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:57.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:57.257 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:57.358 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:57.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:57.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:57.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:57.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:57.503 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:57.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:57.504 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:57.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:57.605 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:28:57.608 INFO:tasks.rados.rados.0.smithi137.stdout:1310: done (4 left) 2024-09-21T08:28:57.609 INFO:tasks.rados.rados.0.smithi137.stdout:1311: done (3 left) 2024-09-21T08:28:57.609 INFO:tasks.rados.rados.0.smithi137.stdout:1312: done (2 left) 2024-09-21T08:28:57.609 INFO:tasks.rados.rados.0.smithi137.stdout:1313: done (1 left) 2024-09-21T08:28:57.609 INFO:tasks.rados.rados.0.smithi137.stdout:1314: done (0 left) 2024-09-21T08:28:57.609 INFO:tasks.rados.rados.0.smithi137.stdout:1315: rollback oid 10 current snap is 132 2024-09-21T08:28:57.609 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 10 to 131 2024-09-21T08:28:57.609 INFO:tasks.rados.rados.0.smithi137.stdout:1316: snap_create 2024-09-21T08:28:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:1315: finishing rollback tid 0 to smithi13779221-10 2024-09-21T08:28:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:1315: finishing rollback tid 1 to smithi13779221-10 2024-09-21T08:28:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:1315: finishing rollback tid 2 to smithi13779221-10 2024-09-21T08:28:57.618 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 331 (ObjNum 385 snap 94 seq_num 385) dirty exists 2024-09-21T08:28:57.627 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:57.631 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:57.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:57.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:57.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:28:57.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:57.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:57.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:57.754 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:57.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:57.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:57.878 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:57.878 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:57.959 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:57 smithi137 ceph-mon[29634]: pgmap v419: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 265 MiB data, 832 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 15 op/s; 242/960 objects degraded (25.208%) 2024-09-21T08:28:57.959 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:57 smithi137 ceph-mon[29634]: Health check failed: Degraded data redundancy: 242/960 objects degraded (25.208%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:28:57.959 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:57 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2739247191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:57.960 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:57 smithi137 ceph-mon[41044]: pgmap v419: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 265 MiB data, 832 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 15 op/s; 242/960 objects degraded (25.208%) 2024-09-21T08:28:57.960 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:57 smithi137 ceph-mon[41044]: Health check failed: Degraded data redundancy: 242/960 objects degraded (25.208%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:28:57.960 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:57 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2739247191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:57.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:57 smithi165 ceph-mon[32893]: pgmap v419: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 265 MiB data, 832 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 15 op/s; 242/960 objects degraded (25.208%) 2024-09-21T08:28:57.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:57 smithi165 ceph-mon[32893]: Health check failed: Degraded data redundancy: 242/960 objects degraded (25.208%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:28:57.964 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:57 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2739247191' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:28:57.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:58.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:58.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:58.003 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:58.072 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:28:58.072 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:28:58.072 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:28:58.072 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44417.0:46 5.1f e70/69 5:f95f44c2:::notify.0:head v 70'1, mlcod=0'0)", 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.569323+0000", 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 323.49512031500001, 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021424550000000001, 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.073 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569323+0000", 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569323+0000", 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569324+0000", 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.009e-06 2024-09-21T08:28:58.074 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569331+0000", 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.528e-06 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569343+0000", 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2483999999999999e-05 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.075 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569351+0000", 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5079999999999999e-06 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569398+0000", 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6708000000000001e-05 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569414+0000", 2024-09-21T08:28:58.076 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6829000000000001e-05 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571435+0000", 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020206299999999998 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571466+0000", 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0759000000000001e-05 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.077 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44387.0:46 5.1f e70/69 5:f95f44c2:::notify.0:head v 70'2, mlcod=0'0)", 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.569550+0000", 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 323.494893268, 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019394410000000001, 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.078 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569550+0000", 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569550+0000", 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.079 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569551+0000", 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2500000000000005e-07 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569557+0000", 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.412e-06 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.080 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569564+0000", 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1420000000000004e-06 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569570+0000", 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6250000000000001e-06 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.570055+0000", 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048423599999999998 2024-09-21T08:28:58.081 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.570072+0000", 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7007000000000002e-05 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571468+0000", 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013962219999999999 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.082 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571490+0000", 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2072000000000001e-05 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:52 5.e 5:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.576342+0000", 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 323.48810177299998, 2024-09-21T08:28:58.083 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011620549999999999, 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 52 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576342+0000", 2024-09-21T08:28:58.084 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576342+0000", 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576344+0000", 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3659999999999999e-06 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.085 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576351+0000", 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7129999999999999e-06 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576362+0000", 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0740999999999999e-05 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576367+0000", 2024-09-21T08:28:58.086 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5670000000000002e-06 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576412+0000", 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5079e-05 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576503+0000", 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0703000000000002e-05 2024-09-21T08:28:58.087 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576585+0000", 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1724999999999994e-05 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577152+0000", 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000567054 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.088 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577187+0000", 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5212000000000001e-05 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577426+0000", 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023901099999999999 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577463+0000", 2024-09-21T08:28:58.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6820000000000003e-05 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577504+0000", 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1063999999999999e-05 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:52 5.e 5:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:28:58.090 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.576451+0000", 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 323.48799253700003, 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001253133, 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 52 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576451+0000", 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576451+0000", 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576453+0000", 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8050000000000001e-06 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576458+0000", 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3299999999999998e-06 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576468+0000", 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8700000000000004e-06 2024-09-21T08:28:58.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576476+0000", 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6729999999999992e-06 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576767+0000", 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029066200000000002 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576845+0000", 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8517999999999996e-05 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576961+0000", 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011566900000000001 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.095 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577297+0000", 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033612900000000001 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577569+0000", 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027198899999999999 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577623+0000", 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3841000000000002e-05 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577667+0000", 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4152e-05 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577704+0000", 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7494999999999999e-05 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:53 5.e 5:7759931f:::notify.3:head [watch watch cookie 94074441556624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.577867+0000", 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 323.486577092, 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001803189, 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:28:58.098 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 53 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577867+0000", 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577867+0000", 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577868+0000", 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1289999999999999e-06 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577872+0000", 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4050000000000002e-06 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577880+0000", 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1130000000000001e-06 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577885+0000", 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8080000000000004e-06 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578144+0000", 2024-09-21T08:28:58.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025872800000000001 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578215+0000", 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1330999999999999e-05 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578696+0000", 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048043800000000002 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578769+0000", 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3302999999999999e-05 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578911+0000", 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014194000000000001 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579286+0000", 2024-09-21T08:28:58.103 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037537600000000002 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579511+0000", 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022471700000000001 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579547+0000", 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5947e-05 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579599+0000", 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1929999999999999e-05 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579670+0000", 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1024000000000003e-05 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:206 4.18 4:1e85421f:gc::gc.1:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.590771+0000", 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.473672943, 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023079490000000001, 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.106 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 206 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590771+0000", 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590771+0000", 2024-09-21T08:28:58.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590773+0000", 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4739999999999999e-06 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590782+0000", 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1290000000000002e-06 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590795+0000", 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2456000000000001e-05 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590805+0000", 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0111e-05 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590876+0000", 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1111000000000004e-05 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590965+0000", 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9061999999999995e-05 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.591134+0000", 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000169044 2024-09-21T08:28:58.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.591516+0000", 2024-09-21T08:28:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000381362 2024-09-21T08:28:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.591767+0000", 2024-09-21T08:28:58.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025140299999999998 2024-09-21T08:28:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.593025+0000", 2024-09-21T08:28:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012583200000000001 2024-09-21T08:28:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.593054+0000", 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8863000000000001e-05 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.593079+0000", 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4613999999999998e-05 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:210 4.9 4:954be8d2:gc::gc.5:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.113 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.600055+0000", 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.46438854600001, 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012045479999999999, 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 210 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600055+0000", 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600055+0000", 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600057+0000", 2024-09-21T08:28:58.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9149999999999999e-06 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600063+0000", 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6230000000000002e-06 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600076+0000", 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3128e-05 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600083+0000", 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2439999999999996e-06 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600148+0000", 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5118999999999998e-05 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600224+0000", 2024-09-21T08:28:58.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6283000000000001e-05 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600400+0000", 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000175794 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600700+0000", 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029931899999999998 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601121+0000", 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042133399999999999 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601205+0000", 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4358999999999996e-05 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601232+0000", 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6902000000000001e-05 2024-09-21T08:28:58.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601260+0000", 2024-09-21T08:28:58.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7528000000000001e-05 2024-09-21T08:28:58.120 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.120 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.120 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:58.122 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:238 4.18 4:1a734c59:::obj_delete_at_hint.0000000000:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.122 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.633924+0000", 2024-09-21T08:28:58.122 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.43051967500003, 2024-09-21T08:28:58.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001234052, 2024-09-21T08:28:58.122 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.122 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 238 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633924+0000", 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633924+0000", 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633925+0000", 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1600000000000001e-07 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633928+0000", 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1810000000000001e-06 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633933+0000", 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4399999999999996e-06 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633937+0000", 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4249999999999998e-06 2024-09-21T08:28:58.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633973+0000", 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6247e-05 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634046+0000", 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2575000000000002e-05 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:28:58.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634151+0000", 2024-09-21T08:28:58.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010584199999999999 2024-09-21T08:28:58.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634527+0000", 2024-09-21T08:28:58.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037577699999999999 2024-09-21T08:28:58.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634630+0000", 2024-09-21T08:28:58.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102846 2024-09-21T08:28:58.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.635070+0000", 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000439796 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.635121+0000", 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0847999999999998e-05 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.635158+0000", 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7358999999999999e-05 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:255 4.9 4:921ecdb1:gc::gc.25:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.636496+0000", 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.42794802200001, 2024-09-21T08:28:58.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015126289999999999, 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 255 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636496+0000", 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636496+0000", 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636497+0000", 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2099999999999995e-07 2024-09-21T08:28:58.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636507+0000", 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9620000000000005e-06 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636514+0000", 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5270000000000001e-06 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636519+0000", 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7589999999999997e-06 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636564+0000", 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4919999999999997e-05 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636637+0000", 2024-09-21T08:28:58.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3219999999999994e-05 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636792+0000", 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015530599999999999 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637091+0000", 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029869500000000003 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637625+0000", 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053441199999999995 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637927+0000", 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030120199999999998 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637974+0000", 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6999999999999997e-05 2024-09-21T08:28:58.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.638008+0000", 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4705000000000001e-05 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:287 4.e 4:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.640319+0000", 2024-09-21T08:28:58.138 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.424125135, 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013081729999999999, 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 287 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640319+0000", 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640319+0000", 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640320+0000", 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4300000000000001e-07 2024-09-21T08:28:58.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640323+0000", 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.715e-06 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640331+0000", 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5619999999999998e-06 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640337+0000", 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8579999999999996e-06 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640379+0000", 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2290000000000003e-05 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640452+0000", 2024-09-21T08:28:58.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2948999999999996e-05 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640582+0000", 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013043000000000001 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640890+0000", 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000307614 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641132+0000", 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000241728 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641472+0000", 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034040600000000002 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641530+0000", 2024-09-21T08:28:58.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8112999999999997e-05 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641627+0000", 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6564999999999994e-05 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:300 4.1 4:856b9b26:reshard::reshard.0000000002:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.145 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.642024+0000", 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.422419768, 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011698909999999999, 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 300 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642024+0000", 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642024+0000", 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642025+0000", 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642030+0000", 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0370000000000002e-06 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642040+0000", 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3929999999999997e-06 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642045+0000", 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0470000000000001e-06 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642083+0000", 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8195000000000003e-05 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642167+0000", 2024-09-21T08:28:58.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3720000000000005e-05 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642315+0000", 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014880500000000001 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642777+0000", 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046160999999999999 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643089+0000", 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031199199999999998 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643132+0000", 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3343999999999997e-05 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643171+0000", 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9088999999999998e-05 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643194+0000", 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2489000000000001e-05 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.152 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:302 4.e 4:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.642059+0000", 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.42238472100001, 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012335429999999999, 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 302 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642059+0000", 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642059+0000", 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642060+0000", 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2900000000000004e-07 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642062+0000", 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5160000000000001e-06 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642066+0000", 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7979999999999999e-06 2024-09-21T08:28:58.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642069+0000", 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0010000000000002e-06 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642105+0000", 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6066999999999999e-05 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642153+0000", 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7642000000000002e-05 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642252+0000", 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9345000000000005e-05 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642627+0000", 2024-09-21T08:28:58.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000374745 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642847+0000", 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022007999999999999 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643187+0000", 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034015599999999999 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643250+0000", 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2829000000000004e-05 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643293+0000", 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2834999999999999e-05 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:316 4.1 4:856b9b26:reshard::reshard.0000000002:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.643653+0000", 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.420790291, 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011504950000000001, 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 316 2024-09-21T08:28:58.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643653+0000", 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643653+0000", 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643655+0000", 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2309999999999999e-06 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643659+0000", 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3669999999999999e-06 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643669+0000", 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0081999999999999e-05 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643675+0000", 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8039999999999998e-06 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643727+0000", 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2144000000000001e-05 2024-09-21T08:28:58.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643822+0000", 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4899000000000001e-05 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643982+0000", 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159887 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644499+0000", 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051742300000000004 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644668+0000", 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016911000000000001 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644716+0000", 2024-09-21T08:28:58.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7833000000000001e-05 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644768+0000", 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1634000000000001e-05 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644804+0000", 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6081000000000002e-05 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.166 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:329 4.1a 4:5909e671:gc::gc.27:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.644844+0000", 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.41959984900001, 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012130229999999999, 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 329 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644844+0000", 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644844+0000", 2024-09-21T08:28:58.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644845+0000", 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3030000000000001e-06 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644850+0000", 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9200000000000003e-06 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644859+0000", 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0529999999999996e-06 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644864+0000", 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6949999999999999e-06 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644905+0000", 2024-09-21T08:28:58.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1433999999999997e-05 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644949+0000", 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4024999999999999e-05 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645045+0000", 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5297999999999995e-05 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645301+0000", 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025601699999999999 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645800+0000", 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000499834 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645965+0000", 2024-09-21T08:28:58.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016433600000000001 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.646021+0000", 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6687999999999997e-05 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.646057+0000", 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5420000000000003e-05 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.173 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:453 4.1 4:80b6fe23:reshard::reshard.0000000011:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.670680+0000", 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.39376393399999, 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011835179999999999, 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.174 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 453 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670680+0000", 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670680+0000", 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670681+0000", 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2049999999999999e-06 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670690+0000", 2024-09-21T08:28:58.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3230000000000004e-06 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670698+0000", 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7810000000000005e-06 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670703+0000", 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6879999999999998e-06 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670757+0000", 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4660999999999997e-05 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670860+0000", 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102471 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:28:58.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671030+0000", 2024-09-21T08:28:58.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016985599999999999 2024-09-21T08:28:58.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671457+0000", 2024-09-21T08:28:58.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042770500000000001 2024-09-21T08:28:58.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671715+0000", 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025777299999999999 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671761+0000", 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5836e-05 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671812+0000", 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0719e-05 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671863+0000", 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1499999999999998e-05 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.183 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:462 4.1 4:80b6fe23:reshard::reshard.0000000011:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.672625+0000", 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.39181910600001, 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014703749999999999, 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 462 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.184 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672625+0000", 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672625+0000", 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672626+0000", 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.139e-06 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672630+0000", 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7190000000000001e-06 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672640+0000", 2024-09-21T08:28:58.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0459999999999994e-06 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672644+0000", 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7609999999999998e-06 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672684+0000", 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9431000000000002e-05 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672739+0000", 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4994999999999998e-05 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672855+0000", 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115975 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.673270+0000", 2024-09-21T08:28:58.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041560499999999999 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.673495+0000", 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022474600000000001 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.674004+0000", 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050865299999999997 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.674061+0000", 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6925999999999997e-05 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.674095+0000", 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4378999999999997e-05 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:501 4.1a 4:5c7e8a6d:reshard::reshard.0000000015:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.684553+0000", 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.37989095299997, 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015740999999999999, 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.191 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 501 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684553+0000", 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684553+0000", 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684554+0000", 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6000000000000002e-07 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684563+0000", 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0790000000000007e-06 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684570+0000", 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8379999999999996e-06 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684575+0000", 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5910000000000003e-06 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684605+0000", 2024-09-21T08:28:58.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0256999999999998e-05 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684651+0000", 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5160999999999997e-05 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684747+0000", 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6220999999999999e-05 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.685024+0000", 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027716299999999998 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.685367+0000", 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034277100000000002 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.686053+0000", 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00068657400000000002 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.686090+0000", 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6801000000000002e-05 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.686127+0000", 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6783999999999999e-05 2024-09-21T08:28:58.197 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:697 4.1b 4:db9d130d:gc::gc.24:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.750453+0000", 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.31399095199998, 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012349640000000001, 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.198 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 697 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750453+0000", 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750453+0000", 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750454+0000", 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3999999999999999e-06 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750462+0000", 2024-09-21T08:28:58.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4279999999999999e-06 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750473+0000", 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1362999999999999e-05 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750480+0000", 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8360000000000003e-06 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750531+0000", 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1008999999999999e-05 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750610+0000", 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9122000000000006e-05 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:28:58.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750784+0000", 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174329 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751214+0000", 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042964799999999998 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751607+0000", 2024-09-21T08:28:58.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039261899999999998 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751636+0000", 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9655999999999999e-05 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751663+0000", 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7254e-05 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751688+0000", 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4300000000000001e-05 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:780 4.1b 4:dc490aaa:::obj_delete_at_hint.0000000067:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.810765+0000", 2024-09-21T08:28:58.205 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 320.25367910300002, 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023807759999999998, 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 780 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810765+0000", 2024-09-21T08:28:58.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810765+0000", 2024-09-21T08:28:58.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810766+0000", 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.497e-06 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810773+0000", 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8689999999999998e-06 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810784+0000", 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0675e-05 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810789+0000", 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7490000000000004e-06 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810832+0000", 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2174e-05 2024-09-21T08:28:58.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810916+0000", 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4040000000000002e-05 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.811045+0000", 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012987299999999999 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.811286+0000", 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000240743 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.811623+0000", 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033676300000000001 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:28:58.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.813096+0000", 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014725059999999999 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.813122+0000", 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6744e-05 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.813145+0000", 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3142999999999998e-05 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.212 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDScrubReserve(3.f,QREQUEST,e:79,from:7,reservation_nonce:3)", 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 266.32211197599997, 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003249558, 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:28:58.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742334+0000", 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6809999999999998e-06 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742342+0000", 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6930000000000007e-06 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742355+0000", 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2556e-05 2024-09-21T08:28:58.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742366+0000", 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0994e-05 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742426+0000", 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0509e-05 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742436+0000", 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8060000000000006e-06 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745540+0000", 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031038630000000001 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745553+0000", 2024-09-21T08:28:58.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2669e-05 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745581+0000", 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8787000000000001e-05 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:28:58.218 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:28:58.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:58.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:58.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:58.341 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:58.357 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:28:58.362 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:58.363 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:58.363 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:58.464 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:58.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:58.487 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:58.487 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:28:58.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:58.601 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:58 smithi137 ceph-mon[41044]: 7.e scrub starts 2024-09-21T08:28:58.612 INFO:tasks.rados.rados.0.smithi137.stdout:1315: done (1 left) 2024-09-21T08:28:58.613 INFO:tasks.rados.rados.0.smithi137.stdout:1316: done (0 left) 2024-09-21T08:28:58.613 INFO:tasks.rados.rados.0.smithi137.stdout:1317: rollback oid 20 current snap is 133 2024-09-21T08:28:58.613 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 20 to 125 2024-09-21T08:28:58.613 INFO:tasks.rados.rados.0.smithi137.stdout:1318: rollback oid 29 current snap is 133 2024-09-21T08:28:58.613 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 29 to 122 2024-09-21T08:28:58.613 INFO:tasks.rados.rados.0.smithi137.stdout:1319: write (excl) oid 37 current snap is 133 2024-09-21T08:28:58.613 INFO:tasks.rados.rados.0.smithi137.stdout:1319: seq_num 479 ranges {450553=574510,1613463=437090} 2024-09-21T08:28:58.618 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:58.619 INFO:tasks.rados.rados.0.smithi137.stdout:1319: writing smithi13779221-37 from 450553 to 1025063 tid 1 2024-09-21T08:28:58.619 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:58.619 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:58.626 INFO:tasks.rados.rados.0.smithi137.stdout:1319: writing smithi13779221-37 from 1613463 to 2050553 tid 2 2024-09-21T08:28:58.626 INFO:tasks.rados.rados.0.smithi137.stdout:1320: write oid 8 current snap is 133 2024-09-21T08:28:58.626 INFO:tasks.rados.rados.0.smithi137.stdout:1320: seq_num 480 ranges {467360=649759,1671945=395415} 2024-09-21T08:28:58.632 INFO:tasks.rados.rados.0.smithi137.stdout:1320: writing smithi13779221-8 from 467360 to 1117119 tid 1 2024-09-21T08:28:58.638 INFO:tasks.rados.rados.0.smithi137.stdout:1320: writing smithi13779221-8 from 1671945 to 2067360 tid 2 2024-09-21T08:28:58.638 INFO:tasks.rados.rados.0.smithi137.stdout:1321: read oid 28 snap -1 2024-09-21T08:28:58.638 INFO:tasks.rados.rados.0.smithi137.stdout:1321: expect (ObjNum 464 snap 125 seq_num 464) 2024-09-21T08:28:58.638 INFO:tasks.rados.rados.0.smithi137.stdout:1322: read oid 50 snap 124 2024-09-21T08:28:58.638 INFO:tasks.rados.rados.0.smithi137.stdout:1322: expect (ObjNum 463 snap 124 seq_num 463) 2024-09-21T08:28:58.638 INFO:tasks.rados.rados.0.smithi137.stdout:1323: delete oid 34 current snap is 133 2024-09-21T08:28:58.638 INFO:tasks.rados.rados.0.smithi137.stdout:1317: finishing rollback tid 0 to smithi13779221-20 2024-09-21T08:28:58.638 INFO:tasks.rados.rados.0.smithi137.stdout:1317: finishing rollback tid 1 to smithi13779221-20 2024-09-21T08:28:58.638 INFO:tasks.rados.rados.0.smithi137.stdout:1317: finishing rollback tid 2 to smithi13779221-20 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 406 (ObjNum 450 snap 118 seq_num 450) dirty exists 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1318: finishing rollback tid 0 to smithi13779221-29 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1318: finishing rollback tid 1 to smithi13779221-29 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1318: finishing rollback tid 2 to smithi13779221-29 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 531 (ObjNum 435 snap 110 seq_num 435) dirty exists 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1319: finishing write tid 1 to smithi13779221-37 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1319: finishing write tid 2 to smithi13779221-37 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1319: finishing write tid 3 to smithi13779221-37 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1319: finishing write tid 4 to smithi13779221-37 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1319: oid 37 updating version 0 to 546 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1319: oid 37 updating version 546 to 547 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1319: oid 37 updating version 547 to 548 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 548 (ObjNum 479 snap 133 seq_num 479) dirty exists 2024-09-21T08:28:58.639 INFO:tasks.rados.rados.0.smithi137.stdout:1319: left oid 37 (ObjNum 479 snap 133 seq_num 479) 2024-09-21T08:28:58.640 INFO:tasks.rados.rados.0.smithi137.stdout:1320: finishing write tid 1 to smithi13779221-8 2024-09-21T08:28:58.640 INFO:tasks.rados.rados.0.smithi137.stdout:1317: done (6 left) 2024-09-21T08:28:58.640 INFO:tasks.rados.rados.0.smithi137.stdout:1318: done (5 left) 2024-09-21T08:28:58.640 INFO:tasks.rados.rados.0.smithi137.stdout:1319: done (4 left) 2024-09-21T08:28:58.640 INFO:tasks.rados.rados.0.smithi137.stdout:1323: done (3 left) 2024-09-21T08:28:58.640 INFO:tasks.rados.rados.0.smithi137.stdout:1324: read oid 36 snap -1 2024-09-21T08:28:58.641 INFO:tasks.rados.rados.0.smithi137.stdout:1324: expect deleted 2024-09-21T08:28:58.641 INFO:tasks.rados.rados.0.smithi137.stdout:1325: delete oid 9 current snap is 133 2024-09-21T08:28:58.649 INFO:tasks.rados.rados.0.smithi137.stdout:1322: expect (ObjNum 463 snap 124 seq_num 463) 2024-09-21T08:28:58.720 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:58.741 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:58.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:58.742 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:58.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:58 smithi165 ceph-mon[32893]: 7.e scrub starts 2024-09-21T08:28:58.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:58 smithi165 ceph-mon[32893]: 7.e scrub ok 2024-09-21T08:28:58.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:58 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:28:58.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:58 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2739247191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:58.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:58 smithi165 ceph-mon[32893]: osdmap e344: 8 total, 7 up, 4 in 2024-09-21T08:28:58.842 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:58.864 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:58.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:58.865 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:58.871 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:28:58.871 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:28:58.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:58 smithi137 ceph-mon[29634]: 7.e scrub starts 2024-09-21T08:28:58.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:58 smithi137 ceph-mon[29634]: 7.e scrub ok 2024-09-21T08:28:58.871 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:58 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:28:58.872 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:58 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2739247191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:58.872 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:58 smithi137 ceph-mon[29634]: osdmap e344: 8 total, 7 up, 4 in 2024-09-21T08:28:58.872 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:58 smithi137 ceph-mon[41044]: 7.e scrub ok 2024-09-21T08:28:58.872 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:58 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:28:58.872 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:58 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2739247191' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:28:58.872 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:58 smithi137 ceph-mon[41044]: osdmap e344: 8 total, 7 up, 4 in 2024-09-21T08:28:58.888 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:28:58.888 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:28:58.904 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:28:58.904 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:28:58.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T08:28:58.919 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:28:58.919 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:28:58.936 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:28:58.936 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:1322: done (4 left) 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:1324: done (3 left) 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:1325: done (2 left) 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:1326: delete oid 35 current snap is 133 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:1320: finishing write tid 2 to smithi13779221-8 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:1320: finishing write tid 3 to smithi13779221-8 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:1320: finishing write tid 4 to smithi13779221-8 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:1320: oid 8 updating version 0 to 318 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:1320: oid 8 version 318 is already newer than 317 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:1320: oid 8 updating version 318 to 319 2024-09-21T08:28:58.938 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 319 (ObjNum 480 snap 133 seq_num 480) dirty exists 2024-09-21T08:28:58.939 INFO:tasks.rados.rados.0.smithi137.stdout:1320: left oid 8 (ObjNum 480 snap 133 seq_num 480) 2024-09-21T08:28:58.939 INFO:tasks.rados.rados.0.smithi137.stdout:1321: expect (ObjNum 464 snap 125 seq_num 464) 2024-09-21T08:28:58.954 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:28:58.954 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:28:58.966 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:28:58.970 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:28:58.970 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:28:58.970 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:28:58.989 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:58.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:58.990 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:28:59.091 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:59.112 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:59.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:59.113 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:59.187 INFO:tasks.rados.rados.0.smithi137.stdout:1320: done (2 left) 2024-09-21T08:28:59.188 INFO:tasks.rados.rados.0.smithi137.stdout:1321: done (1 left) 2024-09-21T08:28:59.188 INFO:tasks.rados.rados.0.smithi137.stdout:1326: done (0 left) 2024-09-21T08:28:59.188 INFO:tasks.rados.rados.0.smithi137.stdout:1327: write oid 21 current snap is 133 2024-09-21T08:28:59.188 INFO:tasks.rados.rados.0.smithi137.stdout:1327: seq_num 484 ranges {534588=550755,1789520=345068} 2024-09-21T08:28:59.193 INFO:tasks.rados.rados.0.smithi137.stdout:1327: writing smithi13779221-21 from 534588 to 1085343 tid 1 2024-09-21T08:28:59.199 INFO:tasks.rados.rados.0.smithi137.stdout:1327: writing smithi13779221-21 from 1789520 to 2134588 tid 2 2024-09-21T08:28:59.199 INFO:tasks.rados.rados.0.smithi137.stdout:1328: snap_remove snap 122 2024-09-21T08:28:59.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:59.235 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:59.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:59.236 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:59.337 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:28:59.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:59.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:59.360 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:28:59.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:28:59.481 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:59.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:59.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:28:59.528 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T08:28:59.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:59.604 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:59.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:59.605 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:1328: done (1 left) 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:1329: rollback oid 22 current snap is 133 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 22 to 131 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:1330: read oid 18 snap -1 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:1330: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:1331: read oid 32 snap -1 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:1331: expect (ObjNum 458 snap 124 seq_num 458) 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:1332: copy_from oid 49 from oid 14 current snap is 133 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:1333: copy_from oid 50 from oid 17 current snap is 133 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:1334: write (excl) oid 27 current snap is 133 2024-09-21T08:28:59.616 INFO:tasks.rados.rados.0.smithi137.stdout:1334: seq_num 485 ranges {551395=626004,1848002=303393} 2024-09-21T08:28:59.623 INFO:tasks.rados.rados.0.smithi137.stdout:1334: writing smithi13779221-27 from 551395 to 1177399 tid 1 2024-09-21T08:28:59.629 INFO:tasks.rados.rados.0.smithi137.stdout:1334: writing smithi13779221-27 from 1848002 to 2151395 tid 2 2024-09-21T08:28:59.630 INFO:tasks.rados.rados.0.smithi137.stdout:1335: read oid 29 snap -1 2024-09-21T08:28:59.630 INFO:tasks.rados.rados.0.smithi137.stdout:1335: expect (ObjNum 435 snap 110 seq_num 435) 2024-09-21T08:28:59.630 INFO:tasks.rados.rados.0.smithi137.stdout:1336: snap_remove snap 117 2024-09-21T08:28:59.657 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:28:59.657 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:28:59.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:28:59.727 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:28:59.728 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:59.728 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:28:59.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:59 smithi165 ceph-mon[32893]: pgmap v422: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 265 MiB data, 832 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 15 op/s; 242/960 objects degraded (25.208%) 2024-09-21T08:28:59.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:59 smithi165 ceph-mon[32893]: osdmap e345: 8 total, 7 up, 4 in 2024-09-21T08:28:59.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:59 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/257223213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:28:59.750 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:59 smithi165 ceph-mon[32893]: 3.c scrub starts 2024-09-21T08:28:59.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:28:59 smithi165 ceph-mon[32893]: 3.c scrub ok 2024-09-21T08:28:59.829 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:28:59.851 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:59.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:59.852 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:28:59.947 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:28:59.953 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:28:59.975 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:28:59.976 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:28:59.976 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:59 smithi137 ceph-mon[29634]: pgmap v422: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 265 MiB data, 832 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 15 op/s; 242/960 objects degraded (25.208%) 2024-09-21T08:29:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:59 smithi137 ceph-mon[29634]: osdmap e345: 8 total, 7 up, 4 in 2024-09-21T08:29:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:59 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/257223213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:59 smithi137 ceph-mon[29634]: 3.c scrub starts 2024-09-21T08:29:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:28:59 smithi137 ceph-mon[29634]: 3.c scrub ok 2024-09-21T08:29:00.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:59 smithi137 ceph-mon[41044]: pgmap v422: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 265 MiB data, 832 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 2.7 MiB/s wr, 15 op/s; 242/960 objects degraded (25.208%) 2024-09-21T08:29:00.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:59 smithi137 ceph-mon[41044]: osdmap e345: 8 total, 7 up, 4 in 2024-09-21T08:29:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:59 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/257223213' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:59 smithi137 ceph-mon[41044]: 3.c scrub starts 2024-09-21T08:29:00.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:28:59 smithi137 ceph-mon[41044]: 3.c scrub ok 2024-09-21T08:29:00.077 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:00.099 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:00.099 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:00.100 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:00.200 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:00.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:00.223 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:00.223 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:00.324 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:00.335 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:29:00.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:00.346 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:00.346 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:00.437 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:00.437 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:00.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:00.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:00.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:00.469 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:00.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:00.589 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:00.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:00.594 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:00.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:00.627 INFO:tasks.rados.rados.0.smithi137.stdout:1336: done (8 left) 2024-09-21T08:29:00.627 INFO:tasks.rados.rados.0.smithi137.stdout:1337: copy_from oid 1 from oid 3 current snap is 133 2024-09-21T08:29:00.627 INFO:tasks.rados.rados.0.smithi137.stdout:1327: finishing write tid 1 to smithi13779221-21 2024-09-21T08:29:00.627 INFO:tasks.rados.rados.0.smithi137.stdout:1327: finishing write tid 2 to smithi13779221-21 2024-09-21T08:29:00.627 INFO:tasks.rados.rados.0.smithi137.stdout:1327: finishing write tid 3 to smithi13779221-21 2024-09-21T08:29:00.627 INFO:tasks.rados.rados.0.smithi137.stdout:1327: finishing write tid 4 to smithi13779221-21 2024-09-21T08:29:00.628 INFO:tasks.rados.rados.0.smithi137.stdout:1327: oid 21 updating version 0 to 119 2024-09-21T08:29:00.628 INFO:tasks.rados.rados.0.smithi137.stdout:1327: oid 21 version 119 is already newer than 118 2024-09-21T08:29:00.628 INFO:tasks.rados.rados.0.smithi137.stdout:1327: oid 21 version 119 is already newer than 117 2024-09-21T08:29:00.628 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 119 (ObjNum 484 snap 133 seq_num 484) dirty exists 2024-09-21T08:29:00.628 INFO:tasks.rados.rados.0.smithi137.stdout:1327: left oid 21 (ObjNum 484 snap 133 seq_num 484) 2024-09-21T08:29:00.628 INFO:tasks.rados.rados.0.smithi137.stdout:1331: expect (ObjNum 458 snap 124 seq_num 458) 2024-09-21T08:29:00.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:00.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:00.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:00.718 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:00.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:00 smithi165 ceph-mon[32893]: 4.d deep-scrub starts 2024-09-21T08:29:00.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:00 smithi165 ceph-mon[32893]: 4.d deep-scrub ok 2024-09-21T08:29:00.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:00 smithi165 ceph-mon[32893]: osdmap e346: 8 total, 7 up, 4 in 2024-09-21T08:29:00.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:00.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:00.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:00 smithi165 ceph-mon[32893]: 4.19 scrub starts 2024-09-21T08:29:00.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:00 smithi165 ceph-mon[32893]: 4.19 scrub ok 2024-09-21T08:29:00.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:7.f 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:7.d 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:7.9 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:5.14 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:3.b 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:5.11 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:4.f 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:4.a 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:4.15 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:5.1b 2024-09-21T08:29:00.820 INFO:teuthology.orchestra.run.smithi165.stdout:7.1f 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:4.8 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:6.15 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:4.6 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:4.4 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:3.16 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:4.1f 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:7.18 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:4.1e 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:7.11 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:4.1b 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:3.1d 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:4.11 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:3.f 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:4.0 2024-09-21T08:29:00.821 INFO:teuthology.orchestra.run.smithi165.stdout:3.e 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:3.1 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:7.21 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:4.c 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:4.b 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:3.1a 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:4.3 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:3.8 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:5.a 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:5.10 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:3.18 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:3.15 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:3.a 2024-09-21T08:29:00.822 INFO:teuthology.orchestra.run.smithi165.stdout:6.5 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:4.1 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:3.13 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:3.1e 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:3.11 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:5.16 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:3.7 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:5.12 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:6.1 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:6.1c 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:6.7 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:4.17 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:7.8 2024-09-21T08:29:00.823 INFO:teuthology.orchestra.run.smithi165.stdout:3.1c 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:5.17 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:5.13 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:2.0 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:7.23 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:7.7 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:3.19 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:3.1b 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:4.e 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:7.16 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:7.1e 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:2.2 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:6.c 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:3.6 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:1.0 2024-09-21T08:29:00.824 INFO:teuthology.orchestra.run.smithi165.stdout:2.1 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:3.1f 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:6.9 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:4.5 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:3.12 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:3.2 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:3.d 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:5.19 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:4.16 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:5.5 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:3.9 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:5.1 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:5.1a 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:6.8 2024-09-21T08:29:00.825 INFO:teuthology.orchestra.run.smithi165.stdout:5.1c 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:7.15 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:5.1d 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:6.12 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:3.5 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:5.1f 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:5.3 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:7.14 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:7.2 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:5.4 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:5.6 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:6.3 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:5.7 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:5.d 2024-09-21T08:29:00.826 INFO:teuthology.orchestra.run.smithi165.stdout:4.7 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:5.e 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:6.14 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:6.16 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:6.17 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:5.2 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:6.1a 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:6.1b 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:6.1e 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:5.9 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:6.2 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:5.18 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:5.1e 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:6.6 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:6.18 2024-09-21T08:29:00.827 INFO:teuthology.orchestra.run.smithi165.stdout:7.17 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:7.22 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:6.b 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:3.14 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:7.10 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:4.14 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:7.12 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:7.13 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:5.b 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:7.19 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:7.1 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:7.1a 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:7.1c 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:7.1d 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:4.9 2024-09-21T08:29:00.828 INFO:teuthology.orchestra.run.smithi165.stdout:7.20 2024-09-21T08:29:00.829 INFO:teuthology.orchestra.run.smithi165.stdout:5.c 2024-09-21T08:29:00.829 INFO:teuthology.orchestra.run.smithi165.stdout:7.3 2024-09-21T08:29:00.829 INFO:teuthology.orchestra.run.smithi165.stdout:5.15 2024-09-21T08:29:00.829 INFO:teuthology.orchestra.run.smithi165.stdout:7.5 2024-09-21T08:29:00.829 INFO:teuthology.orchestra.run.smithi165.stdout:7.6 2024-09-21T08:29:00.851 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:00.852 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:00.852 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:00.857 INFO:tasks.rados.rados.0.smithi137.stdout:1330: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:29:00.954 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:00.976 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:00.977 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:00.977 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:00.988 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[29634]: 4.d deep-scrub starts 2024-09-21T08:29:00.988 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[29634]: 4.d deep-scrub ok 2024-09-21T08:29:00.988 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[29634]: osdmap e346: 8 total, 7 up, 4 in 2024-09-21T08:29:00.988 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:00.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:00.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[29634]: 4.19 scrub starts 2024-09-21T08:29:00.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[29634]: 4.19 scrub ok 2024-09-21T08:29:00.989 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:29:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:29:00.989 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[41044]: 4.d deep-scrub starts 2024-09-21T08:29:00.989 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[41044]: 4.d deep-scrub ok 2024-09-21T08:29:00.989 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[41044]: osdmap e346: 8 total, 7 up, 4 in 2024-09-21T08:29:00.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:00.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:00.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[41044]: 4.19 scrub starts 2024-09-21T08:29:00.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:00 smithi137 ceph-mon[41044]: 4.19 scrub ok 2024-09-21T08:29:01.070 INFO:tasks.rados.rados.0.smithi137.stdout:1329: finishing rollback tid 0 to smithi13779221-22 2024-09-21T08:29:01.070 INFO:tasks.rados.rados.0.smithi137.stdout:1329: finishing rollback tid 1 to smithi13779221-22 2024-09-21T08:29:01.070 INFO:tasks.rados.rados.0.smithi137.stdout:1329: finishing rollback tid 2 to smithi13779221-22 2024-09-21T08:29:01.070 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 335 (ObjNum 469 snap 127 seq_num 469) dirty exists 2024-09-21T08:29:01.070 INFO:tasks.rados.rados.0.smithi137.stdout:1334: finishing write tid 1 to smithi13779221-27 2024-09-21T08:29:01.071 INFO:tasks.rados.rados.0.smithi137.stdout:1334: finishing write tid 2 to smithi13779221-27 2024-09-21T08:29:01.071 INFO:tasks.rados.rados.0.smithi137.stdout:1334: finishing write tid 3 to smithi13779221-27 2024-09-21T08:29:01.071 INFO:tasks.rados.rados.0.smithi137.stdout:1334: finishing write tid 4 to smithi13779221-27 2024-09-21T08:29:01.071 INFO:tasks.rados.rados.0.smithi137.stdout:1334: oid 27 updating version 0 to 296 2024-09-21T08:29:01.071 INFO:tasks.rados.rados.0.smithi137.stdout:1334: oid 27 updating version 296 to 297 2024-09-21T08:29:01.071 INFO:tasks.rados.rados.0.smithi137.stdout:1334: oid 27 updating version 297 to 298 2024-09-21T08:29:01.071 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 298 (ObjNum 485 snap 133 seq_num 485) dirty exists 2024-09-21T08:29:01.071 INFO:tasks.rados.rados.0.smithi137.stdout:1334: left oid 27 (ObjNum 485 snap 133 seq_num 485) 2024-09-21T08:29:01.071 INFO:tasks.rados.rados.0.smithi137.stdout:1333: finishing copy_from racing read to smithi13779221-50 2024-09-21T08:29:01.072 INFO:tasks.rados.rados.0.smithi137.stdout:1332: finishing copy_from racing read to smithi13779221-49 2024-09-21T08:29:01.072 INFO:tasks.rados.rados.0.smithi137.stdout:1327: done (8 left) 2024-09-21T08:29:01.072 INFO:tasks.rados.rados.0.smithi137.stdout:1329: done (7 left) 2024-09-21T08:29:01.072 INFO:tasks.rados.rados.0.smithi137.stdout:1330: done (6 left) 2024-09-21T08:29:01.072 INFO:tasks.rados.rados.0.smithi137.stdout:1331: done (5 left) 2024-09-21T08:29:01.072 INFO:tasks.rados.rados.0.smithi137.stdout:1334: done (4 left) 2024-09-21T08:29:01.072 INFO:tasks.rados.rados.0.smithi137.stdout:1338: read oid 13 snap -1 2024-09-21T08:29:01.072 INFO:tasks.rados.rados.0.smithi137.stdout:1338: expect (ObjNum 340 snap 80 seq_num 340) 2024-09-21T08:29:01.072 INFO:tasks.rados.rados.0.smithi137.stdout:1339: copy_from oid 30 from oid 42 current snap is 133 2024-09-21T08:29:01.072 INFO:tasks.rados.rados.0.smithi137.stdout:1335: expect (ObjNum 435 snap 110 seq_num 435) 2024-09-21T08:29:01.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:01.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:01.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:01.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:01.160 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T08:29:01.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:01.225 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:01.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:01.226 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:01.227 INFO:tasks.rados.rados.0.smithi137.stdout:1332: finishing copy_from to smithi13779221-49 2024-09-21T08:29:01.227 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 305 (ObjNum 456 snap 123 seq_num 456) dirty exists 2024-09-21T08:29:01.227 INFO:tasks.rados.rados.0.smithi137.stdout:1333: finishing copy_from to smithi13779221-50 2024-09-21T08:29:01.227 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 183 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:29:01.227 INFO:tasks.rados.rados.0.smithi137.stdout:1332: done (5 left) 2024-09-21T08:29:01.227 INFO:tasks.rados.rados.0.smithi137.stdout:1333: done (4 left) 2024-09-21T08:29:01.227 INFO:tasks.rados.rados.0.smithi137.stdout:1335: done (3 left) 2024-09-21T08:29:01.227 INFO:tasks.rados.rados.0.smithi137.stdout:1340: snap_create 2024-09-21T08:29:01.228 INFO:tasks.rados.rados.0.smithi137.stdout:1337: finishing copy_from racing read to smithi13779221-1 2024-09-21T08:29:01.228 INFO:tasks.rados.rados.0.smithi137.stdout:1337: finishing copy_from to smithi13779221-1 2024-09-21T08:29:01.228 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 186 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:29:01.228 INFO:tasks.rados.rados.0.smithi137.stdout:1338: expect (ObjNum 340 snap 80 seq_num 340) 2024-09-21T08:29:01.326 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:01.348 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:01.348 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:01.348 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:01.442 INFO:tasks.rados.rados.0.smithi137.stdout:1339: finishing copy_from racing read to smithi13779221-30 2024-09-21T08:29:01.442 INFO:tasks.rados.rados.0.smithi137.stdout:1339: finishing copy_from to smithi13779221-30 2024-09-21T08:29:01.442 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 291 (ObjNum 445 snap 115 seq_num 445) dirty exists 2024-09-21T08:29:01.450 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:01.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:01.471 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:01.472 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:01.550 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:29:01.553 INFO:tasks.rados.rados.0.smithi137.stdout:1337: done (3 left) 2024-09-21T08:29:01.554 INFO:tasks.rados.rados.0.smithi137.stdout:1338: done (2 left) 2024-09-21T08:29:01.554 INFO:tasks.rados.rados.0.smithi137.stdout:1339: done (1 left) 2024-09-21T08:29:01.554 INFO:tasks.rados.rados.0.smithi137.stdout:1340: done (0 left) 2024-09-21T08:29:01.554 INFO:tasks.rados.rados.0.smithi137.stdout:1341: delete oid 13 current snap is 134 2024-09-21T08:29:01.563 INFO:tasks.rados.rados.0.smithi137.stdout:1341: done (0 left) 2024-09-21T08:29:01.563 INFO:tasks.rados.rados.0.smithi137.stdout:1342: snap_remove snap 127 2024-09-21T08:29:01.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:01.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:01.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:01.596 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:01.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:29:01.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:01.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:01.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:01.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:01.820 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:01.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:01.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:01.843 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:01.859 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[29634]: pgmap v425: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 265 MiB data, 832 MiB used, 357 GiB / 358 GiB avail; 242/960 objects degraded (25.208%) 2024-09-21T08:29:01.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[29634]: osdmap e347: 8 total, 7 up, 4 in 2024-09-21T08:29:01.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/185062465' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:01.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:01.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/185062465' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:29:01.860 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[29634]: osdmap e348: 8 total, 7 up, 4 in 2024-09-21T08:29:01.860 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[41044]: pgmap v425: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 265 MiB data, 832 MiB used, 357 GiB / 358 GiB avail; 242/960 objects degraded (25.208%) 2024-09-21T08:29:01.860 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[41044]: osdmap e347: 8 total, 7 up, 4 in 2024-09-21T08:29:01.860 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/185062465' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:01.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:01.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/185062465' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:29:01.861 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:01 smithi137 ceph-mon[41044]: osdmap e348: 8 total, 7 up, 4 in 2024-09-21T08:29:01.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:01.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:01 smithi165 ceph-mon[32893]: pgmap v425: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 265 MiB data, 832 MiB used, 357 GiB / 358 GiB avail; 242/960 objects degraded (25.208%) 2024-09-21T08:29:01.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:01 smithi165 ceph-mon[32893]: osdmap e347: 8 total, 7 up, 4 in 2024-09-21T08:29:01.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:01 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/185062465' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:01.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:01 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:01.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:01 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/185062465' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:29:01.949 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:01 smithi165 ceph-mon[32893]: osdmap e348: 8 total, 7 up, 4 in 2024-09-21T08:29:01.949 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:29:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:29:01.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:01.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:01.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:02.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:02.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:02.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:02.090 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:02.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:02.214 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:02.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:02.215 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:02.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:02.334 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:02.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:02.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:02.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:02.433 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.7 --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 3.2 --file /var/log/ceph/exp.3.2.7 2024-09-21T08:29:02.439 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:02.461 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:02.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:02.462 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:02.473 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:02.555 INFO:tasks.rados.rados.0.smithi137.stdout:1342: done (0 left) 2024-09-21T08:29:02.556 INFO:tasks.rados.rados.0.smithi137.stdout:1343: delete oid 43 current snap is 134 2024-09-21T08:29:02.562 INFO:tasks.rados.rados.0.smithi137.stdout:1343: done (0 left) 2024-09-21T08:29:02.562 INFO:tasks.rados.rados.0.smithi137.stdout:1344: read oid 5 snap -1 2024-09-21T08:29:02.562 INFO:tasks.rados.rados.0.smithi137.stdout:1344: expect (ObjNum 419 snap 103 seq_num 419) 2024-09-21T08:29:02.563 INFO:tasks.rados.rados.0.smithi137.stdout:1345: snap_remove snap 125 2024-09-21T08:29:02.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:02.585 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:02.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:02.586 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:02.686 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:02.708 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:02.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:02.709 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:02.736 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T08:29:02.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:02.830 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:29:02.831 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:29:02.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:02.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:02.833 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:02.847 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:02.848 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:02.863 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:02.864 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:02.879 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:02.879 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:02.895 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:02.896 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:02.912 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:02.912 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:02.928 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:02.929 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:02.929 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:02.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:02.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:02.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:02.955 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:03.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:03.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:03.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:03.079 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:03.134 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:29:03.134 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:29:03.135 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:29:03.135 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:29:03.135 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:29:03.180 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:03.201 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:03.202 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:03.202 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:03.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:03.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:03.326 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:03.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:03.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:03.449 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:03.450 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:03.450 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:03.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:03.563 INFO:tasks.rados.rados.0.smithi137.stdout:1345: done (1 left) 2024-09-21T08:29:03.563 INFO:tasks.rados.rados.0.smithi137.stdout:1346: read oid 42 snap -1 2024-09-21T08:29:03.563 INFO:tasks.rados.rados.0.smithi137.stdout:1346: expect (ObjNum 445 snap 115 seq_num 445) 2024-09-21T08:29:03.563 INFO:tasks.rados.rados.0.smithi137.stdout:1347: write (excl) oid 6 current snap is 134 2024-09-21T08:29:03.563 INFO:tasks.rados.rados.0.smithi137.stdout:1347: seq_num 488 ranges {601816=568104,1623448=451823} 2024-09-21T08:29:03.570 INFO:tasks.rados.rados.0.smithi137.stdout:1347: writing smithi13779221-6 from 601816 to 1169920 tid 1 2024-09-21T08:29:03.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:03.573 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:03.573 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:03.576 INFO:tasks.rados.rados.0.smithi137.stdout:1347: writing smithi13779221-6 from 1623448 to 2075271 tid 2 2024-09-21T08:29:03.576 INFO:tasks.rados.rados.0.smithi137.stdout:1348: snap_create 2024-09-21T08:29:03.576 INFO:tasks.rados.rados.0.smithi137.stdout:1344: expect (ObjNum 419 snap 103 seq_num 419) 2024-09-21T08:29:03.646 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:03.647 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:03.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:03.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:03.696 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:03.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:03.713 INFO:tasks.rados.rados.0.smithi137.stdout:1346: expect (ObjNum 445 snap 115 seq_num 445) 2024-09-21T08:29:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:03 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:03 smithi165 ceph-mon[32893]: pgmap v428: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 288 MiB data, 857 MiB used, 357 GiB / 358 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 30 op/s; 255/1011 objects degraded (25.223%) 2024-09-21T08:29:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:03 smithi165 ceph-mon[32893]: osdmap e349: 8 total, 7 up, 4 in 2024-09-21T08:29:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:03 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 255/1011 objects degraded (25.223%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:03 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1968864123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:03.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:03.819 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:03.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:03.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:03.897 INFO:tasks.rados.rados.0.smithi137.stdout:1347: finishing write tid 1 to smithi13779221-6 2024-09-21T08:29:03.898 INFO:tasks.rados.rados.0.smithi137.stdout:1347: finishing write tid 2 to smithi13779221-6 2024-09-21T08:29:03.898 INFO:tasks.rados.rados.0.smithi137.stdout:1347: finishing write tid 3 to smithi13779221-6 2024-09-21T08:29:03.898 INFO:tasks.rados.rados.0.smithi137.stdout:1347: finishing write tid 4 to smithi13779221-6 2024-09-21T08:29:03.898 INFO:tasks.rados.rados.0.smithi137.stdout:1347: oid 6 updating version 0 to 307 2024-09-21T08:29:03.898 INFO:tasks.rados.rados.0.smithi137.stdout:1347: oid 6 version 307 is already newer than 306 2024-09-21T08:29:03.898 INFO:tasks.rados.rados.0.smithi137.stdout:1347: oid 6 updating version 307 to 308 2024-09-21T08:29:03.898 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 308 (ObjNum 488 snap 134 seq_num 488) dirty exists 2024-09-21T08:29:03.898 INFO:tasks.rados.rados.0.smithi137.stdout:1347: left oid 6 (ObjNum 488 snap 134 seq_num 488) 2024-09-21T08:29:03.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:03.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:03.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:03.944 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:03 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:03 smithi137 ceph-mon[29634]: pgmap v428: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 288 MiB data, 857 MiB used, 357 GiB / 358 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 30 op/s; 255/1011 objects degraded (25.223%) 2024-09-21T08:29:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:03 smithi137 ceph-mon[29634]: osdmap e349: 8 total, 7 up, 4 in 2024-09-21T08:29:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:03 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 255/1011 objects degraded (25.223%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:04.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:03 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1968864123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:04.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:03 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:04.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:03 smithi137 ceph-mon[41044]: pgmap v428: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 288 MiB data, 857 MiB used, 357 GiB / 358 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 30 op/s; 255/1011 objects degraded (25.223%) 2024-09-21T08:29:04.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:03 smithi137 ceph-mon[41044]: osdmap e349: 8 total, 7 up, 4 in 2024-09-21T08:29:04.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:03 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 255/1011 objects degraded (25.223%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:04.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:03 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1968864123' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:04.044 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:04.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:04.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:04.067 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:04.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:04.171 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T08:29:04.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:04.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:04.190 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:04.238 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:29:04.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:04.312 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:04.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:04.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:04.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:04.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:04.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:04.436 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:04.487 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:04.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:04.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:04.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:04.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:04.571 INFO:tasks.rados.rados.0.smithi137.stdout:1344: done (3 left) 2024-09-21T08:29:04.571 INFO:tasks.rados.rados.0.smithi137.stdout:1346: done (2 left) 2024-09-21T08:29:04.571 INFO:tasks.rados.rados.0.smithi137.stdout:1347: done (1 left) 2024-09-21T08:29:04.571 INFO:tasks.rados.rados.0.smithi137.stdout:1348: done (0 left) 2024-09-21T08:29:04.571 INFO:tasks.rados.rados.0.smithi137.stdout:1349: read oid 9 snap -1 2024-09-21T08:29:04.571 INFO:tasks.rados.rados.0.smithi137.stdout:1349: expect deleted 2024-09-21T08:29:04.571 INFO:tasks.rados.rados.0.smithi137.stdout:1350: write oid 6 current snap is 135 2024-09-21T08:29:04.572 INFO:tasks.rados.rados.0.smithi137.stdout:1350: seq_num 489 ranges {618623=643353,1681930=536693} 2024-09-21T08:29:04.577 INFO:tasks.rados.rados.0.smithi137.stdout:1350: writing smithi13779221-6 from 618623 to 1261976 tid 1 2024-09-21T08:29:04.584 INFO:tasks.rados.rados.0.smithi137.stdout:1350: writing smithi13779221-6 from 1681930 to 2218623 tid 2 2024-09-21T08:29:04.584 INFO:tasks.rados.rados.0.smithi137.stdout:1351: rollback oid 40 current snap is 135 2024-09-21T08:29:04.584 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 40 to 103 2024-09-21T08:29:04.584 INFO:tasks.rados.rados.0.smithi137.stdout:1349: done (2 left) 2024-09-21T08:29:04.584 INFO:tasks.rados.rados.0.smithi137.stdout:1352: write (excl) oid 38 current snap is 135 2024-09-21T08:29:04.584 INFO:tasks.rados.rados.0.smithi137.stdout:1352: seq_num 490 ranges {635430=718602,2140412=95018} 2024-09-21T08:29:04.592 INFO:tasks.rados.rados.0.smithi137.stdout:1352: writing smithi13779221-38 from 635430 to 1354032 tid 1 2024-09-21T08:29:04.597 INFO:tasks.rados.rados.0.smithi137.stdout:1352: writing smithi13779221-38 from 2140412 to 2235430 tid 2 2024-09-21T08:29:04.597 INFO:tasks.rados.rados.0.smithi137.stdout:1353: snap_create 2024-09-21T08:29:04.597 INFO:tasks.rados.rados.0.smithi137.stdout:1351: finishing rollback tid 0 to smithi13779221-40 2024-09-21T08:29:04.597 INFO:tasks.rados.rados.0.smithi137.stdout:1351: finishing rollback tid 1 to smithi13779221-40 2024-09-21T08:29:04.597 INFO:tasks.rados.rados.0.smithi137.stdout:1351: finishing rollback tid 2 to smithi13779221-40 2024-09-21T08:29:04.597 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 321 (ObjNum 287 snap 66 seq_num 287) dirty exists 2024-09-21T08:29:04.597 INFO:tasks.rados.rados.0.smithi137.stdout:1350: finishing write tid 1 to smithi13779221-6 2024-09-21T08:29:04.598 INFO:tasks.rados.rados.0.smithi137.stdout:1350: finishing write tid 2 to smithi13779221-6 2024-09-21T08:29:04.598 INFO:tasks.rados.rados.0.smithi137.stdout:1350: finishing write tid 3 to smithi13779221-6 2024-09-21T08:29:04.598 INFO:tasks.rados.rados.0.smithi137.stdout:1350: finishing write tid 4 to smithi13779221-6 2024-09-21T08:29:04.598 INFO:tasks.rados.rados.0.smithi137.stdout:1350: oid 6 updating version 0 to 309 2024-09-21T08:29:04.598 INFO:tasks.rados.rados.0.smithi137.stdout:1350: oid 6 updating version 309 to 310 2024-09-21T08:29:04.598 INFO:tasks.rados.rados.0.smithi137.stdout:1350: oid 6 updating version 310 to 311 2024-09-21T08:29:04.598 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 311 (ObjNum 489 snap 135 seq_num 489) dirty exists 2024-09-21T08:29:04.598 INFO:tasks.rados.rados.0.smithi137.stdout:1350: left oid 6 (ObjNum 489 snap 135 seq_num 489) 2024-09-21T08:29:04.598 INFO:tasks.rados.rados.0.smithi137.stdout:1352: finishing write tid 1 to smithi13779221-38 2024-09-21T08:29:04.599 INFO:tasks.rados.rados.0.smithi137.stdout:1352: finishing write tid 2 to smithi13779221-38 2024-09-21T08:29:04.599 INFO:tasks.rados.rados.0.smithi137.stdout:1352: finishing write tid 3 to smithi13779221-38 2024-09-21T08:29:04.599 INFO:tasks.rados.rados.0.smithi137.stdout:1352: finishing write tid 4 to smithi13779221-38 2024-09-21T08:29:04.599 INFO:tasks.rados.rados.0.smithi137.stdout:1352: oid 38 updating version 0 to 562 2024-09-21T08:29:04.599 INFO:tasks.rados.rados.0.smithi137.stdout:1352: oid 38 version 562 is already newer than 561 2024-09-21T08:29:04.599 INFO:tasks.rados.rados.0.smithi137.stdout:1352: oid 38 version 562 is already newer than 560 2024-09-21T08:29:04.599 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 562 (ObjNum 490 snap 135 seq_num 490) dirty exists 2024-09-21T08:29:04.599 INFO:tasks.rados.rados.0.smithi137.stdout:1352: left oid 38 (ObjNum 490 snap 135 seq_num 490) 2024-09-21T08:29:04.653 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T08:29:04.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:04.682 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:04.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:04.682 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:04.783 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:04.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:04.806 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:04.807 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:04.906 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:04 smithi137 ceph-mon[29634]: osdmap e350: 8 total, 7 up, 4 in 2024-09-21T08:29:04.906 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:04 smithi137 ceph-mon[41044]: osdmap e350: 8 total, 7 up, 4 in 2024-09-21T08:29:04.907 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:04.908 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:04 smithi165 ceph-mon[32893]: osdmap e350: 8 total, 7 up, 4 in 2024-09-21T08:29:04.929 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:04.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:04.930 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:05.030 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:05.051 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:05.052 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:05.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:05.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:05.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:05.175 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:05.175 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:05.213 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:05.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:05.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:05.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:05.299 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:05.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:05.483 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:05.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:05.575 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:29:05.584 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:05.584 INFO:tasks.rados.rados.0.smithi137.stdout:1350: done (3 left) 2024-09-21T08:29:05.584 INFO:tasks.rados.rados.0.smithi137.stdout:1351: done (2 left) 2024-09-21T08:29:05.585 INFO:tasks.rados.rados.0.smithi137.stdout:1352: done (1 left) 2024-09-21T08:29:05.585 INFO:tasks.rados.rados.0.smithi137.stdout:1353: done (0 left) 2024-09-21T08:29:05.585 INFO:tasks.rados.rados.0.smithi137.stdout:1354: delete oid 34 current snap is 136 2024-09-21T08:29:05.589 INFO:tasks.rados.rados.0.smithi137.stdout:1354: done (0 left) 2024-09-21T08:29:05.589 INFO:tasks.rados.rados.0.smithi137.stdout:1355: rollback oid 43 current snap is 136 2024-09-21T08:29:05.589 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 43 to 118 2024-09-21T08:29:05.589 INFO:tasks.rados.rados.0.smithi137.stdout:1356: delete oid 1 current snap is 136 2024-09-21T08:29:05.590 INFO:tasks.rados.rados.0.smithi137.stdout:1355: finishing rollback tid 1 to smithi13779221-43 2024-09-21T08:29:05.595 INFO:tasks.rados.rados.0.smithi137.stdout:1355: finishing rollback tid 2 to smithi13779221-43 2024-09-21T08:29:05.595 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 359 (ObjNum 417 snap 103 seq_num 417) dirty exists 2024-09-21T08:29:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1355: done (1 left) 2024-09-21T08:29:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1356: done (0 left) 2024-09-21T08:29:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1357: write (excl) oid 30 current snap is 136 2024-09-21T08:29:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1357: seq_num 493 ranges {685851=544349,1799505=486346} 2024-09-21T08:29:05.603 INFO:tasks.rados.rados.0.smithi137.stdout:1357: writing smithi13779221-30 from 685851 to 1230200 tid 1 2024-09-21T08:29:05.609 INFO:tasks.rados.rados.0.smithi137.stdout:1357: writing smithi13779221-30 from 1799505 to 2285851 tid 2 2024-09-21T08:29:05.609 INFO:tasks.rados.rados.0.smithi137.stdout:1358: copy_from oid 3 from oid 13 current snap is 136 2024-09-21T08:29:05.609 INFO:tasks.rados.rados.0.smithi137.stdout:1357: finishing write tid 1 to smithi13779221-30 2024-09-21T08:29:05.609 INFO:tasks.rados.rados.0.smithi137.stdout:1359: write oid 49 current snap is 136 2024-09-21T08:29:05.609 INFO:tasks.rados.rados.0.smithi137.stdout:1359: seq_num 494 ranges {702658=619598,1857987=444671} 2024-09-21T08:29:05.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:05.610 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:05.610 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:05.616 INFO:tasks.rados.rados.0.smithi137.stdout:1359: writing smithi13779221-49 from 702658 to 1322256 tid 1 2024-09-21T08:29:05.623 INFO:tasks.rados.rados.0.smithi137.stdout:1359: writing smithi13779221-49 from 1857987 to 2302658 tid 2 2024-09-21T08:29:05.623 INFO:tasks.rados.rados.0.smithi137.stdout:1360: write oid 2 current snap is 136 2024-09-21T08:29:05.623 INFO:tasks.rados.rados.0.smithi137.stdout:1358: finishing copy_from to smithi13779221-3 2024-09-21T08:29:05.623 INFO:tasks.rados.rados.0.smithi137.stdout:1358: got expected ENOENT (src dne) 2024-09-21T08:29:05.623 INFO:tasks.rados.rados.0.smithi137.stdout:1360: seq_num 495 ranges {719465=411200,1632822=686643} 2024-09-21T08:29:05.629 INFO:tasks.rados.rados.0.smithi137.stdout:1360: writing smithi13779221-2 from 719465 to 1130665 tid 1 2024-09-21T08:29:05.635 INFO:tasks.rados.rados.0.smithi137.stdout:1360: writing smithi13779221-2 from 1632822 to 2319465 tid 2 2024-09-21T08:29:05.635 INFO:tasks.rados.rados.0.smithi137.stdout:1361: snap_create 2024-09-21T08:29:05.635 INFO:tasks.rados.rados.0.smithi137.stdout:1358: finishing copy_from racing read to smithi13779221-3 2024-09-21T08:29:05.635 INFO:tasks.rados.rados.0.smithi137.stdout:1357: finishing write tid 2 to smithi13779221-30 2024-09-21T08:29:05.635 INFO:tasks.rados.rados.0.smithi137.stdout:1357: finishing write tid 3 to smithi13779221-30 2024-09-21T08:29:05.635 INFO:tasks.rados.rados.0.smithi137.stdout:1357: finishing write tid 4 to smithi13779221-30 2024-09-21T08:29:05.635 INFO:tasks.rados.rados.0.smithi137.stdout:1357: oid 30 updating version 0 to 564 2024-09-21T08:29:05.635 INFO:tasks.rados.rados.0.smithi137.stdout:1357: oid 30 updating version 564 to 566 2024-09-21T08:29:05.635 INFO:tasks.rados.rados.0.smithi137.stdout:1357: oid 30 version 566 is already newer than 565 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 566 (ObjNum 493 snap 136 seq_num 493) dirty exists 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:1357: left oid 30 (ObjNum 493 snap 136 seq_num 493) 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:1359: finishing write tid 1 to smithi13779221-49 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:1359: finishing write tid 2 to smithi13779221-49 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:1359: finishing write tid 3 to smithi13779221-49 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:1359: finishing write tid 4 to smithi13779221-49 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:1359: oid 49 updating version 0 to 414 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:1359: oid 49 version 414 is already newer than 412 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:1359: oid 49 version 414 is already newer than 413 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 414 (ObjNum 494 snap 136 seq_num 494) dirty exists 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:1359: left oid 49 (ObjNum 494 snap 136 seq_num 494) 2024-09-21T08:29:05.636 INFO:tasks.rados.rados.0.smithi137.stdout:1360: finishing write tid 1 to smithi13779221-2 2024-09-21T08:29:05.645 INFO:tasks.rados.rados.0.smithi137.stdout:1360: finishing write tid 2 to smithi13779221-2 2024-09-21T08:29:05.645 INFO:tasks.rados.rados.0.smithi137.stdout:1360: finishing write tid 3 to smithi13779221-2 2024-09-21T08:29:05.645 INFO:tasks.rados.rados.0.smithi137.stdout:1360: finishing write tid 4 to smithi13779221-2 2024-09-21T08:29:05.645 INFO:tasks.rados.rados.0.smithi137.stdout:1360: oid 2 updating version 0 to 154 2024-09-21T08:29:05.645 INFO:tasks.rados.rados.0.smithi137.stdout:1360: oid 2 updating version 154 to 156 2024-09-21T08:29:05.645 INFO:tasks.rados.rados.0.smithi137.stdout:1360: oid 2 version 156 is already newer than 155 2024-09-21T08:29:05.645 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 156 (ObjNum 495 snap 136 seq_num 495) dirty exists 2024-09-21T08:29:05.645 INFO:tasks.rados.rados.0.smithi137.stdout:1360: left oid 2 (ObjNum 495 snap 136 seq_num 495) 2024-09-21T08:29:05.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:29:05.683 INFO:teuthology.orchestra.run.smithi165.stderr:Exporting 3.2 info 3.2( empty local-lis/les=290/291 n=0 ec=65/65 lis/c=290/290 les/c/f=291/291/0 sis=290) 2024-09-21T08:29:05.683 INFO:teuthology.orchestra.run.smithi165.stderr:Export successful 2024-09-21T08:29:05.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:05 smithi165 ceph-mon[32893]: pgmap v431: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 288 MiB data, 857 MiB used, 357 GiB / 358 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 30 op/s; 255/1011 objects degraded (25.223%) 2024-09-21T08:29:05.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:05 smithi165 ceph-mon[32893]: osdmap e351: 8 total, 7 up, 4 in 2024-09-21T08:29:05.708 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:05 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/340454797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:05.710 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:29:05.710 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:29:05.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1083 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94837804037760 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 267.54629298999998, 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.395092834, 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:05.711 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1083 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159586+0000", 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.48e-06 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159599+0000", 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3699e-05 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159630+0000", 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0629000000000001e-05 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159647+0000", 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7030000000000001e-05 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159767+0000", 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119571 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159781+0000", 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3844e-05 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.549248+0000", 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.38946733 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.549559+0000", 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000311057 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552031+0000", 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024720720000000001 2024-09-21T08:29:05.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552119+0000", 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8261000000000001e-05 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.554673+0000", 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025538610000000002 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.716 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1159 7.b 7:df2b42d9:::smithi13779221-36:head [write 642067~696188,stat] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 194.81575461, 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62427366799999995, 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1159 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890120+0000", 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6950000000000001e-06 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890849+0000", 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072891299999999998 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890861+0000", 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1899000000000001e-05 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890868+0000", 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2110000000000001e-06 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890947+0000", 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8598999999999997e-05 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890954+0000", 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1180000000000003e-06 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508089+0000", 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61713446299999997 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508263+0000", 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174671 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508366+0000", 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102786 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511111+0000", 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027452710000000001 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.513304+0000", 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021930560000000001 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514317+0000", 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010129959999999999 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514355+0000", 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7404000000000003e-05 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514392+0000", 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7586000000000002e-05 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1160 7.b 7:df2b42d9:::smithi13779221-36:head [write 2025476~459397] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 194.80930672100001, 2024-09-21T08:29:05.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61820804500000004, 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1160 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:29:05.724 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896568+0000", 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2610000000000001e-06 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.725 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896958+0000", 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039041199999999997 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896984+0000", 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6115e-05 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896989+0000", 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0379999999999999e-06 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.897069+0000", 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9196000000000005e-05 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.897075+0000", 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.28e-06 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509576+0000", 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61250132700000004 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509636+0000", 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9753999999999997e-05 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509725+0000", 2024-09-21T08:29:05.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9524000000000005e-05 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511333+0000", 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001607619 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.513767+0000", 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024336819999999999 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514703+0000", 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093633599999999996 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514736+0000", 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2438999999999999e-05 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514775+0000", 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9062e-05 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.730 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.731 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1161 7.b 7:df2b42d9:::smithi13779221-36:head [write 2971091~70976] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:29:05.731 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:29:05.731 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 194.80642779600001, 2024-09-21T08:29:05.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61638660599999995, 2024-09-21T08:29:05.731 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.731 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.731 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.731 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1161 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:29:05.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899446+0000", 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0800000000000004e-07 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899537+0000", 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0414999999999994e-05 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899544+0000", 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8580000000000002e-06 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899547+0000", 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8970000000000001e-06 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899597+0000", 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9058999999999999e-05 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899605+0000", 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3990000000000004e-06 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510251+0000", 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61064600599999996 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510324+0000", 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2961999999999997e-05 2024-09-21T08:29:05.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510438+0000", 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113793 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511337+0000", 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089906399999999998 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514615+0000", 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00327853 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515770+0000", 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001154715 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515802+0000", 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1956999999999999e-05 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515832+0000", 2024-09-21T08:29:05.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0142999999999999e-05 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1162 7.b 7:df2b42d9:::smithi13779221-36:head [setxattr _header (57) in=7b,truncate 3042067] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 194.80631872800001, 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61641711799999999, 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1162 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9100000000000004e-07 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899557+0000", 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3989999999999998e-06 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899561+0000", 2024-09-21T08:29:05.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9349999999999996e-06 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899564+0000", 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7700000000000002e-06 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899633+0000", 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9268000000000002e-05 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899637+0000", 2024-09-21T08:29:05.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9650000000000002e-06 2024-09-21T08:29:05.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510758+0000", 2024-09-21T08:29:05.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61112027999999996 2024-09-21T08:29:05.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510819+0000", 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0866999999999998e-05 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510929+0000", 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110914 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511340+0000", 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041042100000000001 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514641+0000", 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003301511 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515865+0000", 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012240319999999999 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515898+0000", 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3036e-05 2024-09-21T08:29:05.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515972+0000", 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3229e-05 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1163 7.b 7:df2b42d9:::smithi13779221-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 194.80628731499999, 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61652938899999998, 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.745 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1163 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8599999999999998e-07 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899589+0000", 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1299999999999999e-06 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899592+0000", 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9190000000000001e-06 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899595+0000", 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9639999999999999e-06 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899654+0000", 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8958e-05 2024-09-21T08:29:05.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899658+0000", 2024-09-21T08:29:05.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2669999999999998e-06 2024-09-21T08:29:05.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511140+0000", 2024-09-21T08:29:05.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61148218399999998 2024-09-21T08:29:05.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:29:05.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511235+0000", 2024-09-21T08:29:05.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4927999999999996e-05 2024-09-21T08:29:05.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516004+0000", 2024-09-21T08:29:05.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047691640000000002 2024-09-21T08:29:05.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516060+0000", 2024-09-21T08:29:05.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5726999999999997e-05 2024-09-21T08:29:05.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516115+0000", 2024-09-21T08:29:05.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5661999999999997e-05 2024-09-21T08:29:05.752 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.752 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1214 7.0 7:00f4c0d4:::smithi13779221-18:head [copy-from ver 91] snapc 27=[25] ondisk+write+known_if_redirected+supports_pool_eio e161)", 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:54.540239+0000", 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 191.165634686, 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016093084000000001, 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1214 2024-09-21T08:29:05.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.754 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540239+0000", 2024-09-21T08:29:05.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.754 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540239+0000", 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540241+0000", 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0499999999999999e-06 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540249+0000", 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8049999999999996e-06 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540263+0000", 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3329999999999999e-05 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540268+0000", 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6470000000000003e-06 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540325+0000", 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6918e-05 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.540389+0000", 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3990000000000002e-05 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.547214+0000", 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068248240000000002 2024-09-21T08:29:05.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.551118+0000", 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039037669999999998 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556219+0000", 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051015089999999997 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556258+0000", 2024-09-21T08:29:05.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8606999999999998e-05 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556298+0000", 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0096999999999999e-05 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:54.556332+0000", 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3540000000000001e-05 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2116 7.b 7:df2b42d9:::smithi13779221-36:head [copy-from ver 318] snapc 4a=[4a,49,48,47,45,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 124.182648289, 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030972873000000001, 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2116 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523228+0000", 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6469999999999998e-06 2024-09-21T08:29:05.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523237+0000", 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4590000000000004e-06 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523251+0000", 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3856000000000001e-05 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523260+0000", 2024-09-21T08:29:05.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523351+0000", 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0787999999999999e-05 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523359+0000", 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5860000000000005e-06 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.526688+0000", 2024-09-21T08:29:05.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033287339999999999 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.526800+0000", 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111566 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.531948+0000", 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051486199999999996 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.538302+0000", 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063539729999999997 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.553780+0000", 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015477451 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554118+0000", 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000338549 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554160+0000", 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2218000000000001e-05 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554198+0000", 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7475999999999998e-05 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2323 7.b 7:df2b42d9:::smithi13779221-36:head [rollback 48] snapc 4f=[4f,4e,4d,48,3d] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T08:29:05.767 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:13.572505+0000", 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.13336801, 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014608846, 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2323 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572505+0000", 2024-09-21T08:29:05.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572505+0000", 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572506+0000", 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.99e-07 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572528+0000", 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2022e-05 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572534+0000", 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9669999999999998e-06 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572540+0000", 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7899999999999996e-06 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572599+0000", 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9120999999999999e-05 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572604+0000", 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7359999999999998e-06 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.575509+0000", 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029042540000000002 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.575560+0000", 2024-09-21T08:29:05.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0889e-05 2024-09-21T08:29:05.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:29:05.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.575675+0000", 2024-09-21T08:29:05.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011489600000000001 2024-09-21T08:29:05.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.576417+0000", 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00074203300000000002 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.586511+0000", 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010094792999999999 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587046+0000", 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053456399999999996 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587082+0000", 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6531e-05 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587114+0000", 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1650999999999999e-05 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2324 7.b 7:df2b42d9:::smithi13779221-36:head [append 0~0] snapc 4f=[4f,4e,4d,48,3d] ondisk+write+known_if_redirected+supports_pool_eio e240)", 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:13.572548+0000", 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 112.133325363, 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014668708000000001, 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2324 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572548+0000", 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572548+0000", 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572549+0000", 2024-09-21T08:29:05.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9500000000000002e-07 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572551+0000", 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.779e-06 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572556+0000", 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1909999999999999e-06 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572562+0000", 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3170000000000001e-06 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572620+0000", 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8405000000000001e-05 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.572623+0000", 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2619999999999999e-06 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.575994+0000", 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033708710000000001 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.576058+0000", 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3571000000000005e-05 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.576214+0000", 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000155771 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.577231+0000", 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010176829999999999 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.586539+0000", 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093072429999999998 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587152+0000", 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00061338899999999999 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587187+0000", 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4817999999999998e-05 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:13.587217+0000", 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9813000000000001e-05 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2577 7.1c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 174] snapc 59=[59,56] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-21T08:29:05.782 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 88.143122091999999, 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0145323420000001, 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2577 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.783 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:29:05.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:29:05.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562753+0000", 2024-09-21T08:29:05.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.745e-06 2024-09-21T08:29:05.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562759+0000", 2024-09-21T08:29:05.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1330000000000004e-06 2024-09-21T08:29:05.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562770+0000", 2024-09-21T08:29:05.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0617e-05 2024-09-21T08:29:05.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562775+0000", 2024-09-21T08:29:05.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4500000000000003e-06 2024-09-21T08:29:05.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562831+0000", 2024-09-21T08:29:05.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6233999999999997e-05 2024-09-21T08:29:05.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562838+0000", 2024-09-21T08:29:05.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7379999999999997e-06 2024-09-21T08:29:05.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552540+0000", 2024-09-21T08:29:05.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98970149799999996 2024-09-21T08:29:05.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552665+0000", 2024-09-21T08:29:05.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012560800000000001 2024-09-21T08:29:05.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:29:05.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.562825+0000", 2024-09-21T08:29:05.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010159916 2024-09-21T08:29:05.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.569448+0000", 2024-09-21T08:29:05.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066225670000000002 2024-09-21T08:29:05.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577182+0000", 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077346680000000001 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577203+0000", 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0619999999999999e-05 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577240+0000", 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6613999999999999e-05 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577284+0000", 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3934000000000003e-05 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2578 7.1c 7:3cae00e7:::smithi13779221-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 88.143076214999994, 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0041372150000001, 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.793 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2578 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562798+0000", 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5000000000000003e-07 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562800+0000", 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2460000000000002e-06 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562806+0000", 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6620000000000002e-06 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562809+0000", 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8500000000000004e-06 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562869+0000", 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9892999999999997e-05 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562879+0000", 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0054999999999999e-05 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552788+0000", 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.989908287 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552833+0000", 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5427999999999999e-05 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566793+0000", 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013959875 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566858+0000", 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4870999999999996e-05 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566934+0000", 2024-09-21T08:29:05.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6497999999999998e-05 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3147 7.c 7:361ee62b:::smithi13779221-40:head [write 712729~784777] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:32.559659+0000", 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 33.146214374000003, 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016831009000000001, 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.799 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3147 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.559659+0000", 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.559659+0000", 2024-09-21T08:29:05.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.559661+0000", 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1440000000000001e-06 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.560606+0000", 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00094477600000000004 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.560615+0000", 2024-09-21T08:29:05.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4630000000000007e-06 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.560621+0000", 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7749999999999998e-06 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.560671+0000", 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9904999999999997e-05 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.560795+0000", 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012356499999999999 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.560970+0000", 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00017508500000000001 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.562904+0000", 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019340959999999999 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.569517+0000", 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066131419999999998 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.576333+0000", 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068162850000000001 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.576424+0000", 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1365000000000004e-05 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.576490+0000", 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5407999999999994e-05 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:3157 7.12 e319/318 7:4ccee2c4:::smithi13779221-50:head v 319'333, mlcod=0'0)", 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:32.587281+0000", 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 33.11859218, 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014673215, 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.587281+0000", 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.587281+0000", 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.587282+0000", 2024-09-21T08:29:05.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8199999999999999e-07 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.594617+0000", 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0073354839999999998 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.594628+0000", 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0747999999999999e-05 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.594634+0000", 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0900000000000001e-06 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.594693+0000", 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8495999999999997e-05 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.594719+0000", 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5834e-05 2024-09-21T08:29:05.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.601908+0000", 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071895209999999999 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:32.601954+0000", 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6260000000000001e-05 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.809 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3197 7.0 7:00f4c0d4:::smithi13779221-18:head [delete] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.553696+0000", 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 29.152177672000001, 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022024709999999999, 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3197 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553696+0000", 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553696+0000", 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553699+0000", 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9639999999999999e-06 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553706+0000", 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0140000000000001e-06 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553718+0000", 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2369e-05 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553729+0000", 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1454000000000001e-05 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553819+0000", 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9420999999999996e-05 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553826+0000", 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2679999999999997e-06 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.556946+0000", 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003119798 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.557067+0000", 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121544 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.557495+0000", 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042718500000000001 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.560546+0000", 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030512069999999998 2024-09-21T08:29:05.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.561411+0000", 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00086473699999999997 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.575597+0000", 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014186025 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.575668+0000", 2024-09-21T08:29:05.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1898999999999997e-05 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.575720+0000", 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1824999999999999e-05 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3198 7.8 7:15ddb887:::smithi13779221-8:head [append 0~0] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.713165+0000", 2024-09-21T08:29:05.816 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.992708574000002, 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85552278699999995, 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3198 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713165+0000", 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713165+0000", 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713169+0000", 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.177e-06 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713182+0000", 2024-09-21T08:29:05.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3593e-05 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713212+0000", 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9292999999999999e-05 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713227+0000", 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5630000000000001e-05 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713409+0000", 2024-09-21T08:29:05.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018163700000000001 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713424+0000", 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5366000000000002e-05 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.562792+0000", 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.84936760099999997 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.563075+0000", 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000282945 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.563308+0000", 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023297899999999999 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566597+0000", 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003289541 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.567690+0000", 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001092906 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568570+0000", 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087925999999999998 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568631+0000", 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1511000000000001e-05 2024-09-21T08:29:05.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568687+0000", 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6348000000000003e-05 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3199 7.8 7:15ddb887:::smithi13779221-8:head [rollback 69] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.713255+0000", 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.992618141000001, 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85558638099999995, 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.823 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3199 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713255+0000", 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713255+0000", 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713257+0000", 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5260000000000001e-06 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713264+0000", 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2599999999999999e-06 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713287+0000", 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3470000000000001e-05 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713302+0000", 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4195999999999999e-05 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713470+0000", 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016798400000000001 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.826 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713478+0000", 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3059999999999998e-06 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.564578+0000", 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85110020600000003 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.564669+0000", 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1321000000000002e-05 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.564879+0000", 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020947899999999999 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566604+0000", 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017247409999999999 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.567857+0000", 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001253852 2024-09-21T08:29:05.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568733+0000", 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087562999999999996 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568793+0000", 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9429000000000003e-05 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568842+0000", 2024-09-21T08:29:05.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8980999999999997e-05 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3200 7.8 7:15ddb887:::smithi13779221-8:head [append 0~0] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.713346+0000", 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 28.992527431999999, 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85569378200000001, 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.830 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3200 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713346+0000", 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713346+0000", 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713347+0000", 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.626e-06 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713354+0000", 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.567e-06 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713369+0000", 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5216e-05 2024-09-21T08:29:05.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713386+0000", 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6286e-05 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713509+0000", 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012365299999999999 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713520+0000", 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0536e-05 2024-09-21T08:29:05.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566084+0000", 2024-09-21T08:29:05.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85256443900000001 2024-09-21T08:29:05.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566173+0000", 2024-09-21T08:29:05.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8936000000000004e-05 2024-09-21T08:29:05.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:29:05.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566334+0000", 2024-09-21T08:29:05.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160941 2024-09-21T08:29:05.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.567127+0000", 2024-09-21T08:29:05.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079256800000000001 2024-09-21T08:29:05.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568004+0000", 2024-09-21T08:29:05.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087773799999999998 2024-09-21T08:29:05.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568895+0000", 2024-09-21T08:29:05.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089024700000000004 2024-09-21T08:29:05.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568975+0000", 2024-09-21T08:29:05.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0527999999999999e-05 2024-09-21T08:29:05.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.569040+0000", 2024-09-21T08:29:05.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4500999999999998e-05 2024-09-21T08:29:05.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.839 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.840 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.840 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3391 7.12 7:4b20cb2f:::smithi13779221-12:head [copy-from ver 108] snapc 81=[81,80,7f,7e,7d,7c,7a,79,78,77,76,72,71,69] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T08:29:05.840 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:29:05.840 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.119669594000001, 2024-09-21T08:29:05.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026028484000000001, 2024-09-21T08:29:05.841 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.841 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.841 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:05.841 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.841 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.841 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.842 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3391 2024-09-21T08:29:05.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.842 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:29:05.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:29:05.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:29:05.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499999999999998e-07 2024-09-21T08:29:05.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586207+0000", 2024-09-21T08:29:05.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.588e-06 2024-09-21T08:29:05.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586216+0000", 2024-09-21T08:29:05.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1079999999999999e-06 2024-09-21T08:29:05.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.846 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.846 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586220+0000", 2024-09-21T08:29:05.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5319999999999998e-06 2024-09-21T08:29:05.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586265+0000", 2024-09-21T08:29:05.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5469999999999997e-05 2024-09-21T08:29:05.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586369+0000", 2024-09-21T08:29:05.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103698 2024-09-21T08:29:05.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:29:05.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.597860+0000", 2024-09-21T08:29:05.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011490887999999999 2024-09-21T08:29:05.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.607396+0000", 2024-09-21T08:29:05.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095365120000000005 2024-09-21T08:29:05.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:05.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.607414+0000", 2024-09-21T08:29:05.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8102e-05 2024-09-21T08:29:05.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:05.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.612155+0000", 2024-09-21T08:29:05.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047404739999999997 2024-09-21T08:29:05.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.612196+0000", 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1007999999999998e-05 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.612232+0000", 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6449000000000003e-05 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.851 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3392 7.12 7:4b20cb2f:::smithi13779221-12:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e336)", 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:49.586226+0000", 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 16.119646779, 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017159666000000001, 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:05.852 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3392 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586226+0000", 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586226+0000", 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:05.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586227+0000", 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3500000000000002e-07 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586229+0000", 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.934e-06 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586233+0000", 2024-09-21T08:29:05.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7799999999999998e-06 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586242+0000", 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7990000000000005e-06 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586502+0000", 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025981499999999999 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586557+0000", 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4978000000000001e-05 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.603246+0000", 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016689246000000001 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.603313+0000", 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6730999999999995e-05 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:05.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:05.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:05.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.603386+0000", 2024-09-21T08:29:05.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2947999999999993e-05 2024-09-21T08:29:05.857 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.857 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.857 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.857 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:05.857 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:05.857 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:29:05.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:05.882 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:05.882 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:05.901 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:05 smithi137 ceph-mon[29634]: pgmap v431: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 288 MiB data, 857 MiB used, 357 GiB / 358 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 30 op/s; 255/1011 objects degraded (25.223%) 2024-09-21T08:29:05.902 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:05 smithi137 ceph-mon[29634]: osdmap e351: 8 total, 7 up, 4 in 2024-09-21T08:29:05.902 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:05 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/340454797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:05.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:05 smithi137 ceph-mon[41044]: pgmap v431: 168 pgs: 72 active+undersized, 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 58 active+undersized+degraded, 32 active+clean; 288 MiB data, 857 MiB used, 357 GiB / 358 GiB avail; 25 MiB/s rd, 4.1 MiB/s wr, 30 op/s; 255/1011 objects degraded (25.223%) 2024-09-21T08:29:05.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:05 smithi137 ceph-mon[41044]: osdmap e351: 8 total, 7 up, 4 in 2024-09-21T08:29:05.902 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:05 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/340454797' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:05.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:06.006 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:06.006 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:06.006 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:06.107 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:06.129 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:06.129 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:06.129 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:06.230 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:06.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:06.251 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:06.252 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:06.353 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:06.374 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:06.375 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:06.375 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:06.427 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:06.476 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:06.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T08:29:06.497 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:06.498 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:06.498 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:06.554 INFO:tasks.rados.rados.0.smithi137.stdout:1357: done (4 left) 2024-09-21T08:29:06.555 INFO:tasks.rados.rados.0.smithi137.stdout:1358: done (3 left) 2024-09-21T08:29:06.555 INFO:tasks.rados.rados.0.smithi137.stdout:1359: done (2 left) 2024-09-21T08:29:06.555 INFO:tasks.rados.rados.0.smithi137.stdout:1360: done (1 left) 2024-09-21T08:29:06.555 INFO:tasks.rados.rados.0.smithi137.stdout:1361: done (0 left) 2024-09-21T08:29:06.555 INFO:tasks.rados.rados.0.smithi137.stdout:1362: rollback oid 27 current snap is 137 2024-09-21T08:29:06.555 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 27 to 112 2024-09-21T08:29:06.555 INFO:tasks.rados.rados.0.smithi137.stdout:1363: read oid 44 snap -1 2024-09-21T08:29:06.555 INFO:tasks.rados.rados.0.smithi137.stdout:1363: expect (ObjNum 471 snap 128 seq_num 471) 2024-09-21T08:29:06.555 INFO:tasks.rados.rados.0.smithi137.stdout:1364: write (excl) oid 49 current snap is 137 2024-09-21T08:29:06.555 INFO:tasks.rados.rados.0.smithi137.stdout:1364: seq_num 496 ranges {736272=486449,1974951=361321} 2024-09-21T08:29:06.561 INFO:tasks.rados.rados.0.smithi137.stdout:1364: writing smithi13779221-49 from 736272 to 1222721 tid 1 2024-09-21T08:29:06.568 INFO:tasks.rados.rados.0.smithi137.stdout:1364: writing smithi13779221-49 from 1974951 to 2336272 tid 2 2024-09-21T08:29:06.568 INFO:tasks.rados.rados.0.smithi137.stdout:1365: read oid 10 snap 123 2024-09-21T08:29:06.569 INFO:tasks.rados.rados.0.smithi137.stdout:1365: expect (ObjNum 385 snap 94 seq_num 385) 2024-09-21T08:29:06.569 INFO:tasks.rados.rados.0.smithi137.stdout:1366: write oid 11 current snap is 137 2024-09-21T08:29:06.569 INFO:tasks.rados.rados.0.smithi137.stdout:1366: seq_num 497 ranges {753079=561698,2033433=319646} 2024-09-21T08:29:06.576 INFO:tasks.rados.rados.0.smithi137.stdout:1366: writing smithi13779221-11 from 753079 to 1314777 tid 1 2024-09-21T08:29:06.583 INFO:tasks.rados.rados.0.smithi137.stdout:1366: writing smithi13779221-11 from 2033433 to 2353079 tid 2 2024-09-21T08:29:06.583 INFO:tasks.rados.rados.0.smithi137.stdout:1367: copy_from oid 26 from oid 4 current snap is 137 2024-09-21T08:29:06.583 INFO:tasks.rados.rados.0.smithi137.stdout:1362: finishing rollback tid 0 to smithi13779221-27 2024-09-21T08:29:06.583 INFO:tasks.rados.rados.0.smithi137.stdout:1362: finishing rollback tid 1 to smithi13779221-27 2024-09-21T08:29:06.583 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 300 (ObjNum 379559136 snap 32534 seq_num 304) dirty dne 2024-09-21T08:29:06.583 INFO:tasks.rados.rados.0.smithi137.stdout:1363: expect (ObjNum 471 snap 128 seq_num 471) 2024-09-21T08:29:06.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:06.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:06.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:06.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:06.712 INFO:tasks.rados.rados.0.smithi137.stdout:1364: finishing write tid 1 to smithi13779221-49 2024-09-21T08:29:06.712 INFO:tasks.rados.rados.0.smithi137.stdout:1364: finishing write tid 2 to smithi13779221-49 2024-09-21T08:29:06.712 INFO:tasks.rados.rados.0.smithi137.stdout:1362: done (5 left) 2024-09-21T08:29:06.712 INFO:tasks.rados.rados.0.smithi137.stdout:1363: done (4 left) 2024-09-21T08:29:06.712 INFO:tasks.rados.rados.0.smithi137.stdout:1368: copy_from oid 47 from oid 45 current snap is 137 2024-09-21T08:29:06.712 INFO:tasks.rados.rados.0.smithi137.stdout:1364: finishing write tid 3 to smithi13779221-49 2024-09-21T08:29:06.713 INFO:tasks.rados.rados.0.smithi137.stdout:1364: finishing write tid 4 to smithi13779221-49 2024-09-21T08:29:06.713 INFO:tasks.rados.rados.0.smithi137.stdout:1364: oid 49 updating version 0 to 416 2024-09-21T08:29:06.713 INFO:tasks.rados.rados.0.smithi137.stdout:1364: oid 49 updating version 416 to 417 2024-09-21T08:29:06.713 INFO:tasks.rados.rados.0.smithi137.stdout:1364: oid 49 updating version 417 to 418 2024-09-21T08:29:06.713 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 418 (ObjNum 496 snap 137 seq_num 496) dirty exists 2024-09-21T08:29:06.713 INFO:tasks.rados.rados.0.smithi137.stdout:1364: left oid 49 (ObjNum 496 snap 137 seq_num 496) 2024-09-21T08:29:06.713 INFO:tasks.rados.rados.0.smithi137.stdout:1365: expect (ObjNum 385 snap 94 seq_num 385) 2024-09-21T08:29:06.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:06.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:06.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:06.744 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:06.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:06 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:29:06.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:06 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:29:06.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:06 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/340454797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:06.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:06 smithi165 ceph-mon[32893]: osdmap e352: 8 total, 7 up, 4 in 2024-09-21T08:29:06.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:06 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:06.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:06 smithi165 ceph-mon[32893]: osdmap e353: 8 total, 7 up, 4 in 2024-09-21T08:29:06.831 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:29:06.831 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:29:06.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/340454797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:06.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[29634]: osdmap e352: 8 total, 7 up, 4 in 2024-09-21T08:29:06.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:06.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[29634]: osdmap e353: 8 total, 7 up, 4 in 2024-09-21T08:29:06.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:29:06.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:29:06.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/340454797' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:06.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[41044]: osdmap e352: 8 total, 7 up, 4 in 2024-09-21T08:29:06.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:06.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:06 smithi137 ceph-mon[41044]: osdmap e353: 8 total, 7 up, 4 in 2024-09-21T08:29:06.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:06.847 INFO:tasks.rados.rados.0.smithi137.stdout:1366: finishing write tid 1 to smithi13779221-11 2024-09-21T08:29:06.847 INFO:tasks.rados.rados.0.smithi137.stdout:1364: done (4 left) 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1365: done (3 left) 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1369: read oid 25 snap 135 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1369: expect (ObjNum 459 snap 124 seq_num 459) 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1370: read oid 34 snap -1 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1370: expect deleted 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1371: rollback oid 2 current snap is 137 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 2 to 131 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1366: finishing write tid 2 to smithi13779221-11 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1366: finishing write tid 3 to smithi13779221-11 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1366: finishing write tid 4 to smithi13779221-11 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1366: oid 11 updating version 0 to 516 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1366: oid 11 updating version 516 to 517 2024-09-21T08:29:06.848 INFO:tasks.rados.rados.0.smithi137.stdout:1366: oid 11 updating version 517 to 518 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 518 (ObjNum 497 snap 137 seq_num 497) dirty exists 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1366: left oid 11 (ObjNum 497 snap 137 seq_num 497) 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1367: finishing copy_from racing read to smithi13779221-26 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1367: finishing copy_from to smithi13779221-26 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 305 (ObjNum 462 snap 124 seq_num 462) dirty exists 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1368: finishing copy_from racing read to smithi13779221-47 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1366: done (5 left) 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1367: done (4 left) 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1372: copy_from oid 36 from oid 18 current snap is 137 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1368: finishing copy_from to smithi13779221-47 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 309 (ObjNum 467 snap 126 seq_num 467) dirty exists 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1368: done (4 left) 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1373: read oid 26 snap -1 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1373: expect (ObjNum 462 snap 124 seq_num 462) 2024-09-21T08:29:06.849 INFO:tasks.rados.rados.0.smithi137.stdout:1374: snap_create 2024-09-21T08:29:06.850 INFO:tasks.rados.rados.0.smithi137.stdout:1371: finishing rollback tid 0 to smithi13779221-2 2024-09-21T08:29:06.851 INFO:tasks.rados.rados.0.smithi137.stdout:1371: finishing rollback tid 1 to smithi13779221-2 2024-09-21T08:29:06.851 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 159 (ObjNum 379559136 snap 32534 seq_num 304) dirty dne 2024-09-21T08:29:06.856 INFO:tasks.rados.rados.0.smithi137.stdout:1373: expect (ObjNum 462 snap 124 seq_num 462) 2024-09-21T08:29:06.859 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:06.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:06.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:06.871 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:06.938 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:29:06.939 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:29:06.956 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:06.957 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:06.971 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:06.972 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:06.972 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:06.989 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:06.990 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:06.992 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:06.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:06.993 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:07.006 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:07.007 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:07.023 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:07.023 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:07.035 INFO:tasks.rados.rados.0.smithi137.stdout:1369: expect (ObjNum 459 snap 124 seq_num 459) 2024-09-21T08:29:07.039 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:07.039 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:07.040 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:07.094 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:07.116 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:07.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:07.117 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:07.217 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:07.238 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:07.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:07.239 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:07.245 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.7 --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 3.2 2024-09-21T08:29:07.250 INFO:tasks.rados.rados.0.smithi137.stdout:1372: finishing copy_from racing read to smithi13779221-36 2024-09-21T08:29:07.250 INFO:tasks.rados.rados.0.smithi137.stdout:1372: finishing copy_from to smithi13779221-36 2024-09-21T08:29:07.250 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 217 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:29:07.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:07.361 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:07.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:07.362 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:07.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:07.484 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:07.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:07.485 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:07.513 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T08:29:07.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:07.593 INFO:tasks.rados.rados.0.smithi137.stdout:1369: done (5 left) 2024-09-21T08:29:07.593 INFO:tasks.rados.rados.0.smithi137.stdout:1370: done (4 left) 2024-09-21T08:29:07.593 INFO:tasks.rados.rados.0.smithi137.stdout:1371: done (3 left) 2024-09-21T08:29:07.593 INFO:tasks.rados.rados.0.smithi137.stdout:1372: done (2 left) 2024-09-21T08:29:07.593 INFO:tasks.rados.rados.0.smithi137.stdout:1373: done (1 left) 2024-09-21T08:29:07.594 INFO:tasks.rados.rados.0.smithi137.stdout:1374: done (0 left) 2024-09-21T08:29:07.594 INFO:tasks.rados.rados.0.smithi137.stdout:1375: read oid 40 snap -1 2024-09-21T08:29:07.594 INFO:tasks.rados.rados.0.smithi137.stdout:1375: expect (ObjNum 287 snap 66 seq_num 287) 2024-09-21T08:29:07.594 INFO:tasks.rados.rados.0.smithi137.stdout:1376: copy_from oid 1 from oid 49 current snap is 138 2024-09-21T08:29:07.594 INFO:tasks.rados.rados.0.smithi137.stdout:1377: read oid 37 snap -1 2024-09-21T08:29:07.594 INFO:tasks.rados.rados.0.smithi137.stdout:1377: expect (ObjNum 479 snap 133 seq_num 479) 2024-09-21T08:29:07.594 INFO:tasks.rados.rados.0.smithi137.stdout:1378: rollback oid 39 current snap is 138 2024-09-21T08:29:07.594 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 39 to 124 2024-09-21T08:29:07.594 INFO:tasks.rados.rados.0.smithi137.stdout:1379: snap_create 2024-09-21T08:29:07.604 INFO:tasks.rados.rados.0.smithi137.stdout:1378: finishing rollback tid 0 to smithi13779221-39 2024-09-21T08:29:07.606 INFO:tasks.rados.rados.0.smithi137.stdout:1377: expect (ObjNum 479 snap 133 seq_num 479) 2024-09-21T08:29:07.619 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:07 smithi137 ceph-mon[29634]: pgmap v434: 168 pgs: 72 active+undersized, 2 active+clean+snaptrim, 58 active+undersized+degraded, 36 active+clean; 304 MiB data, 875 MiB used, 357 GiB / 358 GiB avail; 6.6 MiB/s rd, 2.3 MiB/s wr, 17 op/s; 260/1032 objects degraded (25.194%) 2024-09-21T08:29:07.619 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:07 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3778163939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:07.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:07 smithi137 ceph-mon[41044]: pgmap v434: 168 pgs: 72 active+undersized, 2 active+clean+snaptrim, 58 active+undersized+degraded, 36 active+clean; 304 MiB data, 875 MiB used, 357 GiB / 358 GiB avail; 6.6 MiB/s rd, 2.3 MiB/s wr, 17 op/s; 260/1032 objects degraded (25.194%) 2024-09-21T08:29:07.620 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:07 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3778163939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:07.621 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:07.621 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:07.645 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:07.646 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:1375: expect (ObjNum 287 snap 66 seq_num 287) 2024-09-21T08:29:07.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:07.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:07.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:07.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:07.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:07.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:07 smithi165 ceph-mon[32893]: pgmap v434: 168 pgs: 72 active+undersized, 2 active+clean+snaptrim, 58 active+undersized+degraded, 36 active+clean; 304 MiB data, 875 MiB used, 357 GiB / 358 GiB avail; 6.6 MiB/s rd, 2.3 MiB/s wr, 17 op/s; 260/1032 objects degraded (25.194%) 2024-09-21T08:29:07.851 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:07 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3778163939' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:07.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:07.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:07.867 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:07.880 INFO:tasks.rados.rados.0.smithi137.stdout:1378: finishing rollback tid 1 to smithi13779221-39 2024-09-21T08:29:07.880 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 229 (ObjNum 225 snap 0 seq_num 379559344) dirty dne 2024-09-21T08:29:07.880 INFO:tasks.rados.rados.0.smithi137.stdout:1376: finishing copy_from racing read to smithi13779221-1 2024-09-21T08:29:07.880 INFO:tasks.rados.rados.0.smithi137.stdout:1376: finishing copy_from to smithi13779221-1 2024-09-21T08:29:07.880 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 418 (ObjNum 496 snap 137 seq_num 496) dirty exists 2024-09-21T08:29:07.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:07.982 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:07.982 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:07.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:07.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:07.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:08.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:08.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:08.113 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:08.113 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:08.214 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:08.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:08.236 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:08.237 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:08.337 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:08.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:29:08.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:08.363 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:08.363 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:08.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:08.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:08.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:08.486 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:08.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:08.594 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:08.595 INFO:tasks.rados.rados.0.smithi137.stdout:1375: done (4 left) 2024-09-21T08:29:08.595 INFO:tasks.rados.rados.0.smithi137.stdout:1376: done (3 left) 2024-09-21T08:29:08.595 INFO:tasks.rados.rados.0.smithi137.stdout:1377: done (2 left) 2024-09-21T08:29:08.595 INFO:tasks.rados.rados.0.smithi137.stdout:1378: done (1 left) 2024-09-21T08:29:08.595 INFO:tasks.rados.rados.0.smithi137.stdout:1379: done (0 left) 2024-09-21T08:29:08.596 INFO:tasks.rados.rados.0.smithi137.stdout:1380: read oid 24 snap 132 2024-09-21T08:29:08.596 INFO:tasks.rados.rados.0.smithi137.stdout:1380: expect (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:29:08.596 INFO:tasks.rados.rados.0.smithi137.stdout:1381: snap_create 2024-09-21T08:29:08.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T08:29:08.603 INFO:tasks.rados.rados.0.smithi137.stdout:1380: expect (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:29:08.608 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:08.609 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:08.609 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:08.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:08.722 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:08 smithi137 ceph-mon[29634]: osdmap e354: 8 total, 7 up, 4 in 2024-09-21T08:29:08.723 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:08 smithi137 ceph-mon[41044]: osdmap e354: 8 total, 7 up, 4 in 2024-09-21T08:29:08.736 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:08.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:08.737 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:08.838 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:08.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:08.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:08.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:08.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:08.982 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:08.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:08.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:08 smithi165 ceph-mon[32893]: osdmap e354: 8 total, 7 up, 4 in 2024-09-21T08:29:09.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:09.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:09.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:09.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:09.206 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:09.226 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:09.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:09.227 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:09.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:09.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:09.351 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:09.351 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:09.374 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T08:29:09.452 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:09.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:09.475 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:09.475 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:09.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:09.601 INFO:tasks.rados.rados.0.smithi137.stdout:1380: done (1 left) 2024-09-21T08:29:09.601 INFO:tasks.rados.rados.0.smithi137.stdout:1381: done (0 left) 2024-09-21T08:29:09.601 INFO:tasks.rados.rados.0.smithi137.stdout:1382: write (excl) oid 19 current snap is 140 2024-09-21T08:29:09.601 INFO:tasks.rados.rados.0.smithi137.stdout:1382: seq_num 498 ranges {769886=636947,2091915=277971} 2024-09-21T08:29:09.602 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:09.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:09.603 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:09.603 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:29:09.609 INFO:tasks.rados.rados.0.smithi137.stdout:1382: writing smithi13779221-19 from 769886 to 1406833 tid 1 2024-09-21T08:29:09.615 INFO:tasks.rados.rados.0.smithi137.stdout:1382: writing smithi13779221-19 from 2091915 to 2369886 tid 2 2024-09-21T08:29:09.615 INFO:tasks.rados.rados.0.smithi137.stdout:1383: snap_remove snap 139 2024-09-21T08:29:09.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:29:09.704 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:09.725 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:09.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:09.726 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:09.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:09.842 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:09.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:09.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:09.848 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:09.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:09 smithi137 ceph-mon[29634]: pgmap v437: 168 pgs: 72 active+undersized, 2 active+clean+snaptrim, 58 active+undersized+degraded, 36 active+clean; 304 MiB data, 875 MiB used, 357 GiB / 358 GiB avail; 6.6 MiB/s rd, 2.3 MiB/s wr, 17 op/s; 260/1032 objects degraded (25.194%) 2024-09-21T08:29:09.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:09 smithi137 ceph-mon[29634]: osdmap e355: 8 total, 7 up, 4 in 2024-09-21T08:29:09.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:09 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4011000222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:09.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:09 smithi137 ceph-mon[29634]: from='client.45521 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:09.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:09 smithi137 ceph-mon[41044]: pgmap v437: 168 pgs: 72 active+undersized, 2 active+clean+snaptrim, 58 active+undersized+degraded, 36 active+clean; 304 MiB data, 875 MiB used, 357 GiB / 358 GiB avail; 6.6 MiB/s rd, 2.3 MiB/s wr, 17 op/s; 260/1032 objects degraded (25.194%) 2024-09-21T08:29:09.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:09 smithi137 ceph-mon[41044]: osdmap e355: 8 total, 7 up, 4 in 2024-09-21T08:29:09.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:09 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4011000222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:09.911 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:09 smithi137 ceph-mon[41044]: from='client.45521 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:09.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:09 smithi165 ceph-mon[32893]: pgmap v437: 168 pgs: 72 active+undersized, 2 active+clean+snaptrim, 58 active+undersized+degraded, 36 active+clean; 304 MiB data, 875 MiB used, 357 GiB / 358 GiB avail; 6.6 MiB/s rd, 2.3 MiB/s wr, 17 op/s; 260/1032 objects degraded (25.194%) 2024-09-21T08:29:09.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:09 smithi165 ceph-mon[32893]: osdmap e355: 8 total, 7 up, 4 in 2024-09-21T08:29:09.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:09 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4011000222' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:09.940 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:09 smithi165 ceph-mon[32893]: from='client.45521 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:09.949 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:09.971 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:09.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:09.972 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:10.072 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:10.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:10.095 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:10.095 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:10.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:10.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:10.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:10.219 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:10.287 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:10.287 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:10.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:10.342 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:10.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:10.343 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:10.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:10.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:10.466 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:10.467 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:10.527 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:10.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:10.576 INFO:teuthology.orchestra.run.smithi165.stdout: marking collection for removal 2024-09-21T08:29:10.576 INFO:teuthology.orchestra.run.smithi165.stdout:setting '_remove' omap key 2024-09-21T08:29:10.576 INFO:teuthology.orchestra.run.smithi165.stdout:finish_remove_pgs 3.2_head removing 3.2 2024-09-21T08:29:10.577 INFO:teuthology.orchestra.run.smithi165.stdout:Remove successful 2024-09-21T08:29:10.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:10.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:10.590 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:10.604 INFO:tasks.rados.rados.0.smithi137.stdout:1383: done (1 left) 2024-09-21T08:29:10.605 INFO:tasks.rados.rados.0.smithi137.stdout:1384: write oid 1 current snap is 140 2024-09-21T08:29:10.605 INFO:tasks.rados.rados.0.smithi137.stdout:1382: finishing write tid 1 to smithi13779221-19 2024-09-21T08:29:10.605 INFO:tasks.rados.rados.0.smithi137.stdout:1384: seq_num 499 ranges {786693=712196,2150397=236296} 2024-09-21T08:29:10.613 INFO:tasks.rados.rados.0.smithi137.stdout:1384: writing smithi13779221-1 from 786693 to 1498889 tid 1 2024-09-21T08:29:10.618 INFO:tasks.rados.rados.0.smithi137.stdout:1384: writing smithi13779221-1 from 2150397 to 2386693 tid 2 2024-09-21T08:29:10.618 INFO:tasks.rados.rados.0.smithi137.stdout:1385: snap_create 2024-09-21T08:29:10.618 INFO:tasks.rados.rados.0.smithi137.stdout:1382: finishing write tid 2 to smithi13779221-19 2024-09-21T08:29:10.618 INFO:tasks.rados.rados.0.smithi137.stdout:1382: finishing write tid 3 to smithi13779221-19 2024-09-21T08:29:10.618 INFO:tasks.rados.rados.0.smithi137.stdout:1382: finishing write tid 4 to smithi13779221-19 2024-09-21T08:29:10.618 INFO:tasks.rados.rados.0.smithi137.stdout:1382: oid 19 updating version 0 to 122 2024-09-21T08:29:10.618 INFO:tasks.rados.rados.0.smithi137.stdout:1382: oid 19 version 122 is already newer than 120 2024-09-21T08:29:10.618 INFO:tasks.rados.rados.0.smithi137.stdout:1382: oid 19 version 122 is already newer than 121 2024-09-21T08:29:10.618 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 122 (ObjNum 498 snap 140 seq_num 498) dirty exists 2024-09-21T08:29:10.618 INFO:tasks.rados.rados.0.smithi137.stdout:1382: left oid 19 (ObjNum 498 snap 140 seq_num 498) 2024-09-21T08:29:10.624 INFO:tasks.rados.rados.0.smithi137.stdout:1384: finishing write tid 1 to smithi13779221-1 2024-09-21T08:29:10.624 INFO:tasks.rados.rados.0.smithi137.stdout:1384: finishing write tid 2 to smithi13779221-1 2024-09-21T08:29:10.624 INFO:tasks.rados.rados.0.smithi137.stdout:1384: finishing write tid 3 to smithi13779221-1 2024-09-21T08:29:10.624 INFO:tasks.rados.rados.0.smithi137.stdout:1384: finishing write tid 4 to smithi13779221-1 2024-09-21T08:29:10.625 INFO:tasks.rados.rados.0.smithi137.stdout:1384: oid 1 updating version 0 to 521 2024-09-21T08:29:10.625 INFO:tasks.rados.rados.0.smithi137.stdout:1384: oid 1 updating version 521 to 522 2024-09-21T08:29:10.625 INFO:tasks.rados.rados.0.smithi137.stdout:1384: oid 1 version 522 is already newer than 520 2024-09-21T08:29:10.625 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 522 (ObjNum 499 snap 140 seq_num 499) dirty exists 2024-09-21T08:29:10.625 INFO:tasks.rados.rados.0.smithi137.stdout:1384: left oid 1 (ObjNum 499 snap 140 seq_num 499) 2024-09-21T08:29:10.677 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:10 smithi137 ceph-mon[29634]: from='client.45521 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:29:10.677 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:10 smithi137 ceph-mon[29634]: osdmap e356: 8 total, 7 up, 4 in 2024-09-21T08:29:10.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:10 smithi137 ceph-mon[41044]: from='client.45521 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:29:10.678 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:10 smithi137 ceph-mon[41044]: osdmap e356: 8 total, 7 up, 4 in 2024-09-21T08:29:10.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:10.711 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:10.712 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:10.712 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:10.813 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:10.835 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:10.835 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:10.835 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:10.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:10 smithi165 ceph-mon[32893]: from='client.45521 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:29:10.897 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:10 smithi165 ceph-mon[32893]: osdmap e356: 8 total, 7 up, 4 in 2024-09-21T08:29:10.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T08:29:10.936 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:10.959 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:10.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:10.974 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:29:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:29:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:29:11.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:11.082 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:11.083 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:11.083 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:11.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:11.205 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:11.206 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:11.206 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:11.222 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:29:11.223 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:29:11.239 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:11.239 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:11.256 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:11.256 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:11.271 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:11.271 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:11.288 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:11.288 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:11.305 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:11.305 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:11.307 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:11.322 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:11.322 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:11.322 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:11.329 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:11.329 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:11.330 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:11.430 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:11.452 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:11.452 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:11.453 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:11.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:11.559 INFO:tasks.rados.rados.0.smithi137.stdout:1382: done (2 left) 2024-09-21T08:29:11.560 INFO:tasks.rados.rados.0.smithi137.stdout:1384: done (1 left) 2024-09-21T08:29:11.560 INFO:tasks.rados.rados.0.smithi137.stdout:1385: done (0 left) 2024-09-21T08:29:11.560 INFO:tasks.rados.rados.0.smithi137.stdout:1386: delete oid 26 current snap is 141 2024-09-21T08:29:11.568 INFO:tasks.rados.rados.0.smithi137.stdout:1386: done (0 left) 2024-09-21T08:29:11.568 INFO:tasks.rados.rados.0.smithi137.stdout:1387: delete oid 5 current snap is 141 2024-09-21T08:29:11.574 INFO:tasks.rados.rados.0.smithi137.stdout:1387: done (0 left) 2024-09-21T08:29:11.574 INFO:tasks.rados.rados.0.smithi137.stdout:1388: rollback oid 35 current snap is 141 2024-09-21T08:29:11.574 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 35 to 134 2024-09-21T08:29:11.574 INFO:tasks.rados.rados.0.smithi137.stdout:1389: write (excl) oid 21 current snap is 141 2024-09-21T08:29:11.574 INFO:tasks.rados.rados.0.smithi137.stdout:1389: seq_num 502 ranges {437114=654296,1525843=477506} 2024-09-21T08:29:11.577 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:11.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:11.578 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:11.579 INFO:tasks.rados.rados.0.smithi137.stdout:1389: writing smithi13779221-21 from 437114 to 1091410 tid 1 2024-09-21T08:29:11.585 INFO:tasks.rados.rados.0.smithi137.stdout:1389: writing smithi13779221-21 from 1525843 to 2003349 tid 2 2024-09-21T08:29:11.585 INFO:tasks.rados.rados.0.smithi137.stdout:1390: copy_from oid 28 from oid 9 current snap is 141 2024-09-21T08:29:11.585 INFO:tasks.rados.rados.0.smithi137.stdout:1388: finishing rollback tid 1 to smithi13779221-35 2024-09-21T08:29:11.585 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 319 (ObjNum 161 snap 0 seq_num 142) dirty dne 2024-09-21T08:29:11.585 INFO:tasks.rados.rados.0.smithi137.stdout:1388: done (2 left) 2024-09-21T08:29:11.585 INFO:tasks.rados.rados.0.smithi137.stdout:1391: write oid 26 current snap is 141 2024-09-21T08:29:11.585 INFO:tasks.rados.rados.0.smithi137.stdout:1391: seq_num 503 ranges {453921=729545,1584325=621164} 2024-09-21T08:29:11.591 INFO:tasks.rados.rados.0.smithi137.stdout:1391: writing smithi13779221-26 from 453921 to 1183466 tid 1 2024-09-21T08:29:11.597 INFO:tasks.rados.rados.0.smithi137.stdout:1391: writing smithi13779221-26 from 1584325 to 2205489 tid 2 2024-09-21T08:29:11.597 INFO:tasks.rados.rados.0.smithi137.stdout:1392: delete oid 12 current snap is 141 2024-09-21T08:29:11.597 INFO:tasks.rados.rados.0.smithi137.stdout:1390: finishing copy_from to smithi13779221-28 2024-09-21T08:29:11.597 INFO:tasks.rados.rados.0.smithi137.stdout:1390: got expected ENOENT (src dne) 2024-09-21T08:29:11.597 INFO:tasks.rados.rados.0.smithi137.stdout:1390: finishing copy_from racing read to smithi13779221-28 2024-09-21T08:29:11.597 INFO:tasks.rados.rados.0.smithi137.stdout:1389: finishing write tid 1 to smithi13779221-21 2024-09-21T08:29:11.598 INFO:tasks.rados.rados.0.smithi137.stdout:1389: finishing write tid 2 to smithi13779221-21 2024-09-21T08:29:11.598 INFO:tasks.rados.rados.0.smithi137.stdout:1389: finishing write tid 3 to smithi13779221-21 2024-09-21T08:29:11.598 INFO:tasks.rados.rados.0.smithi137.stdout:1389: finishing write tid 4 to smithi13779221-21 2024-09-21T08:29:11.598 INFO:tasks.rados.rados.0.smithi137.stdout:1389: oid 21 updating version 0 to 123 2024-09-21T08:29:11.599 INFO:tasks.rados.rados.0.smithi137.stdout:1389: oid 21 updating version 123 to 124 2024-09-21T08:29:11.599 INFO:tasks.rados.rados.0.smithi137.stdout:1389: oid 21 updating version 124 to 125 2024-09-21T08:29:11.599 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 125 (ObjNum 502 snap 141 seq_num 502) dirty exists 2024-09-21T08:29:11.599 INFO:tasks.rados.rados.0.smithi137.stdout:1389: left oid 21 (ObjNum 502 snap 141 seq_num 502) 2024-09-21T08:29:11.600 INFO:tasks.rados.rados.0.smithi137.stdout:1391: finishing write tid 1 to smithi13779221-26 2024-09-21T08:29:11.601 INFO:tasks.rados.rados.0.smithi137.stdout:1391: finishing write tid 2 to smithi13779221-26 2024-09-21T08:29:11.601 INFO:tasks.rados.rados.0.smithi137.stdout:1391: finishing write tid 3 to smithi13779221-26 2024-09-21T08:29:11.601 INFO:tasks.rados.rados.0.smithi137.stdout:1389: done (3 left) 2024-09-21T08:29:11.601 INFO:tasks.rados.rados.0.smithi137.stdout:1390: done (2 left) 2024-09-21T08:29:11.601 INFO:tasks.rados.rados.0.smithi137.stdout:1392: done (1 left) 2024-09-21T08:29:11.601 INFO:tasks.rados.rados.0.smithi137.stdout:1393: read oid 14 snap -1 2024-09-21T08:29:11.601 INFO:tasks.rados.rados.0.smithi137.stdout:1393: expect (ObjNum 456 snap 123 seq_num 456) 2024-09-21T08:29:11.601 INFO:tasks.rados.rados.0.smithi137.stdout:1394: write (excl) oid 20 current snap is 141 2024-09-21T08:29:11.601 INFO:tasks.rados.rados.0.smithi137.stdout:1394: seq_num 505 ranges {487535=480043,1584936=624833} 2024-09-21T08:29:11.606 INFO:tasks.rados.rados.0.smithi137.stdout:1394: writing smithi13779221-20 from 487535 to 967578 tid 1 2024-09-21T08:29:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:1394: writing smithi13779221-20 from 1584936 to 2209769 tid 2 2024-09-21T08:29:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:1395: read oid 34 snap -1 2024-09-21T08:29:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:1395: expect deleted 2024-09-21T08:29:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:1396: write (excl) oid 50 current snap is 141 2024-09-21T08:29:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:1391: finishing write tid 4 to smithi13779221-26 2024-09-21T08:29:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:1391: oid 26 updating version 0 to 361 2024-09-21T08:29:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:1391: oid 26 updating version 361 to 362 2024-09-21T08:29:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:1391: oid 26 updating version 362 to 363 2024-09-21T08:29:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 363 (ObjNum 503 snap 141 seq_num 503) dirty exists 2024-09-21T08:29:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:1391: left oid 26 (ObjNum 503 snap 141 seq_num 503) 2024-09-21T08:29:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:1396: seq_num 506 ranges {504342=555292,1643418=484844} 2024-09-21T08:29:11.619 INFO:tasks.rados.rados.0.smithi137.stdout:1396: writing smithi13779221-50 from 504342 to 1059634 tid 1 2024-09-21T08:29:11.626 INFO:tasks.rados.rados.0.smithi137.stdout:1396: writing smithi13779221-50 from 1643418 to 2128262 tid 2 2024-09-21T08:29:11.626 INFO:tasks.rados.rados.0.smithi137.stdout:1391: done (4 left) 2024-09-21T08:29:11.626 INFO:tasks.rados.rados.0.smithi137.stdout:1397: rollback oid 18 current snap is 141 2024-09-21T08:29:11.626 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 18 to 111 2024-09-21T08:29:11.626 INFO:tasks.rados.rados.0.smithi137.stdout:1398: write (excl) oid 43 current snap is 141 2024-09-21T08:29:11.626 INFO:tasks.rados.rados.0.smithi137.stdout:1398: seq_num 507 ranges {521149=630541,1701900=628502} 2024-09-21T08:29:11.632 INFO:tasks.rados.rados.0.smithi137.stdout:1398: writing smithi13779221-43 from 521149 to 1151690 tid 1 2024-09-21T08:29:11.639 INFO:tasks.rados.rados.0.smithi137.stdout:1398: writing smithi13779221-43 from 1701900 to 2330402 tid 2 2024-09-21T08:29:11.639 INFO:tasks.rados.rados.0.smithi137.stdout:1399: read oid 35 snap -1 2024-09-21T08:29:11.639 INFO:tasks.rados.rados.0.smithi137.stdout:1399: expect deleted 2024-09-21T08:29:11.639 INFO:tasks.rados.rados.0.smithi137.stdout:1400: snap_create 2024-09-21T08:29:11.639 INFO:tasks.rados.rados.0.smithi137.stdout:1393: expect (ObjNum 456 snap 123 seq_num 456) 2024-09-21T08:29:11.679 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:11.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:11.701 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:11.701 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:11.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:11 smithi165 ceph-mon[32893]: pgmap v440: 168 pgs: 72 active+undersized, 2 active+clean+snaptrim, 58 active+undersized+degraded, 36 active+clean; 304 MiB data, 875 MiB used, 357 GiB / 358 GiB avail; 260/1032 objects degraded (25.194%) 2024-09-21T08:29:11.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:11 smithi165 ceph-mon[32893]: osdmap e357: 8 total, 7 up, 4 in 2024-09-21T08:29:11.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:11 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/848421134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:11.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:11 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:11.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:11 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 260/1032 objects degraded (25.194%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:11.760 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:11 smithi165 ceph-mon[32893]: osdmap e358: 8 total, 7 up, 4 in 2024-09-21T08:29:11.785 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.7 --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.3.2.7 2024-09-21T08:29:11.802 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:11.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:11.823 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:11.824 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:11.884 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1394: finishing write tid 1 to smithi13779221-20 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1394: finishing write tid 2 to smithi13779221-20 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1394: finishing write tid 3 to smithi13779221-20 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1394: finishing write tid 4 to smithi13779221-20 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1394: oid 20 updating version 0 to 422 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1394: oid 20 updating version 422 to 423 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1394: oid 20 updating version 423 to 424 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 424 (ObjNum 505 snap 141 seq_num 505) dirty exists 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1394: left oid 20 (ObjNum 505 snap 141 seq_num 505) 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1396: finishing write tid 1 to smithi13779221-50 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1396: finishing write tid 2 to smithi13779221-50 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1397: finishing rollback tid 0 to smithi13779221-18 2024-09-21T08:29:11.885 INFO:tasks.rados.rados.0.smithi137.stdout:1397: finishing rollback tid 1 to smithi13779221-18 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1397: finishing rollback tid 2 to smithi13779221-18 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 220 (ObjNum 406 snap 99 seq_num 406) dirty exists 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1396: finishing write tid 3 to smithi13779221-50 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1396: finishing write tid 4 to smithi13779221-50 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1396: oid 50 updating version 0 to 377 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1396: oid 50 version 377 is already newer than 376 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1396: oid 50 updating version 377 to 378 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 378 (ObjNum 506 snap 141 seq_num 506) dirty exists 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1396: left oid 50 (ObjNum 506 snap 141 seq_num 506) 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1398: finishing write tid 1 to smithi13779221-43 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1398: finishing write tid 2 to smithi13779221-43 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1398: finishing write tid 3 to smithi13779221-43 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1398: finishing write tid 4 to smithi13779221-43 2024-09-21T08:29:11.886 INFO:tasks.rados.rados.0.smithi137.stdout:1398: oid 43 updating version 0 to 364 2024-09-21T08:29:11.887 INFO:tasks.rados.rados.0.smithi137.stdout:1398: oid 43 updating version 364 to 365 2024-09-21T08:29:11.887 INFO:tasks.rados.rados.0.smithi137.stdout:1398: oid 43 updating version 365 to 366 2024-09-21T08:29:11.887 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 366 (ObjNum 507 snap 141 seq_num 507) dirty exists 2024-09-21T08:29:11.887 INFO:tasks.rados.rados.0.smithi137.stdout:1398: left oid 43 (ObjNum 507 snap 141 seq_num 507) 2024-09-21T08:29:11.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:11.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:11.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:11.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:12.010 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:29:11 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:29:12.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[29634]: pgmap v440: 168 pgs: 72 active+undersized, 2 active+clean+snaptrim, 58 active+undersized+degraded, 36 active+clean; 304 MiB data, 875 MiB used, 357 GiB / 358 GiB avail; 260/1032 objects degraded (25.194%) 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[29634]: osdmap e357: 8 total, 7 up, 4 in 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/848421134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 260/1032 objects degraded (25.194%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[29634]: osdmap e358: 8 total, 7 up, 4 in 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[41044]: pgmap v440: 168 pgs: 72 active+undersized, 2 active+clean+snaptrim, 58 active+undersized+degraded, 36 active+clean; 304 MiB data, 875 MiB used, 357 GiB / 358 GiB avail; 260/1032 objects degraded (25.194%) 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[41044]: osdmap e357: 8 total, 7 up, 4 in 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/848421134' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 260/1032 objects degraded (25.194%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:12.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:11 smithi137 ceph-mon[41044]: osdmap e358: 8 total, 7 up, 4 in 2024-09-21T08:29:12.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:12.054 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:12.055 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:12.068 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:12.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:12.069 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:12.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:12.191 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:12.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:12.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:12.264 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T08:29:12.292 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:12.313 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:12.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:12.315 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:12.364 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 337.79465503699998, 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049090319999999998, 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:12.365 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.366 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567959+0000", 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.762e-06 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567965+0000", 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.367 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567980+0000", 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4742e-05 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567990+0000", 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8800000000000003e-06 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568114+0000", 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012444499999999999 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568250+0000", 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135205 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568390+0000", 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140335 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569250+0000", 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085971599999999995 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572132+0000", 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028820819999999998 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572759+0000", 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062750100000000004 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572810+0000", 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0488000000000002e-05 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572867+0000", 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6999000000000001e-05 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 337.794188228, 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045637589999999997, 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.371 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568426+0000", 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6470000000000001e-06 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568432+0000", 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.395e-06 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568443+0000", 2024-09-21T08:29:12.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1070000000000001e-05 2024-09-21T08:29:12.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568456+0000", 2024-09-21T08:29:12.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3862e-05 2024-09-21T08:29:12.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.374 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.374 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568688+0000", 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023171699999999999 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568773+0000", 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5307000000000002e-05 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568890+0000", 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116082 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:12.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569279+0000", 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038986899999999999 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572238+0000", 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029581009999999999 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572906+0000", 2024-09-21T08:29:12.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000668945 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572955+0000", 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8455000000000003e-05 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572988+0000", 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3309000000000002e-05 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:47 5.1f 5:f95f44c2:::notify.0:head [watch watch cookie 94074441540560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 337.78940714800001, 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026954729999999999, 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 47 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:29:12.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573208+0000", 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9970000000000001e-06 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573213+0000", 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5679999999999999e-06 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573224+0000", 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1270000000000001e-05 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573233+0000", 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8939999999999999e-06 2024-09-21T08:29:12.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573770+0000", 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053668100000000001 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573841+0000", 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.135e-05 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.574869+0000", 2024-09-21T08:29:12.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010281719999999999 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.574911+0000", 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1631000000000002e-05 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575071+0000", 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015977 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575463+0000", 2024-09-21T08:29:12.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039199299999999999 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575737+0000", 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027415300000000002 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575785+0000", 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7970999999999999e-05 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:12.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575832+0000", 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7120999999999998e-05 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575901+0000", 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8901999999999999e-05 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:192 4.5 4:a2ace8b3:::data_log.0:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:29:12.385 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 337.74678646199999, 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031080779999999998, 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.386 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615827+0000", 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1629999999999999e-06 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615831+0000", 2024-09-21T08:29:12.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0790000000000002e-06 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615841+0000", 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8279999999999998e-06 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615847+0000", 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2660000000000001e-06 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615879+0000", 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2512999999999999e-05 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615924+0000", 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4709999999999997e-05 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.618934+0000", 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030105190000000001 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1084 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94837804042368 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 274.20299511500002, 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40449157200000002, 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1084 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159622+0000", 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1400000000000002e-06 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159631+0000", 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9670000000000007e-06 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159655+0000", 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4037999999999999e-05 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159667+0000", 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2427e-05 2024-09-21T08:29:12.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159756+0000", 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9103000000000004e-05 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159772+0000", 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5877000000000001e-05 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.554979+0000", 2024-09-21T08:29:12.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39520667999999998 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.555262+0000", 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028320500000000002 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.560260+0000", 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049979630000000002 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.560331+0000", 2024-09-21T08:29:12.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1048000000000001e-05 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.564109+0000", 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037781239999999999 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1085 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94837808833664 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 274.20285588299998, 2024-09-21T08:29:12.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40437364799999997, 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1085 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159758+0000", 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.539e-06 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159764+0000", 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1330000000000004e-06 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159773+0000", 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3250000000000008e-06 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159779+0000", 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5400000000000001e-06 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159844+0000", 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4580999999999997e-05 2024-09-21T08:29:12.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159854+0000", 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.893e-06 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552880+0000", 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.393026184 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.553195+0000", 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031526299999999998 2024-09-21T08:29:12.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559454+0000", 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062587809999999997 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559541+0000", 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7379999999999993e-05 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.564130+0000", 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045890289999999997 2024-09-21T08:29:12.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:265 7.b e93/80 7:dd97af7e:::smithi13779221-30:head v 93'26, mlcod=86'23)", 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 265.79886082899998, 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036488330000000002, 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563753+0000", 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4000000000000001e-07 2024-09-21T08:29:12.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565430+0000", 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016774439999999999 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565446+0000", 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5540999999999999e-05 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565455+0000", 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9580000000000002e-06 2024-09-21T08:29:12.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565513+0000", 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7312000000000003e-05 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565542+0000", 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8856999999999998e-05 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.567363+0000", 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018212300000000001 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.567401+0000", 2024-09-21T08:29:12.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7851000000000001e-05 2024-09-21T08:29:12.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.405 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.405 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.405 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:321 7.c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 6] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:29:12.405 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 265.55298546400002, 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013109218000000001, 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 321 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809632+0000", 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9559999999999996e-06 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.407 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809648+0000", 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5814000000000001e-05 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809680+0000", 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1732e-05 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809696+0000", 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6293000000000001e-05 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809815+0000", 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119442 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809903+0000", 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7033999999999994e-05 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.815434+0000", 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055318420000000004 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.820591+0000", 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051568229999999996 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.821125+0000", 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053413400000000002 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822656+0000", 2024-09-21T08:29:12.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001531035 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822697+0000", 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0488999999999998e-05 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822736+0000", 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9623999999999999e-05 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:322 7.c 7:3cae00e7:::smithi13779221-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 265.55288721199997, 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093384190000000006, 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 322 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809727+0000", 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.13e-06 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809734+0000", 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9569999999999996e-06 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809744+0000", 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0633e-05 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809753+0000", 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7090000000000008e-06 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.810046+0000", 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029301499999999999 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.810079+0000", 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3293000000000001e-05 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.818920+0000", 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088405299999999992 2024-09-21T08:29:12.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.818982+0000", 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999000000000001e-05 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.819064+0000", 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2152999999999998e-05 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.416 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:329 7.c 7:361ee62b:::smithi13779221-40:head [sparse-read 0~1272916,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 264.67643435999997, 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90863698400000004, 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 329 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:29:12.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686182+0000", 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0459999999999999e-06 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686196+0000", 2024-09-21T08:29:12.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4097000000000001e-05 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686225+0000", 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8846000000000001e-05 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686240+0000", 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4256e-05 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686362+0000", 2024-09-21T08:29:12.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012248200000000001 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686390+0000", 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7546e-05 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.591277+0000", 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90488744700000001 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.591416+0000", 2024-09-21T08:29:12.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013935900000000001 2024-09-21T08:29:12.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.594815+0000", 2024-09-21T08:29:12.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033989049999999998 2024-09-21T08:29:12.421 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:330 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2655473,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 264.67634542100001, 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91026091200000003, 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.422 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 330 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:29:12.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686269+0000", 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7060000000000001e-06 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686277+0000", 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4470000000000006e-06 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686294+0000", 2024-09-21T08:29:12.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6099999999999998e-05 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686308+0000", 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4516e-05 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686428+0000", 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119834 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686436+0000", 2024-09-21T08:29:12.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1450000000000008e-06 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.595025+0000", 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90858932599999997 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.595096+0000", 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0821999999999994e-05 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596528+0000", 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014320159999999999 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:331 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2655473,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 264.67628592900002, 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91109795900000001, 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.427 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 331 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686328+0000", 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.387e-06 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686336+0000", 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4030000000000001e-06 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686350+0000", 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4713e-05 2024-09-21T08:29:12.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686364+0000", 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3716999999999999e-05 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686455+0000", 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0737999999999998e-05 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686460+0000", 2024-09-21T08:29:12.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1290000000000004e-06 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596614+0000", 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91015405299999996 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596687+0000", 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2700000000000005e-05 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.597425+0000", 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00073811899999999997 2024-09-21T08:29:12.431 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:338 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 263.794780989, 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038704049999999999, 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:12.432 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 338 2024-09-21T08:29:12.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.433 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:29:12.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:29:12.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:12.434 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:29:12.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567835+0000", 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0019999999999998e-06 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567843+0000", 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0090000000000007e-06 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567861+0000", 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8748000000000001e-05 2024-09-21T08:29:12.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567872+0000", 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0173000000000001e-05 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567975+0000", 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103398 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568085+0000", 2024-09-21T08:29:12.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109691 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571702+0000", 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003617384 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:339 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 263.79470588999999, 2024-09-21T08:29:12.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054334199999999996, 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 339 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567908+0000", 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-06 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567913+0000", 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.536e-06 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567922+0000", 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0730000000000005e-06 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567928+0000", 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4489999999999996e-06 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571741+0000", 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038125590000000001 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571859+0000", 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118249 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.573340+0000", 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001481472 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:340 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:29:12.442 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 263.79450736500002, 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082399729999999994, 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 340 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:29:12.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568107+0000", 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1850000000000001e-06 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568112+0000", 2024-09-21T08:29:12.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8019999999999997e-06 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568122+0000", 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6970000000000005e-06 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568129+0000", 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2119999999999997e-06 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.575080+0000", 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069511850000000003 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.575168+0000", 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7613000000000003e-05 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.576345+0000", 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001177279 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.446 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.7 95/94 [PushReplyOp(7:e1536981:::smithi13779221-32:head)])", 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 263.70592187, 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030062309999999998, 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:29:12.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656692+0000", 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0690000000000001e-06 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656696+0000", 2024-09-21T08:29:12.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3590000000000001e-06 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656705+0000", 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5639999999999997e-06 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656710+0000", 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656749+0000", 2024-09-21T08:29:12.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9026999999999998e-05 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.659697+0000", 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029482829999999999 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.c 95/94 [PushReplyOp(7:3cae00e7:::smithi13779221-28:5)])", 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 263.68448784200001, 2024-09-21T08:29:12.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028000960000000002, 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678126+0000", 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.327e-06 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678131+0000", 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4920000000000004e-06 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678141+0000", 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9839999999999996e-06 2024-09-21T08:29:12.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678146+0000", 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.446e-06 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678184+0000", 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8259000000000001e-05 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.680925+0000", 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002740588 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.453 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.c 95/94 [PushReplyOp(7:3cae00e7:::smithi13779221-28:head)])", 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 263.67418479700001, 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027171439999999999, 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688429+0000", 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688434+0000", 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999996e-06 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688443+0000", 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4769999999999992e-06 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688448+0000", 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3499999999999996e-06 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688491+0000", 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2216999999999999e-05 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.691145+0000", 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026543550000000002 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:353 7.c 7:361ee62b:::smithi13779221-40:head [copy-from ver 7] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 262.46032124700002, 2024-09-21T08:29:12.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014212783, 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 353 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902296+0000", 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1509999999999997e-06 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902323+0000", 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7491000000000002e-05 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902342+0000", 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8522999999999999e-05 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902350+0000", 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8119999999999997e-06 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902448+0000", 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7867000000000003e-05 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902544+0000", 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5612999999999994e-05 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.907790+0000", 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052465360000000004 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.913037+0000", 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052469129999999998 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916372+0000", 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003334914 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916404+0000", 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1327e-05 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916459+0000", 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4917000000000003e-05 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916504+0000", 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5719000000000001e-05 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:354 7.c 7:361ee62b:::smithi13779221-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 262.46024030199999, 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089418770000000009, 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:12.463 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 354 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:12.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902373+0000", 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6499999999999998e-07 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902376+0000", 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.616e-06 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:12.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902402+0000", 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5890000000000001e-05 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902407+0000", 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0019999999999997e-06 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902704+0000", 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029720499999999998 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:29:12.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902740+0000", 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6383e-05 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911116+0000", 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.008375136 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911208+0000", 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2791e-05 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:12.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911314+0000", 2024-09-21T08:29:12.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105989 2024-09-21T08:29:12.468 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.468 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.468 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.468 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:12.468 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:12.468 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:29:12.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:12.470 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:12.470 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:12.571 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:12.595 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:12.595 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:12.596 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:12.613 INFO:tasks.rados.rados.0.smithi137.stdout:1393: done (7 left) 2024-09-21T08:29:12.613 INFO:tasks.rados.rados.0.smithi137.stdout:1394: done (6 left) 2024-09-21T08:29:12.613 INFO:tasks.rados.rados.0.smithi137.stdout:1395: done (5 left) 2024-09-21T08:29:12.614 INFO:tasks.rados.rados.0.smithi137.stdout:1396: done (4 left) 2024-09-21T08:29:12.614 INFO:tasks.rados.rados.0.smithi137.stdout:1397: done (3 left) 2024-09-21T08:29:12.614 INFO:tasks.rados.rados.0.smithi137.stdout:1398: done (2 left) 2024-09-21T08:29:12.614 INFO:tasks.rados.rados.0.smithi137.stdout:1399: done (1 left) 2024-09-21T08:29:12.614 INFO:tasks.rados.rados.0.smithi137.stdout:1400: done (0 left) 2024-09-21T08:29:12.614 INFO:tasks.rados.rados.0.smithi137.stdout:1401: read oid 6 snap 137 2024-09-21T08:29:12.615 INFO:tasks.rados.rados.0.smithi137.stdout:1401: expect (ObjNum 489 snap 135 seq_num 489) 2024-09-21T08:29:12.615 INFO:tasks.rados.rados.0.smithi137.stdout:1402: snap_remove snap 133 2024-09-21T08:29:12.685 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:12.696 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:12.720 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:12.720 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:12.720 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:12.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:12 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:29:12.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:12 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:29:12.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:12 smithi137 ceph-mon[29634]: 6.d scrub starts 2024-09-21T08:29:12.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:12 smithi137 ceph-mon[29634]: 6.d scrub ok 2024-09-21T08:29:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:12 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:29:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:12 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:29:12.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:12 smithi137 ceph-mon[41044]: 6.d scrub starts 2024-09-21T08:29:12.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:12 smithi137 ceph-mon[41044]: 6.d scrub ok 2024-09-21T08:29:12.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:12.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:12.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:12.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:12.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:12.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:12.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:12.967 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:13.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:12 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:29:13.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:12 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:29:13.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:12 smithi165 ceph-mon[32893]: 6.d scrub starts 2024-09-21T08:29:13.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:12 smithi165 ceph-mon[32893]: 6.d scrub ok 2024-09-21T08:29:13.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:13.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:13.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:13.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:13.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:13.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:13.213 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:13.214 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:13.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:13.336 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:13.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:13.337 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:13.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T08:29:13.438 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:13.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:13.461 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:13.461 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:13.562 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:13.584 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:13.585 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:13.585 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:13.609 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:29:13.614 INFO:tasks.rados.rados.0.smithi137.stdout:1402: done (1 left) 2024-09-21T08:29:13.615 INFO:tasks.rados.rados.0.smithi137.stdout:1403: write (excl) oid 2 current snap is 142 2024-09-21T08:29:13.615 INFO:tasks.rados.rados.0.smithi137.stdout:1403: seq_num 508 ranges {537956=705790,1644029=772160} 2024-09-21T08:29:13.622 INFO:tasks.rados.rados.0.smithi137.stdout:1403: writing smithi13779221-2 from 537956 to 1243746 tid 1 2024-09-21T08:29:13.631 INFO:tasks.rados.rados.0.smithi137.stdout:1403: writing smithi13779221-2 from 1644029 to 2416189 tid 2 2024-09-21T08:29:13.631 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:13.632 INFO:tasks.rados.rados.0.smithi137.stdout:1401: expect (ObjNum 489 snap 135 seq_num 489) 2024-09-21T08:29:13.686 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:13.707 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:13.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:13.708 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:13.738 INFO:tasks.rados.rados.0.smithi137.stdout:1403: finishing write tid 1 to smithi13779221-2 2024-09-21T08:29:13.738 INFO:tasks.rados.rados.0.smithi137.stdout:1401: done (1 left) 2024-09-21T08:29:13.738 INFO:tasks.rados.rados.0.smithi137.stdout:1404: read oid 4 snap -1 2024-09-21T08:29:13.738 INFO:tasks.rados.rados.0.smithi137.stdout:1404: expect (ObjNum 462 snap 124 seq_num 462) 2024-09-21T08:29:13.738 INFO:tasks.rados.rados.0.smithi137.stdout:1405: write (excl) oid 17 current snap is 142 2024-09-21T08:29:13.738 INFO:tasks.rados.rados.0.smithi137.stdout:1403: finishing write tid 2 to smithi13779221-2 2024-09-21T08:29:13.738 INFO:tasks.rados.rados.0.smithi137.stdout:1403: finishing write tid 3 to smithi13779221-2 2024-09-21T08:29:13.738 INFO:tasks.rados.rados.0.smithi137.stdout:1403: finishing write tid 4 to smithi13779221-2 2024-09-21T08:29:13.739 INFO:tasks.rados.rados.0.smithi137.stdout:1403: oid 2 updating version 0 to 161 2024-09-21T08:29:13.739 INFO:tasks.rados.rados.0.smithi137.stdout:1403: oid 2 version 161 is already newer than 160 2024-09-21T08:29:13.739 INFO:tasks.rados.rados.0.smithi137.stdout:1403: oid 2 updating version 161 to 162 2024-09-21T08:29:13.739 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 162 (ObjNum 508 snap 142 seq_num 508) dirty exists 2024-09-21T08:29:13.739 INFO:tasks.rados.rados.0.smithi137.stdout:1403: left oid 2 (ObjNum 508 snap 142 seq_num 508) 2024-09-21T08:29:13.739 INFO:tasks.rados.rados.0.smithi137.stdout:1405: seq_num 509 ranges {554763=781039,2102511=452252} 2024-09-21T08:29:13.745 INFO:tasks.rados.rados.0.smithi137.stdout:1405: writing smithi13779221-17 from 554763 to 1335802 tid 1 2024-09-21T08:29:13.752 INFO:tasks.rados.rados.0.smithi137.stdout:1405: writing smithi13779221-17 from 2102511 to 2554763 tid 2 2024-09-21T08:29:13.752 INFO:tasks.rados.rados.0.smithi137.stdout:1403: done (2 left) 2024-09-21T08:29:13.752 INFO:tasks.rados.rados.0.smithi137.stdout:1406: read oid 23 snap -1 2024-09-21T08:29:13.752 INFO:tasks.rados.rados.0.smithi137.stdout:1406: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:29:13.752 INFO:tasks.rados.rados.0.smithi137.stdout:1407: snap_remove snap 140 2024-09-21T08:29:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:29:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:29:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[29634]: pgmap v443: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 321 MiB data, 915 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 4.0 MiB/s wr, 24 op/s; 267/1056 objects degraded (25.284%) 2024-09-21T08:29:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[29634]: osdmap e359: 8 total, 7 up, 4 in 2024-09-21T08:29:13.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1990371286' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:29:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:29:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[41044]: pgmap v443: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 321 MiB data, 915 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 4.0 MiB/s wr, 24 op/s; 267/1056 objects degraded (25.284%) 2024-09-21T08:29:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[41044]: osdmap e359: 8 total, 7 up, 4 in 2024-09-21T08:29:13.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:13 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1990371286' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:13.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:13.830 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:13.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:13.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:13.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:13.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:13.955 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:13.955 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:13.967 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:13 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:13.968 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:13 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:29:13.968 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:13 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:29:13.968 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:13 smithi165 ceph-mon[32893]: pgmap v443: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 321 MiB data, 915 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 4.0 MiB/s wr, 24 op/s; 267/1056 objects degraded (25.284%) 2024-09-21T08:29:13.968 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:13 smithi165 ceph-mon[32893]: osdmap e359: 8 total, 7 up, 4 in 2024-09-21T08:29:13.968 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:13 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1990371286' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:14.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:29:14.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:14.077 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:14.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:14.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:14.179 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:14.200 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:14.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:14.201 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:14.291 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T08:29:14.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:14.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:14.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:14.326 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:14.426 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:14.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:14.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:14.449 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:14.550 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:14.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:14.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:14.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:14.631 INFO:tasks.rados.rados.0.smithi137.stdout:1407: done (3 left) 2024-09-21T08:29:14.631 INFO:tasks.rados.rados.0.smithi137.stdout:1408: copy_from oid 3 from oid 40 current snap is 142 2024-09-21T08:29:14.632 INFO:tasks.rados.rados.0.smithi137.stdout:1409: delete oid 49 current snap is 142 2024-09-21T08:29:14.632 INFO:tasks.rados.rados.0.smithi137.stdout:1405: finishing write tid 1 to smithi13779221-17 2024-09-21T08:29:14.632 INFO:tasks.rados.rados.0.smithi137.stdout:1404: expect (ObjNum 462 snap 124 seq_num 462) 2024-09-21T08:29:14.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:14.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:14.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:14.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:14.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:14 smithi137 ceph-mon[29634]: 7.4 scrub starts 2024-09-21T08:29:14.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:14 smithi137 ceph-mon[29634]: 7.4 scrub ok 2024-09-21T08:29:14.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:14 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:29:14.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:14 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1990371286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:14.741 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:14 smithi137 ceph-mon[29634]: osdmap e360: 8 total, 7 up, 4 in 2024-09-21T08:29:14.741 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:14 smithi137 ceph-mon[41044]: 7.4 scrub starts 2024-09-21T08:29:14.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:14 smithi137 ceph-mon[41044]: 7.4 scrub ok 2024-09-21T08:29:14.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:14 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:29:14.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:14 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1990371286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:14.742 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:14 smithi137 ceph-mon[41044]: osdmap e360: 8 total, 7 up, 4 in 2024-09-21T08:29:14.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:14.813 INFO:tasks.rados.rados.0.smithi137.stdout:1405: finishing write tid 2 to smithi13779221-17 2024-09-21T08:29:14.813 INFO:tasks.rados.rados.0.smithi137.stdout:1405: finishing write tid 3 to smithi13779221-17 2024-09-21T08:29:14.813 INFO:tasks.rados.rados.0.smithi137.stdout:1405: finishing write tid 4 to smithi13779221-17 2024-09-21T08:29:14.813 INFO:tasks.rados.rados.0.smithi137.stdout:1405: oid 17 updating version 0 to 322 2024-09-21T08:29:14.813 INFO:tasks.rados.rados.0.smithi137.stdout:1405: oid 17 updating version 322 to 324 2024-09-21T08:29:14.813 INFO:tasks.rados.rados.0.smithi137.stdout:1405: oid 17 version 324 is already newer than 323 2024-09-21T08:29:14.813 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 324 (ObjNum 509 snap 142 seq_num 509) dirty exists 2024-09-21T08:29:14.814 INFO:tasks.rados.rados.0.smithi137.stdout:1405: left oid 17 (ObjNum 509 snap 142 seq_num 509) 2024-09-21T08:29:14.814 INFO:tasks.rados.rados.0.smithi137.stdout:1406: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:29:14.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:14.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:14.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:14.848 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:14.848 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:29:14.849 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:29:14.849 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:29:14.849 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:29:14.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:14.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:14.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:14.942 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:14 smithi165 ceph-mon[32893]: 7.4 scrub starts 2024-09-21T08:29:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:14 smithi165 ceph-mon[32893]: 7.4 scrub ok 2024-09-21T08:29:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:14 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:29:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:14 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1990371286' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:14 smithi165 ceph-mon[32893]: osdmap e360: 8 total, 7 up, 4 in 2024-09-21T08:29:15.040 INFO:tasks.rados.rados.0.smithi137.stdout:1408: finishing copy_from racing read to smithi13779221-3 2024-09-21T08:29:15.041 INFO:tasks.rados.rados.0.smithi137.stdout:1408: finishing copy_from to smithi13779221-3 2024-09-21T08:29:15.041 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 321 (ObjNum 287 snap 66 seq_num 287) dirty exists 2024-09-21T08:29:15.041 INFO:tasks.rados.rados.0.smithi137.stdout:1404: done (4 left) 2024-09-21T08:29:15.041 INFO:tasks.rados.rados.0.smithi137.stdout:1405: done (3 left) 2024-09-21T08:29:15.041 INFO:tasks.rados.rados.0.smithi137.stdout:1406: done (2 left) 2024-09-21T08:29:15.041 INFO:tasks.rados.rados.0.smithi137.stdout:1408: done (1 left) 2024-09-21T08:29:15.041 INFO:tasks.rados.rados.0.smithi137.stdout:1409: done (0 left) 2024-09-21T08:29:15.041 INFO:tasks.rados.rados.0.smithi137.stdout:1410: rollback oid 35 current snap is 142 2024-09-21T08:29:15.041 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 35 to 132 2024-09-21T08:29:15.041 INFO:tasks.rados.rados.0.smithi137.stdout:1411: snap_remove snap 112 2024-09-21T08:29:15.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:15.065 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:15.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:15.065 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:15.166 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:15.188 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:15.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:15.189 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:15.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:15.312 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:15.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:15.313 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:15.320 INFO:teuthology.orchestra.run.smithi165.stdout:get_pg_num_history pg_num_history pg_num_history(e338 pg_nums {1={19=1},2={64=3},3={65=32},4={67=32},5={69=32},6={71=32},7={80=16,181=26,259=36}} deleted_pools ) 2024-09-21T08:29:15.320 INFO:teuthology.orchestra.run.smithi165.stdout:Importing pgid 3.2 2024-09-21T08:29:15.320 INFO:teuthology.orchestra.run.smithi165.stdout:write_pg epoch 332 info 3.2( empty local-lis/les=290/291 n=0 ec=65/65 lis/c=290/290 les/c/f=291/291/0 sis=290) 2024-09-21T08:29:15.320 INFO:teuthology.orchestra.run.smithi165.stdout:Import successful 2024-09-21T08:29:15.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:15.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:15.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:15.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:15.470 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:29:15.470 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:29:15.486 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:15.487 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:15.502 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:15.502 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:15.518 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:15.518 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:15.534 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:15.534 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:15.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:15.551 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:15.551 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:15.558 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:15.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:15.559 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:15.568 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:15.569 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:15.569 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:15.634 INFO:tasks.rados.rados.0.smithi137.stdout:1411: done (1 left) 2024-09-21T08:29:15.635 INFO:tasks.rados.rados.0.smithi137.stdout:1412: read oid 28 snap -1 2024-09-21T08:29:15.635 INFO:tasks.rados.rados.0.smithi137.stdout:1412: expect (ObjNum 464 snap 125 seq_num 464) 2024-09-21T08:29:15.635 INFO:tasks.rados.rados.0.smithi137.stdout:1413: delete oid 8 current snap is 142 2024-09-21T08:29:15.635 INFO:tasks.rados.rados.0.smithi137.stdout:1410: finishing rollback tid 1 to smithi13779221-35 2024-09-21T08:29:15.635 INFO:tasks.rados.rados.0.smithi137.stdout:1410: finishing rollback tid 2 to smithi13779221-35 2024-09-21T08:29:15.635 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 326 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:29:15.647 INFO:tasks.rados.rados.0.smithi137.stdout:1412: expect (ObjNum 464 snap 125 seq_num 464) 2024-09-21T08:29:15.659 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:15.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:15.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:15.681 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:15 smithi137 ceph-mon[29634]: pgmap v446: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 321 MiB data, 915 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 4.0 MiB/s wr, 24 op/s; 267/1056 objects degraded (25.284%) 2024-09-21T08:29:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:15 smithi137 ceph-mon[29634]: osdmap e361: 8 total, 7 up, 4 in 2024-09-21T08:29:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:15 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:15 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3676531990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:15 smithi137 ceph-mon[41044]: pgmap v446: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 321 MiB data, 915 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 4.0 MiB/s wr, 24 op/s; 267/1056 objects degraded (25.284%) 2024-09-21T08:29:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:15 smithi137 ceph-mon[41044]: osdmap e361: 8 total, 7 up, 4 in 2024-09-21T08:29:15.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:15 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:15.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:15 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3676531990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:15.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:15.805 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:15.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:15.806 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:15.898 INFO:tasks.rados.rados.0.smithi137.stdout:1410: done (2 left) 2024-09-21T08:29:15.898 INFO:tasks.rados.rados.0.smithi137.stdout:1412: done (1 left) 2024-09-21T08:29:15.898 INFO:tasks.rados.rados.0.smithi137.stdout:1413: done (0 left) 2024-09-21T08:29:15.898 INFO:tasks.rados.rados.0.smithi137.stdout:1414: rollback oid 27 current snap is 142 2024-09-21T08:29:15.898 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 27 to 132 2024-09-21T08:29:15.899 INFO:tasks.rados.rados.0.smithi137.stdout:1415: copy_from oid 47 from oid 9 current snap is 142 2024-09-21T08:29:15.899 INFO:tasks.rados.rados.0.smithi137.stdout:1416: snap_remove snap 103 2024-09-21T08:29:15.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:15.929 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:15.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:15.930 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:15 smithi165 ceph-mon[32893]: pgmap v446: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 321 MiB data, 915 MiB used, 357 GiB / 358 GiB avail; 22 MiB/s rd, 4.0 MiB/s wr, 24 op/s; 267/1056 objects degraded (25.284%) 2024-09-21T08:29:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:15 smithi165 ceph-mon[32893]: osdmap e361: 8 total, 7 up, 4 in 2024-09-21T08:29:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:15 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:15 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3676531990' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:16.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:16.054 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:16.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:16.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:16.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:16.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:16.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:16.179 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:16.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_blocked_ops 2024-09-21T08:29:16.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:16.305 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:16.305 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:16.306 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:16.406 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:16.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /var/log/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/exp.3.2.7 2024-09-21T08:29:16.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:16.429 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:16.430 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:16.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:29:16.531 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:16.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:16.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:16.554 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:16.641 INFO:tasks.rados.rados.0.smithi137.stdout:1416: done (2 left) 2024-09-21T08:29:16.641 INFO:tasks.rados.rados.0.smithi137.stdout:1417: read oid 7 snap -1 2024-09-21T08:29:16.641 INFO:tasks.rados.rados.0.smithi137.stdout:1417: expect (ObjNum 443 snap 114 seq_num 443) 2024-09-21T08:29:16.641 INFO:tasks.rados.rados.0.smithi137.stdout:1418: snap_create 2024-09-21T08:29:16.641 INFO:tasks.rados.rados.0.smithi137.stdout:1415: finishing copy_from to smithi13779221-47 2024-09-21T08:29:16.641 INFO:tasks.rados.rados.0.smithi137.stdout:1415: got expected ENOENT (src dne) 2024-09-21T08:29:16.641 INFO:tasks.rados.rados.0.smithi137.stdout:1415: finishing copy_from racing read to smithi13779221-47 2024-09-21T08:29:16.641 INFO:tasks.rados.rados.0.smithi137.stdout:1414: finishing rollback tid 1 to smithi13779221-27 2024-09-21T08:29:16.642 INFO:tasks.rados.rados.0.smithi137.stdout:1414: finishing rollback tid 2 to smithi13779221-27 2024-09-21T08:29:16.642 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 302 (ObjNum 384 snap 94 seq_num 384) dirty exists 2024-09-21T08:29:16.651 INFO:tasks.rados.rados.0.smithi137.stdout:1417: expect (ObjNum 443 snap 114 seq_num 443) 2024-09-21T08:29:16.654 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:16.675 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:16.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:16.675 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:16.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:16 smithi165 ceph-mon[32893]: 4.1d scrub starts 2024-09-21T08:29:16.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:16 smithi165 ceph-mon[32893]: 4.1d scrub ok 2024-09-21T08:29:16.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:16 smithi165 ceph-mon[32893]: osdmap e362: 8 total, 7 up, 4 in 2024-09-21T08:29:16.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:16 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 267/1056 objects degraded (25.284%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:16.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:16 smithi137 ceph-mon[29634]: 4.1d scrub starts 2024-09-21T08:29:16.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:16 smithi137 ceph-mon[29634]: 4.1d scrub ok 2024-09-21T08:29:16.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:16 smithi137 ceph-mon[29634]: osdmap e362: 8 total, 7 up, 4 in 2024-09-21T08:29:16.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:16 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 267/1056 objects degraded (25.284%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:16.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:16 smithi137 ceph-mon[41044]: 4.1d scrub starts 2024-09-21T08:29:16.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:16 smithi137 ceph-mon[41044]: 4.1d scrub ok 2024-09-21T08:29:16.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:16 smithi137 ceph-mon[41044]: osdmap e362: 8 total, 7 up, 4 in 2024-09-21T08:29:16.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:16 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 267/1056 objects degraded (25.284%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:16.777 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:16.799 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:16.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:16.799 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:16.900 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:16.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:16.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:16.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:17.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:17.033 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:17.033 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:17.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:17.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:17.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:17.109 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:17.146 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:17.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:17.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:17.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:17.171 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:17.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:17.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:17.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:17.294 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:17.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:17.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:17.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:17.418 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:17.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:17.539 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:17.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:17.540 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:17.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:17.643 INFO:tasks.rados.rados.0.smithi137.stdout:1414: done (3 left) 2024-09-21T08:29:17.643 INFO:tasks.rados.rados.0.smithi137.stdout:1415: done (2 left) 2024-09-21T08:29:17.643 INFO:tasks.rados.rados.0.smithi137.stdout:1417: done (1 left) 2024-09-21T08:29:17.643 INFO:tasks.rados.rados.0.smithi137.stdout:1418: done (0 left) 2024-09-21T08:29:17.643 INFO:tasks.rados.rados.0.smithi137.stdout:1419: write oid 41 current snap is 143 2024-09-21T08:29:17.643 INFO:tasks.rados.rados.0.smithi137.stdout:1419: seq_num 512 ranges {605184=723139,1877957=727227} 2024-09-21T08:29:17.651 INFO:tasks.rados.rados.0.smithi137.stdout:1419: writing smithi13779221-41 from 605184 to 1328323 tid 1 2024-09-21T08:29:17.659 INFO:tasks.rados.rados.0.smithi137.stdout:1419: writing smithi13779221-41 from 1877957 to 2605184 tid 2 2024-09-21T08:29:17.659 INFO:tasks.rados.rados.0.smithi137.stdout:1420: snap_create 2024-09-21T08:29:17.661 INFO:tasks.rados.rados.0.smithi137.stdout:1419: finishing write tid 1 to smithi13779221-41 2024-09-21T08:29:17.661 INFO:tasks.rados.rados.0.smithi137.stdout:1419: finishing write tid 2 to smithi13779221-41 2024-09-21T08:29:17.661 INFO:tasks.rados.rados.0.smithi137.stdout:1419: finishing write tid 3 to smithi13779221-41 2024-09-21T08:29:17.662 INFO:tasks.rados.rados.0.smithi137.stdout:1419: finishing write tid 4 to smithi13779221-41 2024-09-21T08:29:17.662 INFO:tasks.rados.rados.0.smithi137.stdout:1419: oid 41 updating version 0 to 368 2024-09-21T08:29:17.662 INFO:tasks.rados.rados.0.smithi137.stdout:1419: oid 41 version 368 is already newer than 367 2024-09-21T08:29:17.662 INFO:tasks.rados.rados.0.smithi137.stdout:1419: oid 41 updating version 368 to 369 2024-09-21T08:29:17.662 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 369 (ObjNum 512 snap 143 seq_num 512) dirty exists 2024-09-21T08:29:17.662 INFO:tasks.rados.rados.0.smithi137.stdout:1419: left oid 41 (ObjNum 512 snap 143 seq_num 512) 2024-09-21T08:29:17.666 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:17.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:17.667 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:17.668 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:17.668 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:17.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:17.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:17.794 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:17.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:17.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:17.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:17.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:17.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:17.965 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:17 smithi137 ceph-mon[29634]: pgmap v449: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 340 MiB data, 939 MiB used, 357 GiB / 358 GiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 25 op/s; 278/1092 objects degraded (25.458%) 2024-09-21T08:29:17.965 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:17 smithi137 ceph-mon[29634]: osdmap e363: 8 total, 7 up, 4 in 2024-09-21T08:29:17.965 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:17 smithi137 ceph-mon[41044]: pgmap v449: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 340 MiB data, 939 MiB used, 357 GiB / 358 GiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 25 op/s; 278/1092 objects degraded (25.458%) 2024-09-21T08:29:17.965 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:17 smithi137 ceph-mon[41044]: osdmap e363: 8 total, 7 up, 4 in 2024-09-21T08:29:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:17 smithi165 ceph-mon[32893]: pgmap v449: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 340 MiB data, 939 MiB used, 357 GiB / 358 GiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 25 op/s; 278/1092 objects degraded (25.458%) 2024-09-21T08:29:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:17 smithi165 ceph-mon[32893]: osdmap e363: 8 total, 7 up, 4 in 2024-09-21T08:29:18.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:18.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:18.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:18.043 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:18.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:18.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:18.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:18.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:18.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_historic_ops 2024-09-21T08:29:18.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:18.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:18.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:18.307 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:18.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:18.430 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:18.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:18.431 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:18.532 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:18.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:18.559 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:18.559 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:18.657 INFO:tasks.rados.rados.0.smithi137.stdout:1419: done (1 left) 2024-09-21T08:29:18.657 INFO:tasks.rados.rados.0.smithi137.stdout:1420: done (0 left) 2024-09-21T08:29:18.658 INFO:tasks.rados.rados.0.smithi137.stdout:1421: read oid 23 snap -1 2024-09-21T08:29:18.658 INFO:tasks.rados.rados.0.smithi137.stdout:1421: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:29:18.658 INFO:tasks.rados.rados.0.smithi137.stdout:1422: read oid 8 snap -1 2024-09-21T08:29:18.658 INFO:tasks.rados.rados.0.smithi137.stdout:1422: expect deleted 2024-09-21T08:29:18.658 INFO:tasks.rados.rados.0.smithi137.stdout:1423: snap_create 2024-09-21T08:29:18.660 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:18.675 INFO:tasks.rados.rados.0.smithi137.stdout:1421: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:29:18.682 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:18.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:18.684 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:18.716 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:18.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:18 smithi137 ceph-mon[29634]: osdmap e364: 8 total, 7 up, 4 in 2024-09-21T08:29:18.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:18 smithi137 ceph-mon[41044]: osdmap e364: 8 total, 7 up, 4 in 2024-09-21T08:29:18.785 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:18.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:18.806 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:18.807 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:18.907 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:18.933 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:18.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:18.934 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:19.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:18 smithi165 ceph-mon[32893]: osdmap e364: 8 total, 7 up, 4 in 2024-09-21T08:29:19.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:29:19.035 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:19.061 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:19.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:19.062 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:19.163 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:19.184 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:19.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:19.185 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:19.232 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:19.232 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:19.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:19.307 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:19.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:19.309 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:19.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:19.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:19.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:19.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:19.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:19.562 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:19.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:19.563 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:19.659 INFO:tasks.rados.rados.0.smithi137.stdout:1421: done (2 left) 2024-09-21T08:29:19.659 INFO:tasks.rados.rados.0.smithi137.stdout:1422: done (1 left) 2024-09-21T08:29:19.660 INFO:tasks.rados.rados.0.smithi137.stdout:1423: done (0 left) 2024-09-21T08:29:19.660 INFO:tasks.rados.rados.0.smithi137.stdout:1424: snap_remove snap 118 2024-09-21T08:29:19.660 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:29:19.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:19.686 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:19.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:19.687 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:19.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:19.810 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:19.810 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:19.810 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:19.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:19.918 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:19 smithi137 ceph-mon[29634]: pgmap v452: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 340 MiB data, 939 MiB used, 357 GiB / 358 GiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 25 op/s; 278/1092 objects degraded (25.458%) 2024-09-21T08:29:19.918 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:19 smithi137 ceph-mon[29634]: osdmap e365: 8 total, 7 up, 4 in 2024-09-21T08:29:19.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:19 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1212414011' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:19.919 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:19 smithi137 ceph-mon[29634]: from='client.45548 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:19.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:19 smithi137 ceph-mon[41044]: pgmap v452: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 340 MiB data, 939 MiB used, 357 GiB / 358 GiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 25 op/s; 278/1092 objects degraded (25.458%) 2024-09-21T08:29:19.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:19 smithi137 ceph-mon[41044]: osdmap e365: 8 total, 7 up, 4 in 2024-09-21T08:29:19.919 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:19 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1212414011' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:19.920 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:19 smithi137 ceph-mon[41044]: from='client.45548 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:19.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:19.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:19.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:19.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T08:29:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:19 smithi165 ceph-mon[32893]: pgmap v452: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 340 MiB data, 939 MiB used, 357 GiB / 358 GiB avail; 18 MiB/s rd, 2.7 MiB/s wr, 25 op/s; 278/1092 objects degraded (25.458%) 2024-09-21T08:29:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:19 smithi165 ceph-mon[32893]: osdmap e365: 8 total, 7 up, 4 in 2024-09-21T08:29:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:19 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1212414011' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:20.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:19 smithi165 ceph-mon[32893]: from='client.45548 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:20.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:20.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:20.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:20.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:20.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:20.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:20.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:20.190 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:20.282 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:20.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:20.317 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:20.317 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:20.318 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:20.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:20.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:20.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:20.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:20.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:20.567 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:20.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:20.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:20.664 INFO:tasks.rados.rados.0.smithi137.stdout:1424: done (0 left) 2024-09-21T08:29:20.664 INFO:tasks.rados.rados.0.smithi137.stdout:1425: read oid 26 snap -1 2024-09-21T08:29:20.664 INFO:tasks.rados.rados.0.smithi137.stdout:1425: expect (ObjNum 503 snap 141 seq_num 503) 2024-09-21T08:29:20.664 INFO:tasks.rados.rados.0.smithi137.stdout:1426: read oid 13 snap -1 2024-09-21T08:29:20.664 INFO:tasks.rados.rados.0.smithi137.stdout:1426: expect deleted 2024-09-21T08:29:20.664 INFO:tasks.rados.rados.0.smithi137.stdout:1427: copy_from oid 17 from oid 46 current snap is 145 2024-09-21T08:29:20.664 INFO:tasks.rados.rados.0.smithi137.stdout:1428: delete oid 14 current snap is 145 2024-09-21T08:29:20.669 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:20.681 INFO:tasks.rados.rados.0.smithi137.stdout:1426: done (3 left) 2024-09-21T08:29:20.682 INFO:tasks.rados.rados.0.smithi137.stdout:1428: done (2 left) 2024-09-21T08:29:20.682 INFO:tasks.rados.rados.0.smithi137.stdout:1429: copy_from oid 48 from oid 11 current snap is 145 2024-09-21T08:29:20.682 INFO:tasks.rados.rados.0.smithi137.stdout:1430: read oid 38 snap -1 2024-09-21T08:29:20.682 INFO:tasks.rados.rados.0.smithi137.stdout:1430: expect (ObjNum 490 snap 135 seq_num 490) 2024-09-21T08:29:20.682 INFO:tasks.rados.rados.0.smithi137.stdout:1431: delete oid 15 current snap is 145 2024-09-21T08:29:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1425: expect (ObjNum 503 snap 141 seq_num 503) 2024-09-21T08:29:20.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:20.700 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:20.700 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:20.763 INFO:tasks.rados.rados.0.smithi137.stdout:1427: finishing copy_from racing read to smithi13779221-17 2024-09-21T08:29:20.763 INFO:tasks.rados.rados.0.smithi137.stdout:1427: finishing copy_from to smithi13779221-17 2024-09-21T08:29:20.763 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 495 (ObjNum 403 snap 99 seq_num 403) dirty exists 2024-09-21T08:29:20.763 INFO:tasks.rados.rados.0.smithi137.stdout:1425: done (4 left) 2024-09-21T08:29:20.763 INFO:tasks.rados.rados.0.smithi137.stdout:1427: done (3 left) 2024-09-21T08:29:20.763 INFO:tasks.rados.rados.0.smithi137.stdout:1431: done (2 left) 2024-09-21T08:29:20.763 INFO:tasks.rados.rados.0.smithi137.stdout:1432: rollback oid 3 current snap is 145 2024-09-21T08:29:20.763 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 3 to 143 2024-09-21T08:29:20.764 INFO:tasks.rados.rados.0.smithi137.stdout:1430: expect (ObjNum 490 snap 135 seq_num 490) 2024-09-21T08:29:20.801 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:20.832 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:20.832 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:20.832 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:20.839 INFO:tasks.rados.rados.0.smithi137.stdout:1429: finishing copy_from racing read to smithi13779221-48 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:1429: finishing copy_from to smithi13779221-48 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 518 (ObjNum 497 snap 137 seq_num 497) dirty exists 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:1429: done (2 left) 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:1430: done (1 left) 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:1433: rollback oid 49 current snap is 145 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 49 to 137 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:1432: finishing rollback tid 0 to smithi13779221-3 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:1432: finishing rollback tid 1 to smithi13779221-3 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:1432: finishing rollback tid 2 to smithi13779221-3 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 324 (ObjNum 287 snap 66 seq_num 287) dirty exists 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:1432: done (1 left) 2024-09-21T08:29:20.840 INFO:tasks.rados.rados.0.smithi137.stdout:1434: snap_remove snap 128 2024-09-21T08:29:20.931 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:20 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:20.932 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:20 smithi137 ceph-mon[29634]: from='client.45548 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:29:20.932 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:20 smithi137 ceph-mon[29634]: osdmap e366: 8 total, 7 up, 4 in 2024-09-21T08:29:20.932 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:20 smithi137 ceph-mon[29634]: osdmap e367: 8 total, 7 up, 4 in 2024-09-21T08:29:20.932 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:29:20 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:29:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:29:20.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:20 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:20.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:20 smithi137 ceph-mon[41044]: from='client.45548 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:29:20.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:20 smithi137 ceph-mon[41044]: osdmap e366: 8 total, 7 up, 4 in 2024-09-21T08:29:20.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:20 smithi137 ceph-mon[41044]: osdmap e367: 8 total, 7 up, 4 in 2024-09-21T08:29:20.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:29:20.957 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:20.958 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:20.958 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:29:21.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:20 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:21.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:20 smithi165 ceph-mon[32893]: from='client.45548 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:29:21.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:20 smithi165 ceph-mon[32893]: osdmap e366: 8 total, 7 up, 4 in 2024-09-21T08:29:21.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:20 smithi165 ceph-mon[32893]: osdmap e367: 8 total, 7 up, 4 in 2024-09-21T08:29:21.058 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:21.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:21.088 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:21.088 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:21.189 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:21.216 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:21.217 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:21.217 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:21.298 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:29:21.299 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:29:21.318 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:21.318 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:21.318 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:21.333 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:21.333 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:21.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:21.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:21.343 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:21.351 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:21.351 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:21.358 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:21.369 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:21.369 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:21.390 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:21.391 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:21.407 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:21.407 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:21.408 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:21.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:21.455 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1] out_osds: [4, 6, 3, 0] dead_osds: [7] live_osds: [0, 1, 2, 3, 4, 5, 6] 2024-09-21T08:29:21.455 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:29:21.455 INFO:tasks.thrashosds.thrasher:Killing osd 0, live_osds are [0, 1, 2, 3, 4, 5, 6] 2024-09-21T08:29:21.456 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-09-21T08:29:21.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl stop ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.0 2024-09-21T08:29:21.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:21.467 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:21.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:21.562 INFO:tasks.rados.rados.0.smithi137.stdout:1434: done (1 left) 2024-09-21T08:29:21.562 INFO:tasks.rados.rados.0.smithi137.stdout:1435: snap_create 2024-09-21T08:29:21.562 INFO:tasks.rados.rados.0.smithi137.stdout:1433: finishing rollback tid 1 to smithi13779221-49 2024-09-21T08:29:21.563 INFO:tasks.rados.rados.0.smithi137.stdout:1433: finishing rollback tid 2 to smithi13779221-49 2024-09-21T08:29:21.563 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 428 (ObjNum 496 snap 137 seq_num 496) dirty exists 2024-09-21T08:29:21.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:21.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:21.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:21.594 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:21.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:21.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:21.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:21.717 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:21.768 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:21 smithi137 systemd[1]: Stopping Ceph osd.0 for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:29:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:21 smithi137 ceph-mon[29634]: pgmap v455: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 340 MiB data, 939 MiB used, 357 GiB / 358 GiB avail; 278/1092 objects degraded (25.458%) 2024-09-21T08:29:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:21 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2236392750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:21 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 278/1092 objects degraded (25.458%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:21 smithi137 ceph-mon[29634]: osdmap e368: 8 total, 7 up, 4 in 2024-09-21T08:29:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:21 smithi137 ceph-mon[41044]: pgmap v455: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 340 MiB data, 939 MiB used, 357 GiB / 358 GiB avail; 278/1092 objects degraded (25.458%) 2024-09-21T08:29:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:21 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2236392750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:21 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 278/1092 objects degraded (25.458%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:21.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:21 smithi137 ceph-mon[41044]: osdmap e368: 8 total, 7 up, 4 in 2024-09-21T08:29:21.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:21.845 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:21.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:21.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:21.946 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:21.972 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:21.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:21.973 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:22.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:29:21 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:29:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:21 smithi165 ceph-mon[32893]: pgmap v455: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 340 MiB data, 939 MiB used, 357 GiB / 358 GiB avail; 278/1092 objects degraded (25.458%) 2024-09-21T08:29:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:21 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2236392750' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:21 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 278/1092 objects degraded (25.458%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:22.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:21 smithi165 ceph-mon[32893]: osdmap e368: 8 total, 7 up, 4 in 2024-09-21T08:29:22.074 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:22.095 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:22.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:22.096 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:22.135 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:29:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:29:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:29:22.136 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:29:22.197 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:22.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:22.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:22.219 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:22.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:22.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:22.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:22.347 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:22.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:22.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:22.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:22.473 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:22.561 INFO:tasks.rados.rados.0.smithi137.stdout:1433: done (1 left) 2024-09-21T08:29:22.561 INFO:tasks.rados.rados.0.smithi137.stdout:1435: done (0 left) 2024-09-21T08:29:22.561 INFO:tasks.rados.rados.0.smithi137.stdout:1436: snap_remove snap 141 2024-09-21T08:29:22.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:22.598 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:22.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:22.599 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:22.700 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:22.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:22.723 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:22.723 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:22.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:22 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:29:22.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:22 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:29:22.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:22 smithi137 ceph-mon[29634]: osdmap e369: 8 total, 7 up, 4 in 2024-09-21T08:29:22.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:22 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:29:22.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:22 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:29:22.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:22 smithi137 ceph-mon[41044]: osdmap e369: 8 total, 7 up, 4 in 2024-09-21T08:29:22.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:29:22.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:22.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T08:29:22.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:22.884 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:22.884 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:22.985 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:23.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:23.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:23.008 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:22 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:29:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:22 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:29:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:22 smithi165 ceph-mon[32893]: osdmap e369: 8 total, 7 up, 4 in 2024-09-21T08:29:23.110 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:23.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:23.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:23.136 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:23.237 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:23.262 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:23.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:23.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:23.288 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:23.289 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:23.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:23.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:23.391 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:23.392 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:23.424 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:23.428 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:23.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:23.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:23.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:23.515 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:23.517 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0[45185]: 2024-09-21T08:29:23.391+0000 7fd247815640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-09-21T08:29:23.517 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0[45185]: 2024-09-21T08:29:23.391+0000 7fd247815640 -1 osd.0 369 *** Got signal Terminated *** 2024-09-21T08:29:23.518 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0[45185]: 2024-09-21T08:29:23.391+0000 7fd247815640 -1 osd.0 369 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-09-21T08:29:23.566 INFO:tasks.rados.rados.0.smithi137.stdout:1436: done (0 left) 2024-09-21T08:29:23.566 INFO:tasks.rados.rados.0.smithi137.stdout:1437: delete oid 35 current snap is 146 2024-09-21T08:29:23.575 INFO:tasks.rados.rados.0.smithi137.stdout:1437: done (0 left) 2024-09-21T08:29:23.575 INFO:tasks.rados.rados.0.smithi137.stdout:1438: rollback oid 30 current snap is 146 2024-09-21T08:29:23.575 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 30 to 134 2024-09-21T08:29:23.576 INFO:tasks.rados.rados.0.smithi137.stdout:1439: write oid 50 current snap is 146 2024-09-21T08:29:23.576 INFO:tasks.rados.rados.0.smithi137.stdout:1439: seq_num 516 ranges {672412=624135,1995532=676880} 2024-09-21T08:29:23.582 INFO:tasks.rados.rados.0.smithi137.stdout:1439: writing smithi13779221-50 from 672412 to 1296547 tid 1 2024-09-21T08:29:23.590 INFO:tasks.rados.rados.0.smithi137.stdout:1439: writing smithi13779221-50 from 1995532 to 2672412 tid 2 2024-09-21T08:29:23.590 INFO:tasks.rados.rados.0.smithi137.stdout:1440: write oid 10 current snap is 146 2024-09-21T08:29:23.590 INFO:tasks.rados.rados.0.smithi137.stdout:1440: seq_num 517 ranges {689219=415737,1770367=646847} 2024-09-21T08:29:23.596 INFO:tasks.rados.rados.0.smithi137.stdout:1440: writing smithi13779221-10 from 689219 to 1104956 tid 1 2024-09-21T08:29:23.603 INFO:tasks.rados.rados.0.smithi137.stdout:1440: writing smithi13779221-10 from 1770367 to 2417214 tid 2 2024-09-21T08:29:23.603 INFO:tasks.rados.rados.0.smithi137.stdout:1441: snap_create 2024-09-21T08:29:23.603 INFO:tasks.rados.rados.0.smithi137.stdout:1438: finishing rollback tid 0 to smithi13779221-30 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1438: finishing rollback tid 1 to smithi13779221-30 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1438: finishing rollback tid 2 to smithi13779221-30 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 589 (ObjNum 445 snap 115 seq_num 445) dirty exists 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1439: finishing write tid 1 to smithi13779221-50 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1439: finishing write tid 2 to smithi13779221-50 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1439: finishing write tid 3 to smithi13779221-50 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1439: finishing write tid 4 to smithi13779221-50 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1439: oid 50 updating version 0 to 382 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1439: oid 50 version 382 is already newer than 381 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1439: oid 50 version 382 is already newer than 380 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 382 (ObjNum 516 snap 146 seq_num 516) dirty exists 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1439: left oid 50 (ObjNum 516 snap 146 seq_num 516) 2024-09-21T08:29:23.604 INFO:tasks.rados.rados.0.smithi137.stdout:1440: finishing write tid 1 to smithi13779221-10 2024-09-21T08:29:23.605 INFO:tasks.rados.rados.0.smithi137.stdout:1440: finishing write tid 2 to smithi13779221-10 2024-09-21T08:29:23.605 INFO:tasks.rados.rados.0.smithi137.stdout:1440: finishing write tid 3 to smithi13779221-10 2024-09-21T08:29:23.606 INFO:tasks.rados.rados.0.smithi137.stdout:1440: finishing write tid 4 to smithi13779221-10 2024-09-21T08:29:23.606 INFO:tasks.rados.rados.0.smithi137.stdout:1440: oid 10 updating version 0 to 520 2024-09-21T08:29:23.606 INFO:tasks.rados.rados.0.smithi137.stdout:1440: oid 10 updating version 520 to 521 2024-09-21T08:29:23.606 INFO:tasks.rados.rados.0.smithi137.stdout:1440: oid 10 version 521 is already newer than 519 2024-09-21T08:29:23.606 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 521 (ObjNum 517 snap 146 seq_num 517) dirty exists 2024-09-21T08:29:23.606 INFO:tasks.rados.rados.0.smithi137.stdout:1440: left oid 10 (ObjNum 517 snap 146 seq_num 517) 2024-09-21T08:29:23.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:23.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:23.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:23.637 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:23.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:23.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:23.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:23.764 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:23.826 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:23.826 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[29634]: pgmap v458: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 343 MiB data, 928 MiB used, 357 GiB / 358 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 282/1098 objects degraded (25.683%) 2024-09-21T08:29:23.826 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:29:23.826 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:29:23.826 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[29634]: osd.0 marked itself down and dead 2024-09-21T08:29:23.826 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[29634]: osdmap e370: 8 total, 6 up, 4 in 2024-09-21T08:29:23.827 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:23.827 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[41044]: pgmap v458: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 343 MiB data, 928 MiB used, 357 GiB / 358 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 282/1098 objects degraded (25.683%) 2024-09-21T08:29:23.827 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:29:23.827 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:29:23.827 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[41044]: osd.0 marked itself down and dead 2024-09-21T08:29:23.827 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:23 smithi137 ceph-mon[41044]: osdmap e370: 8 total, 6 up, 4 in 2024-09-21T08:29:23.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:23.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:23.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:23.892 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:23.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:23 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:23 smithi165 ceph-mon[32893]: pgmap v458: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 343 MiB data, 928 MiB used, 357 GiB / 358 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 282/1098 objects degraded (25.683%) 2024-09-21T08:29:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:23 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:29:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:23 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:29:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:23 smithi165 ceph-mon[32893]: osd.0 marked itself down and dead 2024-09-21T08:29:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:23 smithi165 ceph-mon[32893]: osdmap e370: 8 total, 6 up, 4 in 2024-09-21T08:29:24.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:24.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:24.016 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:24.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:24.139 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:24.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:24.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:24.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:24.262 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:24.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:24.263 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:24.267 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:24 smithi137 podman[129890]: 2024-09-21 08:29:24.046911181 +0000 UTC m=+0.775098150 container died 32840677591df8ca940eadd5a402a5f5c709aa166abab6a6982a3d8376f00200 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240731, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, ceph=True, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD) 2024-09-21T08:29:24.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:24.390 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:24.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:24.390 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:24.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:24.514 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:24.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:24.515 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:24.562 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:29:24.567 INFO:tasks.rados.rados.0.smithi137.stdout:1438: done (3 left) 2024-09-21T08:29:24.567 INFO:tasks.rados.rados.0.smithi137.stdout:1439: done (2 left) 2024-09-21T08:29:24.568 INFO:tasks.rados.rados.0.smithi137.stdout:1440: done (1 left) 2024-09-21T08:29:24.568 INFO:tasks.rados.rados.0.smithi137.stdout:1441: done (0 left) 2024-09-21T08:29:24.568 INFO:tasks.rados.rados.0.smithi137.stdout:1442: snap_create 2024-09-21T08:29:24.576 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:24 smithi137 podman[129890]: 2024-09-21 08:29:24.471179146 +0000 UTC m=+1.199366144 container cleanup 32840677591df8ca940eadd5a402a5f5c709aa166abab6a6982a3d8376f00200 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=squid-9e2ac70, io.buildah.version=1.37.2, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2) 2024-09-21T08:29:24.576 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:24 smithi137 bash[129890]: ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0 2024-09-21T08:29:24.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:24.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:24.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:24.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:24.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:24.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:24.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:24.761 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:24.862 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:24.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:24.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:24.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:24.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:25.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:25.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:25.012 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:24 smithi165 ceph-mon[32893]: pgmap v461: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 343 MiB data, 928 MiB used, 357 GiB / 358 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 282/1098 objects degraded (25.683%) 2024-09-21T08:29:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:24 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1529465006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:24 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1529465006' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:24 smithi165 ceph-mon[32893]: osdmap e371: 8 total, 6 up, 4 in 2024-09-21T08:29:25.017 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:24 smithi137 podman[130088]: 2024-09-21 08:29:24.706323523 +0000 UTC m=+0.659489630 container remove 32840677591df8ca940eadd5a402a5f5c709aa166abab6a6982a3d8376f00200 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T08:29:25.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:24 smithi137 ceph-mon[29634]: pgmap v461: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 343 MiB data, 928 MiB used, 357 GiB / 358 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 282/1098 objects degraded (25.683%) 2024-09-21T08:29:25.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:24 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1529465006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:25.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:24 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1529465006' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:25.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:24 smithi137 ceph-mon[29634]: osdmap e371: 8 total, 6 up, 4 in 2024-09-21T08:29:25.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:24 smithi137 ceph-mon[41044]: pgmap v461: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 343 MiB data, 928 MiB used, 357 GiB / 358 GiB avail; 12 MiB/s rd, 1.6 MiB/s wr, 13 op/s; 282/1098 objects degraded (25.683%) 2024-09-21T08:29:25.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:24 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1529465006' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:25.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:24 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1529465006' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:25.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:24 smithi137 ceph-mon[41044]: osdmap e371: 8 total, 6 up, 4 in 2024-09-21T08:29:25.114 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:25.139 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:25.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:25.141 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:25.242 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:25.267 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:25.268 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:25.268 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:25.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:29:25.369 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:25.394 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:25.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:25.395 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:25.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:25.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:25.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:25.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:25.568 INFO:tasks.rados.rados.0.smithi137.stdout:1442: done (0 left) 2024-09-21T08:29:25.568 INFO:tasks.rados.rados.0.smithi137.stdout:1443: delete oid 40 current snap is 148 2024-09-21T08:29:25.577 INFO:tasks.rados.rados.0.smithi137.stdout:1443: done (0 left) 2024-09-21T08:29:25.577 INFO:tasks.rados.rados.0.smithi137.stdout:1444: snap_remove snap 129 2024-09-21T08:29:25.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:25.642 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:25.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:25.643 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:25.728 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:29:25.728 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:29:25.728 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:29:25.728 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:29:25.729 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:29:25.744 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:25.767 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:25.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:25.768 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:25.869 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:25.893 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:25.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:25.894 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:25.947 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:25.995 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:26.017 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:26.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:26.018 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:26.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:26.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:26.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:26.141 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:26.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:26.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:26.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:26.264 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:26.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:26.386 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:26.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:26.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:26.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:26.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:26.512 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:26.512 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:26.516 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T08:29:26.571 INFO:tasks.rados.rados.0.smithi137.stdout:1444: done (0 left) 2024-09-21T08:29:26.571 INFO:tasks.rados.rados.0.smithi137.stdout:1445: snap_create 2024-09-21T08:29:26.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:26.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:26.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:26.640 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:26.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:26 smithi137 ceph-mon[29634]: osdmap e372: 8 total, 6 up, 4 in 2024-09-21T08:29:26.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:26 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:26.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:26 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 282/1098 objects degraded (25.683%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:26.735 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:26 smithi137 ceph-mon[41044]: osdmap e372: 8 total, 6 up, 4 in 2024-09-21T08:29:26.735 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:26 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:26.735 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:26 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 282/1098 objects degraded (25.683%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:26.741 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:26.764 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:26.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:26.765 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:26.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:26.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:26.888 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:26.888 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:26.970 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:29:26.970 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:26.971 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:26.989 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:26.992 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:26.992 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:27.010 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:27.011 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:27.011 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:27.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:27.012 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:26 smithi165 ceph-mon[32893]: osdmap e372: 8 total, 6 up, 4 in 2024-09-21T08:29:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:26 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:26 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 282/1098 objects degraded (25.683%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:27.028 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:27.029 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:27.046 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:27.047 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:27.050 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:27.072 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:27.073 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:27.073 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:27.113 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:27.134 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:27.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:27.135 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:27.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:27.262 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:27.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:27.263 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:27.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:27.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:27.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:27.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:27.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:29:27.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:27.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:27.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:27.517 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:27.572 INFO:tasks.rados.rados.0.smithi137.stdout:1445: done (0 left) 2024-09-21T08:29:27.572 INFO:tasks.rados.rados.0.smithi137.stdout:1446: rollback oid 39 current snap is 149 2024-09-21T08:29:27.572 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 39 to 132 2024-09-21T08:29:27.572 INFO:tasks.rados.rados.0.smithi137.stdout:1447: copy_from oid 28 from oid 41 current snap is 149 2024-09-21T08:29:27.572 INFO:tasks.rados.rados.0.smithi137.stdout:1448: write oid 31 current snap is 149 2024-09-21T08:29:27.572 INFO:tasks.rados.rados.0.smithi137.stdout:1448: seq_num 519 ranges {722833=566235,1887331=650516} 2024-09-21T08:29:27.582 INFO:tasks.rados.rados.0.smithi137.stdout:1448: writing smithi13779221-31 from 722833 to 1289068 tid 1 2024-09-21T08:29:27.590 INFO:tasks.rados.rados.0.smithi137.stdout:1448: writing smithi13779221-31 from 1887331 to 2537847 tid 2 2024-09-21T08:29:27.591 INFO:tasks.rados.rados.0.smithi137.stdout:1449: read oid 38 snap -1 2024-09-21T08:29:27.591 INFO:tasks.rados.rados.0.smithi137.stdout:1449: expect (ObjNum 490 snap 135 seq_num 490) 2024-09-21T08:29:27.591 INFO:tasks.rados.rados.0.smithi137.stdout:1448: finishing write tid 1 to smithi13779221-31 2024-09-21T08:29:27.591 INFO:tasks.rados.rados.0.smithi137.stdout:1450: write oid 29 current snap is 149 2024-09-21T08:29:27.591 INFO:tasks.rados.rados.0.smithi137.stdout:1450: seq_num 520 ranges {739640=641484,1829460=794174} 2024-09-21T08:29:27.598 INFO:tasks.rados.rados.0.smithi137.stdout:1450: writing smithi13779221-29 from 739640 to 1381124 tid 1 2024-09-21T08:29:27.605 INFO:tasks.rados.rados.0.smithi137.stdout:1450: writing smithi13779221-29 from 1829460 to 2623634 tid 2 2024-09-21T08:29:27.606 INFO:tasks.rados.rados.0.smithi137.stdout:1451: copy_from oid 32 from oid 47 current snap is 149 2024-09-21T08:29:27.606 INFO:tasks.rados.rados.0.smithi137.stdout:1446: finishing rollback tid 1 to smithi13779221-39 2024-09-21T08:29:27.606 INFO:tasks.rados.rados.0.smithi137.stdout:1446: finishing rollback tid 2 to smithi13779221-39 2024-09-21T08:29:27.606 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 231 (ObjNum 477 snap 131 seq_num 477) dirty exists 2024-09-21T08:29:27.606 INFO:tasks.rados.rados.0.smithi137.stdout:1449: expect (ObjNum 490 snap 135 seq_num 490) 2024-09-21T08:29:27.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:27.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:27.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:27.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:27.675 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:27 smithi137 ceph-mon[29634]: pgmap v464: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 346 MiB data, 932 MiB used, 357 GiB / 358 GiB avail; 1.7 KiB/s rd, 577 KiB/s wr, 5 op/s; 286/1113 objects degraded (25.696%) 2024-09-21T08:29:27.675 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:27 smithi137 ceph-mon[29634]: osdmap e373: 8 total, 6 up, 4 in 2024-09-21T08:29:27.675 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:27 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/675036422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:27.675 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:27 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:29:27.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:27 smithi137 ceph-mon[41044]: pgmap v464: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 346 MiB data, 932 MiB used, 357 GiB / 358 GiB avail; 1.7 KiB/s rd, 577 KiB/s wr, 5 op/s; 286/1113 objects degraded (25.696%) 2024-09-21T08:29:27.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:27 smithi137 ceph-mon[41044]: osdmap e373: 8 total, 6 up, 4 in 2024-09-21T08:29:27.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:27 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/675036422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:27.675 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:27 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:29:27.685 INFO:tasks.rados.rados.0.smithi137.stdout:1448: finishing write tid 2 to smithi13779221-31 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1448: finishing write tid 3 to smithi13779221-31 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1448: finishing write tid 4 to smithi13779221-31 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1448: oid 31 updating version 0 to 338 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1448: oid 31 version 338 is already newer than 337 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1448: oid 31 version 338 is already newer than 336 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 338 (ObjNum 519 snap 149 seq_num 519) dirty exists 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1448: left oid 31 (ObjNum 519 snap 149 seq_num 519) 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1447: finishing copy_from racing read to smithi13779221-28 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1446: done (5 left) 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1448: done (4 left) 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1449: done (3 left) 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1452: write oid 6 current snap is 149 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:1447: finishing copy_from to smithi13779221-28 2024-09-21T08:29:27.686 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 369 (ObjNum 512 snap 143 seq_num 512) dirty exists 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:1450: finishing write tid 1 to smithi13779221-29 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:1450: finishing write tid 2 to smithi13779221-29 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:1450: finishing write tid 3 to smithi13779221-29 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:1450: finishing write tid 4 to smithi13779221-29 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:1450: oid 29 updating version 0 to 571 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:1450: oid 29 version 571 is already newer than 569 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:1450: oid 29 version 571 is already newer than 570 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 571 (ObjNum 520 snap 149 seq_num 520) dirty exists 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:1450: left oid 29 (ObjNum 520 snap 149 seq_num 520) 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:1451: finishing copy_from racing read to smithi13779221-32 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:1451: finishing copy_from to smithi13779221-32 2024-09-21T08:29:27.687 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 309 (ObjNum 467 snap 126 seq_num 467) dirty exists 2024-09-21T08:29:27.688 INFO:tasks.rados.rados.0.smithi137.stdout:1452: seq_num 521 ranges {756447=716733,1887942=537832} 2024-09-21T08:29:27.695 INFO:tasks.rados.rados.0.smithi137.stdout:1452: writing smithi13779221-6 from 756447 to 1473180 tid 1 2024-09-21T08:29:27.702 INFO:tasks.rados.rados.0.smithi137.stdout:1452: writing smithi13779221-6 from 1887942 to 2425774 tid 2 2024-09-21T08:29:27.702 INFO:tasks.rados.rados.0.smithi137.stdout:1447: done (3 left) 2024-09-21T08:29:27.702 INFO:tasks.rados.rados.0.smithi137.stdout:1450: done (2 left) 2024-09-21T08:29:27.702 INFO:tasks.rados.rados.0.smithi137.stdout:1451: done (1 left) 2024-09-21T08:29:27.703 INFO:tasks.rados.rados.0.smithi137.stdout:1453: snap_remove snap 147 2024-09-21T08:29:27.716 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:27.716 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:27.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:27.763 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:27.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:27.764 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:27.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:27.886 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:27.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:27.887 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:27.988 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:28.009 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:28.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:28.011 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:27 smithi165 ceph-mon[32893]: pgmap v464: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 346 MiB data, 932 MiB used, 357 GiB / 358 GiB avail; 1.7 KiB/s rd, 577 KiB/s wr, 5 op/s; 286/1113 objects degraded (25.696%) 2024-09-21T08:29:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:27 smithi165 ceph-mon[32893]: osdmap e373: 8 total, 6 up, 4 in 2024-09-21T08:29:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:27 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/675036422' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:27 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:29:28.111 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:28.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:28.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:28.134 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:28.144 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:28.144 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:28.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:28.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:28.258 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:28.259 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:28.359 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:28.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:28.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:28.382 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:28.483 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:28.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:28.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:28.505 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:28.595 INFO:tasks.rados.rados.0.smithi137.stdout:1453: done (1 left) 2024-09-21T08:29:28.595 INFO:tasks.rados.rados.0.smithi137.stdout:1454: write oid 40 current snap is 149 2024-09-21T08:29:28.595 INFO:tasks.rados.rados.0.smithi137.stdout:1452: finishing write tid 1 to smithi13779221-6 2024-09-21T08:29:28.595 INFO:tasks.rados.rados.0.smithi137.stdout:1454: seq_num 522 ranges {773254=791982,2346424=426830} 2024-09-21T08:29:28.604 INFO:tasks.rados.rados.0.smithi137.stdout:1454: writing smithi13779221-40 from 773254 to 1565236 tid 1 2024-09-21T08:29:28.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:28.611 INFO:tasks.rados.rados.0.smithi137.stdout:1454: writing smithi13779221-40 from 2346424 to 2773254 tid 2 2024-09-21T08:29:28.611 INFO:tasks.rados.rados.0.smithi137.stdout:1455: delete oid 41 current snap is 149 2024-09-21T08:29:28.611 INFO:tasks.rados.rados.0.smithi137.stdout:1452: finishing write tid 2 to smithi13779221-6 2024-09-21T08:29:28.611 INFO:tasks.rados.rados.0.smithi137.stdout:1452: finishing write tid 3 to smithi13779221-6 2024-09-21T08:29:28.611 INFO:tasks.rados.rados.0.smithi137.stdout:1452: finishing write tid 4 to smithi13779221-6 2024-09-21T08:29:28.611 INFO:tasks.rados.rados.0.smithi137.stdout:1452: oid 6 updating version 0 to 315 2024-09-21T08:29:28.611 INFO:tasks.rados.rados.0.smithi137.stdout:1452: oid 6 version 315 is already newer than 313 2024-09-21T08:29:28.611 INFO:tasks.rados.rados.0.smithi137.stdout:1452: oid 6 version 315 is already newer than 314 2024-09-21T08:29:28.611 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 315 (ObjNum 521 snap 149 seq_num 521) dirty exists 2024-09-21T08:29:28.612 INFO:tasks.rados.rados.0.smithi137.stdout:1452: left oid 6 (ObjNum 521 snap 149 seq_num 521) 2024-09-21T08:29:28.615 INFO:tasks.rados.rados.0.smithi137.stdout:1452: done (2 left) 2024-09-21T08:29:28.615 INFO:tasks.rados.rados.0.smithi137.stdout:1455: done (1 left) 2024-09-21T08:29:28.615 INFO:tasks.rados.rados.0.smithi137.stdout:1456: write oid 1 current snap is 149 2024-09-21T08:29:28.616 INFO:tasks.rados.rados.0.smithi137.stdout:1454: finishing write tid 1 to smithi13779221-40 2024-09-21T08:29:28.616 INFO:tasks.rados.rados.0.smithi137.stdout:1456: seq_num 524 ranges {406868=542480,1547035=401512,2690354=116514} 2024-09-21T08:29:28.620 INFO:tasks.rados.rados.0.smithi137.stdout:1456: writing smithi13779221-1 from 406868 to 949348 tid 1 2024-09-21T08:29:28.626 INFO:tasks.rados.rados.0.smithi137.stdout:1456: writing smithi13779221-1 from 1547035 to 1948547 tid 2 2024-09-21T08:29:28.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:28.629 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:28.630 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:28.632 INFO:tasks.rados.rados.0.smithi137.stdout:1456: writing smithi13779221-1 from 2690354 to 2806868 tid 3 2024-09-21T08:29:28.632 INFO:tasks.rados.rados.0.smithi137.stdout:1457: read oid 3 snap 136 2024-09-21T08:29:28.632 INFO:tasks.rados.rados.0.smithi137.stdout:1457: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:29:28.632 INFO:tasks.rados.rados.0.smithi137.stdout:1458: delete oid 4 current snap is 149 2024-09-21T08:29:28.632 INFO:tasks.rados.rados.0.smithi137.stdout:1454: finishing write tid 2 to smithi13779221-40 2024-09-21T08:29:28.632 INFO:tasks.rados.rados.0.smithi137.stdout:1454: finishing write tid 3 to smithi13779221-40 2024-09-21T08:29:28.632 INFO:tasks.rados.rados.0.smithi137.stdout:1454: finishing write tid 4 to smithi13779221-40 2024-09-21T08:29:28.633 INFO:tasks.rados.rados.0.smithi137.stdout:1454: oid 40 updating version 0 to 323 2024-09-21T08:29:28.633 INFO:tasks.rados.rados.0.smithi137.stdout:1454: oid 40 updating version 323 to 325 2024-09-21T08:29:28.633 INFO:tasks.rados.rados.0.smithi137.stdout:1454: oid 40 version 325 is already newer than 324 2024-09-21T08:29:28.633 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 325 (ObjNum 522 snap 149 seq_num 522) dirty exists 2024-09-21T08:29:28.633 INFO:tasks.rados.rados.0.smithi137.stdout:1454: left oid 40 (ObjNum 522 snap 149 seq_num 522) 2024-09-21T08:29:28.633 INFO:tasks.rados.rados.0.smithi137.stdout:1456: finishing write tid 1 to smithi13779221-1 2024-09-21T08:29:28.633 INFO:tasks.rados.rados.0.smithi137.stdout:1456: finishing write tid 2 to smithi13779221-1 2024-09-21T08:29:28.634 INFO:tasks.rados.rados.0.smithi137.stdout:1456: finishing write tid 3 to smithi13779221-1 2024-09-21T08:29:28.634 INFO:tasks.rados.rados.0.smithi137.stdout:1454: done (3 left) 2024-09-21T08:29:28.635 INFO:tasks.rados.rados.0.smithi137.stdout:1458: done (2 left) 2024-09-21T08:29:28.635 INFO:tasks.rados.rados.0.smithi137.stdout:1459: write oid 39 current snap is 149 2024-09-21T08:29:28.635 INFO:tasks.rados.rados.0.smithi137.stdout:1459: seq_num 526 ranges {440482=409331,1380352=405181,2461553=378929} 2024-09-21T08:29:28.638 INFO:tasks.rados.rados.0.smithi137.stdout:1459: writing smithi13779221-39 from 440482 to 849813 tid 1 2024-09-21T08:29:28.643 INFO:tasks.rados.rados.0.smithi137.stdout:1459: writing smithi13779221-39 from 1380352 to 1785533 tid 2 2024-09-21T08:29:28.650 INFO:tasks.rados.rados.0.smithi137.stdout:1459: writing smithi13779221-39 from 2461553 to 2840482 tid 3 2024-09-21T08:29:28.650 INFO:tasks.rados.rados.0.smithi137.stdout:1460: read oid 15 snap -1 2024-09-21T08:29:28.650 INFO:tasks.rados.rados.0.smithi137.stdout:1460: expect deleted 2024-09-21T08:29:28.650 INFO:tasks.rados.rados.0.smithi137.stdout:1461: write oid 43 current snap is 149 2024-09-21T08:29:28.650 INFO:tasks.rados.rados.0.smithi137.stdout:1456: finishing write tid 4 to smithi13779221-1 2024-09-21T08:29:28.650 INFO:tasks.rados.rados.0.smithi137.stdout:1456: finishing write tid 5 to smithi13779221-1 2024-09-21T08:29:28.650 INFO:tasks.rados.rados.0.smithi137.stdout:1456: oid 1 updating version 0 to 525 2024-09-21T08:29:28.650 INFO:tasks.rados.rados.0.smithi137.stdout:1456: oid 1 version 525 is already newer than 523 2024-09-21T08:29:28.651 INFO:tasks.rados.rados.0.smithi137.stdout:1456: oid 1 version 525 is already newer than 524 2024-09-21T08:29:28.651 INFO:tasks.rados.rados.0.smithi137.stdout:1456: oid 1 updating version 525 to 526 2024-09-21T08:29:28.651 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 526 (ObjNum 524 snap 149 seq_num 524) dirty exists 2024-09-21T08:29:28.651 INFO:tasks.rados.rados.0.smithi137.stdout:1456: left oid 1 (ObjNum 524 snap 149 seq_num 524) 2024-09-21T08:29:28.651 INFO:tasks.rados.rados.0.smithi137.stdout:1457: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:29:28.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:28.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:28.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:28.753 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:1459: finishing write tid 1 to smithi13779221-39 2024-09-21T08:29:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:1459: finishing write tid 2 to smithi13779221-39 2024-09-21T08:29:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:1459: finishing write tid 3 to smithi13779221-39 2024-09-21T08:29:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:1459: finishing write tid 4 to smithi13779221-39 2024-09-21T08:29:28.755 INFO:tasks.rados.rados.0.smithi137.stdout:1459: finishing write tid 5 to smithi13779221-39 2024-09-21T08:29:28.755 INFO:tasks.rados.rados.0.smithi137.stdout:1459: oid 39 updating version 0 to 235 2024-09-21T08:29:28.755 INFO:tasks.rados.rados.0.smithi137.stdout:1459: oid 39 version 235 is already newer than 234 2024-09-21T08:29:28.755 INFO:tasks.rados.rados.0.smithi137.stdout:1459: oid 39 version 235 is already newer than 233 2024-09-21T08:29:28.755 INFO:tasks.rados.rados.0.smithi137.stdout:1459: oid 39 version 235 is already newer than 232 2024-09-21T08:29:28.755 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 235 (ObjNum 526 snap 149 seq_num 526) dirty exists 2024-09-21T08:29:28.755 INFO:tasks.rados.rados.0.smithi137.stdout:1459: left oid 39 (ObjNum 526 snap 149 seq_num 526) 2024-09-21T08:29:28.755 INFO:tasks.rados.rados.0.smithi137.stdout:1461: seq_num 527 ranges {457289=484580,1438834=548839,2772623=84666} 2024-09-21T08:29:28.757 INFO:tasks.rados.rados.0.smithi137.stdout:1461: writing smithi13779221-43 from 457289 to 941869 tid 1 2024-09-21T08:29:28.763 INFO:tasks.rados.rados.0.smithi137.stdout:1461: writing smithi13779221-43 from 1438834 to 1987673 tid 2 2024-09-21T08:29:28.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[29634]: osdmap e374: 8 total, 6 up, 4 in 2024-09-21T08:29:28.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:28.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:28.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:29:28.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:29:28.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[41044]: osdmap e374: 8 total, 6 up, 4 in 2024-09-21T08:29:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:29:28.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:29:28.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:28 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:28.769 INFO:tasks.rados.rados.0.smithi137.stdout:1461: writing smithi13779221-43 from 2772623 to 2857289 tid 3 2024-09-21T08:29:28.769 INFO:tasks.rados.rados.0.smithi137.stdout:1456: done (4 left) 2024-09-21T08:29:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:1457: done (3 left) 2024-09-21T08:29:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:1459: done (2 left) 2024-09-21T08:29:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:1460: done (1 left) 2024-09-21T08:29:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:1462: delete oid 19 current snap is 149 2024-09-21T08:29:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:1461: finishing write tid 1 to smithi13779221-43 2024-09-21T08:29:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:1461: finishing write tid 2 to smithi13779221-43 2024-09-21T08:29:28.771 INFO:tasks.rados.rados.0.smithi137.stdout:1461: finishing write tid 3 to smithi13779221-43 2024-09-21T08:29:28.771 INFO:tasks.rados.rados.0.smithi137.stdout:1461: finishing write tid 4 to smithi13779221-43 2024-09-21T08:29:28.771 INFO:tasks.rados.rados.0.smithi137.stdout:1462: done (1 left) 2024-09-21T08:29:28.771 INFO:tasks.rados.rados.0.smithi137.stdout:1463: snap_create 2024-09-21T08:29:28.771 INFO:tasks.rados.rados.0.smithi137.stdout:1461: finishing write tid 5 to smithi13779221-43 2024-09-21T08:29:28.771 INFO:tasks.rados.rados.0.smithi137.stdout:1461: oid 43 updating version 0 to 371 2024-09-21T08:29:28.771 INFO:tasks.rados.rados.0.smithi137.stdout:1461: oid 43 updating version 371 to 372 2024-09-21T08:29:28.771 INFO:tasks.rados.rados.0.smithi137.stdout:1461: oid 43 updating version 372 to 373 2024-09-21T08:29:28.771 INFO:tasks.rados.rados.0.smithi137.stdout:1461: oid 43 updating version 373 to 374 2024-09-21T08:29:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 374 (ObjNum 527 snap 149 seq_num 527) dirty exists 2024-09-21T08:29:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:1461: left oid 43 (ObjNum 527 snap 149 seq_num 527) 2024-09-21T08:29:28.807 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:28.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T08:29:28.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:28.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:28.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:28.876 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:28.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:29.000 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:29.001 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:29.001 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:28 smithi165 ceph-mon[32893]: osdmap e374: 8 total, 6 up, 4 in 2024-09-21T08:29:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:28 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:28 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:28 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:29:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:28 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:29:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:28 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:29.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:29.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:29.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:29.127 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:29.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:29.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:29.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:29.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:29.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:29.373 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:29.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:29.375 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:29.389 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:29 smithi137 podman[131178]: 2024-09-21 08:29:29.032091757 +0000 UTC m=+0.198547905 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T08:29:29.389 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:29 smithi137 podman[131178]: 2024-09-21 08:29:29.177755631 +0000 UTC m=+0.344211772 container create a286dd75c50595f7208e7a05f8f49f540b545a3decaa0c752e78260e7839b1e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux ) 2024-09-21T08:29:29.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:29.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:29.497 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:29.497 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:29.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:29.600 INFO:tasks.rados.rados.0.smithi137.stdout:1461: done (1 left) 2024-09-21T08:29:29.600 INFO:tasks.rados.rados.0.smithi137.stdout:1463: done (0 left) 2024-09-21T08:29:29.600 INFO:tasks.rados.rados.0.smithi137.stdout:1464: read oid 18 snap 131 2024-09-21T08:29:29.600 INFO:tasks.rados.rados.0.smithi137.stdout:1464: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:29:29.600 INFO:tasks.rados.rados.0.smithi137.stdout:1465: write oid 7 current snap is 150 2024-09-21T08:29:29.600 INFO:tasks.rados.rados.0.smithi137.stdout:1465: seq_num 529 ranges {490903=635078,1839445=552508} 2024-09-21T08:29:29.606 INFO:tasks.rados.rados.0.smithi137.stdout:1465: writing smithi13779221-7 from 490903 to 1125981 tid 1 2024-09-21T08:29:29.614 INFO:tasks.rados.rados.0.smithi137.stdout:1465: writing smithi13779221-7 from 1839445 to 2391953 tid 2 2024-09-21T08:29:29.615 INFO:tasks.rados.rados.0.smithi137.stdout:1466: copy_from oid 6 from oid 26 current snap is 150 2024-09-21T08:29:29.615 INFO:tasks.rados.rados.0.smithi137.stdout:1467: read oid 21 snap -1 2024-09-21T08:29:29.615 INFO:tasks.rados.rados.0.smithi137.stdout:1467: expect (ObjNum 502 snap 141 seq_num 502) 2024-09-21T08:29:29.615 INFO:tasks.rados.rados.0.smithi137.stdout:1468: read oid 34 snap -1 2024-09-21T08:29:29.615 INFO:tasks.rados.rados.0.smithi137.stdout:1468: expect deleted 2024-09-21T08:29:29.615 INFO:tasks.rados.rados.0.smithi137.stdout:1469: rollback oid 14 current snap is 150 2024-09-21T08:29:29.615 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 14 to 136 2024-09-21T08:29:29.615 INFO:tasks.rados.rados.0.smithi137.stdout:1470: write oid 17 current snap is 150 2024-09-21T08:29:29.615 INFO:tasks.rados.rados.0.smithi137.stdout:1470: seq_num 530 ranges {507710=710327,1897927=412519,2854892=52818} 2024-09-21T08:29:29.622 INFO:tasks.rados.rados.0.smithi137.stdout:1470: writing smithi13779221-17 from 507710 to 1218037 tid 1 2024-09-21T08:29:29.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:29.623 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:29.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:29.629 INFO:tasks.rados.rados.0.smithi137.stdout:1470: writing smithi13779221-17 from 1897927 to 2310446 tid 2 2024-09-21T08:29:29.632 INFO:tasks.rados.rados.0.smithi137.stdout:1470: writing smithi13779221-17 from 2854892 to 2907710 tid 3 2024-09-21T08:29:29.633 INFO:tasks.rados.rados.0.smithi137.stdout:1471: copy_from oid 32 from oid 23 current snap is 150 2024-09-21T08:29:29.633 INFO:tasks.rados.rados.0.smithi137.stdout:1464: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:29:29.673 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:29 smithi137 ceph-mon[29634]: pgmap v467: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 346 MiB data, 932 MiB used, 357 GiB / 358 GiB avail; 1.7 KiB/s rd, 577 KiB/s wr, 5 op/s; 286/1113 objects degraded (25.696%) 2024-09-21T08:29:29.673 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:29 smithi137 ceph-mon[29634]: osdmap e375: 8 total, 6 up, 4 in 2024-09-21T08:29:29.673 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:29 smithi137 podman[131178]: 2024-09-21 08:29:29.565082005 +0000 UTC m=+0.731538144 container init a286dd75c50595f7208e7a05f8f49f540b545a3decaa0c752e78260e7839b1e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-deactivate, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-09-21T08:29:29.673 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:29 smithi137 podman[131178]: 2024-09-21 08:29:29.570420066 +0000 UTC m=+0.736876210 container start a286dd75c50595f7208e7a05f8f49f540b545a3decaa0c752e78260e7839b1e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-deactivate, GIT_CLEAN=True, RELEASE=squid-9e2ac70, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240731, CEPH_POINT_RELEASE=, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2) 2024-09-21T08:29:29.674 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:29 smithi137 ceph-mon[41044]: pgmap v467: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 346 MiB data, 932 MiB used, 357 GiB / 358 GiB avail; 1.7 KiB/s rd, 577 KiB/s wr, 5 op/s; 286/1113 objects degraded (25.696%) 2024-09-21T08:29:29.674 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:29 smithi137 ceph-mon[41044]: osdmap e375: 8 total, 6 up, 4 in 2024-09-21T08:29:29.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:29:29.723 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:29.783 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:29.784 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:29.785 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:29.785 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:29.877 INFO:tasks.rados.rados.0.smithi137.stdout:1469: finishing rollback tid 1 to smithi13779221-14 2024-09-21T08:29:29.877 INFO:tasks.rados.rados.0.smithi137.stdout:1465: finishing write tid 1 to smithi13779221-7 2024-09-21T08:29:29.878 INFO:tasks.rados.rados.0.smithi137.stdout:1467: expect (ObjNum 502 snap 141 seq_num 502) 2024-09-21T08:29:29.886 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:29.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:29.909 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:29.909 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:30.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:30.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:29 smithi165 ceph-mon[32893]: pgmap v467: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 346 MiB data, 932 MiB used, 357 GiB / 358 GiB avail; 1.7 KiB/s rd, 577 KiB/s wr, 5 op/s; 286/1113 objects degraded (25.696%) 2024-09-21T08:29:30.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:29 smithi165 ceph-mon[32893]: osdmap e375: 8 total, 6 up, 4 in 2024-09-21T08:29:30.017 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:29 smithi137 podman[131178]: 2024-09-21 08:29:29.673100961 +0000 UTC m=+0.839557108 container attach a286dd75c50595f7208e7a05f8f49f540b545a3decaa0c752e78260e7839b1e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.37.2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image) 2024-09-21T08:29:30.018 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:29:29 smithi137 podman[131178]: 2024-09-21 08:29:29.817319139 +0000 UTC m=+0.983775283 container died a286dd75c50595f7208e7a05f8f49f540b545a3decaa0c752e78260e7839b1e0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.37.2) 2024-09-21T08:29:30.027 INFO:tasks.rados.rados.0.smithi137.stdout:1465: finishing write tid 2 to smithi13779221-7 2024-09-21T08:29:30.027 INFO:tasks.rados.rados.0.smithi137.stdout:1465: finishing write tid 3 to smithi13779221-7 2024-09-21T08:29:30.027 INFO:tasks.rados.rados.0.smithi137.stdout:1465: finishing write tid 4 to smithi13779221-7 2024-09-21T08:29:30.027 INFO:tasks.rados.rados.0.smithi137.stdout:1465: oid 7 updating version 0 to 575 2024-09-21T08:29:30.027 INFO:tasks.rados.rados.0.smithi137.stdout:1465: oid 7 updating version 575 to 577 2024-09-21T08:29:30.027 INFO:tasks.rados.rados.0.smithi137.stdout:1465: oid 7 version 577 is already newer than 576 2024-09-21T08:29:30.027 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 577 (ObjNum 529 snap 150 seq_num 529) dirty exists 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1465: left oid 7 (ObjNum 529 snap 150 seq_num 529) 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1466: finishing copy_from racing read to smithi13779221-6 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1469: finishing rollback tid 2 to smithi13779221-14 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 317 (ObjNum 456 snap 123 seq_num 456) dirty exists 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1470: finishing write tid 1 to smithi13779221-17 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1466: finishing copy_from to smithi13779221-6 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 363 (ObjNum 503 snap 141 seq_num 503) dirty exists 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1470: finishing write tid 2 to smithi13779221-17 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1470: finishing write tid 3 to smithi13779221-17 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1470: finishing write tid 4 to smithi13779221-17 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1464: done (7 left) 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1465: done (6 left) 2024-09-21T08:29:30.028 INFO:tasks.rados.rados.0.smithi137.stdout:1466: done (5 left) 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1467: done (4 left) 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1468: done (3 left) 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1469: done (2 left) 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1472: delete oid 37 current snap is 150 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1470: finishing write tid 5 to smithi13779221-17 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1470: oid 17 updating version 0 to 498 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1470: oid 17 updating version 498 to 499 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1470: oid 17 updating version 499 to 500 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1470: oid 17 version 500 is already newer than 497 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 500 (ObjNum 530 snap 150 seq_num 530) dirty exists 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1470: left oid 17 (ObjNum 530 snap 150 seq_num 530) 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1471: finishing copy_from racing read to smithi13779221-32 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:1471: finishing copy_from to smithi13779221-32 2024-09-21T08:29:30.029 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 187 (ObjNum 428 snap 108 seq_num 428) dirty exists 2024-09-21T08:29:30.030 INFO:tasks.rados.rados.0.smithi137.stdout:1470: done (2 left) 2024-09-21T08:29:30.030 INFO:tasks.rados.rados.0.smithi137.stdout:1471: done (1 left) 2024-09-21T08:29:30.030 INFO:tasks.rados.rados.0.smithi137.stdout:1472: done (0 left) 2024-09-21T08:29:30.030 INFO:tasks.rados.rados.0.smithi137.stdout:1473: snap_create 2024-09-21T08:29:30.035 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:30.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:30.036 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:30.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:30.158 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:30.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:30.159 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:30.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:30.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:30.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:30.285 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:30.386 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:30.408 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:30.409 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:30.409 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:30.503 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:29:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:29:30.045+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:29:30.510 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:30.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo pkill -f 'journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.0.service' 2024-09-21T08:29:30.532 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:30.533 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:30.533 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:30.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-09-21T08:29:30.603 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-09-21T08:29:30.603 INFO:tasks.thrashosds.thrasher:Testing ceph-objectstore-tool on down osd.0 2024-09-21T08:29:30.603 INFO:tasks.thrashosds.thrasher:remote for osd 0 is ubuntu@smithi137.front.sepia.ceph.com 2024-09-21T08:29:30.603 INFO:tasks.thrashosds.thrasher:remote for exp osd 7 is ubuntu@smithi165.front.sepia.ceph.com 2024-09-21T08:29:30.604 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.7 --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op list-pgs 2024-09-21T08:29:30.605 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:29:30.608 INFO:tasks.rados.rados.0.smithi137.stdout:1473: done (0 left) 2024-09-21T08:29:30.608 INFO:tasks.rados.rados.0.smithi137.stdout:1474: rollback oid 30 current snap is 151 2024-09-21T08:29:30.608 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 30 to 123 2024-09-21T08:29:30.608 INFO:tasks.rados.rados.0.smithi137.stdout:1475: snap_create 2024-09-21T08:29:30.619 INFO:tasks.rados.rados.0.smithi137.stdout:1474: finishing rollback tid 0 to smithi13779221-30 2024-09-21T08:29:30.619 INFO:tasks.rados.rados.0.smithi137.stdout:1474: finishing rollback tid 1 to smithi13779221-30 2024-09-21T08:29:30.624 INFO:tasks.rados.rados.0.smithi137.stdout:1474: finishing rollback tid 2 to smithi13779221-30 2024-09-21T08:29:30.624 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 605 (ObjNum 448 snap 117 seq_num 448) dirty exists 2024-09-21T08:29:30.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:30.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:30.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:30.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:30.768 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:29:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:29:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:29:30.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:30 smithi137 ceph-mon[29634]: osdmap e376: 8 total, 6 up, 4 in 2024-09-21T08:29:30.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:30 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3737436409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:30.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:30.770 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:30.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:30 smithi137 ceph-mon[41044]: osdmap e376: 8 total, 6 up, 4 in 2024-09-21T08:29:30.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:30 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3737436409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:30.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:30.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:30.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:30.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:30.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:30.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:30.808 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T08:29:30.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:30.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:30.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:30.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:30.930 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:30 smithi165 ceph-mon[32893]: osdmap e376: 8 total, 6 up, 4 in 2024-09-21T08:29:30.930 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:30 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3737436409' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:30.930 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:29:30.930 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:31.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:31.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:31.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:31.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:31.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:31.165 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:31.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:31.166 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:31.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:31.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:31.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:31.289 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:31.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:31.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:31.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:31.412 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:31.513 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:31.535 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:31.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:31.536 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:31.568 INFO:tasks.rados.rados.0.smithi137.stdout:1474: done (1 left) 2024-09-21T08:29:31.569 INFO:tasks.rados.rados.0.smithi137.stdout:1475: done (0 left) 2024-09-21T08:29:31.569 INFO:tasks.rados.rados.0.smithi137.stdout:1476: read oid 34 snap -1 2024-09-21T08:29:31.569 INFO:tasks.rados.rados.0.smithi137.stdout:1476: expect deleted 2024-09-21T08:29:31.569 INFO:tasks.rados.rados.0.smithi137.stdout:1477: write (excl) oid 12 current snap is 152 2024-09-21T08:29:31.569 INFO:tasks.rados.rados.0.smithi137.stdout:1477: seq_num 532 ranges {541324=460825,1498538=416188,2393385=547939} 2024-09-21T08:29:31.575 INFO:tasks.rados.rados.0.smithi137.stdout:1477: writing smithi13779221-12 from 541324 to 1002149 tid 1 2024-09-21T08:29:31.580 INFO:tasks.rados.rados.0.smithi137.stdout:1477: writing smithi13779221-12 from 1498538 to 1914726 tid 2 2024-09-21T08:29:31.587 INFO:tasks.rados.rados.0.smithi137.stdout:1477: writing smithi13779221-12 from 2393385 to 2941324 tid 3 2024-09-21T08:29:31.587 INFO:tasks.rados.rados.0.smithi137.stdout:1478: delete oid 17 current snap is 152 2024-09-21T08:29:31.589 INFO:tasks.rados.rados.0.smithi137.stdout:1477: finishing write tid 1 to smithi13779221-12 2024-09-21T08:29:31.595 INFO:tasks.rados.rados.0.smithi137.stdout:1477: finishing write tid 2 to smithi13779221-12 2024-09-21T08:29:31.596 INFO:tasks.rados.rados.0.smithi137.stdout:1477: finishing write tid 3 to smithi13779221-12 2024-09-21T08:29:31.596 INFO:tasks.rados.rados.0.smithi137.stdout:1476: done (2 left) 2024-09-21T08:29:31.596 INFO:tasks.rados.rados.0.smithi137.stdout:1478: done (1 left) 2024-09-21T08:29:31.596 INFO:tasks.rados.rados.0.smithi137.stdout:1479: read oid 37 snap 138 2024-09-21T08:29:31.596 INFO:tasks.rados.rados.0.smithi137.stdout:1479: expect (ObjNum 479 snap 133 seq_num 479) 2024-09-21T08:29:31.596 INFO:tasks.rados.rados.0.smithi137.stdout:1480: delete oid 26 current snap is 152 2024-09-21T08:29:31.597 INFO:tasks.rados.rados.0.smithi137.stdout:1477: finishing write tid 4 to smithi13779221-12 2024-09-21T08:29:31.597 INFO:tasks.rados.rados.0.smithi137.stdout:1477: finishing write tid 5 to smithi13779221-12 2024-09-21T08:29:31.597 INFO:tasks.rados.rados.0.smithi137.stdout:1477: oid 12 updating version 0 to 384 2024-09-21T08:29:31.597 INFO:tasks.rados.rados.0.smithi137.stdout:1477: oid 12 updating version 384 to 385 2024-09-21T08:29:31.597 INFO:tasks.rados.rados.0.smithi137.stdout:1477: oid 12 updating version 385 to 386 2024-09-21T08:29:31.597 INFO:tasks.rados.rados.0.smithi137.stdout:1477: oid 12 version 386 is already newer than 383 2024-09-21T08:29:31.597 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 386 (ObjNum 532 snap 152 seq_num 532) dirty exists 2024-09-21T08:29:31.597 INFO:tasks.rados.rados.0.smithi137.stdout:1477: left oid 12 (ObjNum 532 snap 152 seq_num 532) 2024-09-21T08:29:31.599 INFO:tasks.rados.rados.0.smithi137.stdout:1477: done (2 left) 2024-09-21T08:29:31.599 INFO:tasks.rados.rados.0.smithi137.stdout:1480: done (1 left) 2024-09-21T08:29:31.599 INFO:tasks.rados.rados.0.smithi137.stdout:1481: snap_create 2024-09-21T08:29:31.604 INFO:tasks.rados.rados.0.smithi137.stdout:1479: expect (ObjNum 479 snap 133 seq_num 479) 2024-09-21T08:29:31.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:31.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:31.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:31.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:31.667 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:31.667 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:31.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:31.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:31.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:31.783 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:31.859 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:31 smithi165 ceph-mon[32893]: pgmap v470: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 346 MiB data, 932 MiB used, 357 GiB / 358 GiB avail; 286/1113 objects degraded (25.696%) 2024-09-21T08:29:31.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:31 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3737436409' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:29:31.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:31 smithi165 ceph-mon[32893]: osdmap e377: 8 total, 6 up, 4 in 2024-09-21T08:29:31.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:31 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:31.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:31 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 286/1113 objects degraded (25.696%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:31.860 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:31 smithi165 ceph-mon[32893]: osdmap e378: 8 total, 6 up, 4 in 2024-09-21T08:29:31.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:31.906 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:31.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:31.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:31.918 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T08:29:32.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[29634]: pgmap v470: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 346 MiB data, 932 MiB used, 357 GiB / 358 GiB avail; 286/1113 objects degraded (25.696%) 2024-09-21T08:29:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3737436409' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:29:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[29634]: osdmap e377: 8 total, 6 up, 4 in 2024-09-21T08:29:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 286/1113 objects degraded (25.696%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:32.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[29634]: osdmap e378: 8 total, 6 up, 4 in 2024-09-21T08:29:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[41044]: pgmap v470: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 346 MiB data, 932 MiB used, 357 GiB / 358 GiB avail; 286/1113 objects degraded (25.696%) 2024-09-21T08:29:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3737436409' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:29:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[41044]: osdmap e377: 8 total, 6 up, 4 in 2024-09-21T08:29:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 286/1113 objects degraded (25.696%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:32.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:31 smithi137 ceph-mon[41044]: osdmap e378: 8 total, 6 up, 4 in 2024-09-21T08:29:32.029 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:32.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:32.030 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:32.130 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:32.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:32.143 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:29:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:29:32.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:32.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:32.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:32.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:32.279 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:32.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:32.280 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:32.380 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:32.404 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:32.404 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:32.405 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:32.505 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:32.527 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:32.528 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:32.528 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:32.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T08:29:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:1479: done (1 left) 2024-09-21T08:29:32.577 INFO:tasks.rados.rados.0.smithi137.stdout:1481: done (0 left) 2024-09-21T08:29:32.578 INFO:tasks.rados.rados.0.smithi137.stdout:1482: snap_create 2024-09-21T08:29:32.629 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:32.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:32.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:32.651 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:32.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:32.775 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:32.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:32.776 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:32.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:32.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:32.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:32.899 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:33.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:33.022 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:33.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:33.024 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:33.046 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:29:33.064 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:33.064 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:33.080 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:33.082 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:33.083 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:33.106 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:33.107 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:33.109 INFO:teuthology.orchestra.run.smithi165.stdout:7.f 2024-09-21T08:29:33.109 INFO:teuthology.orchestra.run.smithi165.stdout:7.d 2024-09-21T08:29:33.109 INFO:teuthology.orchestra.run.smithi165.stdout:7.9 2024-09-21T08:29:33.109 INFO:teuthology.orchestra.run.smithi165.stdout:5.14 2024-09-21T08:29:33.109 INFO:teuthology.orchestra.run.smithi165.stdout:3.b 2024-09-21T08:29:33.109 INFO:teuthology.orchestra.run.smithi165.stdout:5.11 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:4.f 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:4.a 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:4.15 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:5.1b 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:7.1f 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:4.8 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:6.15 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:4.6 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:4.4 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:3.16 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:4.1f 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:7.18 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:4.1e 2024-09-21T08:29:33.110 INFO:teuthology.orchestra.run.smithi165.stdout:7.11 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:4.1b 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:3.1d 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:4.11 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:3.f 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:4.0 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:3.e 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:3.1 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:7.21 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:4.c 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:4.b 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:3.1a 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:4.3 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:3.8 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:5.a 2024-09-21T08:29:33.111 INFO:teuthology.orchestra.run.smithi165.stdout:5.10 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:3.18 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:3.15 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:3.a 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:6.5 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:4.1 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:3.13 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:3.1e 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:3.11 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:5.16 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:3.7 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:5.12 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:6.1 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:6.1c 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:6.7 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:4.17 2024-09-21T08:29:33.112 INFO:teuthology.orchestra.run.smithi165.stdout:7.8 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:3.1c 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:5.17 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:5.13 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:2.0 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:7.23 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:7.7 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:3.19 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:3.1b 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:4.e 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:7.16 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:7.1e 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:2.2 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:6.c 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:3.6 2024-09-21T08:29:33.113 INFO:teuthology.orchestra.run.smithi165.stdout:1.0 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:2.1 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:3.1f 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:6.9 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:4.5 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:3.12 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:3.2 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:3.d 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:5.19 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:4.16 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:5.5 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:3.9 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:5.1 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:5.1a 2024-09-21T08:29:33.114 INFO:teuthology.orchestra.run.smithi165.stdout:6.8 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:5.1c 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:7.15 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:5.1d 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:6.12 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:3.5 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:5.1f 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:5.3 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:7.14 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:7.2 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:5.4 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:5.6 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:6.3 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:5.7 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:5.d 2024-09-21T08:29:33.115 INFO:teuthology.orchestra.run.smithi165.stdout:4.7 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:5.e 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:6.14 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:6.16 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:6.17 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:5.2 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:6.1a 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:6.1b 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:6.1e 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:5.9 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:6.2 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:5.18 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:5.1e 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:6.6 2024-09-21T08:29:33.116 INFO:teuthology.orchestra.run.smithi165.stdout:6.18 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:7.17 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:7.22 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:6.b 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:3.14 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:7.10 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:4.14 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:7.12 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:7.13 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:5.b 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:7.19 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:7.1 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:7.1a 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:7.1c 2024-09-21T08:29:33.117 INFO:teuthology.orchestra.run.smithi165.stdout:7.1d 2024-09-21T08:29:33.118 INFO:teuthology.orchestra.run.smithi165.stdout:4.9 2024-09-21T08:29:33.118 INFO:teuthology.orchestra.run.smithi165.stdout:7.20 2024-09-21T08:29:33.118 INFO:teuthology.orchestra.run.smithi165.stdout:5.c 2024-09-21T08:29:33.118 INFO:teuthology.orchestra.run.smithi165.stdout:7.3 2024-09-21T08:29:33.118 INFO:teuthology.orchestra.run.smithi165.stdout:5.15 2024-09-21T08:29:33.118 INFO:teuthology.orchestra.run.smithi165.stdout:7.5 2024-09-21T08:29:33.118 INFO:teuthology.orchestra.run.smithi165.stdout:7.6 2024-09-21T08:29:33.122 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:33.123 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:33.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:33.138 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:33.139 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:33.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:33.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:33.147 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:33.159 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:33.159 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:33.160 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:33.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:33.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:33.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:33.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:33.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:29:33.372 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:33.393 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:33.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:33.394 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:33.495 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:33.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:33.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:33.518 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:33.581 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:33.581 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[41044]: pgmap v473: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 379 MiB data, 981 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 26 op/s; 298/1161 objects degraded (25.668%) 2024-09-21T08:29:33.584 INFO:tasks.rados.rados.0.smithi137.stdout:1482: done (0 left) 2024-09-21T08:29:33.584 INFO:tasks.rados.rados.0.smithi137.stdout:1483: read oid 31 snap -1 2024-09-21T08:29:33.585 INFO:tasks.rados.rados.0.smithi137.stdout:1483: expect (ObjNum 519 snap 149 seq_num 519) 2024-09-21T08:29:33.585 INFO:tasks.rados.rados.0.smithi137.stdout:1484: copy_from oid 2 from oid 26 current snap is 154 2024-09-21T08:29:33.585 INFO:tasks.rados.rados.0.smithi137.stdout:1485: read oid 3 snap -1 2024-09-21T08:29:33.585 INFO:tasks.rados.rados.0.smithi137.stdout:1485: expect (ObjNum 287 snap 66 seq_num 287) 2024-09-21T08:29:33.585 INFO:tasks.rados.rados.0.smithi137.stdout:1486: read oid 46 snap -1 2024-09-21T08:29:33.585 INFO:tasks.rados.rados.0.smithi137.stdout:1486: expect (ObjNum 403 snap 99 seq_num 403) 2024-09-21T08:29:33.585 INFO:tasks.rados.rados.0.smithi137.stdout:1487: rollback oid 29 current snap is 154 2024-09-21T08:29:33.585 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 29 to 131 2024-09-21T08:29:33.585 INFO:tasks.rados.rados.0.smithi137.stdout:1488: snap_remove snap 149 2024-09-21T08:29:33.618 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:33.644 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:33.644 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:33.645 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:33.731 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:29:33.732 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:29:33.732 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:29:33.732 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:29:33.732 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:29:33.745 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:33.767 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:33.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:33.768 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:33.868 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:33.890 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:33.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:33.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:33.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:33 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:33.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:33 smithi165 ceph-mon[32893]: pgmap v473: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 379 MiB data, 981 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 26 op/s; 298/1161 objects degraded (25.668%) 2024-09-21T08:29:33.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:33 smithi165 ceph-mon[32893]: 7.0 scrub starts 2024-09-21T08:29:33.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:33 smithi165 ceph-mon[32893]: 7.0 scrub ok 2024-09-21T08:29:33.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:33 smithi165 ceph-mon[32893]: 7.b scrub starts 2024-09-21T08:29:33.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:33 smithi165 ceph-mon[32893]: 7.b scrub ok 2024-09-21T08:29:33.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:33 smithi165 ceph-mon[32893]: osdmap e379: 8 total, 6 up, 4 in 2024-09-21T08:29:33.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:33 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1950341237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:33.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:34.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:34.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:34.014 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:34.017 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[41044]: 7.0 scrub starts 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[41044]: 7.0 scrub ok 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[41044]: 7.b scrub starts 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[41044]: 7.b scrub ok 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[41044]: osdmap e379: 8 total, 6 up, 4 in 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1950341237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[29634]: pgmap v473: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 379 MiB data, 981 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 26 op/s; 298/1161 objects degraded (25.668%) 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[29634]: 7.0 scrub starts 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[29634]: 7.0 scrub ok 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[29634]: 7.b scrub starts 2024-09-21T08:29:34.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[29634]: 7.b scrub ok 2024-09-21T08:29:34.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[29634]: osdmap e379: 8 total, 6 up, 4 in 2024-09-21T08:29:34.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:33 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1950341237' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:34.115 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:34.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:34.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:34.138 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:34.181 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.7 --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --op export --pgid 4.16 --file /var/log/ceph/exp.4.16.7 2024-09-21T08:29:34.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:34.260 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:34.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:34.261 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:34.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:34.372 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T08:29:34.376 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:34.385 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:34.386 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:34.386 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:34.406 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T08:29:34.487 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:34.511 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:34.511 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:34.512 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:34.591 INFO:tasks.rados.rados.0.smithi137.stdout:1488: done (5 left) 2024-09-21T08:29:34.591 INFO:tasks.rados.rados.0.smithi137.stdout:1489: snap_create 2024-09-21T08:29:34.591 INFO:tasks.rados.rados.0.smithi137.stdout:1484: finishing copy_from to smithi13779221-2 2024-09-21T08:29:34.591 INFO:tasks.rados.rados.0.smithi137.stdout:1484: got expected ENOENT (src dne) 2024-09-21T08:29:34.591 INFO:tasks.rados.rados.0.smithi137.stdout:1484: finishing copy_from racing read to smithi13779221-2 2024-09-21T08:29:34.591 INFO:tasks.rados.rados.0.smithi137.stdout:1486: expect (ObjNum 403 snap 99 seq_num 403) 2024-09-21T08:29:34.613 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:34.634 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:34.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:34.634 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:34.703 INFO:tasks.rados.rados.0.smithi137.stdout:1487: finishing rollback tid 0 to smithi13779221-29 2024-09-21T08:29:34.703 INFO:tasks.rados.rados.0.smithi137.stdout:1487: finishing rollback tid 1 to smithi13779221-29 2024-09-21T08:29:34.703 INFO:tasks.rados.rados.0.smithi137.stdout:1487: finishing rollback tid 2 to smithi13779221-29 2024-09-21T08:29:34.703 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 581 (ObjNum 385 snap 94 seq_num 385) dirty exists 2024-09-21T08:29:34.703 INFO:tasks.rados.rados.0.smithi137.stdout:1483: expect (ObjNum 519 snap 149 seq_num 519) 2024-09-21T08:29:34.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:34.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:34.757 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:34.757 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:34 smithi165 ceph-mon[32893]: 6.1f scrub starts 2024-09-21T08:29:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:34 smithi165 ceph-mon[32893]: 6.1f scrub ok 2024-09-21T08:29:34.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:34 smithi165 ceph-mon[32893]: osdmap e380: 8 total, 6 up, 4 in 2024-09-21T08:29:34.764 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:34 smithi137 ceph-mon[29634]: 6.1f scrub starts 2024-09-21T08:29:34.764 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:34 smithi137 ceph-mon[29634]: 6.1f scrub ok 2024-09-21T08:29:34.764 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:34 smithi137 ceph-mon[29634]: osdmap e380: 8 total, 6 up, 4 in 2024-09-21T08:29:34.764 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:34 smithi137 ceph-mon[41044]: 6.1f scrub starts 2024-09-21T08:29:34.764 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:34 smithi137 ceph-mon[41044]: 6.1f scrub ok 2024-09-21T08:29:34.764 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:34 smithi137 ceph-mon[41044]: osdmap e380: 8 total, 6 up, 4 in 2024-09-21T08:29:34.792 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:34.792 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:34.859 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:34.877 INFO:tasks.rados.rados.0.smithi137.stdout:1485: expect (ObjNum 287 snap 66 seq_num 287) 2024-09-21T08:29:34.880 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:34.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:34.881 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:34.981 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:35.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:35.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:35.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:35.032 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:35.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:35.126 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:35.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:35.127 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:35.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:35.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:35.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:35.250 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:35.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:35.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:35.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:35.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:35.450 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T08:29:35.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:35.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:35.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:35.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:35.598 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:35.601 INFO:tasks.rados.rados.0.smithi137.stdout:1483: done (5 left) 2024-09-21T08:29:35.601 INFO:tasks.rados.rados.0.smithi137.stdout:1484: done (4 left) 2024-09-21T08:29:35.601 INFO:tasks.rados.rados.0.smithi137.stdout:1485: done (3 left) 2024-09-21T08:29:35.601 INFO:tasks.rados.rados.0.smithi137.stdout:1486: done (2 left) 2024-09-21T08:29:35.601 INFO:tasks.rados.rados.0.smithi137.stdout:1487: done (1 left) 2024-09-21T08:29:35.601 INFO:tasks.rados.rados.0.smithi137.stdout:1489: done (0 left) 2024-09-21T08:29:35.602 INFO:tasks.rados.rados.0.smithi137.stdout:1490: copy_from oid 21 from oid 46 current snap is 155 2024-09-21T08:29:35.602 INFO:tasks.rados.rados.0.smithi137.stdout:1491: snap_create 2024-09-21T08:29:35.612 INFO:tasks.rados.rados.0.smithi137.stdout:1490: finishing copy_from racing read to smithi13779221-21 2024-09-21T08:29:35.615 INFO:tasks.rados.rados.0.smithi137.stdout:1490: finishing copy_from to smithi13779221-21 2024-09-21T08:29:35.615 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 495 (ObjNum 403 snap 99 seq_num 403) dirty exists 2024-09-21T08:29:35.621 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:35.622 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:35.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:35 smithi137 ceph-mon[29634]: pgmap v476: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 379 MiB data, 981 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 26 op/s; 298/1161 objects degraded (25.668%) 2024-09-21T08:29:35.671 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:35 smithi137 ceph-mon[29634]: osdmap e381: 8 total, 6 up, 4 in 2024-09-21T08:29:35.671 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:35 smithi137 ceph-mon[41044]: pgmap v476: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 379 MiB data, 981 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 26 op/s; 298/1161 objects degraded (25.668%) 2024-09-21T08:29:35.672 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:35 smithi137 ceph-mon[41044]: osdmap e381: 8 total, 6 up, 4 in 2024-09-21T08:29:35.674 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:35 smithi165 ceph-mon[32893]: pgmap v476: 168 pgs: 2 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 34 active+clean; 379 MiB data, 981 MiB used, 357 GiB / 358 GiB avail; 11 MiB/s rd, 5.5 MiB/s wr, 26 op/s; 298/1161 objects degraded (25.668%) 2024-09-21T08:29:35.675 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:35 smithi165 ceph-mon[32893]: osdmap e381: 8 total, 6 up, 4 in 2024-09-21T08:29:35.679 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:29:35.679 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:29:35.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:29:35.679 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:29:35.679 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.679 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:1082 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94837804043520 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:29:35.679 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:29:35.679 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 297.51697576200002, 2024-09-21T08:29:35.679 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.39751146399999998, 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1082 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.680 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:29:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.681 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159525+0000", 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159528+0000", 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2749999999999998e-06 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159533+0000", 2024-09-21T08:29:35.682 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2480000000000004e-06 2024-09-21T08:29:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159547+0000", 2024-09-21T08:29:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4516e-05 2024-09-21T08:29:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.683 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159556+0000", 2024-09-21T08:29:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9460000000000004e-06 2024-09-21T08:29:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159617+0000", 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0999000000000003e-05 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.159625+0000", 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240000000000007e-06 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544062+0000", 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.38443652099999998 2024-09-21T08:29:35.686 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:29:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.544359+0000", 2024-09-21T08:29:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000297378 2024-09-21T08:29:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548773+0000", 2024-09-21T08:29:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044136440000000004 2024-09-21T08:29:35.687 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.548858+0000", 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5638999999999998e-05 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:38.557037+0000", 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081783740000000004 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.688 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:255 7.2 7:4031afd1:::smithi13779221-48:head [copy-from ver 3] snapc 4=[] ondisk+write+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 290.16866536800001, 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015259864, 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 255 2024-09-21T08:29:35.689 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507836+0000", 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.690 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507839+0000", 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0089999999999999e-06 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507852+0000", 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3617e-05 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507872+0000", 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9482000000000001e-05 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.691 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507883+0000", 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1036000000000001e-05 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507941+0000", 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7819999999999999e-05 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507946+0000", 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9080000000000003e-06 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.692 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510282+0000", 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023368170000000001 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510376+0000", 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3846999999999998e-05 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.516270+0000", 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005893934 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.693 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522776+0000", 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065057179999999997 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522902+0000", 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012641 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.522990+0000", 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7491000000000006e-05 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.694 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523050+0000", 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0075999999999999e-05 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.523096+0000", 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5698999999999997e-05 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:256 7.2 7:4031afd1:::smithi13779221-48:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e92)", 2024-09-21T08:29:35.695 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:29:35.696 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 290.16859806999997, 2024-09-21T08:29:35.696 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013379356, 2024-09-21T08:29:35.696 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.696 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.696 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:29:35.696 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.697 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.697 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 256 2024-09-21T08:29:35.697 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.697 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.697 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.697 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.697 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:29:35.697 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507903+0000", 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507904+0000", 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8000000000000005e-07 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.698 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507907+0000", 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.258e-06 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507915+0000", 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.571e-06 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507920+0000", 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1189999999999996e-06 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.699 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.507998+0000", 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8181000000000002e-05 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.508004+0000", 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3960000000000003e-06 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.510529+0000", 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0025244989999999999 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.700 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.511053+0000", 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052380199999999997 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521102+0000", 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010049361999999999 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521183+0000", 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0495999999999996e-05 2024-09-21T08:29:35.701 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:45.521282+0000", 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9891999999999998e-05 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:260 7.8 e93/84 7:1fd29771:::smithi13779221-22:head v 93'20, mlcod=86'17)", 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 289.10755173699999, 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0078947310000000003, 2024-09-21T08:29:35.702 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568949+0000", 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.703 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568951+0000", 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1510000000000001e-06 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571221+0000", 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022708279999999999 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571244+0000", 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2759999999999999e-05 2024-09-21T08:29:35.704 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571257+0000", 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2669e-05 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571601+0000", 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034448900000000001 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.705 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.571645+0000", 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3498999999999998e-05 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576787+0000", 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051417010000000003 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576844+0000", 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7633999999999997e-05 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.706 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:294 7.2 7:4d793d31:::smithi13779221-9:head [copy-from ver 3] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 288.913834941, 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017337353, 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 294 2024-09-21T08:29:35.707 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762666+0000", 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.708 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762668+0000", 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.353e-06 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762673+0000", 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1569999999999999e-06 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762683+0000", 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0331e-05 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.709 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762688+0000", 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1410000000000004e-06 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.766904+0000", 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042154899999999997 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767004+0000", 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100135 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.710 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.771465+0000", 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044613170000000002 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779379+0000", 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079137649999999997 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779883+0000", 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050370000000000005 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.711 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779896+0000", 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3458999999999999e-05 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.779950+0000", 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3851000000000003e-05 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.780004+0000", 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3653999999999998e-05 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.712 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:295 7.2 7:4d793d31:::smithi13779221-9:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 288.913804223, 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012332286, 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.713 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 295 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.714 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762697+0000", 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4400000000000001e-07 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762699+0000", 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0099999999999998e-06 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762705+0000", 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0279999999999997e-06 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.715 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.762710+0000", 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8210000000000001e-06 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.767340+0000", 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046298770000000001 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.768012+0000", 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067169199999999999 2024-09-21T08:29:35.716 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774900+0000", 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0068879680000000004 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.774961+0000", 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0980999999999998e-05 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.775029+0000", 2024-09-21T08:29:35.717 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8465000000000003e-05 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1908 7.a 7:53537bd3:::smithi13779221-39:head [append 0~0] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 167.12661426299999, 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.037489017, 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.718 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1908 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549887+0000", 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.719 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549889+0000", 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.199e-06 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549894+0000", 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9450000000000001e-06 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549902+0000", 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9459999999999998e-06 2024-09-21T08:29:35.720 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549910+0000", 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6510000000000001e-06 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549962+0000", 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2741999999999997e-05 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549970+0000", 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0949999999999996e-06 2024-09-21T08:29:35.721 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.556983+0000", 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.007012673 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557124+0000", 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014058199999999999 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583019+0000", 2024-09-21T08:29:35.722 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025894830000000001 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583102+0000", 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3332999999999996e-05 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583767+0000", 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00066536999999999998 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583809+0000", 2024-09-21T08:29:35.723 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1919999999999998e-05 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583935+0000", 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012586800000000001 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584880+0000", 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00094458400000000005 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.724 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585414+0000", 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000533892 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.586605+0000", 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0011918479999999999 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587260+0000", 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00065433099999999997 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.725 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587376+0000", 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116208 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1909 7.a 7:53537bd3:::smithi13779221-39:head [rollback 39] snapc 43=[40,3d,39] ondisk+write+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 167.126518299, 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0375938790000001, 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.726 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1909 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.727 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549983+0000", 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549984+0000", 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5799999999999998e-07 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549987+0000", 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5099999999999999e-06 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.728 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549993+0000", 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6099999999999997e-06 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.549998+0000", 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4890000000000003e-06 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550039+0000", 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1155000000000002e-05 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.729 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.550045+0000", 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7130000000000002e-06 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557614+0000", 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.007568971 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for missing object", 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.557661+0000", 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6726e-05 2024-09-21T08:29:35.730 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583163+0000", 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025501894000000001 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.583309+0000", 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014584299999999999 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.731 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584385+0000", 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010765250000000001 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584430+0000", 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5151000000000002e-05 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.584564+0000", 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133652 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.732 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585131+0000", 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00056728200000000001 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.585577+0000", 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044534199999999999 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587426+0000", 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001849402 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.733 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587490+0000", 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4126000000000002e-05 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.587577+0000", 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6630000000000002e-05 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.734 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.44387.0:1376 5.0 5:05bf5b68:::notify.1:head [watch reconnect cookie 94837804043520 gen 4] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e217)", 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 165.047615431, 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0087414380000000007, 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.44387", 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1376 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.735 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628886+0000", 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628889+0000", 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9299999999999999e-06 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.736 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628895+0000", 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6189999999999996e-06 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628909+0000", 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3963e-05 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628916+0000", 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.131e-06 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.737 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.628998+0000", 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1219999999999998e-05 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.629004+0000", 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7510000000000002e-06 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.634800+0000", 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.005795311 2024-09-21T08:29:35.738 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.634882+0000", 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2025000000000002e-05 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637627+0000", 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027454879999999999 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.739 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2114 7.4 e228/215 7:26d6e516:::smithi13779221-33:head v 228'62, mlcod=211'59)", 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 154.13556090899999, 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012725531, 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.740 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540940+0000", 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.540942+0000", 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3549999999999999e-06 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543680+0000", 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002738379 2024-09-21T08:29:35.741 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543701+0000", 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1072000000000001e-05 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543712+0000", 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0691e-05 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543786+0000", 2024-09-21T08:29:35.742 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4721999999999994e-05 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.543809+0000", 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2731e-05 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553613+0000", 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098038980000000001 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.743 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553666+0000", 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2682999999999997e-05 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2124 7.7 e229/215 7:e55f2143:::smithi13779221-11:head v 229'87, mlcod=224'83)", 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 153.14374043199999, 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.008804704, 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.744 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.532761+0000", 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1900000000000003e-07 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.745 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535432+0000", 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026709979999999999 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535449+0000", 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6888000000000002e-05 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535460+0000", 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0947e-05 2024-09-21T08:29:35.746 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535516+0000", 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5551999999999999e-05 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.535538+0000", 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2135000000000001e-05 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.541520+0000", 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059823330000000003 2024-09-21T08:29:35.747 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:02.541565+0000", 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5331999999999999e-05 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2177 7.2 7:43ab705f:::smithi13779221-10:head [rollback 44] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 146.14234418300001, 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0080117160000000003, 2024-09-21T08:29:35.748 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2177 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.749 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534157+0000", 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534158+0000", 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0580000000000001e-06 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534200+0000", 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1875999999999997e-05 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.750 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534209+0000", 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0899999999999994e-06 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534215+0000", 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9499999999999998e-06 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534288+0000", 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3223e-05 2024-09-21T08:29:35.751 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534294+0000", 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.519e-06 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.539853+0000", 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0055597980000000003 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.539923+0000", 2024-09-21T08:29:35.752 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9115000000000005e-05 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540058+0000", 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013511999999999999 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540864+0000", 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00080602199999999999 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.753 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.541123+0000", 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00025911100000000002 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542066+0000", 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000943541 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542123+0000", 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7108999999999998e-05 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.754 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542169+0000", 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5184e-05 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2178 7.2 7:43ab705f:::smithi13779221-10:head [append 0~0] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 146.14217268100001, 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081322689999999993, 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.755 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2178 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534328+0000", 2024-09-21T08:29:35.756 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534329+0000", 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8299999999999995e-07 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534351+0000", 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1662999999999999e-05 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.757 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534359+0000", 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9480000000000008e-06 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534364+0000", 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1329999999999998e-06 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534415+0000", 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1199999999999998e-05 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:29:35.758 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.534421+0000", 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1200000000000001e-06 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540407+0000", 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0059863010000000003 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540492+0000", 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4715000000000005e-05 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.759 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540616+0000", 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012489700000000001 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.540903+0000", 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00028699500000000001 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.541328+0000", 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00042463800000000002 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.760 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542392+0000", 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0010644199999999999 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542427+0000", 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4338999999999998e-05 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.542461+0000", 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3917000000000001e-05 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.761 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2184 7.c 7:305aef79:::smithi13779221-15:head [copy-from ver 143] snapc 4d=[4d,48,47,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 146.122915769, 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022239622000000001, 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.762 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2184 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553585+0000", 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.763 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553588+0000", 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.021e-06 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553596+0000", 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1400000000000002e-06 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553611+0000", 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5478e-05 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.764 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.765 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553619+0000", 2024-09-21T08:29:35.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:35.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9019999999999999e-06 2024-09-21T08:29:35.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.766 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.766 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553685+0000", 2024-09-21T08:29:35.766 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5977999999999997e-05 2024-09-21T08:29:35.766 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.766 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553801+0000", 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116506 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.565629+0000", 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011827127999999999 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.574989+0000", 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093609639999999994 2024-09-21T08:29:35.767 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575477+0000", 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00048743400000000002 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575700+0000", 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022307499999999999 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575756+0000", 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6125999999999998e-05 2024-09-21T08:29:35.768 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.575825+0000", 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8869999999999996e-05 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2185 7.c 7:305aef79:::smithi13779221-15:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e236)", 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 146.12286215399999, 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016182583, 2024-09-21T08:29:35.769 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2185 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.770 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553639+0000", 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553640+0000", 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.61e-07 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553643+0000", 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1020000000000001e-06 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.771 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553653+0000", 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0249e-05 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553659+0000", 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3319999999999997e-06 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.553963+0000", 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000303617 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.772 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.554000+0000", 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6976e-05 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569623+0000", 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015623076 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569721+0000", 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7398000000000005e-05 2024-09-21T08:29:35.773 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:09.569822+0000", 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101072 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2219 7.c 7:361ee62b:::smithi13779221-40:head [sparse-read 0~3130511] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:29:35.774 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 145.13192820699999, 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095045249999999998, 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2219 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:29:35.775 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544573+0000", 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544574+0000", 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.297e-06 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.776 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544579+0000", 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2830000000000003e-06 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544588+0000", 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4849999999999998e-06 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544596+0000", 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6930000000000007e-06 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.777 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544738+0000", 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014263700000000001 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544744+0000", 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6030000000000004e-06 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.552342+0000", 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0075976259999999997 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.778 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.552431+0000", 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9571e-05 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554077+0000", 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016463300000000001 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.779 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2220 7.c 7:361ee62b:::smithi13779221-40:head [read 0~3546277,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e237)", 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 145.13189623100001, 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011021838000000001, 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2220 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.780 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544605+0000", 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5499999999999998e-07 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.781 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544610+0000", 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4349999999999999e-06 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544617+0000", 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2030000000000003e-06 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544623+0000", 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0260000000000004e-06 2024-09-21T08:29:35.782 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544805+0000", 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000181635 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.544809+0000", 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5950000000000001e-06 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554139+0000", 2024-09-21T08:29:35.783 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.009329225 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.554247+0000", 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010890700000000001 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:10.555627+0000", 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013792570000000001 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.784 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2233 7.8 e238/218 7:17170b1f:::smithi13779221-4:head v 238'212, mlcod=238'208)", 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 144.120586594, 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018069034000000001, 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.785 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555915+0000", 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555917+0000", 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3089999999999998e-06 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556417+0000", 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050039099999999999 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.786 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556437+0000", 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9746e-05 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556446+0000", 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4590000000000004e-06 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556581+0000", 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013473700000000001 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.787 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556614+0000", 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3287000000000002e-05 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573937+0000", 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017322422000000001 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573984+0000", 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6683e-05 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.788 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2234 7.8 e238/218 7:17170b1f:::smithi13779221-4:head v 238'213, mlcod=238'208)", 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 144.12003093000001, 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.017540864, 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.789 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556470+0000", 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556471+0000", 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3099999999999997e-07 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.790 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556477+0000", 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9680000000000003e-06 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556486+0000", 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5719999999999992e-06 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.556492+0000", 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6149999999999996e-06 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.791 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.557226+0000", 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00073403900000000002 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.557411+0000", 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018489400000000001 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.573988+0000", 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016576941000000001 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.792 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574011+0000", 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3104000000000001e-05 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2285 7.c 7:361ee62b:::smithi13779221-40:head [copy-from ver 252] snapc 4f=[4f,4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e239)", 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 142.814984442, 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010853527, 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:29:35.793 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2285 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.794 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861517+0000", 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5999999999999999e-07 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861521+0000", 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7299999999999999e-06 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.795 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861524+0000", 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9510000000000002e-06 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861526+0000", 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5739999999999998e-06 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861577+0000", 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0764999999999999e-05 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.796 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.861629+0000", 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1937000000000001e-05 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.863963+0000", 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002334175 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.869705+0000", 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0057416610000000003 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.797 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872022+0000", 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023169670000000001 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872295+0000", 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027316900000000003 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872331+0000", 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6276999999999998e-05 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:29:35.798 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:29:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:29:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:12.872370+0000", 2024-09-21T08:29:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8961000000000002e-05 2024-09-21T08:29:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:29:35.799 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:29:35.799 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:29:35.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:35.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:35.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:35.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:35.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:35.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:35.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:36.047 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:36.069 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:36.070 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:36.070 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:36.171 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:36.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:36.193 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:36.193 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:36.294 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:36.315 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:36.316 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:36.316 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:36.417 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:36.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T08:29:36.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:36.567 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:36.567 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:36.603 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:29:36.604 INFO:tasks.rados.rados.0.smithi137.stdout:1490: done (1 left) 2024-09-21T08:29:36.604 INFO:tasks.rados.rados.0.smithi137.stdout:1491: done (0 left) 2024-09-21T08:29:36.604 INFO:tasks.rados.rados.0.smithi137.stdout:1492: read oid 24 snap -1 2024-09-21T08:29:36.605 INFO:tasks.rados.rados.0.smithi137.stdout:1492: expect (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:29:36.605 INFO:tasks.rados.rados.0.smithi137.stdout:1493: read oid 50 snap -1 2024-09-21T08:29:36.605 INFO:tasks.rados.rados.0.smithi137.stdout:1493: expect (ObjNum 516 snap 146 seq_num 516) 2024-09-21T08:29:36.605 INFO:tasks.rados.rados.0.smithi137.stdout:1494: delete oid 13 current snap is 156 2024-09-21T08:29:36.613 INFO:tasks.rados.rados.0.smithi137.stdout:1494: done (2 left) 2024-09-21T08:29:36.613 INFO:tasks.rados.rados.0.smithi137.stdout:1495: snap_create 2024-09-21T08:29:36.617 INFO:tasks.rados.rados.0.smithi137.stdout:1492: expect (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:29:36.654 INFO:teuthology.orchestra.run.smithi165.stderr:Exporting 4.16 info 4.16( v 73'9 (0'0,73'9] local-lis/les=315/316 n=4 ec=67/67 lis/c=315/315 les/c/f=316/316/0 sis=315) 2024-09-21T08:29:36.654 INFO:teuthology.orchestra.run.smithi165.stderr:Read #4:6849fd06:gc::gc.17:head# 2024-09-21T08:29:36.654 INFO:teuthology.orchestra.run.smithi165.stderr:Read #4:6a988a9b:reshard::reshard.0000000007:head# 2024-09-21T08:29:36.654 INFO:teuthology.orchestra.run.smithi165.stderr:Read #4:6dc3b799:reshard::reshard.0000000006:head# 2024-09-21T08:29:36.654 INFO:teuthology.orchestra.run.smithi165.stderr:Read #4:6f6aba0e:::obj_delete_at_hint.0000000045:head# 2024-09-21T08:29:36.654 INFO:teuthology.orchestra.run.smithi165.stderr:Export successful 2024-09-21T08:29:36.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:36.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:36.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:36.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:36.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:36 smithi137 ceph-mon[41044]: osdmap e382: 8 total, 6 up, 4 in 2024-09-21T08:29:36.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:36 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2022122704' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:36.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:36 smithi137 ceph-mon[41044]: from='client.45584 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:36.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:36 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 298/1161 objects degraded (25.668%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:36.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:36 smithi137 ceph-mon[29634]: osdmap e382: 8 total, 6 up, 4 in 2024-09-21T08:29:36.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:36 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2022122704' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:36.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:36 smithi137 ceph-mon[29634]: from='client.45584 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:36.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:36 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 298/1161 objects degraded (25.668%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:36.776 INFO:tasks.rados.rados.0.smithi137.stdout:1493: expect (ObjNum 516 snap 146 seq_num 516) 2024-09-21T08:29:36.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:36.793 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:29:36.799 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:36.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:36.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:36.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:36.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:36.919 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:36 smithi165 ceph-mon[32893]: osdmap e382: 8 total, 6 up, 4 in 2024-09-21T08:29:36.920 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:36 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2022122704' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:36.920 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:36 smithi165 ceph-mon[32893]: from='client.45584 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:36.920 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:36 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 298/1161 objects degraded (25.668%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:36.937 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:36.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:36.938 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:37.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:37.060 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:37.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:37.061 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:37.162 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:37.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:37.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:37.187 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:37.221 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:37.288 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:37.309 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:37.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:37.310 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:37.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:37.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:37.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:37.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:37.493 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:29:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:29:37.493 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:29:37.493 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:29:37.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:37.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:37.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:37.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:37.608 INFO:tasks.rados.rados.0.smithi137.stdout:1492: done (2 left) 2024-09-21T08:29:37.608 INFO:tasks.rados.rados.0.smithi137.stdout:1493: done (1 left) 2024-09-21T08:29:37.608 INFO:tasks.rados.rados.0.smithi137.stdout:1495: done (0 left) 2024-09-21T08:29:37.608 INFO:tasks.rados.rados.0.smithi137.stdout:1496: read oid 22 snap -1 2024-09-21T08:29:37.608 INFO:tasks.rados.rados.0.smithi137.stdout:1496: expect (ObjNum 469 snap 127 seq_num 469) 2024-09-21T08:29:37.609 INFO:tasks.rados.rados.0.smithi137.stdout:1497: read oid 23 snap -1 2024-09-21T08:29:37.609 INFO:tasks.rados.rados.0.smithi137.stdout:1497: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:29:37.609 INFO:tasks.rados.rados.0.smithi137.stdout:1498: read oid 30 snap 150 2024-09-21T08:29:37.609 INFO:tasks.rados.rados.0.smithi137.stdout:1498: expect (ObjNum 445 snap 115 seq_num 445) 2024-09-21T08:29:37.609 INFO:tasks.rados.rados.0.smithi137.stdout:1499: write (excl) oid 18 current snap is 157 2024-09-21T08:29:37.609 INFO:tasks.rados.rados.0.smithi137.stdout:1499: seq_num 536 ranges {608552=478174,1848819=707173} 2024-09-21T08:29:37.613 INFO:tasks.rados.rados.0.smithi137.stdout:1499: writing smithi13779221-18 from 608552 to 1086726 tid 1 2024-09-21T08:29:37.622 INFO:tasks.rados.rados.0.smithi137.stdout:1499: writing smithi13779221-18 from 1848819 to 2555992 tid 2 2024-09-21T08:29:37.622 INFO:tasks.rados.rados.0.smithi137.stdout:1500: delete oid 39 current snap is 157 2024-09-21T08:29:37.630 INFO:tasks.rados.rados.0.smithi137.stdout:1497: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:29:37.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:37.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:37.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:37.679 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:37.749 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:29:37.753 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:37.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:37 smithi137 ceph-mon[29634]: pgmap v479: 168 pgs: 2 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 36 active+clean; 387 MiB data, 1001 MiB used, 357 GiB / 358 GiB avail; 9.5 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 301/1173 objects degraded (25.661%) 2024-09-21T08:29:37.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:37 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:29:37.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:37 smithi137 ceph-mon[29634]: from='client.45584 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:37.753 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:37 smithi137 ceph-mon[29634]: osdmap e383: 8 total, 6 up, 4 in 2024-09-21T08:29:37.754 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:37.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:37 smithi137 ceph-mon[41044]: pgmap v479: 168 pgs: 2 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 36 active+clean; 387 MiB data, 1001 MiB used, 357 GiB / 358 GiB avail; 9.5 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 301/1173 objects degraded (25.661%) 2024-09-21T08:29:37.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:37 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:29:37.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:37 smithi137 ceph-mon[41044]: from='client.45584 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:37.754 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:37 smithi137 ceph-mon[41044]: osdmap e383: 8 total, 6 up, 4 in 2024-09-21T08:29:37.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:37 smithi165 ceph-mon[32893]: pgmap v479: 168 pgs: 2 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 36 active+clean; 387 MiB data, 1001 MiB used, 357 GiB / 358 GiB avail; 9.5 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 301/1173 objects degraded (25.661%) 2024-09-21T08:29:37.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:37 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:29:37.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:37 smithi165 ceph-mon[32893]: from='client.45584 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:37.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:37 smithi165 ceph-mon[32893]: osdmap e383: 8 total, 6 up, 4 in 2024-09-21T08:29:37.769 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:37.769 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:37.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:37.785 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:37.785 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:37.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:37.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:37.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:37.803 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:37.803 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:37.818 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:37.819 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:37.835 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:37.835 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:37.835 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:37.860 INFO:tasks.rados.rados.0.smithi137.stdout:1498: expect (ObjNum 445 snap 115 seq_num 445) 2024-09-21T08:29:37.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:37.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:37.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:37.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:38.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:38.048 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:38.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:38.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:38.050 INFO:tasks.rados.rados.0.smithi137.stdout:1496: expect (ObjNum 469 snap 127 seq_num 469) 2024-09-21T08:29:38.107 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.7 --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-7 --journal-path /var/lib/ceph/osd/ceph-7/journal --force --op remove --pgid 4.16 2024-09-21T08:29:38.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:1499: finishing write tid 1 to smithi13779221-18 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:1499: finishing write tid 2 to smithi13779221-18 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:1499: finishing write tid 3 to smithi13779221-18 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:1499: finishing write tid 4 to smithi13779221-18 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:1499: oid 18 updating version 0 to 221 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:1499: oid 18 updating version 221 to 222 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:1499: oid 18 updating version 222 to 223 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 223 (ObjNum 536 snap 157 seq_num 536) dirty exists 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:1499: left oid 18 (ObjNum 536 snap 157 seq_num 536) 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:1496: done (4 left) 2024-09-21T08:29:38.170 INFO:tasks.rados.rados.0.smithi137.stdout:1497: done (3 left) 2024-09-21T08:29:38.171 INFO:tasks.rados.rados.0.smithi137.stdout:1498: done (2 left) 2024-09-21T08:29:38.171 INFO:tasks.rados.rados.0.smithi137.stdout:1499: done (1 left) 2024-09-21T08:29:38.171 INFO:tasks.rados.rados.0.smithi137.stdout:1500: done (0 left) 2024-09-21T08:29:38.171 INFO:tasks.rados.rados.0.smithi137.stdout:1501: snap_remove snap 135 2024-09-21T08:29:38.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:38.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:38.173 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:38.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:38.296 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:38.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:38.297 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:38.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:38.419 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:38.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:38.420 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:38.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:38.532 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:38.533 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:38.543 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:38.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:38.544 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:38.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T08:29:38.614 INFO:tasks.rados.rados.0.smithi137.stdout:1501: done (0 left) 2024-09-21T08:29:38.615 INFO:tasks.rados.rados.0.smithi137.stdout:1502: rollback oid 10 current snap is 157 2024-09-21T08:29:38.615 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 10 to 156 2024-09-21T08:29:38.615 INFO:tasks.rados.rados.0.smithi137.stdout:1503: snap_remove snap 136 2024-09-21T08:29:38.630 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/osd.7/config 2024-09-21T08:29:38.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:38.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:38.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:38.668 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:38.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:38 smithi137 ceph-mon[29634]: 4.1c scrub starts 2024-09-21T08:29:38.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:38 smithi137 ceph-mon[29634]: 4.1c scrub ok 2024-09-21T08:29:38.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:38 smithi137 ceph-mon[29634]: osdmap e384: 8 total, 6 up, 4 in 2024-09-21T08:29:38.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:38 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/513517731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:38.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:38 smithi137 ceph-mon[41044]: 4.1c scrub starts 2024-09-21T08:29:38.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:38 smithi137 ceph-mon[41044]: 4.1c scrub ok 2024-09-21T08:29:38.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:38 smithi137 ceph-mon[41044]: osdmap e384: 8 total, 6 up, 4 in 2024-09-21T08:29:38.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:38 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/513517731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:38.769 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:38.790 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:38.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:38.791 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:38.882 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:38 smithi165 ceph-mon[32893]: 4.1c scrub starts 2024-09-21T08:29:38.882 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:38 smithi165 ceph-mon[32893]: 4.1c scrub ok 2024-09-21T08:29:38.882 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:38 smithi165 ceph-mon[32893]: osdmap e384: 8 total, 6 up, 4 in 2024-09-21T08:29:38.882 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:38 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/513517731' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:38.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:38.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:38.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:38.914 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:39.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:39.036 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:39.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:39.037 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:39.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:39.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:39.159 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:39.159 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:39.177 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:29:39.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:39.281 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:39.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:39.282 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:39.371 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:39.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:39.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:39.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:39.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:39.446 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:39.506 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:39.529 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:39.529 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:39.530 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:39.617 INFO:tasks.rados.rados.0.smithi137.stdout:1503: done (1 left) 2024-09-21T08:29:39.617 INFO:tasks.rados.rados.0.smithi137.stdout:1504: rollback oid 16 current snap is 157 2024-09-21T08:29:39.618 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 16 to 134 2024-09-21T08:29:39.618 INFO:tasks.rados.rados.0.smithi137.stdout:1502: finishing rollback tid 0 to smithi13779221-10 2024-09-21T08:29:39.618 INFO:tasks.rados.rados.0.smithi137.stdout:1502: finishing rollback tid 1 to smithi13779221-10 2024-09-21T08:29:39.618 INFO:tasks.rados.rados.0.smithi137.stdout:1502: finishing rollback tid 2 to smithi13779221-10 2024-09-21T08:29:39.618 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 524 (ObjNum 517 snap 146 seq_num 517) dirty exists 2024-09-21T08:29:39.618 INFO:tasks.rados.rados.0.smithi137.stdout:1502: done (1 left) 2024-09-21T08:29:39.618 INFO:tasks.rados.rados.0.smithi137.stdout:1505: snap_create 2024-09-21T08:29:39.628 INFO:tasks.rados.rados.0.smithi137.stdout:1504: finishing rollback tid 0 to smithi13779221-16 2024-09-21T08:29:39.629 INFO:tasks.rados.rados.0.smithi137.stdout:1504: finishing rollback tid 1 to smithi13779221-16 2024-09-21T08:29:39.629 INFO:tasks.rados.rados.0.smithi137.stdout:1504: finishing rollback tid 2 to smithi13779221-16 2024-09-21T08:29:39.629 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 336 (ObjNum 468 snap 126 seq_num 468) dirty exists 2024-09-21T08:29:39.630 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:39.650 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:39.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:39.651 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:39.752 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:39 smithi165 ceph-mon[32893]: pgmap v482: 168 pgs: 2 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 36 active+clean; 387 MiB data, 1001 MiB used, 357 GiB / 358 GiB avail; 9.5 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 301/1173 objects degraded (25.661%) 2024-09-21T08:29:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:39 smithi165 ceph-mon[32893]: osdmap e385: 8 total, 6 up, 4 in 2024-09-21T08:29:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:39 smithi165 ceph-mon[32893]: 3.10 deep-scrub starts 2024-09-21T08:29:39.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:39 smithi165 ceph-mon[32893]: 3.10 deep-scrub ok 2024-09-21T08:29:39.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:39.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:39.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:39.812 INFO:teuthology.orchestra.run.smithi165.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T08:29:39.842 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:39 smithi137 ceph-mon[29634]: pgmap v482: 168 pgs: 2 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 36 active+clean; 387 MiB data, 1001 MiB used, 357 GiB / 358 GiB avail; 9.5 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 301/1173 objects degraded (25.661%) 2024-09-21T08:29:39.842 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:39 smithi137 ceph-mon[29634]: osdmap e385: 8 total, 6 up, 4 in 2024-09-21T08:29:39.842 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:39 smithi137 ceph-mon[29634]: 3.10 deep-scrub starts 2024-09-21T08:29:39.842 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:39 smithi137 ceph-mon[29634]: 3.10 deep-scrub ok 2024-09-21T08:29:39.842 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:39 smithi137 ceph-mon[41044]: pgmap v482: 168 pgs: 2 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 36 active+clean; 387 MiB data, 1001 MiB used, 357 GiB / 358 GiB avail; 9.5 MiB/s rd, 1.0 MiB/s wr, 13 op/s; 301/1173 objects degraded (25.661%) 2024-09-21T08:29:39.842 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:39 smithi137 ceph-mon[41044]: osdmap e385: 8 total, 6 up, 4 in 2024-09-21T08:29:39.842 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:39 smithi137 ceph-mon[41044]: 3.10 deep-scrub starts 2024-09-21T08:29:39.842 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:39 smithi137 ceph-mon[41044]: 3.10 deep-scrub ok 2024-09-21T08:29:39.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:39.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:39.898 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:39.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:39.999 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:40.021 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:40.022 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:40.022 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:40.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:40.144 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:40.145 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:40.145 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:40.246 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:40.270 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:40.271 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:40.271 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:40.372 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:40.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:40.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:40.394 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:40.494 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:40.495 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:40.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:40.516 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:40.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:40.517 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:40.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:29:40.618 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:40.622 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:29:40.635 INFO:tasks.rados.rados.0.smithi137.stdout:1504: done (1 left) 2024-09-21T08:29:40.635 INFO:tasks.rados.rados.0.smithi137.stdout:1505: done (0 left) 2024-09-21T08:29:40.635 INFO:tasks.rados.rados.0.smithi137.stdout:1506: snap_create 2024-09-21T08:29:40.648 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:40.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:40.649 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:40.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:40.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:40.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:40.772 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:40.873 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:40.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:40.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:40.895 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:40.954 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:40 smithi137 ceph-mon[29634]: 4.10 scrub starts 2024-09-21T08:29:40.954 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:40 smithi137 ceph-mon[29634]: 4.10 scrub ok 2024-09-21T08:29:40.954 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:40 smithi137 ceph-mon[29634]: osdmap e386: 8 total, 6 up, 4 in 2024-09-21T08:29:40.955 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:40 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1046195362' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:40.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:40 smithi137 ceph-mon[41044]: 4.10 scrub starts 2024-09-21T08:29:40.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:40 smithi137 ceph-mon[41044]: 4.10 scrub ok 2024-09-21T08:29:40.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:40 smithi137 ceph-mon[41044]: osdmap e386: 8 total, 6 up, 4 in 2024-09-21T08:29:40.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:40 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1046195362' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:40.955 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:29:40 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:29:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:29:40.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:40 smithi165 ceph-mon[32893]: 4.10 scrub starts 2024-09-21T08:29:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:40 smithi165 ceph-mon[32893]: 4.10 scrub ok 2024-09-21T08:29:41.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:40 smithi165 ceph-mon[32893]: osdmap e386: 8 total, 6 up, 4 in 2024-09-21T08:29:41.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:40 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1046195362' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:29:41.017 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:41.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:41.018 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:41.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T08:29:41.119 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:41.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:41.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:41.141 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:41.242 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:41.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:41.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:41.264 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:41.364 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:41.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:41.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:41.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:41.399 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:41.428 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:41.491 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:41.512 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:41.514 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:41.567 INFO:tasks.rados.rados.0.smithi137.stdout:1506: done (0 left) 2024-09-21T08:29:41.568 INFO:tasks.rados.rados.0.smithi137.stdout:1507: delete oid 5 current snap is 159 2024-09-21T08:29:41.574 INFO:tasks.rados.rados.0.smithi137.stdout:1507: done (0 left) 2024-09-21T08:29:41.574 INFO:tasks.rados.rados.0.smithi137.stdout:1508: snap_create 2024-09-21T08:29:41.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:41.638 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:41.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:41.639 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:41.740 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:41.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:41.763 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:41.763 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:41.817 INFO:teuthology.orchestra.run.smithi165.stdout: marking collection for removal 2024-09-21T08:29:41.817 INFO:teuthology.orchestra.run.smithi165.stdout:setting '_remove' omap key 2024-09-21T08:29:41.817 INFO:teuthology.orchestra.run.smithi165.stdout:finish_remove_pgs 4.16_head removing 4.16 2024-09-21T08:29:41.818 INFO:teuthology.orchestra.run.smithi165.stdout:Remove successful 2024-09-21T08:29:41.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:41.878 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:29:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:29:41.878 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:41 smithi165 ceph-mon[32893]: pgmap v485: 168 pgs: 2 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 36 active+clean; 387 MiB data, 1001 MiB used, 357 GiB / 358 GiB avail; 301/1173 objects degraded (25.661%) 2024-09-21T08:29:41.878 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:41 smithi165 ceph-mon[32893]: 6.e scrub starts 2024-09-21T08:29:41.878 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:41 smithi165 ceph-mon[32893]: 6.e scrub ok 2024-09-21T08:29:41.878 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:41 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:41.878 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:41 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1046195362' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:29:41.878 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:41 smithi165 ceph-mon[32893]: osdmap e387: 8 total, 6 up, 4 in 2024-09-21T08:29:41.878 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:41 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 301/1173 objects degraded (25.661%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:41.879 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:41 smithi165 ceph-mon[32893]: osdmap e388: 8 total, 6 up, 4 in 2024-09-21T08:29:41.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:41.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:41.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:41.897 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:41.897 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:41.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[41044]: pgmap v485: 168 pgs: 2 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 36 active+clean; 387 MiB data, 1001 MiB used, 357 GiB / 358 GiB avail; 301/1173 objects degraded (25.661%) 2024-09-21T08:29:41.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[41044]: 6.e scrub starts 2024-09-21T08:29:41.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[41044]: 6.e scrub ok 2024-09-21T08:29:41.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:41.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1046195362' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:29:41.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[41044]: osdmap e387: 8 total, 6 up, 4 in 2024-09-21T08:29:41.944 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 301/1173 objects degraded (25.661%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:41.944 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[41044]: osdmap e388: 8 total, 6 up, 4 in 2024-09-21T08:29:41.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[29634]: pgmap v485: 168 pgs: 2 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 36 active+clean; 387 MiB data, 1001 MiB used, 357 GiB / 358 GiB avail; 301/1173 objects degraded (25.661%) 2024-09-21T08:29:41.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[29634]: 6.e scrub starts 2024-09-21T08:29:41.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[29634]: 6.e scrub ok 2024-09-21T08:29:41.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:41.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1046195362' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:29:41.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[29634]: osdmap e387: 8 total, 6 up, 4 in 2024-09-21T08:29:41.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 301/1173 objects degraded (25.661%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:41.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:41 smithi137 ceph-mon[29634]: osdmap e388: 8 total, 6 up, 4 in 2024-09-21T08:29:41.957 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:29:41.959 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:41.959 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:41.976 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:41.976 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:41.990 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:41.993 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:41.993 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:42.011 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:42.012 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:42.012 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:42.013 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:42.013 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:42.028 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:42.028 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:42.044 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:42.045 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:42.045 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:42.115 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:42.135 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:42.136 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:42.137 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:42.237 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:42.261 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:42.261 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:42.262 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:42.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:42.384 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:42.384 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:42.385 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:42.485 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:42.507 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:42.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:42.508 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:42.567 INFO:tasks.rados.rados.0.smithi137.stdout:1508: done (0 left) 2024-09-21T08:29:42.567 INFO:tasks.rados.rados.0.smithi137.stdout:1509: read oid 18 snap -1 2024-09-21T08:29:42.567 INFO:tasks.rados.rados.0.smithi137.stdout:1509: expect (ObjNum 536 snap 157 seq_num 536) 2024-09-21T08:29:42.567 INFO:tasks.rados.rados.0.smithi137.stdout:1510: snap_create 2024-09-21T08:29:42.577 INFO:tasks.rados.rados.0.smithi137.stdout:1509: expect (ObjNum 536 snap 157 seq_num 536) 2024-09-21T08:29:42.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:42.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:42.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:42.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:42.754 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:42.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:42.755 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:42.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:42 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1693236008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:42.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:42 smithi165 ceph-mon[32893]: osdmap e389: 8 total, 6 up, 4 in 2024-09-21T08:29:42.764 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:42.764 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:42.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T08:29:42.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:42.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:42.880 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:42.880 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:42.981 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:43.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:43.003 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:43.004 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:43.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:42 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1693236008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:43.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:42 smithi137 ceph-mon[41044]: osdmap e389: 8 total, 6 up, 4 in 2024-09-21T08:29:43.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:42 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1693236008' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:43.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:42 smithi137 ceph-mon[29634]: osdmap e389: 8 total, 6 up, 4 in 2024-09-21T08:29:43.105 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:43.127 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:43.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:43.128 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:43.210 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.0 --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --no-mon-config '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal --op list-pgs 2024-09-21T08:29:43.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:43.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:43.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:43.276 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:43.286 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:43.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:29:43.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:43.407 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/osd.0/config 2024-09-21T08:29:43.428 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:43.429 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:43.429 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:43.530 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:43.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:43.552 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:43.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:43.572 INFO:tasks.rados.rados.0.smithi137.stdout:1509: done (1 left) 2024-09-21T08:29:43.573 INFO:tasks.rados.rados.0.smithi137.stdout:1510: done (0 left) 2024-09-21T08:29:43.573 INFO:tasks.rados.rados.0.smithi137.stdout:1511: snap_remove snap 148 2024-09-21T08:29:43.653 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:43.676 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:43.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:43.677 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:43.772 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:29:43.773 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:29:43.773 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:29:43.773 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:29:43.778 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:43.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:43 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:43.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:43 smithi165 ceph-mon[32893]: pgmap v488: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 388 MiB data, 1010 MiB used, 357 GiB / 358 GiB avail; 19 MiB/s rd, 289 KiB/s wr, 15 op/s; 303/1185 objects degraded (25.570%) 2024-09-21T08:29:43.781 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:43 smithi165 ceph-mon[32893]: osdmap e390: 8 total, 6 up, 4 in 2024-09-21T08:29:43.799 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:43.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:43.800 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:43.808 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:43 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:43.808 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:43 smithi137 ceph-mon[29634]: pgmap v488: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 388 MiB data, 1010 MiB used, 357 GiB / 358 GiB avail; 19 MiB/s rd, 289 KiB/s wr, 15 op/s; 303/1185 objects degraded (25.570%) 2024-09-21T08:29:43.808 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:43 smithi137 ceph-mon[29634]: osdmap e390: 8 total, 6 up, 4 in 2024-09-21T08:29:43.808 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:43 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:43.808 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:43 smithi137 ceph-mon[41044]: pgmap v488: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 388 MiB data, 1010 MiB used, 357 GiB / 358 GiB avail; 19 MiB/s rd, 289 KiB/s wr, 15 op/s; 303/1185 objects degraded (25.570%) 2024-09-21T08:29:43.808 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:43 smithi137 ceph-mon[41044]: osdmap e390: 8 total, 6 up, 4 in 2024-09-21T08:29:43.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:43.922 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:43.923 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:43.923 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:44.024 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:44.045 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:44.046 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:44.046 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:44.147 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:44.159 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:44.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:44.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:44.170 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:44.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:44.292 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:44.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:44.293 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:44.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:44.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:44.418 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:44.418 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:44.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T08:29:44.519 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:44.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:44.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:44.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:44.571 INFO:tasks.rados.rados.0.smithi137.stdout:1511: done (0 left) 2024-09-21T08:29:44.571 INFO:tasks.rados.rados.0.smithi137.stdout:1512: read oid 20 snap -1 2024-09-21T08:29:44.571 INFO:tasks.rados.rados.0.smithi137.stdout:1512: expect (ObjNum 505 snap 141 seq_num 505) 2024-09-21T08:29:44.571 INFO:tasks.rados.rados.0.smithi137.stdout:1513: write (excl) oid 37 current snap is 161 2024-09-21T08:29:44.571 INFO:tasks.rados.rados.0.smithi137.stdout:1513: seq_num 539 ranges {658973=703921,1907912=570853,2985346=73627} 2024-09-21T08:29:44.579 INFO:tasks.rados.rados.0.smithi137.stdout:1513: writing smithi13779221-37 from 658973 to 1362894 tid 1 2024-09-21T08:29:44.585 INFO:tasks.rados.rados.0.smithi137.stdout:1513: writing smithi13779221-37 from 1907912 to 2478765 tid 2 2024-09-21T08:29:44.589 INFO:tasks.rados.rados.0.smithi137.stdout:1513: writing smithi13779221-37 from 2985346 to 3058973 tid 3 2024-09-21T08:29:44.589 INFO:tasks.rados.rados.0.smithi137.stdout:1514: rollback oid 1 current snap is 161 2024-09-21T08:29:44.589 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 1 to 144 2024-09-21T08:29:44.589 INFO:tasks.rados.rados.0.smithi137.stdout:1512: expect (ObjNum 505 snap 141 seq_num 505) 2024-09-21T08:29:44.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:44.663 INFO:teuthology.orchestra.run.smithi137.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T08:29:44.666 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:44.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:44.667 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:44.711 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:44.730 INFO:tasks.rados.rados.0.smithi137.stdout:1513: finishing write tid 1 to smithi13779221-37 2024-09-21T08:29:44.730 INFO:tasks.rados.rados.0.smithi137.stdout:1513: finishing write tid 2 to smithi13779221-37 2024-09-21T08:29:44.730 INFO:tasks.rados.rados.0.smithi137.stdout:1513: finishing write tid 3 to smithi13779221-37 2024-09-21T08:29:44.730 INFO:tasks.rados.rados.0.smithi137.stdout:1513: finishing write tid 4 to smithi13779221-37 2024-09-21T08:29:44.730 INFO:tasks.rados.rados.0.smithi137.stdout:1512: done (2 left) 2024-09-21T08:29:44.730 INFO:tasks.rados.rados.0.smithi137.stdout:1515: delete oid 9 current snap is 161 2024-09-21T08:29:44.730 INFO:tasks.rados.rados.0.smithi137.stdout:1513: finishing write tid 5 to smithi13779221-37 2024-09-21T08:29:44.730 INFO:tasks.rados.rados.0.smithi137.stdout:1513: oid 37 updating version 0 to 617 2024-09-21T08:29:44.731 INFO:tasks.rados.rados.0.smithi137.stdout:1513: oid 37 version 617 is already newer than 616 2024-09-21T08:29:44.731 INFO:tasks.rados.rados.0.smithi137.stdout:1513: oid 37 updating version 617 to 618 2024-09-21T08:29:44.731 INFO:tasks.rados.rados.0.smithi137.stdout:1513: oid 37 updating version 618 to 619 2024-09-21T08:29:44.731 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 619 (ObjNum 539 snap 161 seq_num 539) dirty exists 2024-09-21T08:29:44.731 INFO:tasks.rados.rados.0.smithi137.stdout:1513: left oid 37 (ObjNum 539 snap 161 seq_num 539) 2024-09-21T08:29:44.731 INFO:tasks.rados.rados.0.smithi137.stdout:1514: finishing rollback tid 0 to smithi13779221-1 2024-09-21T08:29:44.731 INFO:tasks.rados.rados.0.smithi137.stdout:1514: finishing rollback tid 1 to smithi13779221-1 2024-09-21T08:29:44.731 INFO:tasks.rados.rados.0.smithi137.stdout:1514: finishing rollback tid 2 to smithi13779221-1 2024-09-21T08:29:44.731 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 529 (ObjNum 499 snap 140 seq_num 499) dirty exists 2024-09-21T08:29:44.732 INFO:tasks.rados.rados.0.smithi137.stdout:1513: done (2 left) 2024-09-21T08:29:44.732 INFO:tasks.rados.rados.0.smithi137.stdout:1514: done (1 left) 2024-09-21T08:29:44.732 INFO:tasks.rados.rados.0.smithi137.stdout:1515: done (0 left) 2024-09-21T08:29:44.732 INFO:tasks.rados.rados.0.smithi137.stdout:1516: snap_remove snap 158 2024-09-21T08:29:44.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:29:44.768 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:44.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:44.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:44.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:44.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:44.913 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:44.914 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:44.914 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:45.015 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:45.037 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:45.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:45.038 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:45.133 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:29:45.133 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:29:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:29:45.134 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:29:45.134 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:29:45.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:45.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:45.161 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:45.161 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:45.165 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:29:45 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:29:45.055+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:29:45.261 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:45.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:45.285 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:45.285 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:45.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:45.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:45.408 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:45.409 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:45.510 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:45.531 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:45.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:45.532 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:45.633 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:45.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:45.639 INFO:tasks.rados.rados.0.smithi137.stdout:1516: done (0 left) 2024-09-21T08:29:45.639 INFO:tasks.rados.rados.0.smithi137.stdout:1517: snap_create 2024-09-21T08:29:45.656 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:45.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:45.657 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:45.758 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:45.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:45 smithi137 ceph-mon[29634]: pgmap v491: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 388 MiB data, 1010 MiB used, 357 GiB / 358 GiB avail; 19 MiB/s rd, 289 KiB/s wr, 15 op/s; 303/1185 objects degraded (25.570%) 2024-09-21T08:29:45.759 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:45 smithi137 ceph-mon[29634]: osdmap e391: 8 total, 6 up, 4 in 2024-09-21T08:29:45.759 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:45 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:45.759 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:45 smithi137 ceph-mon[41044]: pgmap v491: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 388 MiB data, 1010 MiB used, 357 GiB / 358 GiB avail; 19 MiB/s rd, 289 KiB/s wr, 15 op/s; 303/1185 objects degraded (25.570%) 2024-09-21T08:29:45.759 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:45 smithi137 ceph-mon[41044]: osdmap e391: 8 total, 6 up, 4 in 2024-09-21T08:29:45.759 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:45 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:45.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:45 smithi165 ceph-mon[32893]: pgmap v491: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 388 MiB data, 1010 MiB used, 357 GiB / 358 GiB avail; 19 MiB/s rd, 289 KiB/s wr, 15 op/s; 303/1185 objects degraded (25.570%) 2024-09-21T08:29:45.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:45 smithi165 ceph-mon[32893]: osdmap e391: 8 total, 6 up, 4 in 2024-09-21T08:29:45.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:45 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:29:45.780 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:45.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:45.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:45.793 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T08:29:45.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:45.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:45.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:45.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:46.005 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:46.028 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:46.029 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:46.029 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:46.081 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:46.130 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:46.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:46.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:46.154 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:46.230 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:46.231 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:29:46.231 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:46.238 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:46.238 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:46.254 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:46.254 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:46.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:46.271 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:46.271 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:46.275 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:46.276 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:46.276 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:46.288 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:46.289 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:46.305 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:46.305 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:46.305 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:46.377 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:46.399 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:46.399 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:46.399 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:46.500 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:46.523 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:46.524 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:46.524 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:46.527 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:46.527 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:46.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:46.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:46.649 INFO:tasks.rados.rados.0.smithi137.stdout:1517: done (0 left) 2024-09-21T08:29:46.649 INFO:tasks.rados.rados.0.smithi137.stdout:1518: write (excl) oid 6 current snap is 162 2024-09-21T08:29:46.650 INFO:tasks.rados.rados.0.smithi137.stdout:1518: seq_num 541 ranges {692587=570772,2024876=574522,3040192=52395} 2024-09-21T08:29:46.650 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:46.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:46.655 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:29:46.656 INFO:tasks.rados.rados.0.smithi137.stdout:1518: writing smithi13779221-6 from 692587 to 1263359 tid 1 2024-09-21T08:29:46.664 INFO:tasks.rados.rados.0.smithi137.stdout:1518: writing smithi13779221-6 from 2024876 to 2599398 tid 2 2024-09-21T08:29:46.667 INFO:tasks.rados.rados.0.smithi137.stdout:1518: writing smithi13779221-6 from 3040192 to 3092587 tid 3 2024-09-21T08:29:46.667 INFO:tasks.rados.rados.0.smithi137.stdout:1519: write (excl) oid 2 current snap is 162 2024-09-21T08:29:46.667 INFO:tasks.rados.rados.0.smithi137.stdout:1518: finishing write tid 1 to smithi13779221-6 2024-09-21T08:29:46.668 INFO:tasks.rados.rados.0.smithi137.stdout:1519: seq_num 542 ranges {709394=646021,2083358=718180} 2024-09-21T08:29:46.674 INFO:tasks.rados.rados.0.smithi137.stdout:1519: writing smithi13779221-2 from 709394 to 1355415 tid 1 2024-09-21T08:29:46.683 INFO:tasks.rados.rados.0.smithi137.stdout:1519: writing smithi13779221-2 from 2083358 to 2801538 tid 2 2024-09-21T08:29:46.683 INFO:tasks.rados.rados.0.smithi137.stdout:1520: delete oid 22 current snap is 162 2024-09-21T08:29:46.683 INFO:tasks.rados.rados.0.smithi137.stdout:1518: finishing write tid 2 to smithi13779221-6 2024-09-21T08:29:46.683 INFO:tasks.rados.rados.0.smithi137.stdout:1518: finishing write tid 3 to smithi13779221-6 2024-09-21T08:29:46.683 INFO:tasks.rados.rados.0.smithi137.stdout:1518: finishing write tid 4 to smithi13779221-6 2024-09-21T08:29:46.683 INFO:tasks.rados.rados.0.smithi137.stdout:1518: finishing write tid 5 to smithi13779221-6 2024-09-21T08:29:46.683 INFO:tasks.rados.rados.0.smithi137.stdout:1518: oid 6 updating version 0 to 366 2024-09-21T08:29:46.683 INFO:tasks.rados.rados.0.smithi137.stdout:1518: oid 6 version 366 is already newer than 365 2024-09-21T08:29:46.684 INFO:tasks.rados.rados.0.smithi137.stdout:1518: oid 6 updating version 366 to 367 2024-09-21T08:29:46.684 INFO:tasks.rados.rados.0.smithi137.stdout:1518: oid 6 version 367 is already newer than 364 2024-09-21T08:29:46.684 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 367 (ObjNum 541 snap 162 seq_num 541) dirty exists 2024-09-21T08:29:46.684 INFO:tasks.rados.rados.0.smithi137.stdout:1518: left oid 6 (ObjNum 541 snap 162 seq_num 541) 2024-09-21T08:29:46.684 INFO:tasks.rados.rados.0.smithi137.stdout:1519: finishing write tid 1 to smithi13779221-2 2024-09-21T08:29:46.685 INFO:tasks.rados.rados.0.smithi137.stdout:1518: done (2 left) 2024-09-21T08:29:46.685 INFO:tasks.rados.rados.0.smithi137.stdout:1520: done (1 left) 2024-09-21T08:29:46.685 INFO:tasks.rados.rados.0.smithi137.stdout:1521: read oid 29 snap -1 2024-09-21T08:29:46.685 INFO:tasks.rados.rados.0.smithi137.stdout:1521: expect (ObjNum 385 snap 94 seq_num 385) 2024-09-21T08:29:46.685 INFO:tasks.rados.rados.0.smithi137.stdout:1522: write oid 50 current snap is 162 2024-09-21T08:29:46.685 INFO:tasks.rados.rados.0.smithi137.stdout:1519: finishing write tid 2 to smithi13779221-2 2024-09-21T08:29:46.686 INFO:tasks.rados.rados.0.smithi137.stdout:1522: seq_num 544 ranges {743008=796519,2200322=721849} 2024-09-21T08:29:46.694 INFO:tasks.rados.rados.0.smithi137.stdout:1522: writing smithi13779221-50 from 743008 to 1539527 tid 1 2024-09-21T08:29:46.702 INFO:tasks.rados.rados.0.smithi137.stdout:1522: writing smithi13779221-50 from 2200322 to 2922171 tid 2 2024-09-21T08:29:46.702 INFO:tasks.rados.rados.0.smithi137.stdout:1523: rollback oid 33 current snap is 162 2024-09-21T08:29:46.703 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 33 to 138 2024-09-21T08:29:46.703 INFO:tasks.rados.rados.0.smithi137.stdout:1519: finishing write tid 3 to smithi13779221-2 2024-09-21T08:29:46.703 INFO:tasks.rados.rados.0.smithi137.stdout:1519: finishing write tid 4 to smithi13779221-2 2024-09-21T08:29:46.703 INFO:tasks.rados.rados.0.smithi137.stdout:1519: oid 2 updating version 0 to 165 2024-09-21T08:29:46.703 INFO:tasks.rados.rados.0.smithi137.stdout:1519: oid 2 version 165 is already newer than 164 2024-09-21T08:29:46.703 INFO:tasks.rados.rados.0.smithi137.stdout:1519: oid 2 updating version 165 to 166 2024-09-21T08:29:46.703 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 166 (ObjNum 542 snap 162 seq_num 542) dirty exists 2024-09-21T08:29:46.703 INFO:tasks.rados.rados.0.smithi137.stdout:1519: left oid 2 (ObjNum 542 snap 162 seq_num 542) 2024-09-21T08:29:46.704 INFO:tasks.rados.rados.0.smithi137.stdout:1521: expect (ObjNum 385 snap 94 seq_num 385) 2024-09-21T08:29:46.751 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:46 smithi165 ceph-mon[32893]: osdmap e392: 8 total, 6 up, 4 in 2024-09-21T08:29:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:46 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4094157951' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:46 smithi165 ceph-mon[32893]: from='client.45608 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:46 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2575009086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:46 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 303/1185 objects degraded (25.570%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:46.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:46.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:46.773 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:46.839 INFO:tasks.rados.rados.0.smithi137.stdout:1522: finishing write tid 1 to smithi13779221-50 2024-09-21T08:29:46.840 INFO:tasks.rados.rados.0.smithi137.stdout:1522: finishing write tid 2 to smithi13779221-50 2024-09-21T08:29:46.840 INFO:tasks.rados.rados.0.smithi137.stdout:1523: finishing rollback tid 0 to smithi13779221-33 2024-09-21T08:29:46.840 INFO:tasks.rados.rados.0.smithi137.stdout:1523: finishing rollback tid 1 to smithi13779221-33 2024-09-21T08:29:46.840 INFO:tasks.rados.rados.0.smithi137.stdout:1522: finishing write tid 3 to smithi13779221-50 2024-09-21T08:29:46.840 INFO:tasks.rados.rados.0.smithi137.stdout:1523: finishing rollback tid 2 to smithi13779221-33 2024-09-21T08:29:46.840 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 289 (ObjNum 472 snap 128 seq_num 472) dirty exists 2024-09-21T08:29:46.840 INFO:tasks.rados.rados.0.smithi137.stdout:1522: finishing write tid 4 to smithi13779221-50 2024-09-21T08:29:46.840 INFO:tasks.rados.rados.0.smithi137.stdout:1522: oid 50 updating version 0 to 390 2024-09-21T08:29:46.840 INFO:tasks.rados.rados.0.smithi137.stdout:1522: oid 50 updating version 390 to 392 2024-09-21T08:29:46.840 INFO:tasks.rados.rados.0.smithi137.stdout:1522: oid 50 version 392 is already newer than 391 2024-09-21T08:29:46.841 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 392 (ObjNum 544 snap 162 seq_num 544) dirty exists 2024-09-21T08:29:46.841 INFO:tasks.rados.rados.0.smithi137.stdout:1522: left oid 50 (ObjNum 544 snap 162 seq_num 544) 2024-09-21T08:29:46.841 INFO:tasks.rados.rados.0.smithi137.stdout:1519: done (3 left) 2024-09-21T08:29:46.841 INFO:tasks.rados.rados.0.smithi137.stdout:1521: done (2 left) 2024-09-21T08:29:46.841 INFO:tasks.rados.rados.0.smithi137.stdout:1522: done (1 left) 2024-09-21T08:29:46.841 INFO:tasks.rados.rados.0.smithi137.stdout:1523: done (0 left) 2024-09-21T08:29:46.841 INFO:tasks.rados.rados.0.smithi137.stdout:1524: read oid 19 snap -1 2024-09-21T08:29:46.841 INFO:tasks.rados.rados.0.smithi137.stdout:1524: expect deleted 2024-09-21T08:29:46.841 INFO:tasks.rados.rados.0.smithi137.stdout:1525: write oid 34 current snap is 162 2024-09-21T08:29:46.841 INFO:tasks.rados.rados.0.smithi137.stdout:1525: seq_num 545 ranges {759815=471768,1742451=465507,2917178=242637} 2024-09-21T08:29:46.846 INFO:tasks.rados.rados.0.smithi137.stdout:1525: writing smithi13779221-34 from 759815 to 1231583 tid 1 2024-09-21T08:29:46.851 INFO:tasks.rados.rados.0.smithi137.stdout:1525: writing smithi13779221-34 from 1742451 to 2207958 tid 2 2024-09-21T08:29:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:1525: writing smithi13779221-34 from 2917178 to 3159815 tid 3 2024-09-21T08:29:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:1526: read oid 40 snap -1 2024-09-21T08:29:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:1526: expect (ObjNum 522 snap 149 seq_num 522) 2024-09-21T08:29:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:1527: read oid 37 snap -1 2024-09-21T08:29:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:1527: expect (ObjNum 539 snap 161 seq_num 539) 2024-09-21T08:29:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:1528: rollback oid 30 current snap is 162 2024-09-21T08:29:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 30 to 155 2024-09-21T08:29:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:1524: done (4 left) 2024-09-21T08:29:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:1529: snap_create 2024-09-21T08:29:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:1525: finishing write tid 1 to smithi13779221-34 2024-09-21T08:29:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:1525: finishing write tid 2 to smithi13779221-34 2024-09-21T08:29:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:1525: finishing write tid 3 to smithi13779221-34 2024-09-21T08:29:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:1525: finishing write tid 4 to smithi13779221-34 2024-09-21T08:29:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:1525: finishing write tid 5 to smithi13779221-34 2024-09-21T08:29:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:1525: oid 34 updating version 0 to 429 2024-09-21T08:29:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:1525: oid 34 updating version 429 to 430 2024-09-21T08:29:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:1525: oid 34 updating version 430 to 431 2024-09-21T08:29:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:1525: oid 34 updating version 431 to 432 2024-09-21T08:29:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 432 (ObjNum 545 snap 162 seq_num 545) dirty exists 2024-09-21T08:29:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:1525: left oid 34 (ObjNum 545 snap 162 seq_num 545) 2024-09-21T08:29:46.867 INFO:tasks.rados.rados.0.smithi137.stdout:1527: expect (ObjNum 539 snap 161 seq_num 539) 2024-09-21T08:29:46.874 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:46.895 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:46.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:46.897 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:46.952 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:46 smithi137 ceph-mon[29634]: osdmap e392: 8 total, 6 up, 4 in 2024-09-21T08:29:46.952 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:46 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4094157951' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:46.953 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:46 smithi137 ceph-mon[29634]: from='client.45608 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:46.953 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:46 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2575009086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:46.953 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:46 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 303/1185 objects degraded (25.570%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:46.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:46 smithi137 ceph-mon[41044]: osdmap e392: 8 total, 6 up, 4 in 2024-09-21T08:29:46.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:46 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4094157951' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:46.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:46 smithi137 ceph-mon[41044]: from='client.45608 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:46.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:46 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2575009086' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:46.953 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:46 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 303/1185 objects degraded (25.570%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:46.966 INFO:tasks.rados.rados.0.smithi137.stdout:1526: expect (ObjNum 522 snap 149 seq_num 522) 2024-09-21T08:29:46.979 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:46.979 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:46.997 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:47.018 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:47.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:47.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:47.059 INFO:tasks.rados.rados.0.smithi137.stdout:1528: finishing rollback tid 0 to smithi13779221-30 2024-09-21T08:29:47.059 INFO:tasks.rados.rados.0.smithi137.stdout:1528: finishing rollback tid 1 to smithi13779221-30 2024-09-21T08:29:47.059 INFO:tasks.rados.rados.0.smithi137.stdout:1528: finishing rollback tid 2 to smithi13779221-30 2024-09-21T08:29:47.059 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 629 (ObjNum 448 snap 117 seq_num 448) dirty exists 2024-09-21T08:29:47.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:47.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T08:29:47.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:47.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:47.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:47.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:47.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:47.265 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:47.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:47.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:47.384 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:47.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:47.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:47.388 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:47.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:47.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:47.512 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:47.513 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:47.613 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:47.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:47.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:47.636 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:47.648 INFO:tasks.rados.rados.0.smithi137.stdout:1525: done (4 left) 2024-09-21T08:29:47.649 INFO:tasks.rados.rados.0.smithi137.stdout:1526: done (3 left) 2024-09-21T08:29:47.649 INFO:tasks.rados.rados.0.smithi137.stdout:1527: done (2 left) 2024-09-21T08:29:47.649 INFO:tasks.rados.rados.0.smithi137.stdout:1528: done (1 left) 2024-09-21T08:29:47.649 INFO:tasks.rados.rados.0.smithi137.stdout:1529: done (0 left) 2024-09-21T08:29:47.649 INFO:tasks.rados.rados.0.smithi137.stdout:1530: copy_from oid 31 from oid 9 current snap is 163 2024-09-21T08:29:47.649 INFO:tasks.rados.rados.0.smithi137.stdout:1531: write (excl) oid 4 current snap is 163 2024-09-21T08:29:47.649 INFO:tasks.rados.rados.0.smithi137.stdout:1531: seq_num 546 ranges {776622=547017,1800933=725518,2944601=232021} 2024-09-21T08:29:47.657 INFO:tasks.rados.rados.0.smithi137.stdout:1531: writing smithi13779221-4 from 776622 to 1323639 tid 1 2024-09-21T08:29:47.664 INFO:tasks.rados.rados.0.smithi137.stdout:1531: writing smithi13779221-4 from 1800933 to 2526451 tid 2 2024-09-21T08:29:47.668 INFO:tasks.rados.rados.0.smithi137.stdout:1531: writing smithi13779221-4 from 2944601 to 3176622 tid 3 2024-09-21T08:29:47.668 INFO:tasks.rados.rados.0.smithi137.stdout:1532: read oid 43 snap -1 2024-09-21T08:29:47.668 INFO:tasks.rados.rados.0.smithi137.stdout:1532: expect (ObjNum 527 snap 149 seq_num 527) 2024-09-21T08:29:47.668 INFO:tasks.rados.rados.0.smithi137.stdout:1533: snap_remove snap 157 2024-09-21T08:29:47.694 INFO:tasks.thrashosds.thrasher:Moving pg 4.16 from osd.7 to osd.0 2024-09-21T08:29:47.694 INFO:tasks.thrashosds.thrasher:Transfer export file from ubuntu@smithi165.front.sepia.ceph.com to ubuntu@smithi137.front.sepia.ceph.com 2024-09-21T08:29:47.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 777 /var/log/ceph 2024-09-21T08:29:47.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo chmod 777 /var/log/ceph 2024-09-21T08:29:47.737 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:47.762 DEBUG:teuthology.orchestra.run.smithi165:> mktemp 2024-09-21T08:29:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:47 smithi165 ceph-mon[32893]: pgmap v494: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 392 MiB data, 1014 MiB used, 357 GiB / 358 GiB avail; 5.1 MiB/s rd, 329 KiB/s wr, 7 op/s; 304/1188 objects degraded (25.589%) 2024-09-21T08:29:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:47 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:47 smithi165 ceph-mon[32893]: from='client.45608 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:47.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:47 smithi165 ceph-mon[32893]: osdmap e393: 8 total, 6 up, 4 in 2024-09-21T08:29:47.778 INFO:teuthology.orchestra.run.smithi165.stdout:/tmp/tmp.B1GyeW4bi8 2024-09-21T08:29:47.779 DEBUG:teuthology.orchestra.run.smithi165:> sudo cp /var/log/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/exp.4.16.7 /tmp/tmp.B1GyeW4bi8 2024-09-21T08:29:47.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:47.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:47.784 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:47.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo chmod 0666 /tmp/tmp.B1GyeW4bi8 2024-09-21T08:29:50.003 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:50.004 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:29:50.005 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:47 smithi137 ceph-mon[29634]: pgmap v494: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 392 MiB data, 1014 MiB used, 357 GiB / 358 GiB avail; 5.1 MiB/s rd, 329 KiB/s wr, 7 op/s; 304/1188 objects degraded (25.589%) 2024-09-21T08:29:50.005 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:47 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:50.005 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:47 smithi137 ceph-mon[29634]: from='client.45608 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:50.005 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:47 smithi137 ceph-mon[29634]: osdmap e393: 8 total, 6 up, 4 in 2024-09-21T08:29:50.005 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:47 smithi137 ceph-mon[41044]: pgmap v494: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 392 MiB data, 1014 MiB used, 357 GiB / 358 GiB avail; 5.1 MiB/s rd, 329 KiB/s wr, 7 op/s; 304/1188 objects degraded (25.589%) 2024-09-21T08:29:50.005 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:47 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:50.005 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:47 smithi137 ceph-mon[41044]: from='client.45608 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:29:50.006 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:47 smithi137 ceph-mon[41044]: osdmap e393: 8 total, 6 up, 4 in 2024-09-21T08:29:50.007 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:50.007 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:50.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T08:29:50.008 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:48 smithi165 ceph-mon[32893]: osdmap e394: 8 total, 6 up, 4 in 2024-09-21T08:29:50.009 INFO:tasks.rados.rados.0.smithi137.stdout:1530: finishing copy_from to smithi13779221-31 2024-09-21T08:29:50.009 INFO:tasks.rados.rados.0.smithi137.stdout:1530: got expected ENOENT (src dne) 2024-09-21T08:29:50.009 INFO:tasks.rados.rados.0.smithi137.stdout:1530: finishing copy_from racing read to smithi13779221-31 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:1531: finishing write tid 1 to smithi13779221-4 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:1531: finishing write tid 2 to smithi13779221-4 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:1531: finishing write tid 3 to smithi13779221-4 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:1531: finishing write tid 4 to smithi13779221-4 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:1531: finishing write tid 5 to smithi13779221-4 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:1531: oid 4 updating version 0 to 322 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:1531: oid 4 updating version 322 to 323 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:1531: oid 4 updating version 323 to 325 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:1531: oid 4 version 325 is already newer than 324 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 325 (ObjNum 546 snap 163 seq_num 546) dirty exists 2024-09-21T08:29:50.010 INFO:tasks.rados.rados.0.smithi137.stdout:1531: left oid 4 (ObjNum 546 snap 163 seq_num 546) 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1532: expect (ObjNum 527 snap 149 seq_num 527) 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1530: done (3 left) 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1531: done (2 left) 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1532: done (1 left) 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1533: done (0 left) 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1534: write (excl) oid 6 current snap is 163 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1534: seq_num 547 ranges {793429=622266,1859415=469176,2972024=221405} 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1534: writing smithi13779221-6 from 793429 to 1415695 tid 1 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1534: writing smithi13779221-6 from 1859415 to 2328591 tid 2 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1534: writing smithi13779221-6 from 2972024 to 3193429 tid 3 2024-09-21T08:29:50.011 INFO:tasks.rados.rados.0.smithi137.stdout:1535: snap_create 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:1534: finishing write tid 1 to smithi13779221-6 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:1534: finishing write tid 2 to smithi13779221-6 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:1534: finishing write tid 3 to smithi13779221-6 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:1534: finishing write tid 4 to smithi13779221-6 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:1534: finishing write tid 5 to smithi13779221-6 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:1534: oid 6 updating version 0 to 370 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:1534: oid 6 version 370 is already newer than 369 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:1534: oid 6 updating version 370 to 371 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:1534: oid 6 version 371 is already newer than 368 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 371 (ObjNum 547 snap 163 seq_num 547) dirty exists 2024-09-21T08:29:50.012 INFO:tasks.rados.rados.0.smithi137.stdout:1534: left oid 6 (ObjNum 547 snap 163 seq_num 547) 2024-09-21T08:29:50.013 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:48 smithi137 ceph-mon[29634]: osdmap e394: 8 total, 6 up, 4 in 2024-09-21T08:29:50.013 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:48 smithi137 ceph-mon[41044]: osdmap e394: 8 total, 6 up, 4 in 2024-09-21T08:29:50.013 INFO:tasks.rados.rados.0.smithi137.stdout:1534: done (1 left) 2024-09-21T08:29:50.013 INFO:tasks.rados.rados.0.smithi137.stdout:1535: done (0 left) 2024-09-21T08:29:50.013 INFO:tasks.rados.rados.0.smithi137.stdout:1536: snap_create 2024-09-21T08:29:50.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:49 smithi165 ceph-mon[32893]: pgmap v497: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 392 MiB data, 1014 MiB used, 357 GiB / 358 GiB avail; 5.1 MiB/s rd, 329 KiB/s wr, 7 op/s; 304/1188 objects degraded (25.589%) 2024-09-21T08:29:50.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:49 smithi165 ceph-mon[32893]: osdmap e395: 8 total, 6 up, 4 in 2024-09-21T08:29:50.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:49 smithi137 ceph-mon[29634]: pgmap v497: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 392 MiB data, 1014 MiB used, 357 GiB / 358 GiB avail; 5.1 MiB/s rd, 329 KiB/s wr, 7 op/s; 304/1188 objects degraded (25.589%) 2024-09-21T08:29:50.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:49 smithi137 ceph-mon[29634]: osdmap e395: 8 total, 6 up, 4 in 2024-09-21T08:29:50.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:49 smithi137 ceph-mon[41044]: pgmap v497: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 392 MiB data, 1014 MiB used, 357 GiB / 358 GiB avail; 5.1 MiB/s rd, 329 KiB/s wr, 7 op/s; 304/1188 objects degraded (25.589%) 2024-09-21T08:29:50.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:49 smithi137 ceph-mon[41044]: osdmap e395: 8 total, 6 up, 4 in 2024-09-21T08:29:50.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:50.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:50.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:50.074 DEBUG:teuthology.orchestra.remote:smithi165:/tmp/tmp.B1GyeW4bi8 is 14KB 2024-09-21T08:29:50.097 DEBUG:teuthology.orchestra.run.smithi165:> rm -fr /tmp/tmp.B1GyeW4bi8 2024-09-21T08:29:50.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo mv /var/log/ceph/exp.4.16.7 /var/log/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/exp.4.16.7 2024-09-21T08:29:50.144 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:50.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell -n osd.0 --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph-objectstore-tool --err-to-stderr --data-path /var/lib/ceph/osd/ceph-0 --journal-path /var/lib/ceph/osd/ceph-0/journal '--log-file=/var/log/ceph/objectstore_tool.$pid.log' --op import --file /var/log/ceph/exp.4.16.7 2024-09-21T08:29:50.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:50.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:50.174 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:50.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:50.278 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:50.282 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:50.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:50.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:50.297 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:50.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:50.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:50.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:50.422 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:50.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:50.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:50.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:50.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:50.576 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/osd.0/config 2024-09-21T08:29:50.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:50.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:50.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:50.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:50.674 INFO:tasks.rados.rados.0.smithi137.stdout:1536: done (0 left) 2024-09-21T08:29:50.675 INFO:tasks.rados.rados.0.smithi137.stdout:1537: copy_from oid 7 from oid 29 current snap is 165 2024-09-21T08:29:50.675 INFO:tasks.rados.rados.0.smithi137.stdout:1538: copy_from oid 34 from oid 9 current snap is 165 2024-09-21T08:29:50.675 INFO:tasks.rados.rados.0.smithi137.stdout:1539: delete oid 20 current snap is 165 2024-09-21T08:29:50.685 INFO:tasks.rados.rados.0.smithi137.stdout:1539: done (2 left) 2024-09-21T08:29:50.685 INFO:tasks.rados.rados.0.smithi137.stdout:1540: delete oid 2 current snap is 165 2024-09-21T08:29:50.689 INFO:tasks.rados.rados.0.smithi137.stdout:1540: done (2 left) 2024-09-21T08:29:50.689 INFO:tasks.rados.rados.0.smithi137.stdout:1541: write (excl) oid 23 current snap is 165 2024-09-21T08:29:50.689 INFO:tasks.rados.rados.0.smithi137.stdout:1541: seq_num 550 ranges {443850=564366,1634861=732856,2770646=473204} 2024-09-21T08:29:50.694 INFO:tasks.rados.rados.0.smithi137.stdout:1541: writing smithi13779221-23 from 443850 to 1008216 tid 1 2024-09-21T08:29:50.702 INFO:tasks.rados.rados.0.smithi137.stdout:1541: writing smithi13779221-23 from 1634861 to 2367717 tid 2 2024-09-21T08:29:50.707 INFO:tasks.rados.rados.0.smithi137.stdout:1541: writing smithi13779221-23 from 2770646 to 3243850 tid 3 2024-09-21T08:29:50.707 INFO:tasks.rados.rados.0.smithi137.stdout:1542: snap_create 2024-09-21T08:29:50.707 INFO:tasks.rados.rados.0.smithi137.stdout:1537: finishing copy_from racing read to smithi13779221-7 2024-09-21T08:29:50.707 INFO:tasks.rados.rados.0.smithi137.stdout:1538: finishing copy_from to smithi13779221-34 2024-09-21T08:29:50.707 INFO:tasks.rados.rados.0.smithi137.stdout:1538: got expected ENOENT (src dne) 2024-09-21T08:29:50.707 INFO:tasks.rados.rados.0.smithi137.stdout:1538: finishing copy_from racing read to smithi13779221-34 2024-09-21T08:29:50.708 INFO:tasks.rados.rados.0.smithi137.stdout:1537: finishing copy_from to smithi13779221-7 2024-09-21T08:29:50.708 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 581 (ObjNum 385 snap 94 seq_num 385) dirty exists 2024-09-21T08:29:50.708 INFO:tasks.rados.rados.0.smithi137.stdout:1541: finishing write tid 1 to smithi13779221-23 2024-09-21T08:29:50.708 INFO:tasks.rados.rados.0.smithi137.stdout:1541: finishing write tid 2 to smithi13779221-23 2024-09-21T08:29:50.709 INFO:tasks.rados.rados.0.smithi137.stdout:1541: finishing write tid 3 to smithi13779221-23 2024-09-21T08:29:50.709 INFO:tasks.rados.rados.0.smithi137.stdout:1541: finishing write tid 4 to smithi13779221-23 2024-09-21T08:29:50.710 INFO:tasks.rados.rados.0.smithi137.stdout:1541: finishing write tid 5 to smithi13779221-23 2024-09-21T08:29:50.710 INFO:tasks.rados.rados.0.smithi137.stdout:1541: oid 23 updating version 0 to 191 2024-09-21T08:29:50.710 INFO:tasks.rados.rados.0.smithi137.stdout:1541: oid 23 version 191 is already newer than 189 2024-09-21T08:29:50.710 INFO:tasks.rados.rados.0.smithi137.stdout:1541: oid 23 version 191 is already newer than 190 2024-09-21T08:29:50.710 INFO:tasks.rados.rados.0.smithi137.stdout:1541: oid 23 version 191 is already newer than 188 2024-09-21T08:29:50.710 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 191 (ObjNum 550 snap 165 seq_num 550) dirty exists 2024-09-21T08:29:50.710 INFO:tasks.rados.rados.0.smithi137.stdout:1541: left oid 23 (ObjNum 550 snap 165 seq_num 550) 2024-09-21T08:29:50.755 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:50.755 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:50.772 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:50.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:50.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:50.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:50.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:50.897 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:50.898 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:50.899 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:50.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:50.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:50.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:50 smithi165 ceph-mon[32893]: osdmap e396: 8 total, 6 up, 4 in 2024-09-21T08:29:51.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:50 smithi137 ceph-mon[29634]: osdmap e396: 8 total, 6 up, 4 in 2024-09-21T08:29:51.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:50 smithi137 ceph-mon[41044]: osdmap e396: 8 total, 6 up, 4 in 2024-09-21T08:29:51.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:29:50 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:29:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:29:51.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:51.039 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:51.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:51.040 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:51.140 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:51.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:51.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:51.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:51.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:51.287 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:51.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:51.288 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:51.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T08:29:51.389 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:51.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:51.411 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:51.412 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:51.512 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:51.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:51.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:51.534 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:51.561 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:51.562 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:51.568 INFO:tasks.rados.rados.0.smithi137.stdout:1537: done (3 left) 2024-09-21T08:29:51.568 INFO:tasks.rados.rados.0.smithi137.stdout:1538: done (2 left) 2024-09-21T08:29:51.568 INFO:tasks.rados.rados.0.smithi137.stdout:1541: done (1 left) 2024-09-21T08:29:51.568 INFO:tasks.rados.rados.0.smithi137.stdout:1542: done (0 left) 2024-09-21T08:29:51.568 INFO:tasks.rados.rados.0.smithi137.stdout:1543: copy_from oid 46 from oid 1 current snap is 166 2024-09-21T08:29:51.568 INFO:tasks.rados.rados.0.smithi137.stdout:1544: delete oid 8 current snap is 166 2024-09-21T08:29:51.579 INFO:tasks.rados.rados.0.smithi137.stdout:1544: done (1 left) 2024-09-21T08:29:51.579 INFO:tasks.rados.rados.0.smithi137.stdout:1545: read oid 45 snap -1 2024-09-21T08:29:51.579 INFO:tasks.rados.rados.0.smithi137.stdout:1545: expect (ObjNum 467 snap 126 seq_num 467) 2024-09-21T08:29:51.579 INFO:tasks.rados.rados.0.smithi137.stdout:1546: write (excl) oid 17 current snap is 166 2024-09-21T08:29:51.579 INFO:tasks.rados.rados.0.smithi137.stdout:1546: seq_num 552 ranges {477464=714864,1751825=736525,3225492=51972} 2024-09-21T08:29:51.585 INFO:tasks.rados.rados.0.smithi137.stdout:1546: writing smithi13779221-17 from 477464 to 1192328 tid 1 2024-09-21T08:29:51.593 INFO:tasks.rados.rados.0.smithi137.stdout:1546: writing smithi13779221-17 from 1751825 to 2488350 tid 2 2024-09-21T08:29:51.597 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:29:51.598 INFO:tasks.rados.rados.0.smithi137.stdout:1546: writing smithi13779221-17 from 3225492 to 3277464 tid 3 2024-09-21T08:29:51.598 INFO:tasks.rados.rados.0.smithi137.stdout:1547: copy_from oid 14 from oid 43 current snap is 166 2024-09-21T08:29:51.598 INFO:tasks.rados.rados.0.smithi137.stdout:1543: finishing copy_from racing read to smithi13779221-46 2024-09-21T08:29:51.598 INFO:tasks.rados.rados.0.smithi137.stdout:1545: expect (ObjNum 467 snap 126 seq_num 467) 2024-09-21T08:29:51.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:51.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:51.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:51.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:51.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:51.777 INFO:tasks.rados.rados.0.smithi137.stdout:1545: done (3 left) 2024-09-21T08:29:51.777 INFO:tasks.rados.rados.0.smithi137.stdout:1548: copy_from oid 36 from oid 42 current snap is 166 2024-09-21T08:29:51.777 INFO:tasks.rados.rados.0.smithi137.stdout:1543: finishing copy_from to smithi13779221-46 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 529 (ObjNum 499 snap 140 seq_num 499) dirty exists 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1546: finishing write tid 1 to smithi13779221-17 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1547: finishing copy_from racing read to smithi13779221-14 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1546: finishing write tid 2 to smithi13779221-17 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1546: finishing write tid 3 to smithi13779221-17 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1546: finishing write tid 4 to smithi13779221-17 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1546: finishing write tid 5 to smithi13779221-17 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1546: oid 17 updating version 0 to 502 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1546: oid 17 updating version 502 to 504 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1546: oid 17 version 504 is already newer than 503 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1546: oid 17 updating version 504 to 505 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 505 (ObjNum 552 snap 166 seq_num 552) dirty exists 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1546: left oid 17 (ObjNum 552 snap 166 seq_num 552) 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:1547: finishing copy_from to smithi13779221-14 2024-09-21T08:29:51.778 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 374 (ObjNum 527 snap 149 seq_num 527) dirty exists 2024-09-21T08:29:51.779 INFO:tasks.rados.rados.0.smithi137.stdout:1543: done (3 left) 2024-09-21T08:29:51.779 INFO:tasks.rados.rados.0.smithi137.stdout:1546: done (2 left) 2024-09-21T08:29:51.779 INFO:tasks.rados.rados.0.smithi137.stdout:1547: done (1 left) 2024-09-21T08:29:51.779 INFO:tasks.rados.rados.0.smithi137.stdout:1549: read oid 24 snap -1 2024-09-21T08:29:51.779 INFO:tasks.rados.rados.0.smithi137.stdout:1549: expect (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:29:51.779 INFO:tasks.rados.rados.0.smithi137.stdout:1550: snap_remove snap 155 2024-09-21T08:29:51.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:51.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:51.782 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:51.831 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:51 smithi137 ceph-mon[29634]: pgmap v500: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 392 MiB data, 1014 MiB used, 357 GiB / 358 GiB avail; 304/1188 objects degraded (25.589%) 2024-09-21T08:29:51.831 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:51 smithi137 ceph-mon[29634]: osdmap e397: 8 total, 6 up, 4 in 2024-09-21T08:29:51.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:51 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/788487916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:51.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:51 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 304/1188 objects degraded (25.589%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:51.832 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:51 smithi137 ceph-mon[29634]: osdmap e398: 8 total, 6 up, 4 in 2024-09-21T08:29:51.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:51 smithi137 ceph-mon[41044]: pgmap v500: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 392 MiB data, 1014 MiB used, 357 GiB / 358 GiB avail; 304/1188 objects degraded (25.589%) 2024-09-21T08:29:51.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:51 smithi137 ceph-mon[41044]: osdmap e397: 8 total, 6 up, 4 in 2024-09-21T08:29:51.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:51 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/788487916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:51.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:51 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 304/1188 objects degraded (25.589%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:51.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:51 smithi137 ceph-mon[41044]: osdmap e398: 8 total, 6 up, 4 in 2024-09-21T08:29:51.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:51.905 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:51.905 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:51.905 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:52.006 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:52.008 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:29:52.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:29:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:29:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:51 smithi165 ceph-mon[32893]: pgmap v500: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 392 MiB data, 1014 MiB used, 357 GiB / 358 GiB avail; 304/1188 objects degraded (25.589%) 2024-09-21T08:29:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:51 smithi165 ceph-mon[32893]: osdmap e397: 8 total, 6 up, 4 in 2024-09-21T08:29:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:51 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/788487916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:52.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:51 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 304/1188 objects degraded (25.589%), 58 pgs degraded (PG_DEGRADED) 2024-09-21T08:29:52.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:51 smithi165 ceph-mon[32893]: osdmap e398: 8 total, 6 up, 4 in 2024-09-21T08:29:52.027 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:52.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:52.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:52.129 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:52.139 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:29:52.139 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:29:52.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:29:52.139 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:29:52.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 377.56910685499997, 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049090319999999998, 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567959+0000", 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.762e-06 2024-09-21T08:29:52.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567965+0000", 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567980+0000", 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4742e-05 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567990+0000", 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8800000000000003e-06 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568114+0000", 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012444499999999999 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568250+0000", 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135205 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568390+0000", 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140335 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569250+0000", 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085971599999999995 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572132+0000", 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028820819999999998 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572759+0000", 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062750100000000004 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572810+0000", 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0488000000000002e-05 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572867+0000", 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6999000000000001e-05 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.145 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 377.56864004599998, 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045637589999999997, 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:52.146 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568426+0000", 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6470000000000001e-06 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568432+0000", 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.395e-06 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568443+0000", 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1070000000000001e-05 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568456+0000", 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3862e-05 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568688+0000", 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023171699999999999 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568773+0000", 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5307000000000002e-05 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568890+0000", 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116082 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569279+0000", 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038986899999999999 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572238+0000", 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029581009999999999 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572906+0000", 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000668945 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572955+0000", 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8455000000000003e-05 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572988+0000", 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3309000000000002e-05 2024-09-21T08:29:52.151 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:47 5.1f 5:f95f44c2:::notify.0:head [watch watch cookie 94074441540560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 377.563858966, 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026954729999999999, 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:29:52.152 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 47 2024-09-21T08:29:52.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.153 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573208+0000", 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9970000000000001e-06 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573213+0000", 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5679999999999999e-06 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573224+0000", 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1270000000000001e-05 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573233+0000", 2024-09-21T08:29:52.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8939999999999999e-06 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573770+0000", 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053668100000000001 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573841+0000", 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.135e-05 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.574869+0000", 2024-09-21T08:29:52.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010281719999999999 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.574911+0000", 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1631000000000002e-05 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575071+0000", 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015977 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:52.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575463+0000", 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039199299999999999 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575737+0000", 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027415300000000002 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575785+0000", 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7970999999999999e-05 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575832+0000", 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7120999999999998e-05 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575901+0000", 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8901999999999999e-05 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.159 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:192 4.5 4:a2ace8b3:::data_log.0:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 377.52123827999998, 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031080779999999998, 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.160 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615827+0000", 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1629999999999999e-06 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615831+0000", 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0790000000000002e-06 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615841+0000", 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8279999999999998e-06 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615847+0000", 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2660000000000001e-06 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615879+0000", 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2512999999999999e-05 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615924+0000", 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4709999999999997e-05 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.618934+0000", 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030105190000000001 2024-09-21T08:29:52.163 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1084 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94837804042368 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 313.97744693300001, 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40449157200000002, 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1084 2024-09-21T08:29:52.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159622+0000", 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1400000000000002e-06 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159631+0000", 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9670000000000007e-06 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159655+0000", 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4037999999999999e-05 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159667+0000", 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2427e-05 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159756+0000", 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9103000000000004e-05 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159772+0000", 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5877000000000001e-05 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.554979+0000", 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39520667999999998 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.555262+0000", 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028320500000000002 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.560260+0000", 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049979630000000002 2024-09-21T08:29:52.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.560331+0000", 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1048000000000001e-05 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.564109+0000", 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037781239999999999 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.169 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1085 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94837808833664 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 313.97730770099997, 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40437364799999997, 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1085 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159758+0000", 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.539e-06 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159764+0000", 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1330000000000004e-06 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159773+0000", 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3250000000000008e-06 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159779+0000", 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5400000000000001e-06 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159844+0000", 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4580999999999997e-05 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159854+0000", 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.893e-06 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552880+0000", 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.393026184 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.173 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.553195+0000", 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031526299999999998 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559454+0000", 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062587809999999997 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559541+0000", 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7379999999999993e-05 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.564130+0000", 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045890289999999997 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:265 7.b e93/80 7:dd97af7e:::smithi13779221-30:head v 93'26, mlcod=86'23)", 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 305.57331264700002, 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036488330000000002, 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.175 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563753+0000", 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4000000000000001e-07 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565430+0000", 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016774439999999999 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565446+0000", 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5540999999999999e-05 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565455+0000", 2024-09-21T08:29:52.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9580000000000002e-06 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565513+0000", 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7312000000000003e-05 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565542+0000", 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8856999999999998e-05 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:52.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.567363+0000", 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018212300000000001 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.567401+0000", 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7851000000000001e-05 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:321 7.c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 6] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 305.32743728200001, 2024-09-21T08:29:52.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013109218000000001, 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 321 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809632+0000", 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9559999999999996e-06 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809648+0000", 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5814000000000001e-05 2024-09-21T08:29:52.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809680+0000", 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1732e-05 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809696+0000", 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6293000000000001e-05 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809815+0000", 2024-09-21T08:29:52.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119442 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809903+0000", 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7033999999999994e-05 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.815434+0000", 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055318420000000004 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.820591+0000", 2024-09-21T08:29:52.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051568229999999996 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.821125+0000", 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053413400000000002 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822656+0000", 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001531035 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822697+0000", 2024-09-21T08:29:52.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0488999999999998e-05 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822736+0000", 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9623999999999999e-05 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:322 7.c 7:3cae00e7:::smithi13779221-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 305.32733903000002, 2024-09-21T08:29:52.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093384190000000006, 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 322 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809727+0000", 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.13e-06 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809734+0000", 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9569999999999996e-06 2024-09-21T08:29:52.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809744+0000", 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0633e-05 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809753+0000", 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7090000000000008e-06 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.810046+0000", 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029301499999999999 2024-09-21T08:29:52.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.810079+0000", 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3293000000000001e-05 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.818920+0000", 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088405299999999992 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.818982+0000", 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999000000000001e-05 2024-09-21T08:29:52.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.819064+0000", 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2152999999999998e-05 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:329 7.c 7:361ee62b:::smithi13779221-40:head [sparse-read 0~1272916,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 304.45088617800002, 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90863698400000004, 2024-09-21T08:29:52.190 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 329 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686182+0000", 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0459999999999999e-06 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686196+0000", 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4097000000000001e-05 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686225+0000", 2024-09-21T08:29:52.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8846000000000001e-05 2024-09-21T08:29:52.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686240+0000", 2024-09-21T08:29:52.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4256e-05 2024-09-21T08:29:52.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:52.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:52.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:52.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686362+0000", 2024-09-21T08:29:52.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012248200000000001 2024-09-21T08:29:52.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686390+0000", 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7546e-05 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.591277+0000", 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90488744700000001 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.591416+0000", 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013935900000000001 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.594815+0000", 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033989049999999998 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:330 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2655473,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 304.450797239, 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91026091200000003, 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.197 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 330 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686269+0000", 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7060000000000001e-06 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686277+0000", 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4470000000000006e-06 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686294+0000", 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6099999999999998e-05 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686308+0000", 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4516e-05 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686428+0000", 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119834 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:52.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686436+0000", 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1450000000000008e-06 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.595025+0000", 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90858932599999997 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.595096+0000", 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0821999999999994e-05 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596528+0000", 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014320159999999999 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:331 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2655473,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 304.45073774700001, 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91109795900000001, 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.202 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 331 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686328+0000", 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.387e-06 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686336+0000", 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4030000000000001e-06 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686350+0000", 2024-09-21T08:29:52.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4713e-05 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686364+0000", 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3716999999999999e-05 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686455+0000", 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0737999999999998e-05 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686460+0000", 2024-09-21T08:29:52.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1290000000000004e-06 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596614+0000", 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91015405299999996 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596687+0000", 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2700000000000005e-05 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.597425+0000", 2024-09-21T08:29:52.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00073811899999999997 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:338 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 303.56923280699999, 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038704049999999999, 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.207 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 338 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567835+0000", 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0019999999999998e-06 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567843+0000", 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0090000000000007e-06 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567861+0000", 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8748000000000001e-05 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567872+0000", 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0173000000000001e-05 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567975+0000", 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103398 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568085+0000", 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109691 2024-09-21T08:29:52.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571702+0000", 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003617384 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:339 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 303.56915770799998, 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054334199999999996, 2024-09-21T08:29:52.211 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 339 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567908+0000", 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-06 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567913+0000", 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.536e-06 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567922+0000", 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0730000000000005e-06 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567928+0000", 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4489999999999996e-06 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571741+0000", 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038125590000000001 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571859+0000", 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118249 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.573340+0000", 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001481472 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:340 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:29:52.215 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 303.568959183, 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082399729999999994, 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 340 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.216 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568107+0000", 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1850000000000001e-06 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568112+0000", 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8019999999999997e-06 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568122+0000", 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6970000000000005e-06 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568129+0000", 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2119999999999997e-06 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.575080+0000", 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069511850000000003 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.575168+0000", 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7613000000000003e-05 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.576345+0000", 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001177279 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.219 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.7 95/94 [PushReplyOp(7:e1536981:::smithi13779221-32:head)])", 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 303.48037368799999, 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030062309999999998, 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:29:52.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656692+0000", 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0690000000000001e-06 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656696+0000", 2024-09-21T08:29:52.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3590000000000001e-06 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656705+0000", 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5639999999999997e-06 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656710+0000", 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656749+0000", 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9026999999999998e-05 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.659697+0000", 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029482829999999999 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.c 95/94 [PushReplyOp(7:3cae00e7:::smithi13779221-28:5)])", 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:29:52.223 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 303.45893966, 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028000960000000002, 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:29:52.224 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678126+0000", 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.327e-06 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678131+0000", 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4920000000000004e-06 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.225 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678141+0000", 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9839999999999996e-06 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678146+0000", 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.446e-06 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678184+0000", 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8259000000000001e-05 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.226 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.680925+0000", 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002740588 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.c 95/94 [PushReplyOp(7:3cae00e7:::smithi13779221-28:head)])", 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 303.448636615, 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027171439999999999, 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.227 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688429+0000", 2024-09-21T08:29:52.228 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688434+0000", 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999996e-06 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688443+0000", 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4769999999999992e-06 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.229 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688448+0000", 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3499999999999996e-06 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688491+0000", 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2216999999999999e-05 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.691145+0000", 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026543550000000002 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.230 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:353 7.c 7:361ee62b:::smithi13779221-40:head [copy-from ver 7] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 302.23477306500001, 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014212783, 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 353 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.231 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902296+0000", 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1509999999999997e-06 2024-09-21T08:29:52.232 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902323+0000", 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7491000000000002e-05 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902342+0000", 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8522999999999999e-05 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902350+0000", 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8119999999999997e-06 2024-09-21T08:29:52.233 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902448+0000", 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7867000000000003e-05 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902544+0000", 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5612999999999994e-05 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.907790+0000", 2024-09-21T08:29:52.234 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052465360000000004 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.913037+0000", 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052469129999999998 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916372+0000", 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003334914 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916404+0000", 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1327e-05 2024-09-21T08:29:52.235 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916459+0000", 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4917000000000003e-05 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916504+0000", 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5719000000000001e-05 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.236 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:354 7.c 7:361ee62b:::smithi13779221-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 302.23469211999998, 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089418770000000009, 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 354 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:29:52.237 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902373+0000", 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6499999999999998e-07 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.238 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902376+0000", 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.616e-06 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902402+0000", 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5890000000000001e-05 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902407+0000", 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0019999999999997e-06 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.239 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902704+0000", 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029720499999999998 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902740+0000", 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6383e-05 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911116+0000", 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.008375136 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.240 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911208+0000", 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2791e-05 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911314+0000", 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105989 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:29:52.241 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:29:52.296 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:52.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:52.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:52.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:52.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:52.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:52.443 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:52.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:52.544 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:52.566 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:52.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:52.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:52.667 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:52.678 INFO:tasks.rados.rados.0.smithi137.stdout:1550: done (2 left) 2024-09-21T08:29:52.678 INFO:tasks.rados.rados.0.smithi137.stdout:1551: copy_from oid 30 from oid 35 current snap is 166 2024-09-21T08:29:52.678 INFO:tasks.rados.rados.0.smithi137.stdout:1552: read oid 40 snap -1 2024-09-21T08:29:52.678 INFO:tasks.rados.rados.0.smithi137.stdout:1552: expect (ObjNum 522 snap 149 seq_num 522) 2024-09-21T08:29:52.678 INFO:tasks.rados.rados.0.smithi137.stdout:1553: delete oid 20 current snap is 166 2024-09-21T08:29:52.678 INFO:tasks.rados.rados.0.smithi137.stdout:1549: expect (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:29:52.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:52.694 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:52.695 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:52.766 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:52.795 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:52.801 INFO:teuthology.orchestra.run.smithi137.stderr:Creating an OSD daemon form without an OSD FSID value 2024-09-21T08:29:52.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:52.818 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:52.842 INFO:tasks.rados.rados.0.smithi137.stdout:1548: finishing copy_from racing read to smithi13779221-36 2024-09-21T08:29:52.842 INFO:tasks.rados.rados.0.smithi137.stdout:1548: finishing copy_from to smithi13779221-36 2024-09-21T08:29:52.842 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 291 (ObjNum 445 snap 115 seq_num 445) dirty exists 2024-09-21T08:29:52.843 INFO:tasks.rados.rados.0.smithi137.stdout:1548: done (4 left) 2024-09-21T08:29:52.843 INFO:tasks.rados.rados.0.smithi137.stdout:1549: done (3 left) 2024-09-21T08:29:52.843 INFO:tasks.rados.rados.0.smithi137.stdout:1553: done (2 left) 2024-09-21T08:29:52.843 INFO:tasks.rados.rados.0.smithi137.stdout:1554: snap_create 2024-09-21T08:29:52.843 INFO:tasks.rados.rados.0.smithi137.stdout:1551: finishing copy_from to smithi13779221-30 2024-09-21T08:29:52.843 INFO:tasks.rados.rados.0.smithi137.stdout:1551: got expected ENOENT (src dne) 2024-09-21T08:29:52.843 INFO:tasks.rados.rados.0.smithi137.stdout:1551: finishing copy_from racing read to smithi13779221-30 2024-09-21T08:29:52.843 INFO:tasks.rados.rados.0.smithi137.stdout:1552: expect (ObjNum 522 snap 149 seq_num 522) 2024-09-21T08:29:52.919 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:52.934 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T08:29:52.940 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:52.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:52.941 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:53.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:53.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:53.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:53.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:53.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:53.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:53.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:53.187 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:53.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:53.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:53.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:53.310 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:53.411 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:53.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:53.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:53.433 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:53.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:53.555 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:53.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:53.556 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:53.563 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:29:53.657 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:53.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:53.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:53.680 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:53.687 INFO:tasks.rados.rados.0.smithi137.stdout:1551: done (2 left) 2024-09-21T08:29:53.687 INFO:tasks.rados.rados.0.smithi137.stdout:1552: done (1 left) 2024-09-21T08:29:53.687 INFO:tasks.rados.rados.0.smithi137.stdout:1554: done (0 left) 2024-09-21T08:29:53.687 INFO:tasks.rados.rados.0.smithi137.stdout:1555: read oid 7 snap 166 2024-09-21T08:29:53.687 INFO:tasks.rados.rados.0.smithi137.stdout:1555: expect (ObjNum 385 snap 94 seq_num 385) 2024-09-21T08:29:53.687 INFO:tasks.rados.rados.0.smithi137.stdout:1556: rollback oid 1 current snap is 167 2024-09-21T08:29:53.687 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 1 to 137 2024-09-21T08:29:53.687 INFO:tasks.rados.rados.0.smithi137.stdout:1557: read oid 23 snap -1 2024-09-21T08:29:53.687 INFO:tasks.rados.rados.0.smithi137.stdout:1557: expect (ObjNum 550 snap 165 seq_num 550) 2024-09-21T08:29:53.687 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:29:53.688 INFO:tasks.rados.rados.0.smithi137.stdout:1558: copy_from oid 45 from oid 43 current snap is 167 2024-09-21T08:29:53.688 INFO:tasks.rados.rados.0.smithi137.stdout:1559: read oid 42 snap -1 2024-09-21T08:29:53.688 INFO:tasks.rados.rados.0.smithi137.stdout:1559: expect (ObjNum 445 snap 115 seq_num 445) 2024-09-21T08:29:53.688 INFO:tasks.rados.rados.0.smithi137.stdout:1560: delete oid 3 current snap is 167 2024-09-21T08:29:53.690 INFO:tasks.rados.rados.0.smithi137.stdout:1556: finishing rollback tid 0 to smithi13779221-1 2024-09-21T08:29:53.694 INFO:tasks.rados.rados.0.smithi137.stdout:1556: finishing rollback tid 1 to smithi13779221-1 2024-09-21T08:29:53.694 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 531 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-21T08:29:53.701 INFO:tasks.rados.rados.0.smithi137.stdout:1559: expect (ObjNum 445 snap 115 seq_num 445) 2024-09-21T08:29:53.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:53.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:53.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:53.804 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:53.896 INFO:tasks.rados.rados.0.smithi137.stdout:1555: expect (ObjNum 385 snap 94 seq_num 385) 2024-09-21T08:29:53.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:53.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:53.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:53.927 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:53 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:53 smithi165 ceph-mon[32893]: pgmap v503: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 408 MiB data, 1.0 GiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 2.9 MiB/s wr, 23 op/s; 312/1221 objects degraded (25.553%) 2024-09-21T08:29:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:53 smithi165 ceph-mon[32893]: osdmap e399: 8 total, 6 up, 4 in 2024-09-21T08:29:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:53 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4048863235' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:53 smithi165 ceph-mon[32893]: from='client.45626 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:54.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:53 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:54.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:53 smithi137 ceph-mon[29634]: pgmap v503: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 408 MiB data, 1.0 GiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 2.9 MiB/s wr, 23 op/s; 312/1221 objects degraded (25.553%) 2024-09-21T08:29:54.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:53 smithi137 ceph-mon[29634]: osdmap e399: 8 total, 6 up, 4 in 2024-09-21T08:29:54.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:53 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4048863235' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:54.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:53 smithi137 ceph-mon[29634]: from='client.45626 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:54.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:53 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:29:54.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:53 smithi137 ceph-mon[41044]: pgmap v503: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 408 MiB data, 1.0 GiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 2.9 MiB/s wr, 23 op/s; 312/1221 objects degraded (25.553%) 2024-09-21T08:29:54.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:53 smithi137 ceph-mon[41044]: osdmap e399: 8 total, 6 up, 4 in 2024-09-21T08:29:54.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:53 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4048863235' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:54.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:53 smithi137 ceph-mon[41044]: from='client.45626 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:29:54.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:54.038 INFO:tasks.rados.rados.0.smithi137.stdout:1557: expect (ObjNum 550 snap 165 seq_num 550) 2024-09-21T08:29:54.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:54.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:54.050 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:54.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:54.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:54.174 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:54.174 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:54.203 INFO:tasks.rados.rados.0.smithi137.stdout:1558: finishing copy_from racing read to smithi13779221-45 2024-09-21T08:29:54.203 INFO:tasks.rados.rados.0.smithi137.stdout:1558: finishing copy_from to smithi13779221-45 2024-09-21T08:29:54.203 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 374 (ObjNum 527 snap 149 seq_num 527) dirty exists 2024-09-21T08:29:54.203 INFO:tasks.rados.rados.0.smithi137.stdout:1555: done (5 left) 2024-09-21T08:29:54.203 INFO:tasks.rados.rados.0.smithi137.stdout:1556: done (4 left) 2024-09-21T08:29:54.203 INFO:tasks.rados.rados.0.smithi137.stdout:1557: done (3 left) 2024-09-21T08:29:54.203 INFO:tasks.rados.rados.0.smithi137.stdout:1558: done (2 left) 2024-09-21T08:29:54.203 INFO:tasks.rados.rados.0.smithi137.stdout:1559: done (1 left) 2024-09-21T08:29:54.203 INFO:tasks.rados.rados.0.smithi137.stdout:1560: done (0 left) 2024-09-21T08:29:54.203 INFO:tasks.rados.rados.0.smithi137.stdout:1561: snap_remove snap 164 2024-09-21T08:29:54.275 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:54.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:54.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:54.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:54.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:54.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:54.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:54.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:54.482 INFO:teuthology.orchestra.run.smithi137.stdout:get_pg_num_history pg_num_history pg_num_history(e369 pg_nums {1={19=1},2={64=3},3={65=32},4={67=32},5={69=32},6={71=32},7={181=26,259=36}} deleted_pools ) 2024-09-21T08:29:54.482 INFO:teuthology.orchestra.run.smithi137.stdout:Importing pgid 4.16 2024-09-21T08:29:54.483 INFO:teuthology.orchestra.run.smithi137.stdout:Write #4:6849fd06:gc::gc.17:head# 2024-09-21T08:29:54.483 INFO:teuthology.orchestra.run.smithi137.stdout:snapset 0=[]:{} 2024-09-21T08:29:54.483 INFO:teuthology.orchestra.run.smithi137.stdout:Write #4:6a988a9b:reshard::reshard.0000000007:head# 2024-09-21T08:29:54.483 INFO:teuthology.orchestra.run.smithi137.stdout:snapset 0=[]:{} 2024-09-21T08:29:54.483 INFO:teuthology.orchestra.run.smithi137.stdout:Write #4:6dc3b799:reshard::reshard.0000000006:head# 2024-09-21T08:29:54.483 INFO:teuthology.orchestra.run.smithi137.stdout:snapset 0=[]:{} 2024-09-21T08:29:54.483 INFO:teuthology.orchestra.run.smithi137.stdout:Write #4:6f6aba0e:::obj_delete_at_hint.0000000045:head# 2024-09-21T08:29:54.483 INFO:teuthology.orchestra.run.smithi137.stdout:snapset 0=[]:{} 2024-09-21T08:29:54.483 INFO:teuthology.orchestra.run.smithi137.stdout:write_pg epoch 316 info 4.16( v 73'9 (0'0,73'9] local-lis/les=315/316 n=4 ec=67/67 lis/c=315/315 les/c/f=316/316/0 sis=315) 2024-09-21T08:29:54.484 INFO:teuthology.orchestra.run.smithi137.stdout:Import successful 2024-09-21T08:29:54.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:54.527 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:54.544 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:54.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:54.545 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:54.571 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:54.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:54.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:54.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:54.668 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:54.691 INFO:tasks.rados.rados.0.smithi137.stdout:1561: done (0 left) 2024-09-21T08:29:54.691 INFO:tasks.rados.rados.0.smithi137.stdout:1562: write oid 42 current snap is 167 2024-09-21T08:29:54.691 INFO:tasks.rados.rados.0.smithi137.stdout:1562: seq_num 555 ranges {527885=656964,1927271=483852,3191408=136477} 2024-09-21T08:29:54.698 INFO:tasks.rados.rados.0.smithi137.stdout:1562: writing smithi13779221-42 from 527885 to 1184849 tid 1 2024-09-21T08:29:54.705 INFO:tasks.rados.rados.0.smithi137.stdout:1562: writing smithi13779221-42 from 1927271 to 2411123 tid 2 2024-09-21T08:29:54.711 INFO:tasks.rados.rados.0.smithi137.stdout:1562: writing smithi13779221-42 from 3191408 to 3327885 tid 3 2024-09-21T08:29:54.711 INFO:tasks.rados.rados.0.smithi137.stdout:1563: read oid 32 snap -1 2024-09-21T08:29:54.711 INFO:tasks.rados.rados.0.smithi137.stdout:1563: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:29:54.711 INFO:tasks.rados.rados.0.smithi137.stdout:1564: copy_from oid 3 from oid 21 current snap is 167 2024-09-21T08:29:54.711 INFO:tasks.rados.rados.0.smithi137.stdout:1562: finishing write tid 1 to smithi13779221-42 2024-09-21T08:29:54.711 INFO:tasks.rados.rados.0.smithi137.stdout:1562: finishing write tid 2 to smithi13779221-42 2024-09-21T08:29:54.711 INFO:tasks.rados.rados.0.smithi137.stdout:1565: read oid 40 snap -1 2024-09-21T08:29:54.712 INFO:tasks.rados.rados.0.smithi137.stdout:1565: expect (ObjNum 522 snap 149 seq_num 522) 2024-09-21T08:29:54.712 INFO:tasks.rados.rados.0.smithi137.stdout:1566: read oid 10 snap -1 2024-09-21T08:29:54.712 INFO:tasks.rados.rados.0.smithi137.stdout:1566: expect (ObjNum 517 snap 146 seq_num 517) 2024-09-21T08:29:54.712 INFO:tasks.rados.rados.0.smithi137.stdout:1567: copy_from oid 28 from oid 16 current snap is 167 2024-09-21T08:29:54.712 INFO:tasks.rados.rados.0.smithi137.stdout:1568: delete oid 7 current snap is 167 2024-09-21T08:29:54.716 INFO:tasks.rados.rados.0.smithi137.stdout:1565: expect (ObjNum 522 snap 149 seq_num 522) 2024-09-21T08:29:54.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:54 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:54 smithi137 ceph-mon[29634]: from='client.45626 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:29:54.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:54 smithi137 ceph-mon[29634]: osdmap e400: 8 total, 6 up, 4 in 2024-09-21T08:29:54.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:54 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:54.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:54 smithi137 ceph-mon[41044]: from='client.45626 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:29:54.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:54 smithi137 ceph-mon[41044]: osdmap e400: 8 total, 6 up, 4 in 2024-09-21T08:29:54.768 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:54.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:54.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:54.791 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:54.814 INFO:tasks.rados.rados.0.smithi137.stdout:1565: done (6 left) 2024-09-21T08:29:54.814 INFO:tasks.rados.rados.0.smithi137.stdout:1568: done (5 left) 2024-09-21T08:29:54.814 INFO:tasks.rados.rados.0.smithi137.stdout:1569: write oid 11 current snap is 167 2024-09-21T08:29:54.815 INFO:tasks.rados.rados.0.smithi137.stdout:1563: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:29:54.892 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:54.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:54.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:54.915 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:54 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:29:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:54 smithi165 ceph-mon[32893]: from='client.45626 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:29:55.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:54 smithi165 ceph-mon[32893]: osdmap e400: 8 total, 6 up, 4 in 2024-09-21T08:29:55.017 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:55.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:55.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:55.043 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:55.054 INFO:tasks.rados.rados.0.smithi137.stdout:1562: finishing write tid 3 to smithi13779221-42 2024-09-21T08:29:55.054 INFO:tasks.rados.rados.0.smithi137.stdout:1569: seq_num 557 ranges {561499=407462,1527882=487521,2729901=557217} 2024-09-21T08:29:55.059 INFO:tasks.rados.rados.0.smithi137.stdout:1569: writing smithi13779221-11 from 561499 to 968961 tid 1 2024-09-21T08:29:55.065 INFO:tasks.rados.rados.0.smithi137.stdout:1569: writing smithi13779221-11 from 1527882 to 2015403 tid 2 2024-09-21T08:29:55.073 INFO:tasks.rados.rados.0.smithi137.stdout:1569: writing smithi13779221-11 from 2729901 to 3287118 tid 3 2024-09-21T08:29:55.073 INFO:tasks.rados.rados.0.smithi137.stdout:1563: done (5 left) 2024-09-21T08:29:55.073 INFO:tasks.rados.rados.0.smithi137.stdout:1570: write (excl) oid 13 current snap is 167 2024-09-21T08:29:55.073 INFO:tasks.rados.rados.0.smithi137.stdout:1566: expect (ObjNum 517 snap 146 seq_num 517) 2024-09-21T08:29:55.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:55.165 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:55.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:55.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:55.255 INFO:tasks.rados.rados.0.smithi137.stdout:1562: finishing write tid 4 to smithi13779221-42 2024-09-21T08:29:55.255 INFO:tasks.rados.rados.0.smithi137.stdout:1562: finishing write tid 5 to smithi13779221-42 2024-09-21T08:29:55.255 INFO:tasks.rados.rados.0.smithi137.stdout:1562: oid 42 updating version 0 to 297 2024-09-21T08:29:55.255 INFO:tasks.rados.rados.0.smithi137.stdout:1562: oid 42 version 297 is already newer than 296 2024-09-21T08:29:55.255 INFO:tasks.rados.rados.0.smithi137.stdout:1562: oid 42 updating version 297 to 298 2024-09-21T08:29:55.255 INFO:tasks.rados.rados.0.smithi137.stdout:1562: oid 42 version 298 is already newer than 295 2024-09-21T08:29:55.255 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 298 (ObjNum 555 snap 167 seq_num 555) dirty exists 2024-09-21T08:29:55.255 INFO:tasks.rados.rados.0.smithi137.stdout:1562: left oid 42 (ObjNum 555 snap 167 seq_num 555) 2024-09-21T08:29:55.255 INFO:tasks.rados.rados.0.smithi137.stdout:1564: finishing copy_from racing read to smithi13779221-3 2024-09-21T08:29:55.255 INFO:tasks.rados.rados.0.smithi137.stdout:1570: seq_num 558 ranges {578306=482711,1586364=631179,2757324=484842} 2024-09-21T08:29:55.260 INFO:tasks.rados.rados.0.smithi137.stdout:1570: writing smithi13779221-13 from 578306 to 1061017 tid 1 2024-09-21T08:29:55.266 INFO:tasks.rados.rados.0.smithi137.stdout:1570: writing smithi13779221-13 from 1586364 to 2217543 tid 2 2024-09-21T08:29:55.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:55.273 INFO:tasks.rados.rados.0.smithi137.stdout:1570: writing smithi13779221-13 from 2757324 to 3242166 tid 3 2024-09-21T08:29:55.273 INFO:tasks.rados.rados.0.smithi137.stdout:1562: done (5 left) 2024-09-21T08:29:55.273 INFO:tasks.rados.rados.0.smithi137.stdout:1566: done (4 left) 2024-09-21T08:29:55.273 INFO:tasks.rados.rados.0.smithi137.stdout:1571: read oid 50 snap 123 2024-09-21T08:29:55.273 INFO:tasks.rados.rados.0.smithi137.stdout:1571: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:29:55.273 INFO:tasks.rados.rados.0.smithi137.stdout:1572: read oid 24 snap 160 2024-09-21T08:29:55.273 INFO:tasks.rados.rados.0.smithi137.stdout:1572: expect (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:29:55.273 INFO:tasks.rados.rados.0.smithi137.stdout:1573: delete oid 49 current snap is 167 2024-09-21T08:29:55.273 INFO:tasks.rados.rados.0.smithi137.stdout:1564: finishing copy_from to smithi13779221-3 2024-09-21T08:29:55.274 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 495 (ObjNum 403 snap 99 seq_num 403) dirty exists 2024-09-21T08:29:55.274 INFO:tasks.rados.rados.0.smithi137.stdout:1567: finishing copy_from racing read to smithi13779221-28 2024-09-21T08:29:55.274 INFO:tasks.rados.rados.0.smithi137.stdout:1567: finishing copy_from to smithi13779221-28 2024-09-21T08:29:55.274 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 336 (ObjNum 468 snap 126 seq_num 468) dirty exists 2024-09-21T08:29:55.274 INFO:tasks.rados.rados.0.smithi137.stdout:1569: finishing write tid 1 to smithi13779221-11 2024-09-21T08:29:55.274 INFO:tasks.rados.rados.0.smithi137.stdout:1569: finishing write tid 2 to smithi13779221-11 2024-09-21T08:29:55.274 INFO:tasks.rados.rados.0.smithi137.stdout:1569: finishing write tid 3 to smithi13779221-11 2024-09-21T08:29:55.274 INFO:tasks.rados.rados.0.smithi137.stdout:1569: finishing write tid 4 to smithi13779221-11 2024-09-21T08:29:55.274 INFO:tasks.rados.rados.0.smithi137.stdout:1569: finishing write tid 5 to smithi13779221-11 2024-09-21T08:29:55.274 INFO:tasks.rados.rados.0.smithi137.stdout:1569: oid 11 updating version 0 to 522 2024-09-21T08:29:55.275 INFO:tasks.rados.rados.0.smithi137.stdout:1569: oid 11 version 522 is already newer than 520 2024-09-21T08:29:55.275 INFO:tasks.rados.rados.0.smithi137.stdout:1569: oid 11 version 522 is already newer than 519 2024-09-21T08:29:55.275 INFO:tasks.rados.rados.0.smithi137.stdout:1569: oid 11 version 522 is already newer than 521 2024-09-21T08:29:55.275 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 522 (ObjNum 557 snap 167 seq_num 557) dirty exists 2024-09-21T08:29:55.275 INFO:tasks.rados.rados.0.smithi137.stdout:1569: left oid 11 (ObjNum 557 snap 167 seq_num 557) 2024-09-21T08:29:55.275 INFO:tasks.rados.rados.0.smithi137.stdout:1570: finishing write tid 1 to smithi13779221-13 2024-09-21T08:29:55.275 INFO:tasks.rados.rados.0.smithi137.stdout:1570: finishing write tid 2 to smithi13779221-13 2024-09-21T08:29:55.276 INFO:tasks.rados.rados.0.smithi137.stdout:1564: done (6 left) 2024-09-21T08:29:55.276 INFO:tasks.rados.rados.0.smithi137.stdout:1567: done (5 left) 2024-09-21T08:29:55.276 INFO:tasks.rados.rados.0.smithi137.stdout:1569: done (4 left) 2024-09-21T08:29:55.276 INFO:tasks.rados.rados.0.smithi137.stdout:1573: done (3 left) 2024-09-21T08:29:55.276 INFO:tasks.rados.rados.0.smithi137.stdout:1574: snap_remove snap 132 2024-09-21T08:29:55.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:55.290 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:55.290 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:55.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:55.413 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:55.414 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:55.414 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:55.462 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:29:55.463 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:29:55.463 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:29:55.473 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:29:55.473 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:29:55.489 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:29:55.489 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:29:55.505 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:29:55.506 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:29:55.515 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:55.522 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:29:55.522 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:29:55.536 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:55.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:55.537 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:55.538 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:29:55.538 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:29:55.538 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:29:55.638 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:55.660 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:55.661 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:55.661 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:55.690 INFO:tasks.rados.rados.0.smithi137.stdout:1574: done (3 left) 2024-09-21T08:29:55.690 INFO:tasks.rados.rados.0.smithi137.stdout:1575: write (excl) oid 26 current snap is 167 2024-09-21T08:29:55.690 INFO:tasks.rados.rados.0.smithi137.stdout:1575: seq_num 560 ranges {611920=633209,1703328=634848,2812170=507386} 2024-09-21T08:29:55.697 INFO:tasks.rados.rados.0.smithi137.stdout:1575: writing smithi13779221-26 from 611920 to 1245129 tid 1 2024-09-21T08:29:55.704 INFO:tasks.rados.rados.0.smithi137.stdout:1575: writing smithi13779221-26 from 1703328 to 2338176 tid 2 2024-09-21T08:29:55.709 INFO:tasks.rados.rados.0.smithi137.stdout:1575: writing smithi13779221-26 from 2812170 to 3319556 tid 3 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:1576: write (excl) oid 42 current snap is 167 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:1570: finishing write tid 3 to smithi13779221-13 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:1570: finishing write tid 4 to smithi13779221-13 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:1570: finishing write tid 5 to smithi13779221-13 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:1570: oid 13 updating version 0 to 613 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:1570: oid 13 updating version 613 to 614 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:1570: oid 13 updating version 614 to 616 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:1570: oid 13 version 616 is already newer than 615 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 616 (ObjNum 558 snap 167 seq_num 558) dirty exists 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:1570: left oid 13 (ObjNum 558 snap 167 seq_num 558) 2024-09-21T08:29:55.710 INFO:tasks.rados.rados.0.smithi137.stdout:1572: expect (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:29:55.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:55.783 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:55.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:55.785 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:55.861 INFO:tasks.rados.rados.0.smithi137.stdout:1571: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:29:55.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:55.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:55.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:55.908 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:55.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:55 smithi137 ceph-mon[29634]: pgmap v506: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 408 MiB data, 1.0 GiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 2.9 MiB/s wr, 23 op/s; 312/1221 objects degraded (25.553%) 2024-09-21T08:29:55.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:55 smithi137 ceph-mon[29634]: osdmap e401: 8 total, 6 up, 4 in 2024-09-21T08:29:55.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:55 smithi137 ceph-mon[29634]: 5.f scrub starts 2024-09-21T08:29:55.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:55 smithi137 ceph-mon[29634]: 5.f scrub ok 2024-09-21T08:29:55.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:55 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4148153686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:55.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:55 smithi137 ceph-mon[41044]: pgmap v506: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 408 MiB data, 1.0 GiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 2.9 MiB/s wr, 23 op/s; 312/1221 objects degraded (25.553%) 2024-09-21T08:29:55.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:55 smithi137 ceph-mon[41044]: osdmap e401: 8 total, 6 up, 4 in 2024-09-21T08:29:55.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:55 smithi137 ceph-mon[41044]: 5.f scrub starts 2024-09-21T08:29:55.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:55 smithi137 ceph-mon[41044]: 5.f scrub ok 2024-09-21T08:29:55.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:55 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4148153686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:55.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo rm -f /var/log/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/exp.4.16.7 2024-09-21T08:29:55.989 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:55 smithi165 ceph-mon[32893]: pgmap v506: 168 pgs: 72 active+undersized, 58 active+undersized+degraded, 38 active+clean; 408 MiB data, 1.0 GiB used, 357 GiB / 358 GiB avail; 9.6 MiB/s rd, 2.9 MiB/s wr, 23 op/s; 312/1221 objects degraded (25.553%) 2024-09-21T08:29:55.989 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:55 smithi165 ceph-mon[32893]: osdmap e401: 8 total, 6 up, 4 in 2024-09-21T08:29:55.989 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:55 smithi165 ceph-mon[32893]: 5.f scrub starts 2024-09-21T08:29:55.989 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:55 smithi165 ceph-mon[32893]: 5.f scrub ok 2024-09-21T08:29:55.989 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:55 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4148153686' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:29:55.993 DEBUG:teuthology.orchestra.run.smithi137:> sudo rm -f /var/log/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/exp.4.16.7 2024-09-21T08:29:56.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:56.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:56.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:56.055 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:56.075 INFO:tasks.rados.rados.0.smithi137.stdout:1575: finishing write tid 1 to smithi13779221-26 2024-09-21T08:29:56.075 INFO:tasks.rados.rados.0.smithi137.stdout:1575: finishing write tid 2 to smithi13779221-26 2024-09-21T08:29:56.075 INFO:tasks.rados.rados.0.smithi137.stdout:1575: finishing write tid 3 to smithi13779221-26 2024-09-21T08:29:56.075 INFO:tasks.rados.rados.0.smithi137.stdout:1575: finishing write tid 4 to smithi13779221-26 2024-09-21T08:29:56.075 INFO:tasks.rados.rados.0.smithi137.stdout:1576: seq_num 561 ranges {628727=708458,2045457=494859,3123240=305487} 2024-09-21T08:29:56.082 INFO:tasks.rados.rados.0.smithi137.stdout:1576: writing smithi13779221-42 from 628727 to 1337185 tid 1 2024-09-21T08:29:56.088 INFO:tasks.rados.rados.0.smithi137.stdout:1576: writing smithi13779221-42 from 2045457 to 2540316 tid 2 2024-09-21T08:29:56.094 INFO:tasks.rados.rados.0.smithi137.stdout:1576: writing smithi13779221-42 from 3123240 to 3428727 tid 3 2024-09-21T08:29:56.094 INFO:tasks.rados.rados.0.smithi137.stdout:1570: done (4 left) 2024-09-21T08:29:56.094 INFO:tasks.rados.rados.0.smithi137.stdout:1571: done (3 left) 2024-09-21T08:29:56.094 INFO:tasks.rados.rados.0.smithi137.stdout:1572: done (2 left) 2024-09-21T08:29:56.094 INFO:tasks.rados.rados.0.smithi137.stdout:1577: rollback oid 23 current snap is 167 2024-09-21T08:29:56.094 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 23 to 144 2024-09-21T08:29:56.094 INFO:tasks.rados.rados.0.smithi137.stdout:1575: finishing write tid 5 to smithi13779221-26 2024-09-21T08:29:56.094 INFO:tasks.rados.rados.0.smithi137.stdout:1575: oid 26 updating version 0 to 377 2024-09-21T08:29:56.095 INFO:tasks.rados.rados.0.smithi137.stdout:1575: oid 26 version 377 is already newer than 376 2024-09-21T08:29:56.095 INFO:tasks.rados.rados.0.smithi137.stdout:1575: oid 26 updating version 377 to 378 2024-09-21T08:29:56.095 INFO:tasks.rados.rados.0.smithi137.stdout:1575: oid 26 updating version 378 to 379 2024-09-21T08:29:56.095 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 379 (ObjNum 560 snap 167 seq_num 560) dirty exists 2024-09-21T08:29:56.095 INFO:tasks.rados.rados.0.smithi137.stdout:1575: left oid 26 (ObjNum 560 snap 167 seq_num 560) 2024-09-21T08:29:56.095 INFO:tasks.rados.rados.0.smithi137.stdout:1575: done (2 left) 2024-09-21T08:29:56.095 INFO:tasks.rados.rados.0.smithi137.stdout:1578: copy_from oid 20 from oid 49 current snap is 167 2024-09-21T08:29:56.095 INFO:tasks.rados.rados.0.smithi137.stdout:1576: finishing write tid 1 to smithi13779221-42 2024-09-21T08:29:56.095 INFO:tasks.rados.rados.0.smithi137.stdout:1576: finishing write tid 2 to smithi13779221-42 2024-09-21T08:29:56.095 INFO:tasks.rados.rados.0.smithi137.stdout:1579: delete oid 45 current snap is 167 2024-09-21T08:29:56.096 INFO:tasks.rados.rados.0.smithi137.stdout:1576: finishing write tid 3 to smithi13779221-42 2024-09-21T08:29:56.096 INFO:tasks.rados.rados.0.smithi137.stdout:1579: done (3 left) 2024-09-21T08:29:56.096 INFO:tasks.rados.rados.0.smithi137.stdout:1580: write oid 43 current snap is 167 2024-09-21T08:29:56.096 INFO:tasks.rados.rados.0.smithi137.stdout:1580: seq_num 563 ranges {662341=575309,1878774=498528,2894439=457555} 2024-09-21T08:29:56.102 INFO:tasks.rados.rados.0.smithi137.stdout:1580: writing smithi13779221-43 from 662341 to 1237650 tid 1 2024-09-21T08:29:56.108 INFO:tasks.rados.rados.0.smithi137.stdout:1580: writing smithi13779221-43 from 1878774 to 2377302 tid 2 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1580: writing smithi13779221-43 from 2894439 to 3351994 tid 3 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1581: snap_create 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1578: finishing copy_from to smithi13779221-20 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1578: got expected ENOENT (src dne) 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1576: finishing write tid 4 to smithi13779221-42 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1578: finishing copy_from racing read to smithi13779221-20 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1576: finishing write tid 5 to smithi13779221-42 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1576: oid 42 updating version 0 to 299 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1576: oid 42 updating version 299 to 300 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1576: oid 42 updating version 300 to 301 2024-09-21T08:29:56.114 INFO:tasks.rados.rados.0.smithi137.stdout:1576: oid 42 updating version 301 to 302 2024-09-21T08:29:56.115 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 302 (ObjNum 561 snap 167 seq_num 561) dirty exists 2024-09-21T08:29:56.115 INFO:tasks.rados.rados.0.smithi137.stdout:1576: left oid 42 (ObjNum 561 snap 167 seq_num 561) 2024-09-21T08:29:56.115 INFO:tasks.rados.rados.0.smithi137.stdout:1577: finishing rollback tid 0 to smithi13779221-23 2024-09-21T08:29:56.115 INFO:tasks.rados.rados.0.smithi137.stdout:1577: finishing rollback tid 1 to smithi13779221-23 2024-09-21T08:29:56.115 INFO:tasks.rados.rados.0.smithi137.stdout:1577: finishing rollback tid 2 to smithi13779221-23 2024-09-21T08:29:56.115 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 194 (ObjNum 428 snap 108 seq_num 428) dirty exists 2024-09-21T08:29:56.115 INFO:tasks.rados.rados.0.smithi137.stdout:1580: finishing write tid 1 to smithi13779221-43 2024-09-21T08:29:56.115 INFO:tasks.rados.rados.0.smithi137.stdout:1580: finishing write tid 2 to smithi13779221-43 2024-09-21T08:29:56.116 INFO:tasks.rados.rados.0.smithi137.stdout:1580: finishing write tid 3 to smithi13779221-43 2024-09-21T08:29:56.116 INFO:tasks.rados.rados.0.smithi137.stdout:1580: finishing write tid 4 to smithi13779221-43 2024-09-21T08:29:56.116 INFO:tasks.rados.rados.0.smithi137.stdout:1580: finishing write tid 5 to smithi13779221-43 2024-09-21T08:29:56.116 INFO:tasks.rados.rados.0.smithi137.stdout:1580: oid 43 updating version 0 to 382 2024-09-21T08:29:56.116 INFO:tasks.rados.rados.0.smithi137.stdout:1580: oid 43 updating version 382 to 383 2024-09-21T08:29:56.116 INFO:tasks.rados.rados.0.smithi137.stdout:1580: oid 43 version 383 is already newer than 380 2024-09-21T08:29:56.116 INFO:tasks.rados.rados.0.smithi137.stdout:1580: oid 43 version 383 is already newer than 381 2024-09-21T08:29:56.116 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 383 (ObjNum 563 snap 167 seq_num 563) dirty exists 2024-09-21T08:29:56.116 INFO:tasks.rados.rados.0.smithi137.stdout:1580: left oid 43 (ObjNum 563 snap 167 seq_num 563) 2024-09-21T08:29:56.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:56.177 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:56.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:56.178 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:56.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:56.301 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:56.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:56.302 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:56.362 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:56.362 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:56.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:56.424 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:56.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:56.425 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:56.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:56.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:29:56.547 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:56.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:56.548 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:56.649 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:56.672 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:56.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:56.673 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:56.700 INFO:tasks.rados.rados.0.smithi137.stdout:1576: done (4 left) 2024-09-21T08:29:56.700 INFO:tasks.rados.rados.0.smithi137.stdout:1577: done (3 left) 2024-09-21T08:29:56.700 INFO:tasks.rados.rados.0.smithi137.stdout:1578: done (2 left) 2024-09-21T08:29:56.700 INFO:tasks.rados.rados.0.smithi137.stdout:1580: done (1 left) 2024-09-21T08:29:56.700 INFO:tasks.rados.rados.0.smithi137.stdout:1581: done (0 left) 2024-09-21T08:29:56.701 INFO:tasks.rados.rados.0.smithi137.stdout:1582: snap_remove snap 160 2024-09-21T08:29:56.774 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:56.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:56.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:56.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:56.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:56.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:56.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:56.926 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:56 smithi165 ceph-mon[32893]: osdmap e402: 8 total, 6 up, 4 in 2024-09-21T08:29:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:56 smithi165 ceph-mon[32893]: pgmap v509: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 438 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 37 MiB/s rd, 6.2 MiB/s wr, 38 op/s; 320/1254 objects degraded (25.518%) 2024-09-21T08:29:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:56 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 312/1221 objects degraded (25.553%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:29:57.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:56 smithi137 ceph-mon[29634]: osdmap e402: 8 total, 6 up, 4 in 2024-09-21T08:29:57.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:56 smithi137 ceph-mon[29634]: pgmap v509: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 438 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 37 MiB/s rd, 6.2 MiB/s wr, 38 op/s; 320/1254 objects degraded (25.518%) 2024-09-21T08:29:57.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:56 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 312/1221 objects degraded (25.553%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:29:57.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:56 smithi137 ceph-mon[41044]: osdmap e402: 8 total, 6 up, 4 in 2024-09-21T08:29:57.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:56 smithi137 ceph-mon[41044]: pgmap v509: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 438 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 37 MiB/s rd, 6.2 MiB/s wr, 38 op/s; 320/1254 objects degraded (25.518%) 2024-09-21T08:29:57.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:56 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 312/1221 objects degraded (25.553%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:29:57.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:57.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:57.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:57.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:57.107 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:29:57.108 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:29:57.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:57.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:57.174 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:57.174 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:57.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:57.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:57.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:57.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:57.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:57.420 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:57.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:57.421 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:57.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:57.538 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T08:29:57.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:57.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:57.545 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:57.635 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:57.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:57.671 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:57.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:57.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:57.706 INFO:tasks.rados.rados.0.smithi137.stdout:1582: done (0 left) 2024-09-21T08:29:57.706 INFO:tasks.rados.rados.0.smithi137.stdout:1583: snap_remove snap 159 2024-09-21T08:29:57.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:57 smithi137 ceph-mon[29634]: 3.0 scrub starts 2024-09-21T08:29:57.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:57 smithi137 ceph-mon[29634]: 3.0 scrub ok 2024-09-21T08:29:57.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:57 smithi137 ceph-mon[29634]: osdmap e403: 8 total, 6 up, 4 in 2024-09-21T08:29:57.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:57 smithi137 ceph-mon[41044]: 3.0 scrub starts 2024-09-21T08:29:57.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:57 smithi137 ceph-mon[41044]: 3.0 scrub ok 2024-09-21T08:29:57.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:57 smithi137 ceph-mon[41044]: osdmap e403: 8 total, 6 up, 4 in 2024-09-21T08:29:57.773 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:57.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:57.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:57.795 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:57.896 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:57.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:57.918 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:57 smithi165 ceph-mon[32893]: 3.0 scrub starts 2024-09-21T08:29:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:57 smithi165 ceph-mon[32893]: 3.0 scrub ok 2024-09-21T08:29:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:57 smithi165 ceph-mon[32893]: osdmap e403: 8 total, 6 up, 4 in 2024-09-21T08:29:58.019 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:58.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:58.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:58.045 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:58.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:58.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:58.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:58.168 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:29:58.220 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:29:58.268 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:58.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:58.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:58.295 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:58.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:58.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:58.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:58.422 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:58.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:58.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:58.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:58.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:58.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:58.675 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:58.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:58.676 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:58.689 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:29:58.689 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:29:58.712 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:29:58.712 INFO:tasks.rados.rados.0.smithi137.stdout:1583: done (0 left) 2024-09-21T08:29:58.712 INFO:tasks.rados.rados.0.smithi137.stdout:1584: write oid 1 current snap is 168 2024-09-21T08:29:58.712 INFO:tasks.rados.rados.0.smithi137.stdout:1584: seq_num 564 ranges {679148=650558,1937256=642186,3321862=157286} 2024-09-21T08:29:58.720 INFO:tasks.rados.rados.0.smithi137.stdout:1584: writing smithi13779221-1 from 679148 to 1329706 tid 1 2024-09-21T08:29:58.727 INFO:tasks.rados.rados.0.smithi137.stdout:1584: writing smithi13779221-1 from 1937256 to 2579442 tid 2 2024-09-21T08:29:58.732 INFO:tasks.rados.rados.0.smithi137.stdout:1584: writing smithi13779221-1 from 3321862 to 3479148 tid 3 2024-09-21T08:29:58.732 INFO:tasks.rados.rados.0.smithi137.stdout:1585: rollback oid 10 current snap is 168 2024-09-21T08:29:58.732 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 10 to 131 2024-09-21T08:29:58.732 INFO:tasks.rados.rados.0.smithi137.stdout:1584: finishing write tid 1 to smithi13779221-1 2024-09-21T08:29:58.733 INFO:tasks.rados.rados.0.smithi137.stdout:1584: finishing write tid 2 to smithi13779221-1 2024-09-21T08:29:58.733 INFO:tasks.rados.rados.0.smithi137.stdout:1586: read oid 46 snap -1 2024-09-21T08:29:58.733 INFO:tasks.rados.rados.0.smithi137.stdout:1586: expect (ObjNum 499 snap 140 seq_num 499) 2024-09-21T08:29:58.733 INFO:tasks.rados.rados.0.smithi137.stdout:1587: write oid 19 current snap is 168 2024-09-21T08:29:58.733 INFO:tasks.rados.rados.0.smithi137.stdout:1587: seq_num 565 ranges {695955=725807,1879385=502197,2949285=480099} 2024-09-21T08:29:58.740 INFO:tasks.rados.rados.0.smithi137.stdout:1587: writing smithi13779221-19 from 695955 to 1421762 tid 1 2024-09-21T08:29:58.745 INFO:tasks.rados.rados.0.smithi137.stdout:1587: writing smithi13779221-19 from 1879385 to 2381582 tid 2 2024-09-21T08:29:58.751 INFO:tasks.rados.rados.0.smithi137.stdout:1587: writing smithi13779221-19 from 2949285 to 3429384 tid 3 2024-09-21T08:29:58.751 INFO:tasks.rados.rados.0.smithi137.stdout:1584: finishing write tid 3 to smithi13779221-1 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:1584: finishing write tid 4 to smithi13779221-1 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:1584: finishing write tid 5 to smithi13779221-1 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:1584: oid 1 updating version 0 to 533 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:1584: oid 1 updating version 533 to 535 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:1584: oid 1 version 535 is already newer than 534 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:1584: oid 1 version 535 is already newer than 532 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 535 (ObjNum 564 snap 168 seq_num 564) dirty exists 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:1584: left oid 1 (ObjNum 564 snap 168 seq_num 564) 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:1585: finishing rollback tid 0 to smithi13779221-10 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:1585: finishing rollback tid 1 to smithi13779221-10 2024-09-21T08:29:58.752 INFO:tasks.rados.rados.0.smithi137.stdout:1586: expect (ObjNum 499 snap 140 seq_num 499) 2024-09-21T08:29:58.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:58.799 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:58.800 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:58.800 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:58.901 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:58.923 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:58.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:58.924 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:58.942 INFO:tasks.rados.rados.0.smithi137.stdout:1585: finishing rollback tid 2 to smithi13779221-10 2024-09-21T08:29:58.942 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 527 (ObjNum 385 snap 94 seq_num 385) dirty exists 2024-09-21T08:29:58.942 INFO:tasks.rados.rados.0.smithi137.stdout:1587: finishing write tid 1 to smithi13779221-19 2024-09-21T08:29:58.942 INFO:tasks.rados.rados.0.smithi137.stdout:1584: done (3 left) 2024-09-21T08:29:58.943 INFO:tasks.rados.rados.0.smithi137.stdout:1585: done (2 left) 2024-09-21T08:29:58.943 INFO:tasks.rados.rados.0.smithi137.stdout:1586: done (1 left) 2024-09-21T08:29:58.943 INFO:tasks.rados.rados.0.smithi137.stdout:1588: read oid 14 snap -1 2024-09-21T08:29:58.943 INFO:tasks.rados.rados.0.smithi137.stdout:1588: expect (ObjNum 527 snap 149 seq_num 527) 2024-09-21T08:29:58.943 INFO:tasks.rados.rados.0.smithi137.stdout:1587: finishing write tid 2 to smithi13779221-19 2024-09-21T08:29:58.943 INFO:tasks.rados.rados.0.smithi137.stdout:1589: write oid 36 current snap is 168 2024-09-21T08:29:58.943 INFO:tasks.rados.rados.0.smithi137.stdout:1587: finishing write tid 3 to smithi13779221-19 2024-09-21T08:29:58.943 INFO:tasks.rados.rados.0.smithi137.stdout:1587: finishing write tid 4 to smithi13779221-19 2024-09-21T08:29:58.943 INFO:tasks.rados.rados.0.smithi137.stdout:1589: seq_num 566 ranges {712762=401056,1537867=645855,2860355=652407} 2024-09-21T08:29:58.948 INFO:tasks.rados.rados.0.smithi137.stdout:1589: writing smithi13779221-36 from 712762 to 1113818 tid 1 2024-09-21T08:29:58.954 INFO:tasks.rados.rados.0.smithi137.stdout:1589: writing smithi13779221-36 from 1537867 to 2183722 tid 2 2024-09-21T08:29:58.962 INFO:tasks.rados.rados.0.smithi137.stdout:1589: writing smithi13779221-36 from 2860355 to 3512762 tid 3 2024-09-21T08:29:58.962 INFO:tasks.rados.rados.0.smithi137.stdout:1590: read oid 35 snap -1 2024-09-21T08:29:58.962 INFO:tasks.rados.rados.0.smithi137.stdout:1590: expect deleted 2024-09-21T08:29:58.962 INFO:tasks.rados.rados.0.smithi137.stdout:1591: write oid 29 current snap is 168 2024-09-21T08:29:58.962 INFO:tasks.rados.rados.0.smithi137.stdout:1587: finishing write tid 5 to smithi13779221-19 2024-09-21T08:29:58.962 INFO:tasks.rados.rados.0.smithi137.stdout:1587: oid 19 updating version 0 to 499 2024-09-21T08:29:58.962 INFO:tasks.rados.rados.0.smithi137.stdout:1587: oid 19 version 499 is already newer than 498 2024-09-21T08:29:58.962 INFO:tasks.rados.rados.0.smithi137.stdout:1587: oid 19 version 499 is already newer than 497 2024-09-21T08:29:58.962 INFO:tasks.rados.rados.0.smithi137.stdout:1587: oid 19 version 499 is already newer than 496 2024-09-21T08:29:58.963 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 499 (ObjNum 565 snap 168 seq_num 565) dirty exists 2024-09-21T08:29:58.963 INFO:tasks.rados.rados.0.smithi137.stdout:1587: left oid 19 (ObjNum 565 snap 168 seq_num 565) 2024-09-21T08:29:58.963 INFO:tasks.rados.rados.0.smithi137.stdout:1588: expect (ObjNum 527 snap 149 seq_num 527) 2024-09-21T08:29:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:58 smithi165 ceph-mon[32893]: osdmap e404: 8 total, 6 up, 4 in 2024-09-21T08:29:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:58 smithi165 ceph-mon[32893]: pgmap v512: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 438 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 37 MiB/s rd, 6.2 MiB/s wr, 38 op/s; 320/1254 objects degraded (25.518%) 2024-09-21T08:29:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:58 smithi165 ceph-mon[32893]: 6.13 scrub starts 2024-09-21T08:29:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:58 smithi165 ceph-mon[32893]: 6.13 scrub ok 2024-09-21T08:29:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:58 smithi165 ceph-mon[32893]: 4.2 scrub starts 2024-09-21T08:29:59.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:58 smithi165 ceph-mon[32893]: 4.2 scrub ok 2024-09-21T08:29:59.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:58 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1958443426' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:59.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:58 smithi165 ceph-mon[32893]: from='client.45641 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:59.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[29634]: osdmap e404: 8 total, 6 up, 4 in 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[29634]: pgmap v512: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 438 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 37 MiB/s rd, 6.2 MiB/s wr, 38 op/s; 320/1254 objects degraded (25.518%) 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[29634]: 6.13 scrub starts 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[29634]: 6.13 scrub ok 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[29634]: 4.2 scrub starts 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[29634]: 4.2 scrub ok 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1958443426' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[29634]: from='client.45641 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[41044]: osdmap e404: 8 total, 6 up, 4 in 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[41044]: pgmap v512: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 438 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 37 MiB/s rd, 6.2 MiB/s wr, 38 op/s; 320/1254 objects degraded (25.518%) 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[41044]: 6.13 scrub starts 2024-09-21T08:29:59.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[41044]: 6.13 scrub ok 2024-09-21T08:29:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[41044]: 4.2 scrub starts 2024-09-21T08:29:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[41044]: 4.2 scrub ok 2024-09-21T08:29:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1958443426' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:59.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:58 smithi137 ceph-mon[41044]: from='client.45641 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:29:59.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:59.046 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:59.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:59.047 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:59.091 INFO:tasks.rados.rados.0.smithi137.stdout:1589: finishing write tid 1 to smithi13779221-36 2024-09-21T08:29:59.091 INFO:tasks.rados.rados.0.smithi137.stdout:1589: finishing write tid 2 to smithi13779221-36 2024-09-21T08:29:59.091 INFO:tasks.rados.rados.0.smithi137.stdout:1591: seq_num 567 ranges {729569=476305,1996349=505866,3004131=525438} 2024-09-21T08:29:59.097 INFO:tasks.rados.rados.0.smithi137.stdout:1591: writing smithi13779221-29 from 729569 to 1205874 tid 1 2024-09-21T08:29:59.104 INFO:tasks.rados.rados.0.smithi137.stdout:1591: writing smithi13779221-29 from 1996349 to 2502215 tid 2 2024-09-21T08:29:59.108 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:29:59.111 INFO:tasks.rados.rados.0.smithi137.stdout:1591: writing smithi13779221-29 from 3004131 to 3529569 tid 3 2024-09-21T08:29:59.111 INFO:tasks.rados.rados.0.smithi137.stdout:1587: done (4 left) 2024-09-21T08:29:59.111 INFO:tasks.rados.rados.0.smithi137.stdout:1588: done (3 left) 2024-09-21T08:29:59.111 INFO:tasks.rados.rados.0.smithi137.stdout:1592: snap_create 2024-09-21T08:29:59.111 INFO:tasks.rados.rados.0.smithi137.stdout:1589: finishing write tid 3 to smithi13779221-36 2024-09-21T08:29:59.111 INFO:tasks.rados.rados.0.smithi137.stdout:1589: finishing write tid 4 to smithi13779221-36 2024-09-21T08:29:59.111 INFO:tasks.rados.rados.0.smithi137.stdout:1589: finishing write tid 5 to smithi13779221-36 2024-09-21T08:29:59.111 INFO:tasks.rados.rados.0.smithi137.stdout:1589: oid 36 updating version 0 to 644 2024-09-21T08:29:59.111 INFO:tasks.rados.rados.0.smithi137.stdout:1589: oid 36 version 644 is already newer than 641 2024-09-21T08:29:59.112 INFO:tasks.rados.rados.0.smithi137.stdout:1589: oid 36 version 644 is already newer than 643 2024-09-21T08:29:59.112 INFO:tasks.rados.rados.0.smithi137.stdout:1589: oid 36 version 644 is already newer than 642 2024-09-21T08:29:59.112 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 644 (ObjNum 566 snap 168 seq_num 566) dirty exists 2024-09-21T08:29:59.112 INFO:tasks.rados.rados.0.smithi137.stdout:1589: left oid 36 (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:29:59.112 INFO:tasks.rados.rados.0.smithi137.stdout:1591: finishing write tid 1 to smithi13779221-29 2024-09-21T08:29:59.112 INFO:tasks.rados.rados.0.smithi137.stdout:1591: finishing write tid 2 to smithi13779221-29 2024-09-21T08:29:59.112 INFO:tasks.rados.rados.0.smithi137.stdout:1591: finishing write tid 3 to smithi13779221-29 2024-09-21T08:29:59.113 INFO:tasks.rados.rados.0.smithi137.stdout:1591: finishing write tid 4 to smithi13779221-29 2024-09-21T08:29:59.113 INFO:tasks.rados.rados.0.smithi137.stdout:1591: finishing write tid 5 to smithi13779221-29 2024-09-21T08:29:59.113 INFO:tasks.rados.rados.0.smithi137.stdout:1591: oid 29 updating version 0 to 628 2024-09-21T08:29:59.113 INFO:tasks.rados.rados.0.smithi137.stdout:1591: oid 29 version 628 is already newer than 627 2024-09-21T08:29:59.113 INFO:tasks.rados.rados.0.smithi137.stdout:1591: oid 29 version 628 is already newer than 626 2024-09-21T08:29:59.113 INFO:tasks.rados.rados.0.smithi137.stdout:1591: oid 29 updating version 628 to 629 2024-09-21T08:29:59.113 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 629 (ObjNum 567 snap 168 seq_num 567) dirty exists 2024-09-21T08:29:59.113 INFO:tasks.rados.rados.0.smithi137.stdout:1591: left oid 29 (ObjNum 567 snap 168 seq_num 567) 2024-09-21T08:29:59.149 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:59.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:59.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:59.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:59.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:29:59.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:59.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:59.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:29:59.399 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:29:59.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:59.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:59.425 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:29:59.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:29:59.548 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:59.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:59.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:29:59.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:29:59.658 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T08:29:59.672 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:59.672 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:59.672 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:29:59.715 INFO:tasks.rados.rados.0.smithi137.stdout:1589: done (3 left) 2024-09-21T08:29:59.715 INFO:tasks.rados.rados.0.smithi137.stdout:1590: done (2 left) 2024-09-21T08:29:59.716 INFO:tasks.rados.rados.0.smithi137.stdout:1591: done (1 left) 2024-09-21T08:29:59.716 INFO:tasks.rados.rados.0.smithi137.stdout:1592: done (0 left) 2024-09-21T08:29:59.716 INFO:tasks.rados.rados.0.smithi137.stdout:1593: write (excl) oid 19 current snap is 169 2024-09-21T08:29:59.716 INFO:tasks.rados.rados.0.smithi137.stdout:1593: seq_num 568 ranges {746376=551554,2054831=649524,3315201=231175} 2024-09-21T08:29:59.722 INFO:tasks.rados.rados.0.smithi137.stdout:1593: writing smithi13779221-19 from 746376 to 1297930 tid 1 2024-09-21T08:29:59.730 INFO:tasks.rados.rados.0.smithi137.stdout:1593: writing smithi13779221-19 from 2054831 to 2704355 tid 2 2024-09-21T08:29:59.734 INFO:tasks.rados.rados.0.smithi137.stdout:1593: writing smithi13779221-19 from 3315201 to 3546376 tid 3 2024-09-21T08:29:59.734 INFO:tasks.rados.rados.0.smithi137.stdout:1594: snap_remove snap 138 2024-09-21T08:29:59.773 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:29:59.798 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:29:59.798 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:59.799 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:29:59.899 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:29:59.921 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:29:59.922 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:29:59.922 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:59 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:30:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:59 smithi165 ceph-mon[32893]: from='client.45641 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:59 smithi165 ceph-mon[32893]: osdmap e405: 8 total, 6 up, 4 in 2024-09-21T08:30:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:59 smithi165 ceph-mon[32893]: 5.8 scrub starts 2024-09-21T08:30:00.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:29:59 smithi165 ceph-mon[32893]: 5.8 scrub ok 2024-09-21T08:30:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:59 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:30:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:59 smithi137 ceph-mon[29634]: from='client.45641 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:59 smithi137 ceph-mon[29634]: osdmap e405: 8 total, 6 up, 4 in 2024-09-21T08:30:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:59 smithi137 ceph-mon[29634]: 5.8 scrub starts 2024-09-21T08:30:00.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:29:59 smithi137 ceph-mon[29634]: 5.8 scrub ok 2024-09-21T08:30:00.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:59 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:30:00.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:59 smithi137 ceph-mon[41044]: from='client.45641 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:00.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:59 smithi137 ceph-mon[41044]: osdmap e405: 8 total, 6 up, 4 in 2024-09-21T08:30:00.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:59 smithi137 ceph-mon[41044]: 5.8 scrub starts 2024-09-21T08:30:00.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:29:59 smithi137 ceph-mon[41044]: 5.8 scrub ok 2024-09-21T08:30:00.023 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:00.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:00.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:00.044 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:00.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:00.159 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:00.172 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:00.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:00.173 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:00.191 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:00.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:00.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:00.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:00.299 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:00.400 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:00.404 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:30:00.069+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:30:00.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:00.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:00.425 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:00.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:00.549 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:00.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:00.550 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:00.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:00.674 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:00.674 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:00.674 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:00.676 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:30:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:30:00.725 INFO:tasks.rados.rados.0.smithi137.stdout:1594: done (1 left) 2024-09-21T08:30:00.725 INFO:tasks.rados.rados.0.smithi137.stdout:1595: rollback oid 8 current snap is 169 2024-09-21T08:30:00.725 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 8 to 142 2024-09-21T08:30:00.725 INFO:tasks.rados.rados.0.smithi137.stdout:1593: finishing write tid 1 to smithi13779221-19 2024-09-21T08:30:00.725 INFO:tasks.rados.rados.0.smithi137.stdout:1593: finishing write tid 2 to smithi13779221-19 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1593: finishing write tid 3 to smithi13779221-19 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1593: finishing write tid 4 to smithi13779221-19 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1593: finishing write tid 5 to smithi13779221-19 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1593: oid 19 updating version 0 to 503 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1593: oid 19 version 503 is already newer than 500 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1593: oid 19 version 503 is already newer than 501 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1593: oid 19 version 503 is already newer than 502 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 503 (ObjNum 568 snap 169 seq_num 568) dirty exists 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1593: left oid 19 (ObjNum 568 snap 169 seq_num 568) 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1593: done (1 left) 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1596: write (excl) oid 7 current snap is 169 2024-09-21T08:30:00.726 INFO:tasks.rados.rados.0.smithi137.stdout:1596: seq_num 569 ranges {763183=626803,1996960=793182,3226271=336912} 2024-09-21T08:30:00.733 INFO:tasks.rados.rados.0.smithi137.stdout:1596: writing smithi13779221-7 from 763183 to 1389986 tid 1 2024-09-21T08:30:00.742 INFO:tasks.rados.rados.0.smithi137.stdout:1596: writing smithi13779221-7 from 1996960 to 2790142 tid 2 2024-09-21T08:30:00.746 INFO:tasks.rados.rados.0.smithi137.stdout:1596: writing smithi13779221-7 from 3226271 to 3563183 tid 3 2024-09-21T08:30:00.746 INFO:tasks.rados.rados.0.smithi137.stdout:1597: delete oid 18 current snap is 169 2024-09-21T08:30:00.746 INFO:tasks.rados.rados.0.smithi137.stdout:1595: finishing rollback tid 1 to smithi13779221-8 2024-09-21T08:30:00.746 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 320 (ObjNum 81 snap 0 seq_num 1) dirty dne 2024-09-21T08:30:00.747 INFO:tasks.rados.rados.0.smithi137.stdout:1596: finishing write tid 1 to smithi13779221-7 2024-09-21T08:30:00.747 INFO:tasks.rados.rados.0.smithi137.stdout:1596: finishing write tid 2 to smithi13779221-7 2024-09-21T08:30:00.748 INFO:tasks.rados.rados.0.smithi137.stdout:1596: finishing write tid 3 to smithi13779221-7 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1595: done (2 left) 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1597: done (1 left) 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1598: copy_from oid 27 from oid 43 current snap is 169 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1599: delete oid 47 current snap is 169 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1596: finishing write tid 4 to smithi13779221-7 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1596: finishing write tid 5 to smithi13779221-7 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1596: oid 7 updating version 0 to 634 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1596: oid 7 updating version 634 to 635 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1596: oid 7 updating version 635 to 637 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1596: oid 7 version 637 is already newer than 636 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 637 (ObjNum 569 snap 169 seq_num 569) dirty exists 2024-09-21T08:30:00.749 INFO:tasks.rados.rados.0.smithi137.stdout:1596: left oid 7 (ObjNum 569 snap 169 seq_num 569) 2024-09-21T08:30:00.750 INFO:tasks.rados.rados.0.smithi137.stdout:1596: done (2 left) 2024-09-21T08:30:00.750 INFO:tasks.rados.rados.0.smithi137.stdout:1599: done (1 left) 2024-09-21T08:30:00.750 INFO:tasks.rados.rados.0.smithi137.stdout:1600: delete oid 31 current snap is 169 2024-09-21T08:30:00.752 INFO:tasks.rados.rados.0.smithi137.stdout:1600: done (1 left) 2024-09-21T08:30:00.752 INFO:tasks.rados.rados.0.smithi137.stdout:1601: read oid 39 snap -1 2024-09-21T08:30:00.752 INFO:tasks.rados.rados.0.smithi137.stdout:1601: expect deleted 2024-09-21T08:30:00.752 INFO:tasks.rados.rados.0.smithi137.stdout:1602: read oid 46 snap -1 2024-09-21T08:30:00.752 INFO:tasks.rados.rados.0.smithi137.stdout:1602: expect (ObjNum 499 snap 140 seq_num 499) 2024-09-21T08:30:00.753 INFO:tasks.rados.rados.0.smithi137.stdout:1603: write (excl) oid 28 current snap is 169 2024-09-21T08:30:00.753 INFO:tasks.rados.rados.0.smithi137.stdout:1603: seq_num 573 ranges {430411=644152,1830888=400520,2935963=402981} 2024-09-21T08:30:00.758 INFO:tasks.rados.rados.0.smithi137.stdout:1603: writing smithi13779221-28 from 430411 to 1074563 tid 1 2024-09-21T08:30:00.764 INFO:tasks.rados.rados.0.smithi137.stdout:1603: writing smithi13779221-28 from 1830888 to 2231408 tid 2 2024-09-21T08:30:00.770 INFO:tasks.rados.rados.0.smithi137.stdout:1603: writing smithi13779221-28 from 2935963 to 3338944 tid 3 2024-09-21T08:30:00.770 INFO:tasks.rados.rados.0.smithi137.stdout:1604: snap_create 2024-09-21T08:30:00.771 INFO:tasks.rados.rados.0.smithi137.stdout:1602: expect (ObjNum 499 snap 140 seq_num 499) 2024-09-21T08:30:00.775 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:00.796 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:00.797 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:00.797 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:00.858 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:00.868 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:00.868 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:00.885 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:00.885 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:00.898 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:00.901 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:00.901 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:00.918 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:00.918 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:00.919 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:00.920 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:00.920 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:00.934 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:00.935 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:00.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: osdmap e406: 8 total, 6 up, 4 in 2024-09-21T08:30:00.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 320/1254 objects degraded (25.518%), 58 pgs degraded, 130 pgs undersized 2024-09-21T08:30:00.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: [WRN] OSD_DOWN: 1 osds down 2024-09-21T08:30:00.935 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: osd.7 (root=default,host=smithi165) is down 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: [WRN] PG_DEGRADED: Degraded data redundancy: 320/1254 objects degraded (25.518%), 58 pgs degraded, 130 pgs undersized 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.5 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.6 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.7 is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.8 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.19 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.1a is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.1b is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.1c is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.1d is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.936 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.1e is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 3.1f is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 4.0 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 4.1 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 4.3 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,1] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 4.4 is active+undersized+degraded, acting [1,2] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 4.f is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 4.1b is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 4.1e is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.1 is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.2 is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.3 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,1] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.e is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.18 is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.19 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.1a is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: osdmap e406: 8 total, 6 up, 4 in 2024-09-21T08:30:00.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 320/1254 objects degraded (25.518%), 58 pgs degraded, 130 pgs undersized 2024-09-21T08:30:00.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: [WRN] OSD_DOWN: 1 osds down 2024-09-21T08:30:00.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: osd.7 (root=default,host=smithi165) is down 2024-09-21T08:30:00.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: [WRN] PG_DEGRADED: Degraded data redundancy: 320/1254 objects degraded (25.518%), 58 pgs degraded, 130 pgs undersized 2024-09-21T08:30:00.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.5 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.6 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.7 is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.8 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.19 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.1a is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.1b is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.1c is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.1d is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.1e is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 3.1f is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 4.0 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 4.1 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 4.3 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,1] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 4.4 is active+undersized+degraded, acting [1,2] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 4.f is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 4.1b is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 4.1e is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.1 is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.2 is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.3 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,1] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.e is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.18 is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.19 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.1a is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.1b is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.1c is stuck undersized for 64s, current state active+undersized, last acting [2,1] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.1d is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 5.1f is stuck undersized for 64s, current state active+undersized+degraded, last acting [2,1] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 6.1 is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 6.2 is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 6.3 is stuck undersized for 64s, current state active+undersized, last acting [2,1] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 6.18 is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 6.1a is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 6.1b is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 6.1c is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 6.1e is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:00.941 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.1 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.2 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.3 is stuck undersized for 64s, current state active+undersized, last acting [2,1] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.7 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.18 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.19 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.1a is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.1d is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.1e is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.1f is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.20 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.21 is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.22 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pg 7.23 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: pgmap v515: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 438 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 320/1254 objects degraded (25.518%) 2024-09-21T08:30:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: 6.f scrub starts 2024-09-21T08:30:00.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[41044]: 6.f scrub ok 2024-09-21T08:30:00.943 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.1b is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.943 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.1c is stuck undersized for 64s, current state active+undersized, last acting [2,1] 2024-09-21T08:30:00.943 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.1d is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.943 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 5.1f is stuck undersized for 64s, current state active+undersized+degraded, last acting [2,1] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 6.1 is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 6.2 is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 6.3 is stuck undersized for 64s, current state active+undersized, last acting [2,1] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 6.18 is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 6.1a is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 6.1b is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 6.1c is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 6.1e is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.1 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.2 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.3 is stuck undersized for 64s, current state active+undersized, last acting [2,1] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.7 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:00.944 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.18 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.19 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.1a is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.1d is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.1e is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.1f is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.20 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.21 is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.22 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pg 7.23 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: pgmap v515: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 438 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 320/1254 objects degraded (25.518%) 2024-09-21T08:30:00.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: 6.f scrub starts 2024-09-21T08:30:00.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:00 smithi137 ceph-mon[29634]: 6.f scrub ok 2024-09-21T08:30:00.954 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:00.955 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:00.955 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:00.964 INFO:tasks.rados.rados.0.smithi137.stdout:1598: finishing copy_from racing read to smithi13779221-27 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1598: finishing copy_from to smithi13779221-27 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 383 (ObjNum 563 snap 167 seq_num 563) dirty exists 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1603: finishing write tid 1 to smithi13779221-28 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1603: finishing write tid 2 to smithi13779221-28 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1603: finishing write tid 3 to smithi13779221-28 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1603: finishing write tid 4 to smithi13779221-28 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1603: finishing write tid 5 to smithi13779221-28 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1603: oid 28 updating version 0 to 370 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1603: oid 28 updating version 370 to 371 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1603: oid 28 updating version 371 to 372 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1603: oid 28 updating version 372 to 373 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 373 (ObjNum 573 snap 169 seq_num 573) dirty exists 2024-09-21T08:30:00.965 INFO:tasks.rados.rados.0.smithi137.stdout:1603: left oid 28 (ObjNum 573 snap 169 seq_num 573) 2024-09-21T08:30:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: osdmap e406: 8 total, 6 up, 4 in 2024-09-21T08:30:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: Health detail: HEALTH_WARN 1 osds down; Degraded data redundancy: 320/1254 objects degraded (25.518%), 58 pgs degraded, 130 pgs undersized 2024-09-21T08:30:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: [WRN] OSD_DOWN: 1 osds down 2024-09-21T08:30:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: osd.7 (root=default,host=smithi165) is down 2024-09-21T08:30:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: [WRN] PG_DEGRADED: Degraded data redundancy: 320/1254 objects degraded (25.518%), 58 pgs degraded, 130 pgs undersized 2024-09-21T08:30:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.5 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.6 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.7 is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:01.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.8 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:01.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.19 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:01.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.1a is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:01.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.1b is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:01.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.1c is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:01.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.1d is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:01.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.1e is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:01.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 3.1f is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:01.015 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 4.0 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:01.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 4.1 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:01.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 4.3 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,1] 2024-09-21T08:30:01.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 4.4 is active+undersized+degraded, acting [1,2] 2024-09-21T08:30:01.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 4.f is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:01.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 4.1b is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:01.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 4.1e is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:01.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.1 is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:01.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.2 is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:01.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.3 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,1] 2024-09-21T08:30:01.016 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.e is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:01.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.18 is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:01.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.19 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:01.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.1a is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:01.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.1b is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:01.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.1c is stuck undersized for 64s, current state active+undersized, last acting [2,1] 2024-09-21T08:30:01.017 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.1d is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:01.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 5.1f is stuck undersized for 64s, current state active+undersized+degraded, last acting [2,1] 2024-09-21T08:30:01.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 6.1 is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:01.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 6.2 is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:01.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 6.3 is stuck undersized for 64s, current state active+undersized, last acting [2,1] 2024-09-21T08:30:01.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 6.18 is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:01.018 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 6.1a is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:01.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 6.1b is stuck undersized for 64s, current state active+undersized, last acting [1,2] 2024-09-21T08:30:01.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 6.1c is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:01.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 6.1e is stuck undersized for 64s, current state active+undersized, last acting [5,2] 2024-09-21T08:30:01.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.1 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:01.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.2 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:01.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.3 is stuck undersized for 64s, current state active+undersized, last acting [2,1] 2024-09-21T08:30:01.019 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.7 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:01.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.18 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:01.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.19 is stuck undersized for 64s, current state active+undersized+degraded, last acting [5,2] 2024-09-21T08:30:01.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.1a is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:01.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.1d is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:01.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.1e is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,5] 2024-09-21T08:30:01.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.1f is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:01.020 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.20 is stuck undersized for 64s, current state active+undersized+degraded, last acting [1,2] 2024-09-21T08:30:01.021 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.21 is stuck undersized for 64s, current state active+undersized, last acting [5,1] 2024-09-21T08:30:01.021 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.22 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:01.021 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pg 7.23 is stuck undersized for 64s, current state active+undersized, last acting [1,5] 2024-09-21T08:30:01.021 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:01.021 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:01.021 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: pgmap v515: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 438 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 320/1254 objects degraded (25.518%) 2024-09-21T08:30:01.021 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: 6.f scrub starts 2024-09-21T08:30:01.022 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:00 smithi165 ceph-mon[32893]: 6.f scrub ok 2024-09-21T08:30:01.022 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:01.035 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1] out_osds: [4, 6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:30:01.036 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:30:01.036 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T08:30:01.036 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T08:30:01.036 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T08:30:01.036 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T08:30:01.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:01.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:01.049 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:01.150 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:01.171 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:01.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:01.172 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:01.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:01.299 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:01.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:01.300 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:01.304 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:01.400 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:01.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:01.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:01.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:01.523 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:01.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:01.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:01.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:01.576 INFO:tasks.rados.rados.0.smithi137.stdout:1598: done (4 left) 2024-09-21T08:30:01.576 INFO:tasks.rados.rados.0.smithi137.stdout:1601: done (3 left) 2024-09-21T08:30:01.577 INFO:tasks.rados.rados.0.smithi137.stdout:1602: done (2 left) 2024-09-21T08:30:01.577 INFO:tasks.rados.rados.0.smithi137.stdout:1603: done (1 left) 2024-09-21T08:30:01.577 INFO:tasks.rados.rados.0.smithi137.stdout:1604: done (0 left) 2024-09-21T08:30:01.577 INFO:tasks.rados.rados.0.smithi137.stdout:1605: snap_remove snap 166 2024-09-21T08:30:01.648 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:01.670 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:01.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:01.671 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:01.707 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:01.709 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:01.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:01.793 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:01.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:01.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:01.803 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:30:01.803 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T08:30:01.803 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:30:01.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:01.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:01.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:01.917 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:01.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:01 smithi137 ceph-mon[29634]: osdmap e407: 8 total, 6 up, 4 in 2024-09-21T08:30:01.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:01 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2199199105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:01.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:01 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 320/1254 objects degraded (25.518%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:01.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:01 smithi137 ceph-mon[29634]: osdmap e408: 8 total, 6 up, 4 in 2024-09-21T08:30:01.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:01 smithi137 ceph-mon[41044]: osdmap e407: 8 total, 6 up, 4 in 2024-09-21T08:30:01.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:01 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2199199105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:01.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:01 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 320/1254 objects degraded (25.518%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:01.940 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:01 smithi137 ceph-mon[41044]: osdmap e408: 8 total, 6 up, 4 in 2024-09-21T08:30:01.960 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:30:01.960 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:30:01.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:30:01.960 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:30:01.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.960 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T08:30:01.960 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:30:01.960 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 525.48438225999996, 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27973036899999998, 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:01.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473701+0000", 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1320000000000001e-06 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473716+0000", 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5622999999999999e-05 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473730+0000", 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3259e-05 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473740+0000", 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0746e-05 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473802+0000", 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2043000000000002e-05 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473810+0000", 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1500000000000002e-06 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752339+0000", 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27852951199999998 2024-09-21T08:30:01.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752485+0000", 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014628699999999999 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752609+0000", 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012376499999999999 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753145+0000", 2024-09-21T08:30:01.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053571000000000001 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753326+0000", 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018102799999999999 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753367+0000", 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1253e-05 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:01.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753400+0000", 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3065999999999998e-05 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753429+0000", 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8795000000000001e-05 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:422 7.d 7:b31d99ce:::smithi13779221-21:head [write 443718~615603] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:30:01.967 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 305.45310527800001, 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034885076000000001, 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 422 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:30:01.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504978+0000", 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0090000000000002e-06 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506268+0000", 2024-09-21T08:30:01.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001290444 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506279+0000", 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0526999999999999e-05 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506288+0000", 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5470000000000002e-06 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506343+0000", 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5287000000000001e-05 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506354+0000", 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0798999999999999e-05 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.525308+0000", 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018953616 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529390+0000", 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040819029999999996 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533318+0000", 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039282190000000002 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533380+0000", 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1890000000000005e-05 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:01.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533709+0000", 2024-09-21T08:30:01.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032948400000000001 2024-09-21T08:30:01.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:01.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536916+0000", 2024-09-21T08:30:01.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032063399999999998 2024-09-21T08:30:01.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:01.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539553+0000", 2024-09-21T08:30:01.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026370909999999998 2024-09-21T08:30:01.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539658+0000", 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105585 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539723+0000", 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4343000000000004e-05 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539861+0000", 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137992 2024-09-21T08:30:01.975 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:423 7.d 7:b31d99ce:::smithi13779221-21:head [write 1564138~470341] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 305.45171561000001, 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033782404000000002, 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:01.976 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 423 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:01.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506366+0000", 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0449999999999999e-06 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507012+0000", 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064568899999999996 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507021+0000", 2024-09-21T08:30:01.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2550000000000003e-06 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507029+0000", 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2559999999999996e-06 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507079+0000", 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0957e-05 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:01.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507088+0000", 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3650000000000004e-06 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529457+0000", 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022369185999999999 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529526+0000", 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8683999999999995e-05 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.980 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535157+0000", 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056315849999999997 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535244+0000", 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6840000000000002e-05 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535398+0000", 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015404399999999999 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.981 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539569+0000", 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004170361 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539613+0000", 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4101999999999999e-05 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539985+0000", 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037279300000000001 2024-09-21T08:30:01.982 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540037+0000", 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1345999999999999e-05 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540148+0000", 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110896 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.983 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:424 7.d 7:b31d99ce:::smithi13779221-21:head [write 2633066~610652] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 305.44472659600001, 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027015847999999999, 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 424 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.984 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513356+0000", 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8789999999999999e-06 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.985 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513753+0000", 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039670999999999999 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513764+0000", 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.154e-05 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513772+0000", 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9149999999999996e-06 2024-09-21T08:30:01.986 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513833+0000", 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1072000000000001e-05 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513843+0000", 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8349999999999999e-06 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529644+0000", 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015801084 2024-09-21T08:30:01.987 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529688+0000", 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3568e-05 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536031+0000", 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063434659999999999 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536135+0000", 2024-09-21T08:30:01.988 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103599 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536291+0000", 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155668 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539575+0000", 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032840450000000002 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:01.989 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539925+0000", 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034988899999999998 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540200+0000", 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027585000000000001 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540250+0000", 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9413000000000003e-05 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:01.990 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540370+0000", 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120315 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:425 7.d 7:b31d99ce:::smithi13779221-21:head [setxattr _header (56) in=7b,truncate 3243718] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 305.44422395599997, 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030130239, 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:01.991 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 425 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:30:01.992 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513858+0000", 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9199999999999999e-07 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513863+0000", 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7650000000000001e-06 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513869+0000", 2024-09-21T08:30:01.993 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9880000000000001e-06 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513874+0000", 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.643e-06 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515915+0000", 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020407279999999999 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515925+0000", 2024-09-21T08:30:01.994 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0051000000000001e-05 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529722+0000", 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013796780999999999 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529804+0000", 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1663999999999996e-05 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:01.995 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540423+0000", 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010619635000000001 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540671+0000", 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024766600000000001 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540795+0000", 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012383899999999999 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.996 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541324+0000", 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052910200000000004 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.542654+0000", 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013304370000000001 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543785+0000", 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001130603 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.997 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543841+0000", 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5939000000000001e-05 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543987+0000", 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014640599999999999 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:426 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:01.998 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 305.44226501100002, 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028482668999999999, 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 426 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:30:01.999 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515818+0000", 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7999999999999999e-06 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.000 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515825+0000", 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0790000000000003e-06 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515840+0000", 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4765e-05 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515857+0000", 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7414999999999998e-05 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.001 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515968+0000", 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111161 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515975+0000", 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7499999999999997e-06 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529841+0000", 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013866387000000001 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.002 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529951+0000", 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010953100000000001 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541110+0000", 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011159591 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541190+0000", 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9243e-05 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.003 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544032+0000", 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028424050000000001 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544118+0000", 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5798999999999996e-05 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544299+0000", 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180743 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.004 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:430 7.d 7:b0b32695:::smithi13779221-19:head [delete] snapc a=[9] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 305.442094333, 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028508957000000001, 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.005 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 430 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.006 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1900000000000001e-07 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515991+0000", 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6899999999999998e-06 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515997+0000", 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2999999999999998e-06 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.007 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516003+0000", 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0220000000000001e-06 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516045+0000", 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1069000000000003e-05 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516052+0000", 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9890000000000003e-06 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.008 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529980+0000", 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013927999999999999 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.532305+0000", 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023250580000000001 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541672+0000", 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093675080000000001 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.009 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541749+0000", 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6829000000000006e-05 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541934+0000", 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018531 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543560+0000", 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00162525 2024-09-21T08:30:02.010 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543661+0000", 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101604 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544339+0000", 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067745299999999995 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544394+0000", 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5515e-05 2024-09-21T08:30:02.011 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544496+0000", 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010144100000000001 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:552 7.d 7:b0b32695:::smithi13779221-19:head [copy-from ver 34] snapc 13=[13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e116)", 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:09.528772+0000", 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 292.42930930199998, 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022405184000000002, 2024-09-21T08:30:02.012 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 552 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528772+0000", 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.013 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528772+0000", 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528776+0000", 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3510000000000004e-06 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528788+0000", 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2542e-05 2024-09-21T08:30:02.014 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528810+0000", 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1467e-05 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528822+0000", 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1973e-05 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528913+0000", 2024-09-21T08:30:02.015 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1458000000000004e-05 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.528922+0000", 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0780000000000002e-06 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.531438+0000", 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002515541 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.016 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.531531+0000", 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3175000000000002e-05 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.539639+0000", 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0081080649999999994 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.547323+0000", 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076836079999999998 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.017 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.550791+0000", 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0034681719999999998 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.551056+0000", 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000264966 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.551116+0000", 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9905000000000003e-05 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.018 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:09.551177+0000", 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0883e-05 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:571 7.1 7:88bf4ad2:::smithi13779221-6:head [copy-from ver 13] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 291.41102767899997, 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037746406000000003, 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.019 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 571 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.020 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547056+0000", 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.146e-06 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547067+0000", 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0609e-05 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.021 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547090+0000", 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2714999999999999e-05 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547100+0000", 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9920000000000002e-06 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547203+0000", 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102828 2024-09-21T08:30:02.022 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547210+0000", 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3629999999999996e-06 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553577+0000", 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063672829999999996 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553669+0000", 2024-09-21T08:30:02.023 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1856999999999998e-05 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573941+0000", 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020272168 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.583019+0000", 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090777630000000008 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584643+0000", 2024-09-21T08:30:02.024 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016245319999999999 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584686+0000", 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2259999999999999e-05 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584748+0000", 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2049000000000002e-05 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584800+0000", 2024-09-21T08:30:02.025 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1841000000000001e-05 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:572 7.1 7:88bf4ad2:::smithi13779221-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 291.41095865, 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032210599999999999, 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.026 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 572 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.027 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.028 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.028 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:02.029 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:30:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:30:02.029 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:01 smithi165 ceph-mon[32893]: osdmap e407: 8 total, 6 up, 4 in 2024-09-21T08:30:02.029 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:01 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2199199105' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:02.029 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:01 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 320/1254 objects degraded (25.518%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:02.029 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:01 smithi165 ceph-mon[32893]: osdmap e408: 8 total, 6 up, 4 in 2024-09-21T08:30:02.030 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547123+0000", 2024-09-21T08:30:02.030 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5300000000000003e-07 2024-09-21T08:30:02.030 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.030 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547138+0000", 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4718999999999999e-05 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547145+0000", 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2520000000000002e-06 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547151+0000", 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7649999999999999e-06 2024-09-21T08:30:02.031 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547263+0000", 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112001 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547269+0000", 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8919999999999997e-06 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553824+0000", 2024-09-21T08:30:02.032 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065557499999999999 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553873+0000", 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8727999999999998e-05 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579133+0000", 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025259964999999999 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579227+0000", 2024-09-21T08:30:02.033 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3517999999999995e-05 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579333+0000", 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010615699999999999 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:576 7.2 7:4031afd1:::smithi13779221-48:head [append 0~0] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547160+0000", 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 291.41092093899999, 2024-09-21T08:30:02.034 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026627913, 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 576 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547160+0000", 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.035 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547160+0000", 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547161+0000", 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.44e-07 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547163+0000", 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5790000000000002e-06 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.036 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547169+0000", 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7610000000000004e-06 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547174+0000", 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7849999999999999e-06 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547266+0000", 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1978999999999995e-05 2024-09-21T08:30:02.037 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547272+0000", 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3629999999999999e-06 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.562646+0000", 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015374095000000001 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.562751+0000", 2024-09-21T08:30:02.038 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010446 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.562953+0000", 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000202151 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564215+0000", 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012625589999999999 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.039 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567784+0000", 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0035684559999999998 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573649+0000", 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0058654110000000001 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573729+0000", 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9555999999999995e-05 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.040 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573788+0000", 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9113999999999997e-05 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:577 7.2 7:4031afd1:::smithi13779221-48:head [rollback 14] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 291.41089428700002, 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026796044000000002, 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.041 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 577 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.042 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547187+0000", 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3e-07 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547190+0000", 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3980000000000002e-06 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547196+0000", 2024-09-21T08:30:02.043 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5239999999999997e-06 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547201+0000", 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5839999999999996e-06 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547293+0000", 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2613000000000003e-05 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:02.044 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547298+0000", 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6500000000000004e-06 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564248+0000", 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01694959 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564348+0000", 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000100893 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.045 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564547+0000", 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000198278 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.565560+0000", 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001013391 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567845+0000", 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022853090000000001 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.046 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573898+0000", 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.006052094 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573944+0000", 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6928000000000002e-05 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573983+0000", 2024-09-21T08:30:02.047 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8161999999999998e-05 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:578 7.2 7:4031afd1:::smithi13779221-48:head [append 0~0] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 291.41087502599999, 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026976508, 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.048 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 578 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.049 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547206+0000", 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8100000000000003e-07 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547209+0000", 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4549999999999998e-06 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547214+0000", 2024-09-21T08:30:02.050 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2889999999999997e-06 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547219+0000", 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7419999999999997e-06 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547317+0000", 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8029000000000006e-05 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:02.051 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547321+0000", 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2880000000000003e-06 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566078+0000", 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018756951000000001 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566167+0000", 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9068999999999997e-05 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.052 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.566318+0000", 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015045999999999999 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.567184+0000", 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00086585299999999998 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.568942+0000", 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001758103 2024-09-21T08:30:02.053 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574047+0000", 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0051054719999999998 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574143+0000", 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5616000000000001e-05 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.574182+0000", 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.96e-05 2024-09-21T08:30:02.054 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3155 7.9 7:9247b5b1:::smithi13779221-17:head [copy-from ver 183] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 89.385584836000007, 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029797267999999998, 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.055 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3155 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.056 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572497+0000", 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4899999999999995e-07 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572500+0000", 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.05e-06 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.057 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572505+0000", 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2820000000000004e-06 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572510+0000", 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8969999999999999e-06 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573522+0000", 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001012481 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.058 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573635+0000", 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112253 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.583897+0000", 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010262288 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594011+0000", 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010113641 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.059 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594202+0000", 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000191418 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602174+0000", 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079717079999999992 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602233+0000", 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9565999999999999e-05 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.060 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602293+0000", 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0134999999999999e-05 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3441 7.15 7:a9dd6c7d:::smithi13779221-47:head [write 783325~557161,stat] snapc 84=[84,83,82,81,7f,7e,7d,7c,78,77,72,71,69] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 65.394735002000004, 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028447395, 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.061 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3441 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.062 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563352+0000", 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9039999999999997e-06 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563878+0000", 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052587200000000004 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.063 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563908+0000", 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0685999999999997e-05 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563922+0000", 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3738e-05 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.564007+0000", 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5283000000000003e-05 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.564017+0000", 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4909999999999994e-06 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571275+0000", 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072579860000000001 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571419+0000", 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014457700000000001 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571666+0000", 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024687599999999999 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591189+0000", 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019522923000000001 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591633+0000", 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044407499999999999 2024-09-21T08:30:02.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591717+0000", 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4010000000000004e-05 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591793+0000", 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5974000000000001e-05 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3470 7.b 7:d22ba293:::smithi13779221-29:head [rollback 7c] snapc 86=[86,85,84,83,82,81,7f,7e,7d,7c,78,77,72,71,69] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:58.612900+0000", 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 63.345181048999997, 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01966029, 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3470 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.612900+0000", 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.612900+0000", 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.612900+0000", 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3600000000000004e-07 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.612903+0000", 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8820000000000001e-06 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.612908+0000", 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2610000000000004e-06 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.612911+0000", 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9160000000000003e-06 2024-09-21T08:30:02.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.613009+0000", 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.7544e-05 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.613015+0000", 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.57e-06 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.622748+0000", 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097331659999999997 2024-09-21T08:30:02.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.622833+0000", 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5464999999999996e-05 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.623039+0000", 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00020581700000000001 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.624431+0000", 2024-09-21T08:30:02.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00139193 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.628570+0000", 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041386369999999997 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.632462+0000", 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0038928600000000002 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.632508+0000", 2024-09-21T08:30:02.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5803999999999997e-05 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.632560+0000", 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1901999999999999e-05 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3471 7.b 7:d22ba293:::smithi13779221-29:head [append 0~0] snapc 86=[86,85,84,83,82,81,7f,7e,7d,7c,78,77,72,71,69] ondisk+write+known_if_redirected+supports_pool_eio e345)", 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:58.613048+0000", 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 63.345033305000001, 2024-09-21T08:30:02.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019619215999999998, 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3471 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.613048+0000", 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.613048+0000", 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.613049+0000", 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8400000000000002e-07 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.613053+0000", 2024-09-21T08:30:02.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2520000000000001e-06 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.613060+0000", 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0639999999999996e-06 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.613065+0000", 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6030000000000004e-06 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.613100+0000", 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4952e-05 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.613106+0000", 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6590000000000004e-06 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.623614+0000", 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010507403 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.623989+0000", 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000375492 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.624128+0000", 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013902900000000001 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.624930+0000", 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00080240800000000001 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.628678+0000", 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003748028 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.632593+0000", 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039142059999999999 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.632625+0000", 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2493000000000002e-05 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:58.632667+0000", 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1643000000000001e-05 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4000 7.b 7:d6c5a045:::smithi13779221-46:head [copy-from ver 529] snapc a7=[a7,a6,a5,a4,a3,a2,a1,9e,9d,9c,9b,9a,99,98,94,93,92,91,90,8c,8b,88,86,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.389756655999999, 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037495776000000001, 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.081 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4000 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568328+0000", 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.349e-06 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568346+0000", 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8839e-05 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568362+0000", 2024-09-21T08:30:02.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5432999999999999e-05 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568371+0000", 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3009999999999995e-06 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568435+0000", 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4199e-05 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568441+0000", 2024-09-21T08:30:02.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0050000000000001e-06 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.575820+0000", 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073784970000000004 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.575934+0000", 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011455699999999999 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:30:02.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587076+0000", 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011141287 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.595892+0000", 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088164750000000007 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605259+0000", 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093663830000000007 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.086 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605662+0000", 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040333900000000001 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605744+0000", 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2560999999999998e-05 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605820+0000", 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5550999999999995e-05 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.087 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4001 7.b 7:d6c5a045:::smithi13779221-46:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e398)", 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:29:51.568392+0000", 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.389688891, 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022944272000000002, 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4001 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.088 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568392+0000", 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568392+0000", 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568393+0000", 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8299999999999997e-07 2024-09-21T08:30:02.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568395+0000", 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2129999999999998e-06 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568400+0000", 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.13e-06 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568404+0000", 2024-09-21T08:30:02.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7239999999999998e-06 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568495+0000", 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1572999999999999e-05 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568502+0000", 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1520000000000003e-06 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.091 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.576113+0000", 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076106560000000004 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.576166+0000", 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2565999999999998e-05 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.591112+0000", 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014946586 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.591216+0000", 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010344 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.591336+0000", 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120649 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4006 7.9 7:9247b5b1:::smithi13779221-17:head [write 477464~714864,stat] snapc a7=[a7,a6,a5,a4,a3,a2,a1,9e,9d,9c,9b,9a,99,98,94,93,92,91,90,8c,8b,88,86,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:29:51.585189+0000", 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 10.372891444, 2024-09-21T08:30:02.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021995097000000002, 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4006 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.585189+0000", 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.585189+0000", 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.585195+0000", 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4269999999999999e-06 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.585732+0000", 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053664999999999995 2024-09-21T08:30:02.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.585763+0000", 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1390000000000003e-05 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.585779+0000", 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5699999999999999e-05 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.585918+0000", 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013946799999999999 2024-09-21T08:30:02.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586090+0000", 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000171826 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586383+0000", 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029268799999999999 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.604565+0000", 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018182203000000001 2024-09-21T08:30:02.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.606984+0000", 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0024187449999999999 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.607059+0000", 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5386e-05 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.607185+0000", 2024-09-21T08:30:02.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012561399999999999 2024-09-21T08:30:02.099 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.099 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.099 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.099 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:02.099 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:02.099 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:30:02.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:30:02.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:02.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:02.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:02.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:02.244 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:02.245 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:02.245 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:02.346 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:02.368 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:02.368 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:02.368 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:02.469 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:02.491 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:02.491 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:02.492 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:02.574 INFO:tasks.rados.rados.0.smithi137.stdout:1605: done (0 left) 2024-09-21T08:30:02.574 INFO:tasks.rados.rados.0.smithi137.stdout:1606: read oid 28 snap -1 2024-09-21T08:30:02.574 INFO:tasks.rados.rados.0.smithi137.stdout:1606: expect (ObjNum 573 snap 169 seq_num 573) 2024-09-21T08:30:02.574 INFO:tasks.rados.rados.0.smithi137.stdout:1607: write (excl) oid 13 current snap is 170 2024-09-21T08:30:02.574 INFO:tasks.rados.rados.0.smithi137.stdout:1607: seq_num 574 ranges {447218=719401,1889370=660531,2963386=614253} 2024-09-21T08:30:02.581 INFO:tasks.rados.rados.0.smithi137.stdout:1607: writing smithi13779221-13 from 447218 to 1166619 tid 1 2024-09-21T08:30:02.588 INFO:tasks.rados.rados.0.smithi137.stdout:1607: writing smithi13779221-13 from 1889370 to 2549901 tid 2 2024-09-21T08:30:02.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:02.595 INFO:tasks.rados.rados.0.smithi137.stdout:1607: writing smithi13779221-13 from 2963386 to 3577639 tid 3 2024-09-21T08:30:02.595 INFO:tasks.rados.rados.0.smithi137.stdout:1608: read oid 7 snap -1 2024-09-21T08:30:02.595 INFO:tasks.rados.rados.0.smithi137.stdout:1608: expect (ObjNum 569 snap 169 seq_num 569) 2024-09-21T08:30:02.595 INFO:tasks.rados.rados.0.smithi137.stdout:1609: read oid 31 snap -1 2024-09-21T08:30:02.595 INFO:tasks.rados.rados.0.smithi137.stdout:1609: expect deleted 2024-09-21T08:30:02.595 INFO:tasks.rados.rados.0.smithi137.stdout:1610: write (excl) oid 43 current snap is 170 2024-09-21T08:30:02.596 INFO:tasks.rados.rados.0.smithi137.stdout:1610: seq_num 575 ranges {464025=794650,1947852=404189,2990809=425525} 2024-09-21T08:30:02.602 INFO:tasks.rados.rados.0.smithi137.stdout:1610: writing smithi13779221-43 from 464025 to 1258675 tid 1 2024-09-21T08:30:02.608 INFO:tasks.rados.rados.0.smithi137.stdout:1610: writing smithi13779221-43 from 1947852 to 2352041 tid 2 2024-09-21T08:30:02.614 INFO:tasks.rados.rados.0.smithi137.stdout:1610: writing smithi13779221-43 from 2990809 to 3416334 tid 3 2024-09-21T08:30:02.614 INFO:tasks.rados.rados.0.smithi137.stdout:1611: snap_remove snap 142 2024-09-21T08:30:02.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:02.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:02.616 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:02.716 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:02.738 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:02.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:02.739 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:02.840 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:02.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:02.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:02.862 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:02.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:02.984 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:02.984 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:02.984 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:03.085 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:03.110 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:03.111 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:03.111 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:03.212 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:03.236 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:03.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:03.238 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:03.240 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T08:30:03.295 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:03.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:03.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:03.363 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:03.363 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:03.463 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:03.485 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:03.486 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:03.486 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:03.582 INFO:tasks.rados.rados.0.smithi137.stdout:1611: done (5 left) 2024-09-21T08:30:03.582 INFO:tasks.rados.rados.0.smithi137.stdout:1612: copy_from oid 49 from oid 26 current snap is 170 2024-09-21T08:30:03.583 INFO:tasks.rados.rados.0.smithi137.stdout:1613: snap_remove snap 156 2024-09-21T08:30:03.587 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:03.611 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:03.612 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:03.612 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:03.711 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:30:03.713 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:03.734 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:03.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:03.735 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:03.776 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:03.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:03.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:03.859 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:03.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:03.899 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:03 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:03.899 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:03 smithi137 ceph-mon[29634]: 6.10 scrub starts 2024-09-21T08:30:03.899 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:03 smithi137 ceph-mon[29634]: 6.10 scrub ok 2024-09-21T08:30:03.899 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:03 smithi137 ceph-mon[29634]: pgmap v518: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 453 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 5.7 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 324/1272 objects degraded (25.472%) 2024-09-21T08:30:03.899 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:03 smithi137 ceph-mon[29634]: osdmap e409: 8 total, 6 up, 4 in 2024-09-21T08:30:03.899 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:03 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:03.899 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:03 smithi137 ceph-mon[41044]: 6.10 scrub starts 2024-09-21T08:30:03.900 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:03 smithi137 ceph-mon[41044]: 6.10 scrub ok 2024-09-21T08:30:03.900 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:03 smithi137 ceph-mon[41044]: pgmap v518: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 453 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 5.7 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 324/1272 objects degraded (25.472%) 2024-09-21T08:30:03.900 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:03 smithi137 ceph-mon[41044]: osdmap e409: 8 total, 6 up, 4 in 2024-09-21T08:30:03.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:03.981 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:03.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:03.982 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:03 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:03 smithi165 ceph-mon[32893]: 6.10 scrub starts 2024-09-21T08:30:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:03 smithi165 ceph-mon[32893]: 6.10 scrub ok 2024-09-21T08:30:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:03 smithi165 ceph-mon[32893]: pgmap v518: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 453 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 5.7 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 324/1272 objects degraded (25.472%) 2024-09-21T08:30:04.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:03 smithi165 ceph-mon[32893]: osdmap e409: 8 total, 6 up, 4 in 2024-09-21T08:30:04.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:04.104 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:04.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:04.105 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:04.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:04.226 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:04.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:04.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:04.293 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:30:04.294 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:30:04.294 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:30:04.294 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:30:04.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:04.350 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:04.350 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:04.351 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:04.451 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:04.473 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:04.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:04.474 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:04.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:04.581 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:30:04.587 INFO:tasks.rados.rados.0.smithi137.stdout:1613: done (6 left) 2024-09-21T08:30:04.587 INFO:tasks.rados.rados.0.smithi137.stdout:1614: write oid 10 current snap is 170 2024-09-21T08:30:04.587 INFO:tasks.rados.rados.0.smithi137.stdout:1614: seq_num 576 ranges {480832=469899,1606334=664200,3018232=662600} 2024-09-21T08:30:04.592 INFO:tasks.rados.rados.0.smithi137.stdout:1614: writing smithi13779221-10 from 480832 to 950731 tid 1 2024-09-21T08:30:04.600 INFO:tasks.rados.rados.0.smithi137.stdout:1614: writing smithi13779221-10 from 1606334 to 2270534 tid 2 2024-09-21T08:30:04.600 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:04.601 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:04.601 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:04.609 INFO:tasks.rados.rados.0.smithi137.stdout:1614: writing smithi13779221-10 from 3018232 to 3680832 tid 3 2024-09-21T08:30:04.609 INFO:tasks.rados.rados.0.smithi137.stdout:1615: write (excl) oid 35 current snap is 170 2024-09-21T08:30:04.609 INFO:tasks.rados.rados.0.smithi137.stdout:1606: expect (ObjNum 573 snap 169 seq_num 573) 2024-09-21T08:30:04.703 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:04.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:04.725 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:04.725 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:04.826 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:04.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:04.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:04.847 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:04.888 INFO:tasks.rados.rados.0.smithi137.stdout:1608: expect (ObjNum 569 snap 169 seq_num 569) 2024-09-21T08:30:04.948 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:04.949 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:04 smithi137 ceph-mon[29634]: osdmap e410: 8 total, 6 up, 4 in 2024-09-21T08:30:04.949 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:04 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1583383079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:30:04.950 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:04 smithi137 ceph-mon[41044]: osdmap e410: 8 total, 6 up, 4 in 2024-09-21T08:30:04.950 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:04 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1583383079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:30:04.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:04.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:04.973 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:04.978 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T08:30:05.003 INFO:tasks.rados.rados.0.smithi137.stdout:1610: finishing write tid 1 to smithi13779221-43 2024-09-21T08:30:05.003 INFO:tasks.rados.rados.0.smithi137.stdout:1610: finishing write tid 2 to smithi13779221-43 2024-09-21T08:30:05.003 INFO:tasks.rados.rados.0.smithi137.stdout:1610: finishing write tid 3 to smithi13779221-43 2024-09-21T08:30:05.003 INFO:tasks.rados.rados.0.smithi137.stdout:1615: seq_num 577 ranges {497639=545148,1548463=407858,2529302=564422,3528143=169496} 2024-09-21T08:30:05.008 INFO:tasks.rados.rados.0.smithi137.stdout:1615: writing smithi13779221-35 from 497639 to 1042787 tid 1 2024-09-21T08:30:05.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:04 smithi165 ceph-mon[32893]: osdmap e410: 8 total, 6 up, 4 in 2024-09-21T08:30:05.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:04 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1583383079' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:30:05.013 INFO:tasks.rados.rados.0.smithi137.stdout:1615: writing smithi13779221-35 from 1548463 to 1956321 tid 2 2024-09-21T08:30:05.019 INFO:tasks.rados.rados.0.smithi137.stdout:1615: writing smithi13779221-35 from 2529302 to 3093724 tid 3 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1615: writing smithi13779221-35 from 3528143 to 3697639 tid 4 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1606: done (7 left) 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1608: done (6 left) 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1609: done (5 left) 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1616: snap_create 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1610: finishing write tid 4 to smithi13779221-43 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1610: finishing write tid 5 to smithi13779221-43 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1610: oid 43 updating version 0 to 384 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1610: oid 43 updating version 384 to 385 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1610: oid 43 updating version 385 to 387 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:1610: oid 43 version 387 is already newer than 386 2024-09-21T08:30:05.024 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 387 (ObjNum 575 snap 170 seq_num 575) dirty exists 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1610: left oid 43 (ObjNum 575 snap 170 seq_num 575) 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1607: finishing write tid 1 to smithi13779221-13 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1607: finishing write tid 2 to smithi13779221-13 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1607: finishing write tid 3 to smithi13779221-13 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1607: finishing write tid 4 to smithi13779221-13 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1607: finishing write tid 5 to smithi13779221-13 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1607: oid 13 updating version 0 to 639 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1607: oid 13 updating version 639 to 640 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1607: oid 13 updating version 640 to 642 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1607: oid 13 version 642 is already newer than 641 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 642 (ObjNum 574 snap 170 seq_num 574) dirty exists 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1607: left oid 13 (ObjNum 574 snap 170 seq_num 574) 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1612: finishing copy_from racing read to smithi13779221-49 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:1612: finishing copy_from to smithi13779221-49 2024-09-21T08:30:05.025 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 379 (ObjNum 560 snap 167 seq_num 560) dirty exists 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1614: finishing write tid 1 to smithi13779221-10 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1614: finishing write tid 2 to smithi13779221-10 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1614: finishing write tid 3 to smithi13779221-10 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1614: finishing write tid 4 to smithi13779221-10 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1614: finishing write tid 5 to smithi13779221-10 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1614: oid 10 updating version 0 to 529 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1614: oid 10 version 529 is already newer than 528 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1614: oid 10 updating version 529 to 530 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1614: oid 10 updating version 530 to 531 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 531 (ObjNum 576 snap 170 seq_num 576) dirty exists 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1614: left oid 10 (ObjNum 576 snap 170 seq_num 576) 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1615: finishing write tid 1 to smithi13779221-35 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1615: finishing write tid 2 to smithi13779221-35 2024-09-21T08:30:05.026 INFO:tasks.rados.rados.0.smithi137.stdout:1615: finishing write tid 3 to smithi13779221-35 2024-09-21T08:30:05.027 INFO:tasks.rados.rados.0.smithi137.stdout:1615: finishing write tid 4 to smithi13779221-35 2024-09-21T08:30:05.027 INFO:tasks.rados.rados.0.smithi137.stdout:1615: finishing write tid 5 to smithi13779221-35 2024-09-21T08:30:05.027 INFO:tasks.rados.rados.0.smithi137.stdout:1615: finishing write tid 6 to smithi13779221-35 2024-09-21T08:30:05.027 INFO:tasks.rados.rados.0.smithi137.stdout:1615: oid 35 updating version 0 to 509 2024-09-21T08:30:05.027 INFO:tasks.rados.rados.0.smithi137.stdout:1615: oid 35 version 509 is already newer than 508 2024-09-21T08:30:05.027 INFO:tasks.rados.rados.0.smithi137.stdout:1615: oid 35 updating version 509 to 510 2024-09-21T08:30:05.028 INFO:tasks.rados.rados.0.smithi137.stdout:1615: oid 35 version 510 is already newer than 506 2024-09-21T08:30:05.028 INFO:tasks.rados.rados.0.smithi137.stdout:1615: oid 35 version 510 is already newer than 507 2024-09-21T08:30:05.028 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 510 (ObjNum 577 snap 170 seq_num 577) dirty exists 2024-09-21T08:30:05.028 INFO:tasks.rados.rados.0.smithi137.stdout:1615: left oid 35 (ObjNum 577 snap 170 seq_num 577) 2024-09-21T08:30:05.074 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:05.100 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:05.101 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:05.101 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:05.202 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:05.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:05.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:05.229 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:05.329 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:05.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:05.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:05.356 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:05.457 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:05.482 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:05.483 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:05.483 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:05.584 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1607: done (5 left) 2024-09-21T08:30:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1610: done (4 left) 2024-09-21T08:30:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1612: done (3 left) 2024-09-21T08:30:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1614: done (2 left) 2024-09-21T08:30:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1615: done (1 left) 2024-09-21T08:30:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1616: done (0 left) 2024-09-21T08:30:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1617: write (excl) oid 6 current snap is 171 2024-09-21T08:30:05.596 INFO:tasks.rados.rados.0.smithi137.stdout:1617: seq_num 578 ranges {514446=736750,1723298=667869,3073078=641368} 2024-09-21T08:30:05.600 INFO:tasks.rados.rados.0.smithi137.stdout:1617: writing smithi13779221-6 from 514446 to 1251196 tid 1 2024-09-21T08:30:05.607 INFO:tasks.rados.rados.0.smithi137.stdout:1617: writing smithi13779221-6 from 1723298 to 2391167 tid 2 2024-09-21T08:30:05.613 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:05.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:05.614 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:05.616 INFO:tasks.rados.rados.0.smithi137.stdout:1617: writing smithi13779221-6 from 3073078 to 3714446 tid 3 2024-09-21T08:30:05.616 INFO:tasks.rados.rados.0.smithi137.stdout:1618: write oid 31 current snap is 171 2024-09-21T08:30:05.616 INFO:tasks.rados.rados.0.smithi137.stdout:1617: finishing write tid 1 to smithi13779221-6 2024-09-21T08:30:05.616 INFO:tasks.rados.rados.0.smithi137.stdout:1617: finishing write tid 2 to smithi13779221-6 2024-09-21T08:30:05.616 INFO:tasks.rados.rados.0.smithi137.stdout:1618: seq_num 579 ranges {531253=411999,1381780=411527,2300501=586966,3376974=354279} 2024-09-21T08:30:05.621 INFO:tasks.rados.rados.0.smithi137.stdout:1618: writing smithi13779221-31 from 531253 to 943252 tid 1 2024-09-21T08:30:05.626 INFO:tasks.rados.rados.0.smithi137.stdout:1618: writing smithi13779221-31 from 1381780 to 1793307 tid 2 2024-09-21T08:30:05.633 INFO:tasks.rados.rados.0.smithi137.stdout:1618: writing smithi13779221-31 from 2300501 to 2887467 tid 3 2024-09-21T08:30:05.639 INFO:tasks.rados.rados.0.smithi137.stdout:1618: writing smithi13779221-31 from 3376974 to 3731253 tid 4 2024-09-21T08:30:05.639 INFO:tasks.rados.rados.0.smithi137.stdout:1619: delete oid 42 current snap is 171 2024-09-21T08:30:05.639 INFO:tasks.rados.rados.0.smithi137.stdout:1617: finishing write tid 3 to smithi13779221-6 2024-09-21T08:30:05.639 INFO:tasks.rados.rados.0.smithi137.stdout:1617: finishing write tid 4 to smithi13779221-6 2024-09-21T08:30:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:1617: finishing write tid 5 to smithi13779221-6 2024-09-21T08:30:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:1617: oid 6 updating version 0 to 375 2024-09-21T08:30:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:1617: oid 6 updating version 375 to 378 2024-09-21T08:30:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:1617: oid 6 version 378 is already newer than 377 2024-09-21T08:30:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:1617: oid 6 version 378 is already newer than 376 2024-09-21T08:30:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 378 (ObjNum 578 snap 171 seq_num 578) dirty exists 2024-09-21T08:30:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:1617: left oid 6 (ObjNum 578 snap 171 seq_num 578) 2024-09-21T08:30:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:1618: finishing write tid 1 to smithi13779221-31 2024-09-21T08:30:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:1618: finishing write tid 2 to smithi13779221-31 2024-09-21T08:30:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:1618: finishing write tid 3 to smithi13779221-31 2024-09-21T08:30:05.641 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:05.642 INFO:tasks.rados.rados.0.smithi137.stdout:1618: finishing write tid 4 to smithi13779221-31 2024-09-21T08:30:05.642 INFO:tasks.rados.rados.0.smithi137.stdout:1618: finishing write tid 5 to smithi13779221-31 2024-09-21T08:30:05.642 INFO:tasks.rados.rados.0.smithi137.stdout:1617: done (2 left) 2024-09-21T08:30:05.642 INFO:tasks.rados.rados.0.smithi137.stdout:1619: done (1 left) 2024-09-21T08:30:05.642 INFO:tasks.rados.rados.0.smithi137.stdout:1620: rollback oid 4 current snap is 171 2024-09-21T08:30:05.642 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 4 to 169 2024-09-21T08:30:05.642 INFO:tasks.rados.rados.0.smithi137.stdout:1621: write (excl) oid 23 current snap is 171 2024-09-21T08:30:05.642 INFO:tasks.rados.rados.0.smithi137.stdout:1618: finishing write tid 6 to smithi13779221-31 2024-09-21T08:30:05.642 INFO:tasks.rados.rados.0.smithi137.stdout:1618: oid 31 updating version 0 to 344 2024-09-21T08:30:05.642 INFO:tasks.rados.rados.0.smithi137.stdout:1618: oid 31 updating version 344 to 345 2024-09-21T08:30:05.643 INFO:tasks.rados.rados.0.smithi137.stdout:1618: oid 31 version 345 is already newer than 341 2024-09-21T08:30:05.643 INFO:tasks.rados.rados.0.smithi137.stdout:1618: oid 31 version 345 is already newer than 343 2024-09-21T08:30:05.643 INFO:tasks.rados.rados.0.smithi137.stdout:1618: oid 31 version 345 is already newer than 342 2024-09-21T08:30:05.643 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 345 (ObjNum 579 snap 171 seq_num 579) dirty exists 2024-09-21T08:30:05.643 INFO:tasks.rados.rados.0.smithi137.stdout:1618: left oid 31 (ObjNum 579 snap 171 seq_num 579) 2024-09-21T08:30:05.643 INFO:tasks.rados.rados.0.smithi137.stdout:1621: seq_num 581 ranges {564867=562497,1898744=415196,2755347=725863} 2024-09-21T08:30:05.648 INFO:tasks.rados.rados.0.smithi137.stdout:1621: writing smithi13779221-23 from 564867 to 1127364 tid 1 2024-09-21T08:30:05.653 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:05.656 INFO:tasks.rados.rados.0.smithi137.stdout:1621: writing smithi13779221-23 from 1898744 to 2313940 tid 2 2024-09-21T08:30:05.664 INFO:tasks.rados.rados.0.smithi137.stdout:1621: writing smithi13779221-23 from 2755347 to 3481210 tid 3 2024-09-21T08:30:05.664 INFO:tasks.rados.rados.0.smithi137.stdout:1618: done (2 left) 2024-09-21T08:30:05.664 INFO:tasks.rados.rados.0.smithi137.stdout:1622: write oid 22 current snap is 171 2024-09-21T08:30:05.664 INFO:tasks.rados.rados.0.smithi137.stdout:1620: finishing rollback tid 0 to smithi13779221-4 2024-09-21T08:30:05.664 INFO:tasks.rados.rados.0.smithi137.stdout:1622: seq_num 582 ranges {581674=637746,1840873=558854,3066417=537135} 2024-09-21T08:30:05.670 INFO:tasks.rados.rados.0.smithi137.stdout:1622: writing smithi13779221-22 from 581674 to 1219420 tid 1 2024-09-21T08:30:05.677 INFO:tasks.rados.rados.0.smithi137.stdout:1622: writing smithi13779221-22 from 1840873 to 2399727 tid 2 2024-09-21T08:30:05.684 INFO:tasks.rados.rados.0.smithi137.stdout:1622: writing smithi13779221-22 from 3066417 to 3603552 tid 3 2024-09-21T08:30:05.685 INFO:tasks.rados.rados.0.smithi137.stdout:1623: snap_remove snap 131 2024-09-21T08:30:05.715 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:05.739 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:05.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:05.740 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[41044]: 6.19 scrub starts 2024-09-21T08:30:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[41044]: 6.19 scrub ok 2024-09-21T08:30:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[41044]: pgmap v521: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 453 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 5.7 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 324/1272 objects degraded (25.472%) 2024-09-21T08:30:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1583383079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:30:05.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[41044]: osdmap e411: 8 total, 6 up, 4 in 2024-09-21T08:30:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[29634]: 6.19 scrub starts 2024-09-21T08:30:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[29634]: 6.19 scrub ok 2024-09-21T08:30:05.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[29634]: pgmap v521: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 453 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 5.7 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 324/1272 objects degraded (25.472%) 2024-09-21T08:30:05.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:05.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1583383079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:30:05.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:05 smithi137 ceph-mon[29634]: osdmap e411: 8 total, 6 up, 4 in 2024-09-21T08:30:05.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:05.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:05.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:05.867 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:05.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:05.992 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:05.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:05.994 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:05 smithi165 ceph-mon[32893]: 6.19 scrub starts 2024-09-21T08:30:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:05 smithi165 ceph-mon[32893]: 6.19 scrub ok 2024-09-21T08:30:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:05 smithi165 ceph-mon[32893]: pgmap v521: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 453 MiB data, 1.1 GiB used, 357 GiB / 358 GiB avail; 5.7 MiB/s rd, 2.7 MiB/s wr, 18 op/s; 324/1272 objects degraded (25.472%) 2024-09-21T08:30:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:05 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:06.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:05 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1583383079' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:30:06.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:05 smithi165 ceph-mon[32893]: osdmap e411: 8 total, 6 up, 4 in 2024-09-21T08:30:06.095 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:06.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:06.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:06.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:06.218 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:06.246 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:06.246 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:06.246 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:06.282 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:06.283 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:06.284 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:06.347 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:06.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:06.373 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:06.374 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:06.474 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:06.499 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:06.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:06.500 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:06.601 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:1623: done (3 left) 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:1624: read oid 50 snap -1 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:1624: expect (ObjNum 544 snap 162 seq_num 544) 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:1625: delete oid 46 current snap is 171 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:1620: finishing rollback tid 1 to smithi13779221-4 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:1620: finishing rollback tid 2 to smithi13779221-4 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 378 (ObjNum 546 snap 163 seq_num 546) dirty exists 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:1621: finishing write tid 1 to smithi13779221-23 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:1621: finishing write tid 2 to smithi13779221-23 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:1621: finishing write tid 3 to smithi13779221-23 2024-09-21T08:30:06.609 INFO:tasks.rados.rados.0.smithi137.stdout:1621: finishing write tid 4 to smithi13779221-23 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1621: finishing write tid 5 to smithi13779221-23 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1621: oid 23 updating version 0 to 195 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1621: oid 23 updating version 195 to 198 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1621: oid 23 version 198 is already newer than 197 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1621: oid 23 version 198 is already newer than 196 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 198 (ObjNum 581 snap 171 seq_num 581) dirty exists 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1621: left oid 23 (ObjNum 581 snap 171 seq_num 581) 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1622: finishing write tid 1 to smithi13779221-22 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1622: finishing write tid 2 to smithi13779221-22 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1622: finishing write tid 3 to smithi13779221-22 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1622: finishing write tid 4 to smithi13779221-22 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1622: finishing write tid 5 to smithi13779221-22 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1622: oid 22 updating version 0 to 348 2024-09-21T08:30:06.610 INFO:tasks.rados.rados.0.smithi137.stdout:1622: oid 22 updating version 348 to 349 2024-09-21T08:30:06.611 INFO:tasks.rados.rados.0.smithi137.stdout:1622: oid 22 version 349 is already newer than 346 2024-09-21T08:30:06.611 INFO:tasks.rados.rados.0.smithi137.stdout:1622: oid 22 version 349 is already newer than 347 2024-09-21T08:30:06.611 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 349 (ObjNum 582 snap 171 seq_num 582) dirty exists 2024-09-21T08:30:06.611 INFO:tasks.rados.rados.0.smithi137.stdout:1622: left oid 22 (ObjNum 582 snap 171 seq_num 582) 2024-09-21T08:30:06.616 INFO:tasks.rados.rados.0.smithi137.stdout:1620: done (4 left) 2024-09-21T08:30:06.616 INFO:tasks.rados.rados.0.smithi137.stdout:1621: done (3 left) 2024-09-21T08:30:06.616 INFO:tasks.rados.rados.0.smithi137.stdout:1622: done (2 left) 2024-09-21T08:30:06.616 INFO:tasks.rados.rados.0.smithi137.stdout:1625: done (1 left) 2024-09-21T08:30:06.616 INFO:tasks.rados.rados.0.smithi137.stdout:1626: rollback oid 7 current snap is 171 2024-09-21T08:30:06.616 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 7 to 143 2024-09-21T08:30:06.616 INFO:tasks.rados.rados.0.smithi137.stdout:1627: write oid 31 current snap is 171 2024-09-21T08:30:06.616 INFO:tasks.rados.rados.0.smithi137.stdout:1627: seq_num 584 ranges {615288=788244,1957837=562523,3121263=559679} 2024-09-21T08:30:06.626 INFO:tasks.rados.rados.0.smithi137.stdout:1627: writing smithi13779221-31 from 615288 to 1403532 tid 1 2024-09-21T08:30:06.629 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:06.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:06.630 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:06.631 INFO:tasks.rados.rados.0.smithi137.stdout:1627: writing smithi13779221-31 from 1957837 to 2520360 tid 2 2024-09-21T08:30:06.637 INFO:tasks.rados.rados.0.smithi137.stdout:1627: writing smithi13779221-31 from 3121263 to 3680942 tid 3 2024-09-21T08:30:06.637 INFO:tasks.rados.rados.0.smithi137.stdout:1628: read oid 28 snap -1 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:1628: expect (ObjNum 573 snap 169 seq_num 573) 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:1626: finishing rollback tid 0 to smithi13779221-7 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:1629: read oid 34 snap -1 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:1629: expect (ObjNum 545 snap 162 seq_num 545) 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:1630: read oid 4 snap 144 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:1630: expect (ObjNum 462 snap 124 seq_num 462) 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:1631: delete oid 42 current snap is 171 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:1626: finishing rollback tid 1 to smithi13779221-7 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:1626: finishing rollback tid 2 to smithi13779221-7 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 654 (ObjNum 443 snap 114 seq_num 443) dirty exists 2024-09-21T08:30:06.638 INFO:tasks.rados.rados.0.smithi137.stdout:1624: expect (ObjNum 544 snap 162 seq_num 544) 2024-09-21T08:30:06.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:06.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:06.753 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:06.753 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:06.808 INFO:tasks.rados.rados.0.smithi137.stdout:1627: finishing write tid 1 to smithi13779221-31 2024-09-21T08:30:06.808 INFO:tasks.rados.rados.0.smithi137.stdout:1627: finishing write tid 2 to smithi13779221-31 2024-09-21T08:30:06.808 INFO:tasks.rados.rados.0.smithi137.stdout:1624: done (6 left) 2024-09-21T08:30:06.809 INFO:tasks.rados.rados.0.smithi137.stdout:1626: done (5 left) 2024-09-21T08:30:06.809 INFO:tasks.rados.rados.0.smithi137.stdout:1631: done (4 left) 2024-09-21T08:30:06.809 INFO:tasks.rados.rados.0.smithi137.stdout:1632: delete oid 45 current snap is 171 2024-09-21T08:30:06.809 INFO:tasks.rados.rados.0.smithi137.stdout:1629: expect (ObjNum 545 snap 162 seq_num 545) 2024-09-21T08:30:06.853 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:06.878 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:06.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:06.879 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:06.912 INFO:tasks.rados.rados.0.smithi137.stdout:1628: expect (ObjNum 573 snap 169 seq_num 573) 2024-09-21T08:30:06.928 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:06 smithi137 ceph-mon[41044]: osdmap e412: 8 total, 6 up, 4 in 2024-09-21T08:30:06.928 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:06 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2911003392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:06.928 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:06 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 324/1272 objects degraded (25.472%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:06.928 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:06 smithi137 ceph-mon[29634]: osdmap e412: 8 total, 6 up, 4 in 2024-09-21T08:30:06.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:06 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2911003392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:06.929 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:06 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 324/1272 objects degraded (25.472%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:06.953 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:06.954 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:06.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:07.001 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:07.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:07.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:06 smithi165 ceph-mon[32893]: osdmap e412: 8 total, 6 up, 4 in 2024-09-21T08:30:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:06 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2911003392' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:06 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 324/1272 objects degraded (25.472%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:07.102 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:07.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:07.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:07.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:07.194 INFO:tasks.rados.rados.0.smithi137.stdout:1627: finishing write tid 3 to smithi13779221-31 2024-09-21T08:30:07.194 INFO:tasks.rados.rados.0.smithi137.stdout:1627: finishing write tid 4 to smithi13779221-31 2024-09-21T08:30:07.194 INFO:tasks.rados.rados.0.smithi137.stdout:1628: done (4 left) 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1629: done (3 left) 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1632: done (2 left) 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1633: read oid 23 snap 165 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1633: expect (ObjNum 550 snap 165 seq_num 550) 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1634: copy_from oid 11 from oid 33 current snap is 171 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1627: finishing write tid 5 to smithi13779221-31 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1627: oid 31 updating version 0 to 351 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1627: oid 31 updating version 351 to 352 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1627: oid 31 updating version 352 to 353 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1627: oid 31 version 353 is already newer than 350 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 353 (ObjNum 584 snap 171 seq_num 584) dirty exists 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1627: left oid 31 (ObjNum 584 snap 171 seq_num 584) 2024-09-21T08:30:07.195 INFO:tasks.rados.rados.0.smithi137.stdout:1630: expect (ObjNum 462 snap 124 seq_num 462) 2024-09-21T08:30:07.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:07.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:07.249 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:07.250 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:07.344 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:07.344 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:07.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:07.389 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1] out_osds: [4, 6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:30:07.389 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:30:07.389 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap 2024-09-21T08:30:07.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T08:30:07.390 INFO:tasks.rados.rados.0.smithi137.stdout:1627: done (3 left) 2024-09-21T08:30:07.391 INFO:tasks.rados.rados.0.smithi137.stdout:1630: done (2 left) 2024-09-21T08:30:07.391 INFO:tasks.rados.rados.0.smithi137.stdout:1635: snap_create 2024-09-21T08:30:07.391 INFO:tasks.rados.rados.0.smithi137.stdout:1633: expect (ObjNum 550 snap 165 seq_num 550) 2024-09-21T08:30:07.411 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:07.412 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:07.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:07.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:07.536 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:07.553 INFO:tasks.rados.rados.0.smithi137.stdout:1634: finishing copy_from racing read to smithi13779221-11 2024-09-21T08:30:07.553 INFO:tasks.rados.rados.0.smithi137.stdout:1634: finishing copy_from to smithi13779221-11 2024-09-21T08:30:07.553 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 289 (ObjNum 472 snap 128 seq_num 472) dirty exists 2024-09-21T08:30:07.605 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:30:07.615 INFO:tasks.rados.rados.0.smithi137.stdout:1633: done (2 left) 2024-09-21T08:30:07.615 INFO:tasks.rados.rados.0.smithi137.stdout:1634: done (1 left) 2024-09-21T08:30:07.615 INFO:tasks.rados.rados.0.smithi137.stdout:1635: done (0 left) 2024-09-21T08:30:07.615 INFO:tasks.rados.rados.0.smithi137.stdout:1636: delete oid 24 current snap is 172 2024-09-21T08:30:07.621 INFO:tasks.rados.rados.0.smithi137.stdout:1636: done (0 left) 2024-09-21T08:30:07.621 INFO:tasks.rados.rados.0.smithi137.stdout:1637: snap_remove snap 143 2024-09-21T08:30:07.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:07.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:07.660 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:07.660 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:07.761 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:07.782 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:07.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:07.783 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:07.884 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:07.906 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:07.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:07.907 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:07.968 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:07 smithi137 ceph-mon[29634]: pgmap v524: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 483 MiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 7.4 MiB/s rd, 5.7 MiB/s wr, 26 op/s; 334/1308 objects degraded (25.535%) 2024-09-21T08:30:07.968 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:07 smithi137 ceph-mon[29634]: osdmap e413: 8 total, 6 up, 4 in 2024-09-21T08:30:07.968 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:07 smithi137 ceph-mon[41044]: pgmap v524: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 483 MiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 7.4 MiB/s rd, 5.7 MiB/s wr, 26 op/s; 334/1308 objects degraded (25.535%) 2024-09-21T08:30:07.968 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:07 smithi137 ceph-mon[41044]: osdmap e413: 8 total, 6 up, 4 in 2024-09-21T08:30:08.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T08:30:08.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:07 smithi165 ceph-mon[32893]: pgmap v524: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 483 MiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 7.4 MiB/s rd, 5.7 MiB/s wr, 26 op/s; 334/1308 objects degraded (25.535%) 2024-09-21T08:30:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:07 smithi165 ceph-mon[32893]: osdmap e413: 8 total, 6 up, 4 in 2024-09-21T08:30:08.033 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:08.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:08.034 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:08.047 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:08.085 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:08.135 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:08.156 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:08.157 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:08.157 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:08.258 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:08.279 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:08.280 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:08.280 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:08.381 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:08.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:08.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:08.403 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:08.503 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:08.524 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:08.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:08.525 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:08.621 INFO:tasks.rados.rados.0.smithi137.stdout:1637: done (0 left) 2024-09-21T08:30:08.621 INFO:tasks.rados.rados.0.smithi137.stdout:1638: read oid 44 snap -1 2024-09-21T08:30:08.621 INFO:tasks.rados.rados.0.smithi137.stdout:1638: expect (ObjNum 471 snap 128 seq_num 471) 2024-09-21T08:30:08.621 INFO:tasks.rados.rados.0.smithi137.stdout:1639: read oid 8 snap 146 2024-09-21T08:30:08.621 INFO:tasks.rados.rados.0.smithi137.stdout:1639: expect deleted 2024-09-21T08:30:08.621 INFO:tasks.rados.rados.0.smithi137.stdout:1640: rollback oid 30 current snap is 172 2024-09-21T08:30:08.621 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 30 to 145 2024-09-21T08:30:08.622 INFO:tasks.rados.rados.0.smithi137.stdout:1641: snap_remove snap 168 2024-09-21T08:30:08.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:08.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:08.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:08.652 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:08.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:08.775 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:08.776 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:08.776 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:08.877 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:08.903 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:08.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:08.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:08.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:30:08.969 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:08 smithi137 ceph-mon[29634]: osdmap e414: 8 total, 6 up, 4 in 2024-09-21T08:30:08.970 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:08 smithi137 ceph-mon[41044]: osdmap e414: 8 total, 6 up, 4 in 2024-09-21T08:30:09.004 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:08 smithi165 ceph-mon[32893]: osdmap e414: 8 total, 6 up, 4 in 2024-09-21T08:30:09.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:09.031 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:09.031 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:09.132 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:09.155 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:09.156 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:09.224 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:09.257 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:09.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:09.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:09.283 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:09.349 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:30:09.349 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:30:09.349 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 415, 2024-09-21T08:30:09.349 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "ec8566d6-77f1-11ef-baf6-efdc52797490", 2024-09-21T08:30:09.349 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-21T08:18:21.794737+0000", 2024-09-21T08:30:09.349 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-21T08:30:08.611608+0000", 2024-09-21T08:30:09.349 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-21T08:29:23.558248+0000", 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-21T08:28:28.563923+0000", 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5801984, 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "noscrub", 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-21T08:30:09.350 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 7, 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-21T08:30:09.351 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:21:15.037508+0000", 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 1, 2024-09-21T08:30:09.352 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 1, 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 1, 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 1, 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 1, 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "122", 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:30:09.353 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:30:09.354 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:30:09.355 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.356 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 8, 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 8, 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.25, 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2, 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2, 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:30:09.357 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:28.246186+0000", 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:30:09.358 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:30:09.359 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 68, 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:30:09.360 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:30:09.361 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:30:09.362 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 5.2600002288818359, 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 5.2600002288818359, 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2, 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2, 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:30:09.363 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:29.209879+0000", 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:30:09.364 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:30:09.365 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:30:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:30:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:30:09.366 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "67", 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:30:09.367 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:30:09.368 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:30:09.369 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.9500000476837158, 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.9500000476837158, 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5, 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5, 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.370 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:30.851833+0000", 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:30:09.371 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.372 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "69", 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:30:09.373 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:30:09.374 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:30:09.375 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 5.4200000762939453, 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 5.4200000762939453, 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.059999942779541, 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.059999942779541, 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:32.798630+0000", 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:30:09.376 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:30:09.377 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "71", 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:30:09.378 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:30:09.379 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:30:09.380 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.9500000476837158, 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.9500000476837158, 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.5, 2024-09-21T08:30:09.381 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.5, 2024-09-21T08:30:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:30:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:30:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:30:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-21T08:30:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-21T08:30:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:34.858938+0000", 2024-09-21T08:30:09.382 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:30:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:30:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:30:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:30:09.383 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:30:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:30:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:30:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:30:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:30:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:30:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:30:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:30:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:30:09.384 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:30:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:30:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:30:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:30:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:30:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:30:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:30:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:30:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:30:09.385 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "73", 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:30:09.386 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:30:09.387 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:30:09.388 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 4.4499998092651367, 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 4.4499998092651367, 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.690000057220459, 2024-09-21T08:30:09.389 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.690000057220459, 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_0", 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:24:33.441067+0000", 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:30:09.390 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 36, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 36, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 36, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 36, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 36, 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:30:09.391 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "415", 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "259", 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 173, 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 415, 2024-09-21T08:30:09.392 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:30:09.393 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:30:09.394 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:30:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:30:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:30:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:30:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-21T08:30:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:30:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:30:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.7400000095367432, 2024-09-21T08:30:09.395 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.7400000095367432, 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.37999999523162842, 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.4199999570846558, 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.4199999570846558, 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977", 2024-09-21T08:30:09.396 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 0, 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 317, 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 370, 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-21T08:30:09.397 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.398 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:30:09.399 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/3938795530", 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/3938795530", 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/3938795530", 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/3938795530", 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "exists" 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d", 2024-09-21T08:30:09.400 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 339, 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-21T08:30:09.401 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.402 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-21T08:30:09.403 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/2743268361", 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/2743268361", 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/2743268361", 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/2743268361", 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.404 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d", 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0.3052520751953125, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 339, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:30:09.405 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:30:09.406 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.407 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1500397557", 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1500397557", 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1500397557", 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1500397557", 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:30:09.408 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "73f21606-38be-4578-a86b-bb417217c0e0", 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 25, 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 243, 2024-09-21T08:30:09.409 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.410 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.411 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/617155021", 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/617155021", 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/617155021", 2024-09-21T08:30:09.412 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/617155021", 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae", 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:30:09.413 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 31, 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 80, 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6800", 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.414 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6801", 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6803", 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.415 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6802", 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6800/70360958", 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6801/70360958", 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6803/70360958", 2024-09-21T08:30:09.416 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6802/70360958", 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd", 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:30:09.417 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 41, 2024-09-21T08:30:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 339, 2024-09-21T08:30:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:30:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:30:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:30:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.418 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:09.419 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6804", 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6805", 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6807", 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.421 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6806", 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6804/1835779380", 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6805/1835779380", 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6807/1835779380", 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6806/1835779380", 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b", 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 49, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 94, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6808", 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6809", 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.424 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:30:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6811", 2024-09-21T08:30:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:30:09.425 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6810", 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6808/3820354208", 2024-09-21T08:30:09.426 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6809/3820354208", 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6811/3820354208", 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6810/3820354208", 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f", 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 0, 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0.9804229736328125, 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0.2433013916015625, 2024-09-21T08:30:09.427 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 59, 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 323, 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 339, 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6812", 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6813", 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.429 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6815", 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6814", 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6812/1933524273", 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6813/1933524273", 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6815/1933524273", 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6814/1933524273", 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "exists" 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:30:09.431 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:28:33.169954+0000", 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 369 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:29:11.976320+0000", 2024-09-21T08:30:09.432 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:29:22.164751+0000", 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-21T08:30:09.433 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:29:21.595661+0000", 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:30:09.434 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:29:05.572639+0000", 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:28:51.964948+0000", 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.435 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:29:11.833637+0000", 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:30:09.436 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:28:07.349774+0000", 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 338 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [ 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.10", 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: 3, 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: 7 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:30:09.437 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.c", 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 7, 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:30:09.438 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [], 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1478868457": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1738156348": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2643910317": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/705906958": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3936900881": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/223603733": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1718301516": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2580000250": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3166961425": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/700535732": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1900990534": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1825289826": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:30:09.439 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3010106803": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3442439903": "2024-09-22T08:23:59.682278+0000" 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [ 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.440 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 105, 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 109, 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 4 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 114, 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 11 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.441 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 127, 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 5 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 133, 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 3 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 137, 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 2 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 140, 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 6 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 149, 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 3 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 157, 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 6 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 166, 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 168, 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:30:09.443 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [ 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 145, 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:30:09.444 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-21T08:30:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-21T08:30:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-21T08:30:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-21T08:30:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-21T08:30:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-21T08:30:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-21T08:30:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-21T08:30:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-21T08:30:09.445 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:09.445 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:30:09.446 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:09.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:09.447 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:09.547 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:09.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:09.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:09.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:09.621 INFO:tasks.rados.rados.0.smithi137.stdout:1641: done (3 left) 2024-09-21T08:30:09.622 INFO:tasks.rados.rados.0.smithi137.stdout:1642: write (excl) oid 13 current snap is 172 2024-09-21T08:30:09.622 INFO:tasks.rados.rados.0.smithi137.stdout:1642: seq_num 588 ranges {682516=405593,1791765=569861,2830955=721120} 2024-09-21T08:30:09.622 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:30:09.628 INFO:tasks.rados.rados.0.smithi137.stdout:1642: writing smithi13779221-13 from 682516 to 1088109 tid 1 2024-09-21T08:30:09.635 INFO:tasks.rados.rados.0.smithi137.stdout:1642: writing smithi13779221-13 from 1791765 to 2361626 tid 2 2024-09-21T08:30:09.642 INFO:tasks.rados.rados.0.smithi137.stdout:1642: writing smithi13779221-13 from 2830955 to 3552075 tid 3 2024-09-21T08:30:09.642 INFO:tasks.rados.rados.0.smithi137.stdout:1643: delete oid 3 current snap is 172 2024-09-21T08:30:09.642 INFO:tasks.rados.rados.0.smithi137.stdout:1638: expect (ObjNum 471 snap 128 seq_num 471) 2024-09-21T08:30:09.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:09.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:09.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:09.698 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:09.774 INFO:tasks.rados.rados.0.smithi137.stdout:1640: finishing rollback tid 0 to smithi13779221-30 2024-09-21T08:30:09.774 INFO:tasks.rados.rados.0.smithi137.stdout:1640: finishing rollback tid 1 to smithi13779221-30 2024-09-21T08:30:09.774 INFO:tasks.rados.rados.0.smithi137.stdout:1640: finishing rollback tid 2 to smithi13779221-30 2024-09-21T08:30:09.774 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 658 (ObjNum 493 snap 136 seq_num 493) dirty exists 2024-09-21T08:30:09.774 INFO:tasks.rados.rados.0.smithi137.stdout:1642: finishing write tid 1 to smithi13779221-13 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1642: finishing write tid 2 to smithi13779221-13 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1638: done (4 left) 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1639: done (3 left) 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1640: done (2 left) 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1643: done (1 left) 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1644: rollback oid 3 current snap is 172 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 3 to 134 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1642: finishing write tid 3 to smithi13779221-13 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1642: finishing write tid 4 to smithi13779221-13 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1642: finishing write tid 5 to smithi13779221-13 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1642: oid 13 updating version 0 to 664 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1642: oid 13 updating version 664 to 666 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1642: oid 13 updating version 666 to 667 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:1642: oid 13 version 667 is already newer than 665 2024-09-21T08:30:09.775 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 667 (ObjNum 588 snap 172 seq_num 588) dirty exists 2024-09-21T08:30:09.776 INFO:tasks.rados.rados.0.smithi137.stdout:1642: left oid 13 (ObjNum 588 snap 172 seq_num 588) 2024-09-21T08:30:09.776 INFO:tasks.rados.rados.0.smithi137.stdout:1642: done (1 left) 2024-09-21T08:30:09.776 INFO:tasks.rados.rados.0.smithi137.stdout:1645: write oid 18 current snap is 172 2024-09-21T08:30:09.776 INFO:tasks.rados.rados.0.smithi137.stdout:1645: seq_num 590 ranges {716130=556091,1792376=573530,2769448=460017,3821956=94174} 2024-09-21T08:30:09.780 INFO:tasks.rados.rados.0.smithi137.stdout:1645: writing smithi13779221-18 from 716130 to 1272221 tid 1 2024-09-21T08:30:09.786 INFO:tasks.rados.rados.0.smithi137.stdout:1645: writing smithi13779221-18 from 1792376 to 2365906 tid 2 2024-09-21T08:30:09.791 INFO:tasks.rados.rados.0.smithi137.stdout:1645: writing smithi13779221-18 from 2769448 to 3229465 tid 3 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:1645: writing smithi13779221-18 from 3821956 to 3916130 tid 4 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:1646: rollback oid 21 current snap is 172 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 21 to 146 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:1644: finishing rollback tid 1 to smithi13779221-3 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:1644: finishing rollback tid 2 to smithi13779221-3 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 498 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:1645: finishing write tid 1 to smithi13779221-18 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:1644: done (2 left) 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:1647: rollback oid 26 current snap is 172 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 26 to 152 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:1645: finishing write tid 2 to smithi13779221-18 2024-09-21T08:30:09.796 INFO:tasks.rados.rados.0.smithi137.stdout:1645: finishing write tid 3 to smithi13779221-18 2024-09-21T08:30:09.797 INFO:tasks.rados.rados.0.smithi137.stdout:1648: delete oid 46 current snap is 172 2024-09-21T08:30:09.797 INFO:tasks.rados.rados.0.smithi137.stdout:1648: done (3 left) 2024-09-21T08:30:09.797 INFO:tasks.rados.rados.0.smithi137.stdout:1649: snap_remove snap 111 2024-09-21T08:30:09.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:09.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:09.823 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:09.823 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:09.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:09 smithi137 ceph-mon[29634]: pgmap v527: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 483 MiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 7.4 MiB/s rd, 5.7 MiB/s wr, 26 op/s; 334/1308 objects degraded (25.535%) 2024-09-21T08:30:09.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:09 smithi137 ceph-mon[29634]: osdmap e415: 8 total, 6 up, 4 in 2024-09-21T08:30:09.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:09 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2396704741' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:09.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:09 smithi137 ceph-mon[29634]: from='client.45665 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:09.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:09 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3047444269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:30:09.882 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:09 smithi137 ceph-mon[41044]: pgmap v527: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 483 MiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 7.4 MiB/s rd, 5.7 MiB/s wr, 26 op/s; 334/1308 objects degraded (25.535%) 2024-09-21T08:30:09.883 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:09 smithi137 ceph-mon[41044]: osdmap e415: 8 total, 6 up, 4 in 2024-09-21T08:30:09.883 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:09 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2396704741' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:09.883 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:09 smithi137 ceph-mon[41044]: from='client.45665 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:09.883 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:09 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3047444269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:30:09.925 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:09.948 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:09.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:09.949 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:09 smithi165 ceph-mon[32893]: pgmap v527: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 483 MiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 7.4 MiB/s rd, 5.7 MiB/s wr, 26 op/s; 334/1308 objects degraded (25.535%) 2024-09-21T08:30:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:09 smithi165 ceph-mon[32893]: osdmap e415: 8 total, 6 up, 4 in 2024-09-21T08:30:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:09 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2396704741' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:09 smithi165 ceph-mon[32893]: from='client.45665 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:09 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3047444269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:30:10.033 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:10.033 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:10.033 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:10.034 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:10.034 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:10.034 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:10.034 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:10.034 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:10.034 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:10.048 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:10.049 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:10.050 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:10.065 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:10.066 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:10.066 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:10.073 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:10.074 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:10.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:10.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:10.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:10.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:10.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:10.321 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:10.422 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:10.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:10.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:10.444 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:10.545 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:10.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:10.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:10.571 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:10.613 INFO:tasks.thrashosds.thrasher:j is {'epoch': 415, 'fsid': 'ec8566d6-77f1-11ef-baf6-efdc52797490', 'created': '2024-09-21T08:18:21.794737+0000', 'modified': '2024-09-21T08:30:08.611608+0000', 'last_up_change': '2024-09-21T08:29:23.558248+0000', 'last_in_change': '2024-09-21T08:28:28.563923+0000', 'flags': 'noscrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5801984, 'flags_set': ['noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T08:21:15.037508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '122', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T08:23:28.246186+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5.260000228881836, 'score_stable': 5.260000228881836, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T08:23:29.209879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.950000047683716, 'score_stable': 3.950000047683716, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T08:23:30.851833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5.420000076293945, 'score_stable': 5.420000076293945, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 2.059999942779541, 'raw_score_stable': 2.059999942779541, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T08:23:32.798630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.950000047683716, 'score_stable': 3.950000047683716, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T08:23:34.858938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 4.449999809265137, 'score_stable': 4.449999809265137, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 1.690000057220459, 'raw_score_stable': 1.690000057220459, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-21T08:24:33.441067+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 36, 'pg_placement_num': 36, 'pg_placement_num_target': 36, 'pg_num_target': 36, 'pg_num_pending': 36, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '415', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '259', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 173, 'snap_epoch': 415, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.740000009536743, 'score_stable': 3.740000009536743, 'optimal_score': 0.3799999952316284, 'raw_score_acting': 1.4199999570846558, 'raw_score_stable': 1.4199999570846558, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}], 'osds': [{'osd': 0, 'uuid': '1b007d1d-6c6b-4a68-9e68-2c5fd7d26977', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 317, 'down_at': 370, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 3938795530}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 3938795530}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 3938795530}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 3938795530}]}, 'public_addr': '172.21.15.137:6801/3938795530', 'cluster_addr': '172.21.15.137:6802/3938795530', 'heartbeat_back_addr': '172.21.15.137:6804/3938795530', 'heartbeat_front_addr': '172.21.15.137:6803/3938795530', 'state': ['exists']}, {'osd': 1, 'uuid': '8651068e-694f-4bd4-9917-5c55670aca5d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 339, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 2743268361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 2743268361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 2743268361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 2743268361}]}, 'public_addr': '172.21.15.137:6805/2743268361', 'cluster_addr': '172.21.15.137:6806/2743268361', 'heartbeat_back_addr': '172.21.15.137:6808/2743268361', 'heartbeat_front_addr': '172.21.15.137:6807/2743268361', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2a22f0f7-c610-4995-98a7-d4785efb1e5d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3052520751953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 339, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1500397557}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1500397557}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1500397557}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1500397557}]}, 'public_addr': '172.21.15.137:6809/1500397557', 'cluster_addr': '172.21.15.137:6810/1500397557', 'heartbeat_back_addr': '172.21.15.137:6812/1500397557', 'heartbeat_front_addr': '172.21.15.137:6811/1500397557', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '73f21606-38be-4578-a86b-bb417217c0e0', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 617155021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 617155021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 617155021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 617155021}]}, 'public_addr': '172.21.15.137:6813/617155021', 'cluster_addr': '172.21.15.137:6814/617155021', 'heartbeat_back_addr': '172.21.15.137:6816/617155021', 'heartbeat_front_addr': '172.21.15.137:6815/617155021', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e5f6762b-65f5-4dc7-8e54-9087149ae6ae', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 80, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 70360958}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 70360958}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 70360958}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 70360958}]}, 'public_addr': '172.21.15.165:6800/70360958', 'cluster_addr': '172.21.15.165:6801/70360958', 'heartbeat_back_addr': '172.21.15.165:6803/70360958', 'heartbeat_front_addr': '172.21.15.165:6802/70360958', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f7f8c456-b77f-4435-a860-202e64dda8dd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 339, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 1835779380}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 1835779380}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 1835779380}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 1835779380}]}, 'public_addr': '172.21.15.165:6804/1835779380', 'cluster_addr': '172.21.15.165:6805/1835779380', 'heartbeat_back_addr': '172.21.15.165:6807/1835779380', 'heartbeat_front_addr': '172.21.15.165:6806/1835779380', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b8b3df16-73fa-41a5-a93f-117c599d488b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 3820354208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 3820354208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 3820354208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 3820354208}]}, 'public_addr': '172.21.15.165:6808/3820354208', 'cluster_addr': '172.21.15.165:6809/3820354208', 'heartbeat_back_addr': '172.21.15.165:6811/3820354208', 'heartbeat_front_addr': '172.21.15.165:6810/3820354208', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ddf234e4-1494-40bd-a31b-333b01717c3f', 'up': 0, 'in': 1, 'weight': 0.9804229736328125, 'primary_affinity': 0.2433013916015625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 323, 'down_at': 339, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 1933524273}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 1933524273}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 1933524273}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 1933524273}]}, 'public_addr': '172.21.15.165:6812/1933524273', 'cluster_addr': '172.21.15.165:6813/1933524273', 'heartbeat_back_addr': '172.21.15.165:6815/1933524273', 'heartbeat_front_addr': '172.21.15.165:6814/1933524273', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:28:33.169954+0000', 'dead_epoch': 369}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:29:11.976320+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:29:22.164751+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:29:21.595661+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:29:05.572639+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:28:51.964948+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:29:11.833637+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:28:07.349774+0000', 'dead_epoch': 338}], 'pg_upmap': [{'pgid': '7.10', 'osds': [5, 3, 7]}], 'pg_upmap_items': [{'pgid': '7.c', 'mappings': [{'from': 7, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [], 'primary_temp': [], 'blocklist': {'172.21.15.137:0/1478868457': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/1738156348': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/2643910317': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/705906958': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/3936900881': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/223603733': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/1718301516': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:6800/2580000250': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/3166961425': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/700535732': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:6800/1900990534': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:6800/1825289826': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/3010106803': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/3442439903': '2024-09-22T08:23:59.682278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 105, 'length': 1}, {'begin': 109, 'length': 4}, {'begin': 114, 'length': 11}, {'begin': 127, 'length': 5}, {'begin': 133, 'length': 3}, {'begin': 137, 'length': 2}, {'begin': 140, 'length': 6}, {'begin': 149, 'length': 3}, {'begin': 157, 'length': 6}, {'begin': 166, 'length': 1}, {'begin': 168, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 145, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T08:30:10.614 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:30:10.631 INFO:tasks.rados.rados.0.smithi137.stdout:1649: done (3 left) 2024-09-21T08:30:10.631 INFO:tasks.rados.rados.0.smithi137.stdout:1650: rollback oid 37 current snap is 172 2024-09-21T08:30:10.631 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 37 to 124 2024-09-21T08:30:10.632 INFO:tasks.rados.rados.0.smithi137.stdout:1647: finishing rollback tid 0 to smithi13779221-26 2024-09-21T08:30:10.632 INFO:tasks.rados.rados.0.smithi137.stdout:1645: finishing write tid 4 to smithi13779221-18 2024-09-21T08:30:10.632 INFO:tasks.rados.rados.0.smithi137.stdout:1647: finishing rollback tid 1 to smithi13779221-26 2024-09-21T08:30:10.632 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 389 (ObjNum 129 snap 0 seq_num 3761244240) dirty dne 2024-09-21T08:30:10.632 INFO:tasks.rados.rados.0.smithi137.stdout:1645: finishing write tid 5 to smithi13779221-18 2024-09-21T08:30:10.632 INFO:tasks.rados.rados.0.smithi137.stdout:1645: finishing write tid 6 to smithi13779221-18 2024-09-21T08:30:10.632 INFO:tasks.rados.rados.0.smithi137.stdout:1645: oid 18 updating version 0 to 226 2024-09-21T08:30:10.632 INFO:tasks.rados.rados.0.smithi137.stdout:1645: oid 18 updating version 226 to 229 2024-09-21T08:30:10.632 INFO:tasks.rados.rados.0.smithi137.stdout:1645: oid 18 version 229 is already newer than 228 2024-09-21T08:30:10.632 INFO:tasks.rados.rados.0.smithi137.stdout:1645: oid 18 version 229 is already newer than 225 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1645: oid 18 version 229 is already newer than 227 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 229 (ObjNum 590 snap 172 seq_num 590) dirty exists 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1645: left oid 18 (ObjNum 590 snap 172 seq_num 590) 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1646: finishing rollback tid 0 to smithi13779221-21 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1646: finishing rollback tid 1 to smithi13779221-21 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1646: finishing rollback tid 2 to smithi13779221-21 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 506 (ObjNum 502 snap 141 seq_num 502) dirty exists 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1645: done (3 left) 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1646: done (2 left) 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1647: done (1 left) 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1651: read oid 41 snap -1 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1651: expect deleted 2024-09-21T08:30:10.633 INFO:tasks.rados.rados.0.smithi137.stdout:1652: write (excl) oid 13 current snap is 172 2024-09-21T08:30:10.634 INFO:tasks.rados.rados.0.smithi137.stdout:1652: seq_num 592 ranges {749744=706589,1909340=577199,3224294=482561} 2024-09-21T08:30:10.639 INFO:tasks.rados.rados.0.smithi137.stdout:1652: writing smithi13779221-13 from 749744 to 1456333 tid 1 2024-09-21T08:30:10.646 INFO:tasks.rados.rados.0.smithi137.stdout:1652: writing smithi13779221-13 from 1909340 to 2486539 tid 2 2024-09-21T08:30:10.653 INFO:tasks.rados.rados.0.smithi137.stdout:1652: writing smithi13779221-13 from 3224294 to 3706855 tid 3 2024-09-21T08:30:10.653 INFO:tasks.rados.rados.0.smithi137.stdout:1653: write (excl) oid 6 current snap is 172 2024-09-21T08:30:10.653 INFO:tasks.rados.rados.0.smithi137.stdout:1653: seq_num 593 ranges {766551=498191,1684175=720857,2851717=693833} 2024-09-21T08:30:10.660 INFO:tasks.rados.rados.0.smithi137.stdout:1653: writing smithi13779221-6 from 766551 to 1264742 tid 1 2024-09-21T08:30:10.666 INFO:tasks.rados.rados.0.smithi137.stdout:1653: writing smithi13779221-6 from 1684175 to 2405032 tid 2 2024-09-21T08:30:10.672 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:10.674 INFO:tasks.rados.rados.0.smithi137.stdout:1653: writing smithi13779221-6 from 2851717 to 3545550 tid 3 2024-09-21T08:30:10.674 INFO:tasks.rados.rados.0.smithi137.stdout:1654: write (excl) oid 27 current snap is 172 2024-09-21T08:30:10.674 INFO:tasks.rados.rados.0.smithi137.stdout:1654: seq_num 594 ranges {783358=573440,2026304=580868,3279140=621458} 2024-09-21T08:30:10.680 INFO:tasks.rados.rados.0.smithi137.stdout:1654: writing smithi13779221-27 from 783358 to 1356798 tid 1 2024-09-21T08:30:10.687 INFO:tasks.rados.rados.0.smithi137.stdout:1654: writing smithi13779221-27 from 2026304 to 2607172 tid 2 2024-09-21T08:30:10.696 INFO:tasks.rados.rados.0.smithi137.stdout:1654: writing smithi13779221-27 from 3279140 to 3900598 tid 3 2024-09-21T08:30:10.696 INFO:tasks.rados.rados.0.smithi137.stdout:1655: rollback oid 26 current snap is 172 2024-09-21T08:30:10.696 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 26 to 154 2024-09-21T08:30:10.696 INFO:tasks.rados.rados.0.smithi137.stdout:1650: finishing rollback tid 0 to smithi13779221-37 2024-09-21T08:30:10.696 INFO:tasks.rados.rados.0.smithi137.stdout:1650: finishing rollback tid 1 to smithi13779221-37 2024-09-21T08:30:10.696 INFO:tasks.rados.rados.0.smithi137.stdout:1650: finishing rollback tid 2 to smithi13779221-37 2024-09-21T08:30:10.696 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 664 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:30:10.696 INFO:tasks.rados.rados.0.smithi137.stdout:1650: done (5 left) 2024-09-21T08:30:10.696 INFO:tasks.rados.rados.0.smithi137.stdout:1651: done (4 left) 2024-09-21T08:30:10.697 INFO:tasks.rados.rados.0.smithi137.stdout:1656: read oid 1 snap -1 2024-09-21T08:30:10.697 INFO:tasks.rados.rados.0.smithi137.stdout:1656: expect (ObjNum 564 snap 168 seq_num 564) 2024-09-21T08:30:10.697 INFO:tasks.rados.rados.0.smithi137.stdout:1657: read oid 23 snap -1 2024-09-21T08:30:10.697 INFO:tasks.rados.rados.0.smithi137.stdout:1657: expect (ObjNum 581 snap 171 seq_num 581) 2024-09-21T08:30:10.697 INFO:tasks.rados.rados.0.smithi137.stdout:1658: snap_remove snap 134 2024-09-21T08:30:10.697 INFO:tasks.rados.rados.0.smithi137.stdout:1652: finishing write tid 1 to smithi13779221-13 2024-09-21T08:30:10.697 INFO:tasks.rados.rados.0.smithi137.stdout:1652: finishing write tid 2 to smithi13779221-13 2024-09-21T08:30:10.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:10.698 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:10.698 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:10.799 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:10.822 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:10.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:10.823 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:10.924 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:10.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:10.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:10.946 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:10 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:10 smithi165 ceph-mon[32893]: from='client.45665 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:10 smithi165 ceph-mon[32893]: osdmap e416: 8 total, 6 up, 4 in 2024-09-21T08:30:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:10 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3163489335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:11.014 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:10 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:11.014 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:10 smithi137 ceph-mon[29634]: from='client.45665 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:11.014 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:10 smithi137 ceph-mon[29634]: osdmap e416: 8 total, 6 up, 4 in 2024-09-21T08:30:11.014 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:10 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3163489335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:11.014 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:30:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:30:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:30:11.014 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:10 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:11.014 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:10 smithi137 ceph-mon[41044]: from='client.45665 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:11.015 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:10 smithi137 ceph-mon[41044]: osdmap e416: 8 total, 6 up, 4 in 2024-09-21T08:30:11.015 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:10 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3163489335' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:11.038 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:11.038 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:11.047 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:11.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:11.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:11.072 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:11.173 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:11.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:11.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:11.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:11.298 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:11.323 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:11.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:11.324 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:11.424 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:11.446 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:11.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:11.447 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:11.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:11.569 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:11.569 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:11.570 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:11.578 INFO:tasks.rados.rados.0.smithi137.stdout:1658: done (6 left) 2024-09-21T08:30:11.579 INFO:tasks.rados.rados.0.smithi137.stdout:1659: read oid 15 snap -1 2024-09-21T08:30:11.579 INFO:tasks.rados.rados.0.smithi137.stdout:1659: expect deleted 2024-09-21T08:30:11.579 INFO:tasks.rados.rados.0.smithi137.stdout:1660: snap_remove snap 123 2024-09-21T08:30:11.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:11.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:11.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:11.697 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:11.735 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:11.741 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:11.798 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:11.819 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:11.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:11.820 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:11.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[29634]: pgmap v530: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 483 MiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 334/1308 objects degraded (25.535%) 2024-09-21T08:30:11.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:30:11.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:30:11.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[29634]: osdmap e417: 8 total, 6 up, 4 in 2024-09-21T08:30:11.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 334/1308 objects degraded (25.535%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:11.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[29634]: osdmap e418: 8 total, 6 up, 4 in 2024-09-21T08:30:11.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[41044]: pgmap v530: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 483 MiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 334/1308 objects degraded (25.535%) 2024-09-21T08:30:11.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:30:11.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:30:11.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[41044]: osdmap e417: 8 total, 6 up, 4 in 2024-09-21T08:30:11.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 334/1308 objects degraded (25.535%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:11.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:11 smithi137 ceph-mon[41044]: osdmap e418: 8 total, 6 up, 4 in 2024-09-21T08:30:11.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:11 smithi165 ceph-mon[32893]: pgmap v530: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 483 MiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 334/1308 objects degraded (25.535%) 2024-09-21T08:30:11.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:11 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:30:11.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:11 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:30:11.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:11 smithi165 ceph-mon[32893]: osdmap e417: 8 total, 6 up, 4 in 2024-09-21T08:30:11.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:11 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 334/1308 objects degraded (25.535%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:11.902 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:11 smithi165 ceph-mon[32893]: osdmap e418: 8 total, 6 up, 4 in 2024-09-21T08:30:11.921 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:11.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:11.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:11.942 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:12.043 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:12.068 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:12.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:12.069 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:12.170 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:12.196 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:12.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:12.197 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:12.247 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:30:12.247 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:30:12.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:30:12.247 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:30:12.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.247 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:260 7.8 7:1fd29771:::smithi13779221-22:head [copy-from ver 15] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 325.69030749699999, 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025065418999999999, 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 260 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.248 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.249 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554325+0000", 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.667e-06 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554332+0000", 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1600000000000001e-06 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554346+0000", 2024-09-21T08:30:12.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4011e-05 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554355+0000", 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9199999999999993e-06 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554524+0000", 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000169469 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554534+0000", 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5489999999999995e-06 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557746+0000", 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003212164 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.252 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557822+0000", 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6108999999999999e-05 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568552+0000", 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01072972 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574773+0000", 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062208760000000002 2024-09-21T08:30:12.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576999+0000", 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022266059999999999 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579275+0000", 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002276054 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579335+0000", 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9775000000000003e-05 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579388+0000", 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2339000000000002e-05 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.255 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:261 7.8 7:1fd29771:::smithi13779221-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 325.69016293599998, 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020259856, 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 261 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.256 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554468+0000", 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0219999999999999e-06 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554475+0000", 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.81e-06 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554484+0000", 2024-09-21T08:30:12.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0550000000000005e-06 2024-09-21T08:30:12.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554491+0000", 2024-09-21T08:30:12.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2080000000000003e-06 2024-09-21T08:30:12.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554607+0000", 2024-09-21T08:30:12.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011585400000000001 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554612+0000", 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0130000000000001e-06 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557969+0000", 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033576320000000002 2024-09-21T08:30:12.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558007+0000", 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.735e-05 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574538+0000", 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016531694 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.261 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574630+0000", 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1902000000000002e-05 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574727+0000", 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6316000000000004e-05 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.262 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:332 7.3 7:c851e2c3:::smithi13779221-25:head [read 0~2403368,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 324.55848611499999, 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90930888700000001, 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 332 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.263 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686147+0000", 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5370000000000002e-06 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686160+0000", 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2542e-05 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:30:12.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7927999999999998e-05 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686187+0000", 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3610000000000007e-06 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686261+0000", 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4197999999999997e-05 2024-09-21T08:30:12.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686271+0000", 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5259999999999999e-06 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.590700+0000", 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90442903500000005 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.590838+0000", 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137863 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595452+0000", 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046148969999999997 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.268 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:333 7.3 7:c851e2c3:::smithi13779221-25:head [read 0~2403368,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 324.55842258600001, 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91060290700000002, 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 333 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.269 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686208+0000", 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4900000000000005e-07 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686212+0000", 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9940000000000002e-06 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686221+0000", 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8789999999999992e-06 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686225+0000", 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5059999999999997e-06 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686303+0000", 2024-09-21T08:30:12.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7855000000000006e-05 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686310+0000", 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0829999999999998e-06 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595485+0000", 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90917505300000001 2024-09-21T08:30:12.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595590+0000", 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105059 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.596810+0000", 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012197289999999999 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.274 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:334 7.3 7:c851e2c3:::smithi13779221-25:head [sparse-read 0~1282710] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 324.55838557999999, 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91097041300000003, 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.275 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 334 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.276 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686245+0000", 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2600000000000002e-07 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686247+0000", 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7429999999999998e-06 2024-09-21T08:30:12.277 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686255+0000", 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8369999999999997e-06 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686259+0000", 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0509999999999998e-06 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686326+0000", 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6845000000000001e-05 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686331+0000", 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.13e-06 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.596911+0000", 2024-09-21T08:30:12.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.910579845 2024-09-21T08:30:12.280 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:30:11 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:30:12.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.597015+0000", 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103538 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.597215+0000", 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000199798 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:569 7.e 7:78d05d33:::smithi13779221-41:head [copy-from ver 7] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 301.69793685500002, 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033085196999999997, 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.283 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 569 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:30:12.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546695+0000", 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3920000000000001e-06 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546704+0000", 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9330000000000007e-06 2024-09-21T08:30:12.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546720+0000", 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5651999999999999e-05 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546731+0000", 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1147e-05 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546794+0000", 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3214999999999997e-05 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546801+0000", 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3350000000000001e-06 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556379+0000", 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095778679999999998 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556505+0000", 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012529699999999999 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564765+0000", 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082601020000000001 2024-09-21T08:30:12.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576097+0000", 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011331938999999999 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576715+0000", 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00061843800000000002 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579665+0000", 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029497 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579729+0000", 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4337000000000004e-05 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579778+0000", 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8841999999999998e-05 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:570 7.e 7:78d05d33:::smithi13779221-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 301.69779576399998, 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024747969000000002, 2024-09-21T08:30:12.291 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 570 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:30:12.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546835+0000", 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4e-07 2024-09-21T08:30:12.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546838+0000", 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4829999999999999e-06 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546845+0000", 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5810000000000002e-06 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546857+0000", 2024-09-21T08:30:12.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2034e-05 2024-09-21T08:30:12.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546900+0000", 2024-09-21T08:30:12.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2945000000000003e-05 2024-09-21T08:30:12.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546905+0000", 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2700000000000004e-06 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556676+0000", 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097709930000000004 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556717+0000", 2024-09-21T08:30:12.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.104e-05 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571353+0000", 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014636287 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571465+0000", 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111943 2024-09-21T08:30:12.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571582+0000", 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011655299999999999 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1571 7.18 7:1fd29771:::smithi13779221-22:head [copy-from ver 143] snapc 32=[32,31] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:16.499943+0000", 2024-09-21T08:30:12.298 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 235.74468652100001, 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022751851999999999, 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1571 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499943+0000", 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499943+0000", 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499945+0000", 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2230000000000001e-06 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499952+0000", 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.764e-06 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499963+0000", 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1124000000000001e-05 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499969+0000", 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.276e-06 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.500029+0000", 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0927000000000001e-05 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.500130+0000", 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010014900000000001 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.509954+0000", 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098240830000000008 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.518293+0000", 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0083390770000000003 2024-09-21T08:30:12.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522396+0000", 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041036809999999996 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522612+0000", 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021587900000000001 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522652+0000", 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9635000000000003e-05 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522695+0000", 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3034000000000002e-05 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1919 7.8 7:144c2ec5:::smithi13779221-45:head [delete] snapc 43=[40,3d,39] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.572348+0000", 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 203.67228155999999, 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0652029459999999, 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1919 2024-09-21T08:30:12.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572348+0000", 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572348+0000", 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572350+0000", 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.928e-06 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572356+0000", 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9449999999999998e-06 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572366+0000", 2024-09-21T08:30:12.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0237e-05 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572375+0000", 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7940000000000008e-06 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572776+0000", 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040072699999999997 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:12.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572789+0000", 2024-09-21T08:30:12.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2974e-05 2024-09-21T08:30:12.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.526784+0000", 2024-09-21T08:30:12.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95399481799999997 2024-09-21T08:30:12.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.526798+0000", 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4428e-05 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.636113+0000", 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.109315396 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.636265+0000", 2024-09-21T08:30:12.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015202999999999999 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.636448+0000", 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000182957 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637097+0000", 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064855999999999996 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637266+0000", 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016917399999999999 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637374+0000", 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107899 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637451+0000", 2024-09-21T08:30:12.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7486999999999999e-05 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637551+0000", 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9592000000000005e-05 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2116 7.b e228/157 7:df2b42d9:::smithi13779221-36:head v 228'352, mlcod=225'348)", 2024-09-21T08:30:12.314 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.532769+0000", 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 190.711860892, 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020877380000000001, 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.532769+0000", 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.532769+0000", 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.532772+0000", 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2729999999999998e-06 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542209+0000", 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094370849999999996 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542235+0000", 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6072999999999999e-05 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542247+0000", 2024-09-21T08:30:12.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2233e-05 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542328+0000", 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0271000000000004e-05 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542360+0000", 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2166999999999999e-05 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553599+0000", 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011239416 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553646+0000", 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6862e-05 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2233 7.8 7:17170b1f:::smithi13779221-4:head [write 2798226~596838] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.553031+0000", 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 180.69159821700001, 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021618093000000001, 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.320 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2233 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553031+0000", 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553031+0000", 2024-09-21T08:30:12.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553036+0000", 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7879999999999997e-06 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553403+0000", 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036713799999999998 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553435+0000", 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1219e-05 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553451+0000", 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6841e-05 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553587+0000", 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013595600000000001 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553713+0000", 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012593700000000001 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553874+0000", 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160798 2024-09-21T08:30:12.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.558304+0000", 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044295280000000003 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.569637+0000", 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011333609 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574517+0000", 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048793129999999997 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574592+0000", 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5236999999999998e-05 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574650+0000", 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7729000000000002e-05 2024-09-21T08:30:12.326 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.327 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.327 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.327 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2234 7.8 7:17170b1f:::smithi13779221-4:head [setxattr _header (56) in=7b,truncate 3445468] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:30:12.327 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.553487+0000", 2024-09-21T08:30:12.327 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 180.69114271800001, 2024-09-21T08:30:12.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021391090000000001, 2024-09-21T08:30:12.327 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.327 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2234 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553487+0000", 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553487+0000", 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553489+0000", 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.215e-06 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.329 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553495+0000", 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7389999999999996e-06 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553500+0000", 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2360000000000004e-06 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553505+0000", 2024-09-21T08:30:12.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3009999999999997e-06 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.554931+0000", 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00142592 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555047+0000", 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115139 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:30:12.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555183+0000", 2024-09-21T08:30:12.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013642099999999999 2024-09-21T08:30:12.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.558431+0000", 2024-09-21T08:30:12.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032475770000000002 2024-09-21T08:30:12.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.569645+0000", 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011214783000000001 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574711+0000", 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050659449999999996 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574768+0000", 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6415000000000001e-05 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574878+0000", 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110399 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.334 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2235 7.8 7:17170b1f:::smithi13779221-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 180.691116962, 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.02162265, 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2235 2024-09-21T08:30:12.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.336 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3000000000000001e-07 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553516+0000", 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5900000000000002e-06 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553522+0000", 2024-09-21T08:30:12.337 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8980000000000001e-06 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553526+0000", 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7360000000000001e-06 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555581+0000", 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020548799999999998 2024-09-21T08:30:12.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555650+0000", 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9046000000000003e-05 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574919+0000", 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019268464999999999 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.575027+0000", 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108526 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.575135+0000", 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107979 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.340 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2831 7.8 7:144c2ec5:::smithi13779221-45:head [copy-from ver 295] snapc 64=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:01.538037+0000", 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 130.70659259600001, 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020800902, 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2831 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538037+0000", 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538037+0000", 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538039+0000", 2024-09-21T08:30:12.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9099999999999999e-06 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538047+0000", 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.79e-06 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538058+0000", 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1141999999999999e-05 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538068+0000", 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6330000000000008e-06 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538117+0000", 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9360000000000002e-05 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.344 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538218+0000", 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101282 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.548519+0000", 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010300971000000001 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.557646+0000", 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0091271570000000003 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558736+0000", 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001089996 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558773+0000", 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6785000000000001e-05 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558810+0000", 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7058999999999999e-05 2024-09-21T08:30:12.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558838+0000", 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7817000000000002e-05 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3157 7.12 7:4ccee2c4:::smithi13779221-50:head [copy-from ver 217] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.573639+0000", 2024-09-21T08:30:12.347 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 99.670990185999997, 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.02873676, 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3157 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573639+0000", 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573639+0000", 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573642+0000", 2024-09-21T08:30:12.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.041e-06 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573649+0000", 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6340000000000001e-06 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573662+0000", 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3263e-05 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.350 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573670+0000", 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.418e-06 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573726+0000", 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6103000000000001e-05 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573839+0000", 2024-09-21T08:30:12.351 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113186 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.586583+0000", 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012743775000000001 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594268+0000", 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076849559999999997 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.352 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602175+0000", 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079072770000000007 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602261+0000", 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6271000000000001e-05 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.353 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602327+0000", 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5827000000000005e-05 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602376+0000", 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9008999999999998e-05 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.354 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3159 7.12 7:4d793d31:::smithi13779221-9:head [copy-from ver 303] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.573715+0000", 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 99.670914538999995, 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019914487000000002, 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3159 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.355 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573715+0000", 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573715+0000", 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.356 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573716+0000", 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.06e-07 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573719+0000", 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.371e-06 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573724+0000", 2024-09-21T08:30:12.357 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4299999999999997e-06 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573729+0000", 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9400000000000001e-06 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.574567+0000", 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00083730199999999995 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.358 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.574658+0000", 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1775999999999997e-05 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.580114+0000", 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054551759999999999 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.359 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.591315+0000", 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011201825 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593389+0000", 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020731130000000001 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593531+0000", 2024-09-21T08:30:12.360 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000142144 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593575+0000", 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4229e-05 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593630+0000", 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4675000000000001e-05 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.361 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:3441 7.15 e343/339 7:a9dd6c7d:::smithi13779221-47:head v 343'399, mlcod=0'0)", 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:56.572197+0000", 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 75.672432547, 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019269069999999999, 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.362 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572197+0000", 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572197+0000", 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.363 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572198+0000", 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7100000000000002e-07 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572500+0000", 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030198600000000002 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572508+0000", 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8799999999999997e-06 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.364 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572518+0000", 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6269999999999995e-06 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572589+0000", 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0644999999999999e-05 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.365 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572612+0000", 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3623000000000001e-05 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591357+0000", 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018744255000000001 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.366 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591466+0000", 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109583 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:4006 7.9 e398/339 7:9247b5b1:::smithi13779221-17:head v 398'338, mlcod=376'335)", 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:29:51.586649+0000", 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 20.657981073999999, 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020193633999999999, 2024-09-21T08:30:12.367 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586649+0000", 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586649+0000", 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.368 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586650+0000", 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.545e-06 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587156+0000", 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050577 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587177+0000", 2024-09-21T08:30:12.369 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1353999999999999e-05 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587192+0000", 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5214e-05 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587422+0000", 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022968399999999999 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.370 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587586+0000", 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016406799999999999 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.606793+0000", 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019206392999999999 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.606842+0000", 2024-09-21T08:30:12.371 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9605999999999999e-05 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:4168 7.b e409/293 7:d7506421:::smithi13779221-13:head v 409'639, mlcod=407'636)", 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.6622510439999996, 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.083502604999999994, 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.372 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.373 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:30:12.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2e-07 2024-09-21T08:30:12.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.374 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582940+0000", 2024-09-21T08:30:12.374 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00056103800000000003 2024-09-21T08:30:12.374 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.374 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.374 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582952+0000", 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1732999999999999e-05 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582961+0000", 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8000000000000004e-06 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.583031+0000", 2024-09-21T08:30:12.375 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0870000000000004e-05 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.583059+0000", 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7206e-05 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665820+0000", 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.082760995000000004 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.376 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665881+0000", 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1543000000000004e-05 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:4169 7.b e409/293 7:d7506421:::smithi13779221-13:head v 409'640, mlcod=407'636)", 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.589410+0000", 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 9.6552201679999996, 2024-09-21T08:30:12.377 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.080226373000000004, 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589410+0000", 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.378 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589410+0000", 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589412+0000", 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9840000000000001e-06 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:12.379 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589813+0000", 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040080299999999998 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665555+0000", 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.075741557000000001 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665569+0000", 2024-09-21T08:30:12.380 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3961000000000001e-05 2024-09-21T08:30:12.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:12.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.668283+0000", 2024-09-21T08:30:12.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027142099999999999 2024-09-21T08:30:12.381 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.381 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.381 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:12.381 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.668307+0000", 2024-09-21T08:30:12.381 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3487999999999999e-05 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669596+0000", 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012892540000000001 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669636+0000", 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0116e-05 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.382 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.383 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.383 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:12.383 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:12.383 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:30:12.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:12.405 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:12.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:12.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:12.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:12.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:12.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:12.536 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:12.585 INFO:tasks.rados.rados.0.smithi137.stdout:1660: done (7 left) 2024-09-21T08:30:12.585 INFO:tasks.rados.rados.0.smithi137.stdout:1661: read oid 9 snap 161 2024-09-21T08:30:12.585 INFO:tasks.rados.rados.0.smithi137.stdout:1661: expect deleted 2024-09-21T08:30:12.586 INFO:tasks.rados.rados.0.smithi137.stdout:1662: rollback oid 30 current snap is 172 2024-09-21T08:30:12.586 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 30 to 144 2024-09-21T08:30:12.586 INFO:tasks.rados.rados.0.smithi137.stdout:1652: finishing write tid 3 to smithi13779221-13 2024-09-21T08:30:12.586 INFO:tasks.rados.rados.0.smithi137.stdout:1652: finishing write tid 4 to smithi13779221-13 2024-09-21T08:30:12.586 INFO:tasks.rados.rados.0.smithi137.stdout:1652: finishing write tid 5 to smithi13779221-13 2024-09-21T08:30:12.586 INFO:tasks.rados.rados.0.smithi137.stdout:1652: oid 13 updating version 0 to 669 2024-09-21T08:30:12.587 INFO:tasks.rados.rados.0.smithi137.stdout:1652: oid 13 updating version 669 to 670 2024-09-21T08:30:12.587 INFO:tasks.rados.rados.0.smithi137.stdout:1652: oid 13 updating version 670 to 671 2024-09-21T08:30:12.587 INFO:tasks.rados.rados.0.smithi137.stdout:1652: oid 13 updating version 671 to 672 2024-09-21T08:30:12.587 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 672 (ObjNum 592 snap 172 seq_num 592) dirty exists 2024-09-21T08:30:12.587 INFO:tasks.rados.rados.0.smithi137.stdout:1652: left oid 13 (ObjNum 592 snap 172 seq_num 592) 2024-09-21T08:30:12.587 INFO:tasks.rados.rados.0.smithi137.stdout:1653: finishing write tid 1 to smithi13779221-6 2024-09-21T08:30:12.587 INFO:tasks.rados.rados.0.smithi137.stdout:1652: done (8 left) 2024-09-21T08:30:12.588 INFO:tasks.rados.rados.0.smithi137.stdout:1663: write oid 13 current snap is 172 2024-09-21T08:30:12.588 INFO:tasks.rados.rados.0.smithi137.stdout:1653: finishing write tid 2 to smithi13779221-6 2024-09-21T08:30:12.588 INFO:tasks.rados.rados.0.smithi137.stdout:1653: finishing write tid 3 to smithi13779221-6 2024-09-21T08:30:12.588 INFO:tasks.rados.rados.0.smithi137.stdout:1653: finishing write tid 4 to smithi13779221-6 2024-09-21T08:30:12.588 INFO:tasks.rados.rados.0.smithi137.stdout:1663: seq_num 595 ranges {400165=648689,1684786=315379} 2024-09-21T08:30:12.589 INFO:tasks.rados.rados.0.smithi137.stdout:1663: writing smithi13779221-13 from 400165 to 1048854 tid 1 2024-09-21T08:30:12.598 INFO:tasks.rados.rados.0.smithi137.stdout:1663: writing smithi13779221-13 from 1684786 to 2000165 tid 2 2024-09-21T08:30:12.598 INFO:tasks.rados.rados.0.smithi137.stdout:1664: snap_remove snap 150 2024-09-21T08:30:12.636 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:12.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:12.666 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:12.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:12.767 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:12.792 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:12.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:12.794 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:12.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:12.920 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:12.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:12.921 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:13.022 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:13.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:30:13.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:13.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:13.046 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:13.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:13.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:13.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:13.170 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:13.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:13.300 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:13.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:13.301 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:13.325 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T08:30:13.402 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:13.424 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:13.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:13.426 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:13.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:13.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:13.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:13.553 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:13.589 INFO:tasks.rados.rados.0.smithi137.stdout:1664: done (9 left) 2024-09-21T08:30:13.589 INFO:tasks.rados.rados.0.smithi137.stdout:1665: snap_create 2024-09-21T08:30:13.589 INFO:tasks.rados.rados.0.smithi137.stdout:1653: finishing write tid 5 to smithi13779221-6 2024-09-21T08:30:13.589 INFO:tasks.rados.rados.0.smithi137.stdout:1653: oid 6 updating version 0 to 380 2024-09-21T08:30:13.589 INFO:tasks.rados.rados.0.smithi137.stdout:1653: oid 6 updating version 380 to 382 2024-09-21T08:30:13.589 INFO:tasks.rados.rados.0.smithi137.stdout:1653: oid 6 version 382 is already newer than 379 2024-09-21T08:30:13.589 INFO:tasks.rados.rados.0.smithi137.stdout:1653: oid 6 version 382 is already newer than 381 2024-09-21T08:30:13.589 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 382 (ObjNum 593 snap 172 seq_num 593) dirty exists 2024-09-21T08:30:13.589 INFO:tasks.rados.rados.0.smithi137.stdout:1653: left oid 6 (ObjNum 593 snap 172 seq_num 593) 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1654: finishing write tid 1 to smithi13779221-27 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1654: finishing write tid 2 to smithi13779221-27 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1654: finishing write tid 3 to smithi13779221-27 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1654: finishing write tid 4 to smithi13779221-27 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1654: finishing write tid 5 to smithi13779221-27 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1654: oid 27 updating version 0 to 387 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1654: oid 27 version 387 is already newer than 385 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1654: oid 27 version 387 is already newer than 386 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1654: oid 27 version 387 is already newer than 384 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 387 (ObjNum 594 snap 172 seq_num 594) dirty exists 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1654: left oid 27 (ObjNum 594 snap 172 seq_num 594) 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1655: finishing rollback tid 1 to smithi13779221-26 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 389 (ObjNum 129 snap 0 seq_num 3761244240) dirty dne 2024-09-21T08:30:13.590 INFO:tasks.rados.rados.0.smithi137.stdout:1656: expect (ObjNum 564 snap 168 seq_num 564) 2024-09-21T08:30:13.654 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:13.677 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:13.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:13.678 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:13.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:30:13.712 INFO:tasks.rados.rados.0.smithi137.stdout:1657: expect (ObjNum 581 snap 171 seq_num 581) 2024-09-21T08:30:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:13 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:13 smithi137 ceph-mon[29634]: pgmap v533: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 508 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 29 MiB/s rd, 3.5 MiB/s wr, 33 op/s; 341/1341 objects degraded (25.429%) 2024-09-21T08:30:13.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:13 smithi137 ceph-mon[29634]: osdmap e419: 8 total, 6 up, 4 in 2024-09-21T08:30:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:13 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:13 smithi137 ceph-mon[41044]: pgmap v533: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 508 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 29 MiB/s rd, 3.5 MiB/s wr, 33 op/s; 341/1341 objects degraded (25.429%) 2024-09-21T08:30:13.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:13 smithi137 ceph-mon[41044]: osdmap e419: 8 total, 6 up, 4 in 2024-09-21T08:30:13.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:13.802 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:13.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:13.803 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:13.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:13.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:13.931 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:13.931 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:1662: finishing rollback tid 0 to smithi13779221-30 2024-09-21T08:30:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:1663: finishing write tid 1 to smithi13779221-13 2024-09-21T08:30:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:1662: finishing rollback tid 1 to smithi13779221-30 2024-09-21T08:30:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:1662: finishing rollback tid 2 to smithi13779221-30 2024-09-21T08:30:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 667 (ObjNum 493 snap 136 seq_num 493) dirty exists 2024-09-21T08:30:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:1663: finishing write tid 2 to smithi13779221-13 2024-09-21T08:30:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:1663: finishing write tid 3 to smithi13779221-13 2024-09-21T08:30:13.943 INFO:tasks.rados.rados.0.smithi137.stdout:1663: finishing write tid 4 to smithi13779221-13 2024-09-21T08:30:13.944 INFO:tasks.rados.rados.0.smithi137.stdout:1663: oid 13 updating version 0 to 679 2024-09-21T08:30:13.944 INFO:tasks.rados.rados.0.smithi137.stdout:1663: oid 13 version 679 is already newer than 677 2024-09-21T08:30:13.944 INFO:tasks.rados.rados.0.smithi137.stdout:1663: oid 13 version 679 is already newer than 678 2024-09-21T08:30:13.944 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 679 (ObjNum 595 snap 172 seq_num 595) dirty exists 2024-09-21T08:30:13.944 INFO:tasks.rados.rados.0.smithi137.stdout:1663: left oid 13 (ObjNum 595 snap 172 seq_num 595) 2024-09-21T08:30:13.969 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:14.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:13 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:14.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:13 smithi165 ceph-mon[32893]: pgmap v533: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 508 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 29 MiB/s rd, 3.5 MiB/s wr, 33 op/s; 341/1341 objects degraded (25.429%) 2024-09-21T08:30:14.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:13 smithi165 ceph-mon[32893]: osdmap e419: 8 total, 6 up, 4 in 2024-09-21T08:30:14.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:14.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:14.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:14.103 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:30:14.106 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:30:14.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:14.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:14.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:14.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:14.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:14.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:14.307 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:14.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:14.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:14.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:14.434 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:14.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:14.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:14.556 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:14.557 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:14.595 INFO:tasks.rados.rados.0.smithi137.stdout:1653: done (9 left) 2024-09-21T08:30:14.595 INFO:tasks.rados.rados.0.smithi137.stdout:1654: done (8 left) 2024-09-21T08:30:14.596 INFO:tasks.rados.rados.0.smithi137.stdout:1655: done (7 left) 2024-09-21T08:30:14.596 INFO:tasks.rados.rados.0.smithi137.stdout:1656: done (6 left) 2024-09-21T08:30:14.596 INFO:tasks.rados.rados.0.smithi137.stdout:1657: done (5 left) 2024-09-21T08:30:14.596 INFO:tasks.rados.rados.0.smithi137.stdout:1659: done (4 left) 2024-09-21T08:30:14.596 INFO:tasks.rados.rados.0.smithi137.stdout:1661: done (3 left) 2024-09-21T08:30:14.596 INFO:tasks.rados.rados.0.smithi137.stdout:1662: done (2 left) 2024-09-21T08:30:14.596 INFO:tasks.rados.rados.0.smithi137.stdout:1663: done (1 left) 2024-09-21T08:30:14.596 INFO:tasks.rados.rados.0.smithi137.stdout:1665: done (0 left) 2024-09-21T08:30:14.596 INFO:tasks.rados.rados.0.smithi137.stdout:1666: rollback oid 38 current snap is 173 2024-09-21T08:30:14.597 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 38 to 165 2024-09-21T08:30:14.597 INFO:tasks.rados.rados.0.smithi137.stdout:1667: snap_create 2024-09-21T08:30:14.604 INFO:tasks.rados.rados.0.smithi137.stdout:1666: finishing rollback tid 0 to smithi13779221-38 2024-09-21T08:30:14.607 INFO:tasks.rados.rados.0.smithi137.stdout:1666: finishing rollback tid 1 to smithi13779221-38 2024-09-21T08:30:14.607 INFO:tasks.rados.rados.0.smithi137.stdout:1666: finishing rollback tid 2 to smithi13779221-38 2024-09-21T08:30:14.607 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 675 (ObjNum 490 snap 135 seq_num 490) dirty exists 2024-09-21T08:30:14.657 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:14.683 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:14.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:14.684 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:14.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:14 smithi137 ceph-mon[29634]: osdmap e420: 8 total, 6 up, 4 in 2024-09-21T08:30:14.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:14 smithi137 ceph-mon[41044]: osdmap e420: 8 total, 6 up, 4 in 2024-09-21T08:30:14.784 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:14.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:14.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:14.809 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:14.858 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":535,"stamp":"2024-09-21T08:30:13.584764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":532953703,"num_objects":447,"num_object_clones":199,"num_object_copies":1341,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":341,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":447,"num_whiteouts":13,"num_read":2922,"num_read_kb":1819510,"num_write":1740,"num_write_kb":479230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1121,"num_bytes_recovered":210716553,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7967,"ondisk_log_size":7967,"up":374,"acting":374,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1224984,"kb_used_data":1028348,"kb_used_omap":6,"kb_used_meta":195449,"kb_avail":373755624,"statfs":{"total":383980142592,"available":382725758976,"internally_reserved":0,"allocated":1053028352,"data_stored":1533892880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6375,"internal_metadata":200140569},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26006698,"num_objects":11,"num_object_clones":11,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":2,"num_read":87,"num_read_kb":119519,"num_write":46,"num_write_kb":14345,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.966929"},"pg_stats":[{"pgid":"7.23","version":"179'19","reported_seq":455,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.643800+0000","last_change":"2024-09-21T08:28:53.599776+0000","last_active":"2024-09-21T08:30:10.643800+0000","last_peered":"2024-09-21T08:30:10.643800+0000","last_clean":"2024-09-21T08:28:51.585797+0000","last_became_active":"2024-09-21T08:28:53.599776+0000","last_became_peered":"2024-09-21T08:28:53.599776+0000","last_unstale":"2024-09-21T08:30:10.643800+0000","last_undegraded":"2024-09-21T08:30:10.643800+0000","last_fullsized":"2024-09-21T08:28:53.583177+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:28:10.421606+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.421606+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.072212988000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"7.22","version":"257'264","reported_seq":1839,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634591+0000","last_change":"2024-09-21T08:28:53.599720+0000","last_active":"2024-09-21T08:30:10.634591+0000","last_peered":"2024-09-21T08:30:10.634591+0000","last_clean":"2024-09-21T08:28:51.585268+0000","last_became_active":"2024-09-21T08:28:53.599720+0000","last_became_peered":"2024-09-21T08:28:53.599720+0000","last_unstale":"2024-09-21T08:30:10.634591+0000","last_undegraded":"2024-09-21T08:30:10.634591+0000","last_fullsized":"2024-09-21T08:28:53.585695+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":6,"last_scrub":"257'264","last_scrub_stamp":"2024-09-21T08:27:48.727600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:48.727600+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.075805560999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"2"}]},{"pgid":"7.21","version":"179'43","reported_seq":1297,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.632740+0000","last_change":"2024-09-21T08:28:53.594285+0000","last_active":"2024-09-21T08:30:10.632740+0000","last_peered":"2024-09-21T08:30:10.632740+0000","last_clean":"2024-09-21T08:28:51.554218+0000","last_became_active":"2024-09-21T08:28:53.594285+0000","last_became_peered":"2024-09-21T08:28:53.594285+0000","last_unstale":"2024-09-21T08:30:10.632740+0000","last_undegraded":"2024-09-21T08:30:10.632740+0000","last_fullsized":"2024-09-21T08:28:53.573996+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:36.934372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.934372+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.025027727,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"7.20","version":"397'168","reported_seq":1688,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.638672+0000","last_change":"2024-09-21T08:28:53.605282+0000","last_active":"2024-09-21T08:30:10.638672+0000","last_peered":"2024-09-21T08:30:10.638672+0000","last_clean":"2024-09-21T08:28:51.585842+0000","last_became_active":"2024-09-21T08:28:53.605282+0000","last_became_peered":"2024-09-21T08:28:53.605282+0000","last_unstale":"2024-09-21T08:30:10.638672+0000","last_undegraded":"2024-09-21T08:28:53.580783+0000","last_fullsized":"2024-09-21T08:28:53.580670+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"251'134","last_scrub_stamp":"2024-09-21T08:27:49.318143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.318143+0000","objects_scrubbed":1,"log_size":168,"log_dups_size":0,"ondisk_log_size":168,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.076740219999999998,"stat_sum":{"num_bytes":11222104,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":17,"num_read_kb":11328,"num_write":25,"num_write_kb":5151,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1848493,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"2"}]},{"pgid":"7.1b","version":"417'664","reported_seq":2496,"reported_epoch":417,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:30:10.743340+0000","last_change":"2024-09-21T08:30:10.743404+0000","last_active":"2024-09-21T08:30:10.743340+0000","last_peered":"2024-09-21T08:30:10.743340+0000","last_clean":"2024-09-21T08:30:10.743340+0000","last_became_active":"2024-09-21T08:28:37.568752+0000","last_became_peered":"2024-09-21T08:28:37.568752+0000","last_unstale":"2024-09-21T08:30:10.743340+0000","last_undegraded":"2024-09-21T08:30:10.743340+0000","last_fullsized":"2024-09-21T08:30:10.743340+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'499","last_scrub_stamp":"2024-09-21T08:28:09.328624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.328624+0000","objects_scrubbed":5,"log_size":664,"log_dups_size":0,"ondisk_log_size":664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.10920159,"stat_sum":{"num_bytes":44508920,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":106,"num_read_kb":115534,"num_write":55,"num_write_kb":25019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":18585585,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"4"},{"start":"72","length":"b"},{"start":"7f","length":"5"},{"start":"85","length":"3"},{"start":"89","length":"2"},{"start":"8c","length":"4"},{"start":"95","length":"3"},{"start":"9d","length":"1"},{"start":"9f","length":"3"}]},{"pgid":"3.1f","version":"0'0","reported_seq":216,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635742+0000","last_change":"2024-09-21T08:28:53.591551+0000","last_active":"2024-09-21T08:30:10.635742+0000","last_peered":"2024-09-21T08:30:10.635742+0000","last_clean":"2024-09-21T08:28:51.556461+0000","last_became_active":"2024-09-21T08:28:53.591551+0000","last_became_peered":"2024-09-21T08:28:53.591551+0000","last_unstale":"2024-09-21T08:30:10.635742+0000","last_undegraded":"2024-09-21T08:30:10.635742+0000","last_fullsized":"2024-09-21T08:28:53.580526+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":289,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.635542+0000","last_change":"2024-09-21T08:28:29.692402+0000","last_active":"2024-09-21T08:30:10.635542+0000","last_peered":"2024-09-21T08:30:10.635542+0000","last_clean":"2024-09-21T08:30:10.635542+0000","last_became_active":"2024-09-21T08:28:29.635580+0000","last_became_peered":"2024-09-21T08:28:29.635580+0000","last_unstale":"2024-09-21T08:30:10.635542+0000","last_undegraded":"2024-09-21T08:30:10.635542+0000","last_fullsized":"2024-09-21T08:30:10.635542+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:56.303669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:56.303669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":274,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.632859+0000","last_change":"2024-09-21T08:28:53.594621+0000","last_active":"2024-09-21T08:30:10.632859+0000","last_peered":"2024-09-21T08:30:10.632859+0000","last_clean":"2024-09-21T08:28:51.584740+0000","last_became_active":"2024-09-21T08:28:53.594621+0000","last_became_peered":"2024-09-21T08:28:53.594621+0000","last_unstale":"2024-09-21T08:30:10.632859+0000","last_undegraded":"2024-09-21T08:30:10.632859+0000","last_fullsized":"2024-09-21T08:28:53.576425+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.424482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.424482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":700,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634197+0000","last_change":"2024-09-21T08:28:53.601367+0000","last_active":"2024-09-21T08:30:10.634197+0000","last_peered":"2024-09-21T08:30:10.634197+0000","last_clean":"2024-09-21T08:28:51.557847+0000","last_became_active":"2024-09-21T08:28:53.601367+0000","last_became_peered":"2024-09-21T08:28:53.601367+0000","last_unstale":"2024-09-21T08:30:10.634197+0000","last_undegraded":"2024-09-21T08:30:10.634197+0000","last_fullsized":"2024-09-21T08:28:53.577149+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.464969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.464969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:11.999750+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1a","version":"248'68","reported_seq":729,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633298+0000","last_change":"2024-09-21T08:28:53.593471+0000","last_active":"2024-09-21T08:30:10.633298+0000","last_peered":"2024-09-21T08:30:10.633298+0000","last_clean":"2024-09-21T08:28:52.589751+0000","last_became_active":"2024-09-21T08:28:53.593471+0000","last_became_peered":"2024-09-21T08:28:53.593471+0000","last_unstale":"2024-09-21T08:30:10.633298+0000","last_undegraded":"2024-09-21T08:30:10.633298+0000","last_fullsized":"2024-09-21T08:28:53.574504+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'68","last_scrub_stamp":"2024-09-21T08:27:35.919031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:35.919031+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0328630059999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"6.1b","version":"0'0","reported_seq":631,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.637067+0000","last_change":"2024-09-21T08:28:53.606497+0000","last_active":"2024-09-21T08:30:10.637067+0000","last_peered":"2024-09-21T08:30:10.637067+0000","last_clean":"2024-09-21T08:28:51.585547+0000","last_became_active":"2024-09-21T08:28:53.606497+0000","last_became_peered":"2024-09-21T08:28:53.606497+0000","last_unstale":"2024-09-21T08:30:10.637067+0000","last_undegraded":"2024-09-21T08:30:10.637067+0000","last_fullsized":"2024-09-21T08:28:53.581183+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.804294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.804294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":558,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635097+0000","last_change":"2024-09-21T08:28:53.602981+0000","last_active":"2024-09-21T08:30:10.635097+0000","last_peered":"2024-09-21T08:30:10.635097+0000","last_clean":"2024-09-21T08:28:51.587203+0000","last_became_active":"2024-09-21T08:28:53.602981+0000","last_became_peered":"2024-09-21T08:28:53.602981+0000","last_unstale":"2024-09-21T08:30:10.635097+0000","last_undegraded":"2024-09-21T08:30:10.635097+0000","last_fullsized":"2024-09-21T08:28:53.581112+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.146798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.146798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":555,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.632958+0000","last_change":"2024-09-21T08:29:00.355702+0000","last_active":"2024-09-21T08:30:10.632958+0000","last_peered":"2024-09-21T08:30:10.632958+0000","last_clean":"2024-09-21T08:30:10.632958+0000","last_became_active":"2024-09-21T08:28:29.620169+0000","last_became_peered":"2024-09-21T08:28:29.620169+0000","last_unstale":"2024-09-21T08:30:10.632958+0000","last_undegraded":"2024-09-21T08:30:10.632958+0000","last_fullsized":"2024-09-21T08:30:10.632958+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:00.355591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:00.355591+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:11.999911+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":266,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635659+0000","last_change":"2024-09-21T08:28:53.601812+0000","last_active":"2024-09-21T08:30:10.635659+0000","last_peered":"2024-09-21T08:30:10.635659+0000","last_clean":"2024-09-21T08:28:51.585034+0000","last_became_active":"2024-09-21T08:28:53.601812+0000","last_became_peered":"2024-09-21T08:28:53.601812+0000","last_unstale":"2024-09-21T08:30:10.635659+0000","last_undegraded":"2024-09-21T08:30:10.635659+0000","last_fullsized":"2024-09-21T08:28:53.578966+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.340817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.340817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.19","version":"417'205","reported_seq":1412,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.698601+0000","last_change":"2024-09-21T08:28:53.587606+0000","last_active":"2024-09-21T08:30:10.698601+0000","last_peered":"2024-09-21T08:30:10.698601+0000","last_clean":"2024-09-21T08:28:52.589310+0000","last_became_active":"2024-09-21T08:28:53.587606+0000","last_became_peered":"2024-09-21T08:28:53.587606+0000","last_unstale":"2024-09-21T08:30:10.698601+0000","last_undegraded":"2024-09-21T08:28:53.574680+0000","last_fullsized":"2024-09-21T08:28:53.574555+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"266'188","last_scrub_stamp":"2024-09-21T08:27:49.719124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.719124+0000","objects_scrubbed":1,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.034770314,"stat_sum":{"num_bytes":11158040,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":25354,"num_write":16,"num_write_kb":11591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"3.1d","version":"0'0","reported_seq":763,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.637039+0000","last_change":"2024-09-21T08:28:53.601545+0000","last_active":"2024-09-21T08:30:10.637039+0000","last_peered":"2024-09-21T08:30:10.637039+0000","last_clean":"2024-09-21T08:28:51.555695+0000","last_became_active":"2024-09-21T08:28:53.601545+0000","last_became_peered":"2024-09-21T08:28:53.601545+0000","last_unstale":"2024-09-21T08:30:10.637039+0000","last_undegraded":"2024-09-21T08:30:10.637039+0000","last_fullsized":"2024-09-21T08:28:53.577677+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.780545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.780545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908564+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":515,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.634621+0000","last_change":"2024-09-21T08:28:39.292016+0000","last_active":"2024-09-21T08:30:10.634621+0000","last_peered":"2024-09-21T08:30:10.634621+0000","last_clean":"2024-09-21T08:30:10.634621+0000","last_became_active":"2024-09-21T08:28:37.568659+0000","last_became_peered":"2024-09-21T08:28:37.568659+0000","last_unstale":"2024-09-21T08:30:10.634621+0000","last_undegraded":"2024-09-21T08:30:10.634621+0000","last_fullsized":"2024-09-21T08:30:10.634621+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:28:39.291909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.291909+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:11.999817+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":722,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.632930+0000","last_change":"2024-09-21T08:28:53.593695+0000","last_active":"2024-09-21T08:30:10.632930+0000","last_peered":"2024-09-21T08:30:10.632930+0000","last_clean":"2024-09-21T08:28:51.556408+0000","last_became_active":"2024-09-21T08:28:53.593695+0000","last_became_peered":"2024-09-21T08:28:53.593695+0000","last_unstale":"2024-09-21T08:30:10.632930+0000","last_undegraded":"2024-09-21T08:30:10.632930+0000","last_fullsized":"2024-09-21T08:28:53.574067+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":677,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.632664+0000","last_change":"2024-09-21T08:28:53.586342+0000","last_active":"2024-09-21T08:30:10.632664+0000","last_peered":"2024-09-21T08:30:10.632664+0000","last_clean":"2024-09-21T08:28:51.584564+0000","last_became_active":"2024-09-21T08:28:53.586342+0000","last_became_peered":"2024-09-21T08:28:53.586342+0000","last_unstale":"2024-09-21T08:30:10.632664+0000","last_undegraded":"2024-09-21T08:30:10.632664+0000","last_fullsized":"2024-09-21T08:28:53.575506+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.291226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.291226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"413'257","reported_seq":1836,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633070+0000","last_change":"2024-09-21T08:28:53.587953+0000","last_active":"2024-09-21T08:30:10.633070+0000","last_peered":"2024-09-21T08:30:10.633070+0000","last_clean":"2024-09-21T08:28:51.550342+0000","last_became_active":"2024-09-21T08:28:53.587953+0000","last_became_peered":"2024-09-21T08:28:53.587953+0000","last_unstale":"2024-09-21T08:30:10.633070+0000","last_undegraded":"2024-09-21T08:28:53.574363+0000","last_fullsized":"2024-09-21T08:28:53.574234+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"319'218","last_scrub_stamp":"2024-09-21T08:28:32.867460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:32.867460+0000","objects_scrubbed":4,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":35,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027063920000000002,"stat_sum":{"num_bytes":24595783,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":68203,"num_write":55,"num_write_kb":28843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7346792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"3.1c","version":"0'0","reported_seq":748,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634289+0000","last_change":"2024-09-21T08:28:53.600345+0000","last_active":"2024-09-21T08:30:10.634289+0000","last_peered":"2024-09-21T08:30:10.634289+0000","last_clean":"2024-09-21T08:28:51.556880+0000","last_became_active":"2024-09-21T08:28:53.600345+0000","last_became_peered":"2024-09-21T08:28:53.600345+0000","last_unstale":"2024-09-21T08:30:10.634289+0000","last_undegraded":"2024-09-21T08:30:10.634289+0000","last_fullsized":"2024-09-21T08:28:53.576313+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908278+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":272,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.639651+0000","last_change":"2024-09-21T08:28:53.605410+0000","last_active":"2024-09-21T08:30:10.639651+0000","last_peered":"2024-09-21T08:30:10.639651+0000","last_clean":"2024-09-21T08:28:51.585702+0000","last_became_active":"2024-09-21T08:28:53.605410+0000","last_became_peered":"2024-09-21T08:28:53.605410+0000","last_unstale":"2024-09-21T08:30:10.639651+0000","last_undegraded":"2024-09-21T08:28:53.583377+0000","last_fullsized":"2024-09-21T08:28:53.583262+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:27:26.356987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:26.356987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":699,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635242+0000","last_change":"2024-09-21T08:28:53.600104+0000","last_active":"2024-09-21T08:30:10.635242+0000","last_peered":"2024-09-21T08:30:10.635242+0000","last_clean":"2024-09-21T08:28:27.553155+0000","last_became_active":"2024-09-21T08:28:53.600104+0000","last_became_peered":"2024-09-21T08:28:53.600104+0000","last_unstale":"2024-09-21T08:30:10.635242+0000","last_undegraded":"2024-09-21T08:30:10.635242+0000","last_fullsized":"2024-09-21T08:28:53.585955+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.850373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":768,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.633820+0000","last_change":"2024-09-21T08:30:04.034656+0000","last_active":"2024-09-21T08:30:10.633820+0000","last_peered":"2024-09-21T08:30:10.633820+0000","last_clean":"2024-09-21T08:30:10.633820+0000","last_became_active":"2024-09-21T08:28:04.705450+0000","last_became_peered":"2024-09-21T08:28:04.705450+0000","last_unstale":"2024-09-21T08:30:10.633820+0000","last_undegraded":"2024-09-21T08:30:10.633820+0000","last_fullsized":"2024-09-21T08:30:10.633820+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:30:04.034575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:04.034575+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:12.159925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1f","version":"247'32","reported_seq":1533,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634218+0000","last_change":"2024-09-21T08:28:53.601095+0000","last_active":"2024-09-21T08:30:10.634218+0000","last_peered":"2024-09-21T08:30:10.634218+0000","last_clean":"2024-09-21T08:28:53.015413+0000","last_became_active":"2024-09-21T08:28:53.601095+0000","last_became_peered":"2024-09-21T08:28:53.601095+0000","last_unstale":"2024-09-21T08:30:10.634218+0000","last_undegraded":"2024-09-21T08:30:10.634218+0000","last_fullsized":"2024-09-21T08:28:53.577251+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:27:51.078531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.078531+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0098481060000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"6.1e","version":"0'0","reported_seq":699,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.637500+0000","last_change":"2024-09-21T08:28:53.591644+0000","last_active":"2024-09-21T08:30:10.637500+0000","last_peered":"2024-09-21T08:30:10.637500+0000","last_clean":"2024-09-21T08:28:51.556030+0000","last_became_active":"2024-09-21T08:28:53.591644+0000","last_became_peered":"2024-09-21T08:28:53.591644+0000","last_unstale":"2024-09-21T08:30:10.637500+0000","last_undegraded":"2024-09-21T08:30:10.637500+0000","last_fullsized":"2024-09-21T08:28:53.578796+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.855448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.855448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908385+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":587,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.636780+0000","last_change":"2024-09-21T08:28:53.604972+0000","last_active":"2024-09-21T08:30:10.636780+0000","last_peered":"2024-09-21T08:30:10.636780+0000","last_clean":"2024-09-21T08:28:51.585297+0000","last_became_active":"2024-09-21T08:28:53.604972+0000","last_became_peered":"2024-09-21T08:28:53.604972+0000","last_unstale":"2024-09-21T08:30:10.636780+0000","last_undegraded":"2024-09-21T08:30:10.636780+0000","last_fullsized":"2024-09-21T08:28:53.579187+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":789,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.633492+0000","last_change":"2024-09-21T08:29:37.352960+0000","last_active":"2024-09-21T08:30:10.633492+0000","last_peered":"2024-09-21T08:30:10.633492+0000","last_clean":"2024-09-21T08:30:10.633492+0000","last_became_active":"2024-09-21T08:28:37.566225+0000","last_became_peered":"2024-09-21T08:28:37.566225+0000","last_unstale":"2024-09-21T08:30:10.633492+0000","last_undegraded":"2024-09-21T08:30:10.633492+0000","last_fullsized":"2024-09-21T08:30:10.633492+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:37.352879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:37.352879+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:41.677208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":272,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.638095+0000","last_change":"2024-09-21T08:28:53.598624+0000","last_active":"2024-09-21T08:30:10.638095+0000","last_peered":"2024-09-21T08:30:10.638095+0000","last_clean":"2024-09-21T08:28:51.587418+0000","last_became_active":"2024-09-21T08:28:53.598624+0000","last_became_peered":"2024-09-21T08:28:53.598624+0000","last_unstale":"2024-09-21T08:30:10.638095+0000","last_undegraded":"2024-09-21T08:30:10.638095+0000","last_fullsized":"2024-09-21T08:28:53.579380+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:30.310628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.310628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"417'295","reported_seq":1874,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.701882+0000","last_change":"2024-09-21T08:28:53.595615+0000","last_active":"2024-09-21T08:30:10.701882+0000","last_peered":"2024-09-21T08:30:10.701882+0000","last_clean":"2024-09-21T08:28:52.666418+0000","last_became_active":"2024-09-21T08:28:53.595615+0000","last_became_peered":"2024-09-21T08:28:53.595615+0000","last_unstale":"2024-09-21T08:30:10.701882+0000","last_undegraded":"2024-09-21T08:28:53.580612+0000","last_fullsized":"2024-09-21T08:28:53.580483+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'236","last_scrub_stamp":"2024-09-21T08:28:09.464279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.464279+0000","objects_scrubbed":2,"log_size":295,"log_dups_size":0,"ondisk_log_size":295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":41,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.0792350470000001,"stat_sum":{"num_bytes":34900029,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":60,"num_read_kb":64280,"num_write":51,"num_write_kb":20459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6996269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":16}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"6.1f","version":"0'0","reported_seq":616,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.637090+0000","last_change":"2024-09-21T08:29:33.233733+0000","last_active":"2024-09-21T08:30:10.637090+0000","last_peered":"2024-09-21T08:30:10.637090+0000","last_clean":"2024-09-21T08:30:10.637090+0000","last_became_active":"2024-09-21T08:28:04.725392+0000","last_became_peered":"2024-09-21T08:28:04.725392+0000","last_unstale":"2024-09-21T08:30:10.637090+0000","last_undegraded":"2024-09-21T08:30:10.637090+0000","last_fullsized":"2024-09-21T08:30:10.637090+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:33.233665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:33.233665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.550890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":272,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.637282+0000","last_change":"2024-09-21T08:28:53.605497+0000","last_active":"2024-09-21T08:30:10.637282+0000","last_peered":"2024-09-21T08:30:10.637282+0000","last_clean":"2024-09-21T08:28:51.587153+0000","last_became_active":"2024-09-21T08:28:53.605497+0000","last_became_peered":"2024-09-21T08:28:53.605497+0000","last_unstale":"2024-09-21T08:30:10.637282+0000","last_undegraded":"2024-09-21T08:30:10.637282+0000","last_fullsized":"2024-09-21T08:28:53.586766+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:33.237691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.237691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":283,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.634151+0000","last_change":"2024-09-21T08:29:15.349199+0000","last_active":"2024-09-21T08:30:10.634151+0000","last_peered":"2024-09-21T08:30:10.634151+0000","last_clean":"2024-09-21T08:30:10.634151+0000","last_became_active":"2024-09-21T08:28:04.724988+0000","last_became_peered":"2024-09-21T08:28:04.724988+0000","last_unstale":"2024-09-21T08:30:10.634151+0000","last_undegraded":"2024-09-21T08:30:10.634151+0000","last_fullsized":"2024-09-21T08:30:10.634151+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:29:15.349124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:15.349124+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:20.426320+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":643,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.641580+0000","last_change":"2024-09-21T08:28:53.591068+0000","last_active":"2024-09-21T08:30:10.641580+0000","last_peered":"2024-09-21T08:30:10.641580+0000","last_clean":"2024-09-21T08:28:51.554175+0000","last_became_active":"2024-09-21T08:28:53.591068+0000","last_became_peered":"2024-09-21T08:28:53.591068+0000","last_unstale":"2024-09-21T08:30:10.641580+0000","last_undegraded":"2024-09-21T08:30:10.641580+0000","last_fullsized":"2024-09-21T08:28:53.580183+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:09.116412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.116412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:13.631135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"248'84","reported_seq":470,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634485+0000","last_change":"2024-09-21T08:28:53.595944+0000","last_active":"2024-09-21T08:30:10.634485+0000","last_peered":"2024-09-21T08:30:10.634485+0000","last_clean":"2024-09-21T08:28:51.587189+0000","last_became_active":"2024-09-21T08:28:53.595944+0000","last_became_peered":"2024-09-21T08:28:53.595944+0000","last_unstale":"2024-09-21T08:30:10.634485+0000","last_undegraded":"2024-09-21T08:30:10.634485+0000","last_fullsized":"2024-09-21T08:28:53.578716+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'84","last_scrub_stamp":"2024-09-21T08:28:12.352559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.352559+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.072813132000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"6.1c","version":"73'1","reported_seq":260,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.634845+0000","last_change":"2024-09-21T08:28:53.589802+0000","last_active":"2024-09-21T08:30:10.634845+0000","last_peered":"2024-09-21T08:30:10.634845+0000","last_clean":"2024-09-21T08:28:51.557351+0000","last_became_active":"2024-09-21T08:28:53.589802+0000","last_became_peered":"2024-09-21T08:28:53.589802+0000","last_unstale":"2024-09-21T08:30:10.634845+0000","last_undegraded":"2024-09-21T08:28:53.577740+0000","last_fullsized":"2024-09-21T08:28:53.577630+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:27:14.826047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.826047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908626+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"198'2","reported_seq":270,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.632901+0000","last_change":"2024-09-21T08:28:53.601978+0000","last_active":"2024-09-21T08:30:10.632901+0000","last_peered":"2024-09-21T08:30:10.632901+0000","last_clean":"2024-09-21T08:28:51.584776+0000","last_became_active":"2024-09-21T08:28:53.601978+0000","last_became_peered":"2024-09-21T08:28:53.601978+0000","last_unstale":"2024-09-21T08:30:10.632901+0000","last_undegraded":"2024-09-21T08:28:53.576796+0000","last_fullsized":"2024-09-21T08:28:53.576697+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'2","last_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_deep_scrub":"198'2","last_deep_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_clean_scrub_stamp":"2024-09-21T08:26:32.819313+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":771,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.635264+0000","last_change":"2024-09-21T08:28:53.597832+0000","last_active":"2024-09-21T08:30:10.635264+0000","last_peered":"2024-09-21T08:30:10.635264+0000","last_clean":"2024-09-21T08:28:51.587327+0000","last_became_active":"2024-09-21T08:28:53.597832+0000","last_became_peered":"2024-09-21T08:28:53.597832+0000","last_unstale":"2024-09-21T08:30:10.635264+0000","last_undegraded":"2024-09-21T08:28:53.582409+0000","last_fullsized":"2024-09-21T08:28:53.582308+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"340'17","reported_seq":394,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.643424+0000","last_change":"2024-09-21T08:28:53.600523+0000","last_active":"2024-09-21T08:30:10.643424+0000","last_peered":"2024-09-21T08:30:10.643424+0000","last_clean":"2024-09-21T08:28:51.556529+0000","last_became_active":"2024-09-21T08:28:53.600523+0000","last_became_peered":"2024-09-21T08:28:53.600523+0000","last_unstale":"2024-09-21T08:30:10.643424+0000","last_undegraded":"2024-09-21T08:28:53.585227+0000","last_fullsized":"2024-09-21T08:28:53.585102+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'15","last_scrub_stamp":"2024-09-21T08:28:14.153130+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.153130+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:13.631123+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"417'236","reported_seq":1934,"reported_epoch":417,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:10.715855+0000","last_change":"2024-09-21T08:30:10.715855+0000","last_active":"2024-09-21T08:30:10.715855+0000","last_peered":"2024-09-21T08:30:10.715855+0000","last_clean":"2024-09-21T08:30:10.715855+0000","last_became_active":"2024-09-21T08:28:32.557643+0000","last_became_peered":"2024-09-21T08:28:32.557643+0000","last_unstale":"2024-09-21T08:30:10.715855+0000","last_undegraded":"2024-09-21T08:30:10.715855+0000","last_fullsized":"2024-09-21T08:30:10.715855+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'160","last_scrub_stamp":"2024-09-21T08:27:34.925496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:34.925496+0000","objects_scrubbed":2,"log_size":236,"log_dups_size":0,"ondisk_log_size":236,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.083262504000000001,"stat_sum":{"num_bytes":9551024,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":50,"num_read_kb":58168,"num_write":21,"num_write_kb":9777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":6235496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"4"},{"start":"72","length":"b"},{"start":"7f","length":"5"},{"start":"85","length":"3"},{"start":"89","length":"2"},{"start":"8c","length":"4"},{"start":"95","length":"3"},{"start":"9d","length":"1"},{"start":"9f","length":"4"},{"start":"a6","length":"1"}]},{"pgid":"4.f","version":"73'15","reported_seq":282,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.635918+0000","last_change":"2024-09-21T08:28:53.595462+0000","last_active":"2024-09-21T08:30:10.635918+0000","last_peered":"2024-09-21T08:30:10.635918+0000","last_clean":"2024-09-21T08:28:51.587748+0000","last_became_active":"2024-09-21T08:28:53.595462+0000","last_became_peered":"2024-09-21T08:28:53.595462+0000","last_unstale":"2024-09-21T08:30:10.635918+0000","last_undegraded":"2024-09-21T08:28:53.579415+0000","last_fullsized":"2024-09-21T08:28:53.579321+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:26:46.777115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.777115+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":266,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635922+0000","last_change":"2024-09-21T08:28:53.595248+0000","last_active":"2024-09-21T08:30:10.635922+0000","last_peered":"2024-09-21T08:30:10.635922+0000","last_clean":"2024-09-21T08:28:51.585248+0000","last_became_active":"2024-09-21T08:28:53.595248+0000","last_became_peered":"2024-09-21T08:28:53.595248+0000","last_unstale":"2024-09-21T08:30:10.635922+0000","last_undegraded":"2024-09-21T08:30:10.635922+0000","last_fullsized":"2024-09-21T08:28:53.579384+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:45.434055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:45.434055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"340'11","reported_seq":803,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.637415+0000","last_change":"2024-09-21T08:28:53.591354+0000","last_active":"2024-09-21T08:30:10.637415+0000","last_peered":"2024-09-21T08:30:10.637415+0000","last_clean":"2024-09-21T08:28:51.555908+0000","last_became_active":"2024-09-21T08:28:53.591354+0000","last_became_peered":"2024-09-21T08:28:53.591354+0000","last_unstale":"2024-09-21T08:30:10.637415+0000","last_undegraded":"2024-09-21T08:28:53.578763+0000","last_fullsized":"2024-09-21T08:28:53.578651+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:26:44.741275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.741275+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908552+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":735,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.637545+0000","last_change":"2024-09-21T08:29:12.167619+0000","last_active":"2024-09-21T08:30:10.637545+0000","last_peered":"2024-09-21T08:30:10.637545+0000","last_clean":"2024-09-21T08:30:10.637545+0000","last_became_active":"2024-09-21T08:28:29.619967+0000","last_became_peered":"2024-09-21T08:28:29.619967+0000","last_unstale":"2024-09-21T08:30:10.637545+0000","last_undegraded":"2024-09-21T08:30:10.637545+0000","last_fullsized":"2024-09-21T08:30:10.637545+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:12.167545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:12.167545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:18.826968+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"179'19","reported_seq":1412,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.641387+0000","last_change":"2024-09-21T08:28:53.592078+0000","last_active":"2024-09-21T08:30:10.641387+0000","last_peered":"2024-09-21T08:30:10.641387+0000","last_clean":"2024-09-21T08:28:51.554873+0000","last_became_active":"2024-09-21T08:28:53.592078+0000","last_became_peered":"2024-09-21T08:28:53.592078+0000","last_unstale":"2024-09-21T08:30:10.641387+0000","last_undegraded":"2024-09-21T08:30:10.641387+0000","last_fullsized":"2024-09-21T08:28:53.580654+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:27:36.268364+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.268364+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":35,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057177200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"4.0","version":"73'18","reported_seq":700,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.635159+0000","last_change":"2024-09-21T08:28:53.593785+0000","last_active":"2024-09-21T08:30:10.635159+0000","last_peered":"2024-09-21T08:30:10.635159+0000","last_clean":"2024-09-21T08:28:51.584787+0000","last_became_active":"2024-09-21T08:28:53.593785+0000","last_became_peered":"2024-09-21T08:28:53.593785+0000","last_unstale":"2024-09-21T08:30:10.635159+0000","last_undegraded":"2024-09-21T08:28:53.574919+0000","last_fullsized":"2024-09-21T08:28:53.574229+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-09-21T08:26:54.003173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:54.003173+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":475,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634506+0000","last_change":"2024-09-21T08:28:53.602596+0000","last_active":"2024-09-21T08:30:10.634506+0000","last_peered":"2024-09-21T08:30:10.634506+0000","last_clean":"2024-09-21T08:28:51.555756+0000","last_became_active":"2024-09-21T08:28:53.602596+0000","last_became_peered":"2024-09-21T08:28:53.602596+0000","last_unstale":"2024-09-21T08:30:10.634506+0000","last_undegraded":"2024-09-21T08:30:10.634506+0000","last_fullsized":"2024-09-21T08:28:53.579349+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908250+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":575,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.632708+0000","last_change":"2024-09-21T08:28:53.587446+0000","last_active":"2024-09-21T08:30:10.632708+0000","last_peered":"2024-09-21T08:30:10.632708+0000","last_clean":"2024-09-21T08:28:27.553250+0000","last_became_active":"2024-09-21T08:28:53.587446+0000","last_became_peered":"2024-09-21T08:28:53.587446+0000","last_unstale":"2024-09-21T08:30:10.632708+0000","last_undegraded":"2024-09-21T08:30:10.632708+0000","last_fullsized":"2024-09-21T08:28:53.574310+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:32.813973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:32.813973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908835+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":423,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.632834+0000","last_change":"2024-09-21T08:28:53.587280+0000","last_active":"2024-09-21T08:30:10.632834+0000","last_peered":"2024-09-21T08:30:10.632834+0000","last_clean":"2024-09-21T08:26:47.628755+0000","last_became_active":"2024-09-21T08:28:53.587280+0000","last_became_peered":"2024-09-21T08:28:53.587280+0000","last_unstale":"2024-09-21T08:30:10.632834+0000","last_undegraded":"2024-09-21T08:30:10.632834+0000","last_fullsized":"2024-09-21T08:28:53.574372+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:16.341564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.341564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"411'350","reported_seq":1919,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.636936+0000","last_change":"2024-09-21T08:28:53.605477+0000","last_active":"2024-09-21T08:30:10.636936+0000","last_peered":"2024-09-21T08:30:10.636936+0000","last_clean":"2024-09-21T08:28:52.666434+0000","last_became_active":"2024-09-21T08:28:53.605477+0000","last_became_peered":"2024-09-21T08:28:53.605477+0000","last_unstale":"2024-09-21T08:30:10.636936+0000","last_undegraded":"2024-09-21T08:28:53.586515+0000","last_fullsized":"2024-09-21T08:28:53.586400+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'273","last_scrub_stamp":"2024-09-21T08:27:50.318620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.318620+0000","objects_scrubbed":3,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":41,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.073725586999999995,"stat_sum":{"num_bytes":20243096,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":65,"num_read_kb":73718,"num_write":41,"num_write_kb":14479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":3845252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.1","version":"239'21","reported_seq":794,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633116+0000","last_change":"2024-09-21T08:28:53.588019+0000","last_active":"2024-09-21T08:30:10.633116+0000","last_peered":"2024-09-21T08:30:10.633116+0000","last_clean":"2024-09-21T08:28:51.554459+0000","last_became_active":"2024-09-21T08:28:53.588019+0000","last_became_peered":"2024-09-21T08:28:53.588019+0000","last_unstale":"2024-09-21T08:30:10.633116+0000","last_undegraded":"2024-09-21T08:28:53.575510+0000","last_fullsized":"2024-09-21T08:28:53.575377+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21","last_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_deep_scrub":"239'21","last_deep_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.969642+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908668+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":709,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.635332+0000","last_change":"2024-09-21T08:28:53.598446+0000","last_active":"2024-09-21T08:30:10.635332+0000","last_peered":"2024-09-21T08:30:10.635332+0000","last_clean":"2024-09-21T08:28:51.587388+0000","last_became_active":"2024-09-21T08:28:53.598446+0000","last_became_peered":"2024-09-21T08:28:53.598446+0000","last_unstale":"2024-09-21T08:30:10.635332+0000","last_undegraded":"2024-09-21T08:28:53.582469+0000","last_fullsized":"2024-09-21T08:28:53.582373+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"317'16","reported_seq":304,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.635397+0000","last_change":"2024-09-21T08:28:29.687150+0000","last_active":"2024-09-21T08:30:10.635397+0000","last_peered":"2024-09-21T08:30:10.635397+0000","last_clean":"2024-09-21T08:30:10.635397+0000","last_became_active":"2024-09-21T08:28:29.634231+0000","last_became_peered":"2024-09-21T08:28:29.634231+0000","last_unstale":"2024-09-21T08:30:10.635397+0000","last_undegraded":"2024-09-21T08:30:10.635397+0000","last_fullsized":"2024-09-21T08:30:10.635397+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'14","last_scrub_stamp":"2024-09-21T08:27:36.998538+0000","last_deep_scrub":"263'14","last_deep_scrub_stamp":"2024-09-21T08:27:36.998538+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.998538+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":722,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.640982+0000","last_change":"2024-09-21T08:28:53.591306+0000","last_active":"2024-09-21T08:30:10.640982+0000","last_peered":"2024-09-21T08:30:10.640982+0000","last_clean":"2024-09-21T08:28:51.554109+0000","last_became_active":"2024-09-21T08:28:53.591306+0000","last_became_peered":"2024-09-21T08:28:53.591306+0000","last_unstale":"2024-09-21T08:30:10.640982+0000","last_undegraded":"2024-09-21T08:30:10.640982+0000","last_fullsized":"2024-09-21T08:28:53.580235+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.344185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.344185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:10.649257+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"179'43","reported_seq":447,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633425+0000","last_change":"2024-09-21T08:28:53.589976+0000","last_active":"2024-09-21T08:30:10.633425+0000","last_peered":"2024-09-21T08:30:10.633425+0000","last_clean":"2024-09-21T08:28:51.585006+0000","last_became_active":"2024-09-21T08:28:53.589976+0000","last_became_peered":"2024-09-21T08:28:53.589976+0000","last_unstale":"2024-09-21T08:30:10.633425+0000","last_undegraded":"2024-09-21T08:30:10.633425+0000","last_fullsized":"2024-09-21T08:28:53.574347+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:50.158232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.158232+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.075152813999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"4.2","version":"73'10","reported_seq":302,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.637489+0000","last_change":"2024-09-21T08:29:58.300891+0000","last_active":"2024-09-21T08:30:10.637489+0000","last_peered":"2024-09-21T08:30:10.637489+0000","last_clean":"2024-09-21T08:30:10.637489+0000","last_became_active":"2024-09-21T08:28:29.638492+0000","last_became_peered":"2024-09-21T08:28:29.638492+0000","last_unstale":"2024-09-21T08:30:10.637489+0000","last_undegraded":"2024-09-21T08:30:10.637489+0000","last_fullsized":"2024-09-21T08:30:10.637489+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:29:58.300806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.300806+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:17.610082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":771,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.634992+0000","last_change":"2024-09-21T08:28:53.588911+0000","last_active":"2024-09-21T08:30:10.634992+0000","last_peered":"2024-09-21T08:30:10.634992+0000","last_clean":"2024-09-21T08:28:51.554689+0000","last_became_active":"2024-09-21T08:28:53.588911+0000","last_became_peered":"2024-09-21T08:28:53.588911+0000","last_unstale":"2024-09-21T08:30:10.634992+0000","last_undegraded":"2024-09-21T08:28:53.576074+0000","last_fullsized":"2024-09-21T08:28:53.575948+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:16.003225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.003225+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908762+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"340'16","reported_seq":527,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633567+0000","last_change":"2024-09-21T08:28:53.594258+0000","last_active":"2024-09-21T08:30:10.633567+0000","last_peered":"2024-09-21T08:30:10.633567+0000","last_clean":"2024-09-21T08:28:51.556539+0000","last_became_active":"2024-09-21T08:28:53.594258+0000","last_became_peered":"2024-09-21T08:28:53.594258+0000","last_unstale":"2024-09-21T08:30:10.633567+0000","last_undegraded":"2024-09-21T08:28:53.574644+0000","last_fullsized":"2024-09-21T08:28:53.574523+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'13","last_scrub_stamp":"2024-09-21T08:27:27.874952+0000","last_deep_scrub":"176'11","last_deep_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_clean_scrub_stamp":"2024-09-21T08:27:27.874952+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":558,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.632483+0000","last_change":"2024-09-21T08:28:37.565930+0000","last_active":"2024-09-21T08:30:10.632483+0000","last_peered":"2024-09-21T08:30:10.632483+0000","last_clean":"2024-09-21T08:30:10.632483+0000","last_became_active":"2024-09-21T08:28:37.565624+0000","last_became_peered":"2024-09-21T08:28:37.565624+0000","last_unstale":"2024-09-21T08:30:10.632483+0000","last_undegraded":"2024-09-21T08:30:10.632483+0000","last_fullsized":"2024-09-21T08:30:10.632483+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.258570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.258570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:11.999845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"416'193","reported_seq":1980,"reported_epoch":417,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:10.743719+0000","last_change":"2024-09-21T08:30:10.743768+0000","last_active":"2024-09-21T08:30:10.743719+0000","last_peered":"2024-09-21T08:30:10.743719+0000","last_clean":"2024-09-21T08:30:10.743719+0000","last_became_active":"2024-09-21T08:28:07.549600+0000","last_became_peered":"2024-09-21T08:28:07.549600+0000","last_unstale":"2024-09-21T08:30:10.743719+0000","last_undegraded":"2024-09-21T08:30:10.743719+0000","last_fullsized":"2024-09-21T08:30:10.743719+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'166","last_scrub_stamp":"2024-09-21T08:29:32.253921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.253921+0000","objects_scrubbed":3,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:47.879996+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.081630849000000005,"stat_sum":{"num_bytes":14026159,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":38,"num_read_kb":38413,"num_write":24,"num_write_kb":8854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1484609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"4"},{"start":"72","length":"b"},{"start":"7f","length":"5"},{"start":"85","length":"3"},{"start":"89","length":"2"},{"start":"8c","length":"5"},{"start":"95","length":"3"},{"start":"9d","length":"1"},{"start":"9f","length":"4"},{"start":"a6","length":"1"}]},{"pgid":"4.3","version":"73'19","reported_seq":288,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633474+0000","last_change":"2024-09-21T08:28:53.593536+0000","last_active":"2024-09-21T08:30:10.633474+0000","last_peered":"2024-09-21T08:30:10.633474+0000","last_clean":"2024-09-21T08:28:51.557090+0000","last_became_active":"2024-09-21T08:28:53.593536+0000","last_became_peered":"2024-09-21T08:28:53.593536+0000","last_unstale":"2024-09-21T08:30:10.633474+0000","last_undegraded":"2024-09-21T08:28:53.574981+0000","last_fullsized":"2024-09-21T08:28:53.574872+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:27:55.260127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:55.260127+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908864+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":264,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.636674+0000","last_change":"2024-09-21T08:28:04.726932+0000","last_active":"2024-09-21T08:30:10.636674+0000","last_peered":"2024-09-21T08:30:10.636674+0000","last_clean":"2024-09-21T08:30:10.636674+0000","last_became_active":"2024-09-21T08:28:04.723998+0000","last_became_peered":"2024-09-21T08:28:04.723998+0000","last_unstale":"2024-09-21T08:30:10.636674+0000","last_undegraded":"2024-09-21T08:30:10.636674+0000","last_fullsized":"2024-09-21T08:30:10.636674+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:54.289650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:54.289650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":218,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633885+0000","last_change":"2024-09-21T08:28:53.588949+0000","last_active":"2024-09-21T08:30:10.633885+0000","last_peered":"2024-09-21T08:30:10.633885+0000","last_clean":"2024-09-21T08:28:51.557587+0000","last_became_active":"2024-09-21T08:28:53.588949+0000","last_became_peered":"2024-09-21T08:28:53.588949+0000","last_unstale":"2024-09-21T08:30:10.633885+0000","last_undegraded":"2024-09-21T08:30:10.633885+0000","last_fullsized":"2024-09-21T08:28:53.575963+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.339126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.339126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":267,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.632768+0000","last_change":"2024-09-21T08:28:53.600582+0000","last_active":"2024-09-21T08:30:10.632768+0000","last_peered":"2024-09-21T08:30:10.632768+0000","last_clean":"2024-09-21T08:28:51.584621+0000","last_became_active":"2024-09-21T08:28:53.600582+0000","last_became_peered":"2024-09-21T08:28:53.600582+0000","last_unstale":"2024-09-21T08:30:10.632768+0000","last_undegraded":"2024-09-21T08:30:10.632768+0000","last_fullsized":"2024-09-21T08:28:53.576129+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:46.263984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.263984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"413'179","reported_seq":1848,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.634262+0000","last_change":"2024-09-21T08:28:53.602072+0000","last_active":"2024-09-21T08:30:10.634262+0000","last_peered":"2024-09-21T08:30:10.634262+0000","last_clean":"2024-09-21T08:28:51.587562+0000","last_became_active":"2024-09-21T08:28:53.602072+0000","last_became_peered":"2024-09-21T08:28:53.602072+0000","last_unstale":"2024-09-21T08:30:10.634262+0000","last_undegraded":"2024-09-21T08:28:53.578389+0000","last_fullsized":"2024-09-21T08:28:53.578258+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":44,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.074195671000000005,"stat_sum":{"num_bytes":27815826,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":59405,"num_write":60,"num_write_kb":34065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":25647838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"1"}]},{"pgid":"4.4","version":"340'35","reported_seq":316,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.664733+0000","last_change":"2024-09-21T08:28:53.603157+0000","last_active":"2024-09-21T08:30:10.664733+0000","last_peered":"2024-09-21T08:30:10.664733+0000","last_clean":"2024-09-21T08:28:51.586037+0000","last_became_active":"2024-09-21T08:28:53.603157+0000","last_became_peered":"2024-09-21T08:28:53.603157+0000","last_unstale":"2024-09-21T08:30:10.664733+0000","last_undegraded":"2024-09-21T08:28:53.579248+0000","last_fullsized":"2024-09-21T08:28:53.579125+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'32","last_scrub_stamp":"2024-09-21T08:27:11.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:11.407384+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":659,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.636976+0000","last_change":"2024-09-21T08:28:29.625492+0000","last_active":"2024-09-21T08:30:10.636976+0000","last_peered":"2024-09-21T08:30:10.636976+0000","last_clean":"2024-09-21T08:30:10.636976+0000","last_became_active":"2024-09-21T08:28:29.625170+0000","last_became_peered":"2024-09-21T08:28:29.625170+0000","last_unstale":"2024-09-21T08:30:10.636976+0000","last_undegraded":"2024-09-21T08:30:10.636976+0000","last_fullsized":"2024-09-21T08:30:10.636976+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:14.370256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.370256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":795,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633350+0000","last_change":"2024-09-21T08:28:53.594367+0000","last_active":"2024-09-21T08:30:10.633350+0000","last_peered":"2024-09-21T08:30:10.633350+0000","last_clean":"2024-09-21T08:28:51.557303+0000","last_became_active":"2024-09-21T08:28:53.594367+0000","last_became_peered":"2024-09-21T08:28:53.594367+0000","last_unstale":"2024-09-21T08:30:10.633350+0000","last_undegraded":"2024-09-21T08:28:53.574945+0000","last_fullsized":"2024-09-21T08:28:53.574836+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908435+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":706,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.636674+0000","last_change":"2024-09-21T08:28:53.603129+0000","last_active":"2024-09-21T08:30:10.636674+0000","last_peered":"2024-09-21T08:30:10.636674+0000","last_clean":"2024-09-21T08:28:51.587295+0000","last_became_active":"2024-09-21T08:28:53.603129+0000","last_became_peered":"2024-09-21T08:28:53.603129+0000","last_unstale":"2024-09-21T08:30:10.636674+0000","last_undegraded":"2024-09-21T08:30:10.636674+0000","last_fullsized":"2024-09-21T08:28:53.579036+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:45.237804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.237804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":577,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.637643+0000","last_change":"2024-09-21T08:28:53.600966+0000","last_active":"2024-09-21T08:30:10.637643+0000","last_peered":"2024-09-21T08:30:10.637643+0000","last_clean":"2024-09-21T08:28:02.651561+0000","last_became_active":"2024-09-21T08:28:53.600966+0000","last_became_peered":"2024-09-21T08:28:53.600966+0000","last_unstale":"2024-09-21T08:30:10.637643+0000","last_undegraded":"2024-09-21T08:30:10.637643+0000","last_fullsized":"2024-09-21T08:28:53.579074+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.844729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.844729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:11.999781+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"416'123","reported_seq":1769,"reported_epoch":417,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:10.633122+0000","last_change":"2024-09-21T08:30:09.723513+0000","last_active":"2024-09-21T08:30:10.633122+0000","last_peered":"2024-09-21T08:30:10.633122+0000","last_clean":"2024-09-21T08:30:10.633122+0000","last_became_active":"2024-09-21T08:28:29.626340+0000","last_became_peered":"2024-09-21T08:28:29.626340+0000","last_unstale":"2024-09-21T08:30:10.633122+0000","last_undegraded":"2024-09-21T08:30:10.633122+0000","last_fullsized":"2024-09-21T08:30:10.633122+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'93","last_scrub_stamp":"2024-09-21T08:29:13.277953+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:29:13.277953+0000","objects_scrubbed":3,"log_size":123,"log_dups_size":0,"ondisk_log_size":123,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:17.896279+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.103500309,"stat_sum":{"num_bytes":5203142,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":28,"num_read_kb":35599,"num_write":14,"num_write_kb":7300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":11198550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"4"},{"start":"72","length":"b"},{"start":"7f","length":"5"},{"start":"85","length":"3"},{"start":"89","length":"2"},{"start":"8c","length":"5"},{"start":"95","length":"3"},{"start":"9d","length":"6"},{"start":"a6","length":"1"}]},{"pgid":"4.7","version":"73'13","reported_seq":304,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.636575+0000","last_change":"2024-09-21T08:28:53.595533+0000","last_active":"2024-09-21T08:30:10.636575+0000","last_peered":"2024-09-21T08:30:10.636575+0000","last_clean":"2024-09-21T08:28:51.587779+0000","last_became_active":"2024-09-21T08:28:53.595533+0000","last_became_peered":"2024-09-21T08:28:53.595533+0000","last_unstale":"2024-09-21T08:30:10.636575+0000","last_undegraded":"2024-09-21T08:28:53.580029+0000","last_fullsized":"2024-09-21T08:28:53.579372+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-09-21T08:28:16.386950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.386950+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":278,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.635825+0000","last_change":"2024-09-21T08:29:56.368212+0000","last_active":"2024-09-21T08:30:10.635825+0000","last_peered":"2024-09-21T08:30:10.635825+0000","last_clean":"2024-09-21T08:30:10.635825+0000","last_became_active":"2024-09-21T08:28:29.629320+0000","last_became_peered":"2024-09-21T08:28:29.629320+0000","last_unstale":"2024-09-21T08:30:10.635825+0000","last_undegraded":"2024-09-21T08:30:10.635825+0000","last_fullsized":"2024-09-21T08:30:10.635825+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:56.368144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:29:56.368144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:02.232771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":735,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633575+0000","last_change":"2024-09-21T08:28:53.605131+0000","last_active":"2024-09-21T08:30:10.633575+0000","last_peered":"2024-09-21T08:30:10.633575+0000","last_clean":"2024-09-21T08:28:51.585166+0000","last_became_active":"2024-09-21T08:28:53.605131+0000","last_became_peered":"2024-09-21T08:28:53.605131+0000","last_unstale":"2024-09-21T08:30:10.633575+0000","last_undegraded":"2024-09-21T08:28:53.583253+0000","last_fullsized":"2024-09-21T08:28:53.582934+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:59.262463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:27:59.262463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":724,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.642859+0000","last_change":"2024-09-21T08:28:53.598122+0000","last_active":"2024-09-21T08:30:10.642859+0000","last_peered":"2024-09-21T08:30:10.642859+0000","last_clean":"2024-09-21T08:28:51.556093+0000","last_became_active":"2024-09-21T08:28:53.598122+0000","last_became_peered":"2024-09-21T08:28:53.598122+0000","last_unstale":"2024-09-21T08:30:10.642859+0000","last_undegraded":"2024-09-21T08:30:10.642859+0000","last_fullsized":"2024-09-21T08:28:53.584774+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.374076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.374076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:10.649281+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":621,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634919+0000","last_change":"2024-09-21T08:28:53.587481+0000","last_active":"2024-09-21T08:30:10.634919+0000","last_peered":"2024-09-21T08:30:10.634919+0000","last_clean":"2024-09-21T08:28:51.554685+0000","last_became_active":"2024-09-21T08:28:53.587481+0000","last_became_peered":"2024-09-21T08:28:53.587481+0000","last_unstale":"2024-09-21T08:30:10.634919+0000","last_undegraded":"2024-09-21T08:30:10.634919+0000","last_fullsized":"2024-09-21T08:28:53.575943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:08.831386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:08.831386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"413'305","reported_seq":1817,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633758+0000","last_change":"2024-09-21T08:28:53.600711+0000","last_active":"2024-09-21T08:30:10.633758+0000","last_peered":"2024-09-21T08:30:10.633758+0000","last_clean":"2024-09-21T08:28:52.661593+0000","last_became_active":"2024-09-21T08:28:53.600711+0000","last_became_peered":"2024-09-21T08:28:53.600711+0000","last_unstale":"2024-09-21T08:30:10.633758+0000","last_undegraded":"2024-09-21T08:28:53.576301+0000","last_fullsized":"2024-09-21T08:28:53.576176+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'245","last_scrub_stamp":"2024-09-21T08:26:30.864040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.864040+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.075799959,"stat_sum":{"num_bytes":6305048,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":48,"num_read_kb":51018,"num_write":34,"num_write_kb":8053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1853978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.6","version":"73'12","reported_seq":732,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.635441+0000","last_change":"2024-09-21T08:28:53.603118+0000","last_active":"2024-09-21T08:30:10.635441+0000","last_peered":"2024-09-21T08:30:10.635441+0000","last_clean":"2024-09-21T08:28:51.587443+0000","last_became_active":"2024-09-21T08:28:53.603118+0000","last_became_peered":"2024-09-21T08:28:53.603118+0000","last_unstale":"2024-09-21T08:30:10.635441+0000","last_undegraded":"2024-09-21T08:28:53.582699+0000","last_fullsized":"2024-09-21T08:28:53.582597+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:26:07.384640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:07.384640+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":593,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633686+0000","last_change":"2024-09-21T08:28:53.599951+0000","last_active":"2024-09-21T08:30:10.633686+0000","last_peered":"2024-09-21T08:30:10.633686+0000","last_clean":"2024-09-21T08:28:51.555106+0000","last_became_active":"2024-09-21T08:28:53.599951+0000","last_became_peered":"2024-09-21T08:28:53.599951+0000","last_unstale":"2024-09-21T08:30:10.633686+0000","last_undegraded":"2024-09-21T08:30:10.633686+0000","last_fullsized":"2024-09-21T08:28:53.575796+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.917891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.917891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1149,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633180+0000","last_change":"2024-09-21T08:28:53.594845+0000","last_active":"2024-09-21T08:30:10.633180+0000","last_peered":"2024-09-21T08:30:10.633180+0000","last_clean":"2024-09-21T08:28:52.670294+0000","last_became_active":"2024-09-21T08:28:53.594845+0000","last_became_peered":"2024-09-21T08:28:53.594845+0000","last_unstale":"2024-09-21T08:30:10.633180+0000","last_undegraded":"2024-09-21T08:28:53.577170+0000","last_fullsized":"2024-09-21T08:28:53.575736+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":377,"num_read_kb":372,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":735,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635946+0000","last_change":"2024-09-21T08:28:53.602597+0000","last_active":"2024-09-21T08:30:10.635946+0000","last_peered":"2024-09-21T08:30:10.635946+0000","last_clean":"2024-09-21T08:28:51.556542+0000","last_became_active":"2024-09-21T08:28:53.602597+0000","last_became_peered":"2024-09-21T08:28:53.602597+0000","last_unstale":"2024-09-21T08:30:10.635946+0000","last_undegraded":"2024-09-21T08:30:10.635946+0000","last_fullsized":"2024-09-21T08:28:53.580807+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.812789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.812789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":272,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.637010+0000","last_change":"2024-09-21T08:28:17.424621+0000","last_active":"2024-09-21T08:30:10.637010+0000","last_peered":"2024-09-21T08:30:10.637010+0000","last_clean":"2024-09-21T08:30:10.637010+0000","last_became_active":"2024-09-21T08:28:04.723837+0000","last_became_peered":"2024-09-21T08:28:04.723837+0000","last_unstale":"2024-09-21T08:30:10.637010+0000","last_undegraded":"2024-09-21T08:30:10.637010+0000","last_fullsized":"2024-09-21T08:30:10.637010+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:17.424515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:17.424515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"416'157","reported_seq":1722,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.636351+0000","last_change":"2024-09-21T08:28:53.589874+0000","last_active":"2024-09-21T08:30:10.636351+0000","last_peered":"2024-09-21T08:30:10.636351+0000","last_clean":"2024-09-21T08:28:51.554645+0000","last_became_active":"2024-09-21T08:28:53.589874+0000","last_became_peered":"2024-09-21T08:28:53.589874+0000","last_unstale":"2024-09-21T08:30:10.636351+0000","last_undegraded":"2024-09-21T08:28:53.577395+0000","last_fullsized":"2024-09-21T08:28:53.577254+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'138","last_scrub_stamp":"2024-09-21T08:28:41.855754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:41.855754+0000","objects_scrubbed":1,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":35,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027851600000000001,"stat_sum":{"num_bytes":11714710,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":49,"num_read_kb":63283,"num_write":24,"num_write_kb":8369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":9282889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"4.5","version":"340'22","reported_seq":577,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.637120+0000","last_change":"2024-09-21T08:28:53.600773+0000","last_active":"2024-09-21T08:30:10.637120+0000","last_peered":"2024-09-21T08:30:10.637120+0000","last_clean":"2024-09-21T08:27:13.582840+0000","last_became_active":"2024-09-21T08:28:53.600773+0000","last_became_peered":"2024-09-21T08:28:53.600773+0000","last_unstale":"2024-09-21T08:30:10.637120+0000","last_undegraded":"2024-09-21T08:28:53.577472+0000","last_fullsized":"2024-09-21T08:28:53.577362+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-21T08:28:13.956509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.956509+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":632,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634491+0000","last_change":"2024-09-21T08:28:53.600983+0000","last_active":"2024-09-21T08:30:10.634491+0000","last_peered":"2024-09-21T08:30:10.634491+0000","last_clean":"2024-09-21T08:28:51.558067+0000","last_became_active":"2024-09-21T08:28:53.600983+0000","last_became_peered":"2024-09-21T08:28:53.600983+0000","last_unstale":"2024-09-21T08:30:10.634491+0000","last_undegraded":"2024-09-21T08:30:10.634491+0000","last_fullsized":"2024-09-21T08:28:53.577601+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.045353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.045353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908538+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":978,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633661+0000","last_change":"2024-09-21T08:28:53.604882+0000","last_active":"2024-09-21T08:30:10.633661+0000","last_peered":"2024-09-21T08:30:10.633661+0000","last_clean":"2024-09-21T08:28:51.584594+0000","last_became_active":"2024-09-21T08:28:53.604882+0000","last_became_peered":"2024-09-21T08:28:53.604882+0000","last_unstale":"2024-09-21T08:30:10.633661+0000","last_undegraded":"2024-09-21T08:28:53.575806+0000","last_fullsized":"2024-09-21T08:28:53.575705+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":201,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634568+0000","last_change":"2024-09-21T08:28:53.589976+0000","last_active":"2024-09-21T08:30:10.634568+0000","last_peered":"2024-09-21T08:30:10.634568+0000","last_clean":"2024-09-21T08:28:51.557022+0000","last_became_active":"2024-09-21T08:28:53.589976+0000","last_became_peered":"2024-09-21T08:28:53.589976+0000","last_unstale":"2024-09-21T08:30:10.634568+0000","last_undegraded":"2024-09-21T08:30:10.634568+0000","last_fullsized":"2024-09-21T08:28:53.576557+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":749,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.636172+0000","last_change":"2024-09-21T08:28:53.602915+0000","last_active":"2024-09-21T08:30:10.636172+0000","last_peered":"2024-09-21T08:30:10.636172+0000","last_clean":"2024-09-21T08:28:51.556967+0000","last_became_active":"2024-09-21T08:28:53.602915+0000","last_became_peered":"2024-09-21T08:28:53.602915+0000","last_unstale":"2024-09-21T08:30:10.636172+0000","last_undegraded":"2024-09-21T08:30:10.636172+0000","last_fullsized":"2024-09-21T08:28:53.581134+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908237+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"416'140","reported_seq":560,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.638420+0000","last_change":"2024-09-21T08:28:53.596348+0000","last_active":"2024-09-21T08:30:10.638420+0000","last_peered":"2024-09-21T08:30:10.638420+0000","last_clean":"2024-09-21T08:28:52.661631+0000","last_became_active":"2024-09-21T08:28:53.596348+0000","last_became_peered":"2024-09-21T08:28:53.596348+0000","last_unstale":"2024-09-21T08:30:10.638420+0000","last_undegraded":"2024-09-21T08:28:53.582490+0000","last_fullsized":"2024-09-21T08:28:53.579369+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'93","last_scrub_stamp":"2024-09-21T08:28:11.373358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:11.373358+0000","objects_scrubbed":3,"log_size":140,"log_dups_size":0,"ondisk_log_size":140,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":44,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.070749670000000001,"stat_sum":{"num_bytes":23175321,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":21,"num_read_kb":8940,"num_write":32,"num_write_kb":10379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":6571636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"1"}]},{"pgid":"4.e","version":"73'11","reported_seq":286,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.634057+0000","last_change":"2024-09-21T08:28:53.597299+0000","last_active":"2024-09-21T08:30:10.634057+0000","last_peered":"2024-09-21T08:30:10.634057+0000","last_clean":"2024-09-21T08:28:51.585940+0000","last_became_active":"2024-09-21T08:28:53.597299+0000","last_became_peered":"2024-09-21T08:28:53.597299+0000","last_unstale":"2024-09-21T08:30:10.634057+0000","last_undegraded":"2024-09-21T08:28:53.581018+0000","last_fullsized":"2024-09-21T08:28:53.580915+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:28:14.416729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.416729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":707,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.639115+0000","last_change":"2024-09-21T08:28:53.591487+0000","last_active":"2024-09-21T08:30:10.639115+0000","last_peered":"2024-09-21T08:30:10.639115+0000","last_clean":"2024-09-21T08:28:51.552892+0000","last_became_active":"2024-09-21T08:28:53.591487+0000","last_became_peered":"2024-09-21T08:28:53.591487+0000","last_unstale":"2024-09-21T08:30:10.639115+0000","last_undegraded":"2024-09-21T08:30:10.639115+0000","last_fullsized":"2024-09-21T08:28:53.575527+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:28.347474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:28.347474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:13.631108+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":757,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.636931+0000","last_change":"2024-09-21T08:29:55.150130+0000","last_active":"2024-09-21T08:30:10.636931+0000","last_peered":"2024-09-21T08:30:10.636931+0000","last_clean":"2024-09-21T08:30:10.636931+0000","last_became_active":"2024-09-21T08:28:04.724965+0000","last_became_peered":"2024-09-21T08:28:04.724965+0000","last_unstale":"2024-09-21T08:30:10.636931+0000","last_undegraded":"2024-09-21T08:30:10.636931+0000","last_fullsized":"2024-09-21T08:30:10.636931+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:55.150058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:32.961381+0000","last_clean_scrub_stamp":"2024-09-21T08:29:55.150058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:10.970830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":611,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635481+0000","last_change":"2024-09-21T08:28:53.601979+0000","last_active":"2024-09-21T08:30:10.635481+0000","last_peered":"2024-09-21T08:30:10.635481+0000","last_clean":"2024-09-21T08:28:51.557371+0000","last_became_active":"2024-09-21T08:28:53.601979+0000","last_became_peered":"2024-09-21T08:28:53.601979+0000","last_unstale":"2024-09-21T08:30:10.635481+0000","last_undegraded":"2024-09-21T08:30:10.635481+0000","last_fullsized":"2024-09-21T08:28:53.577942+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.977956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908680+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"256'199","reported_seq":1863,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.642994+0000","last_change":"2024-09-21T08:30:10.643427+0000","last_active":"2024-09-21T08:30:10.642994+0000","last_peered":"2024-09-21T08:30:10.642994+0000","last_clean":"2024-09-21T08:30:10.642994+0000","last_became_active":"2024-09-21T08:28:32.558440+0000","last_became_peered":"2024-09-21T08:28:32.558440+0000","last_unstale":"2024-09-21T08:30:10.642994+0000","last_undegraded":"2024-09-21T08:30:10.642994+0000","last_fullsized":"2024-09-21T08:30:10.642994+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'199","last_scrub_stamp":"2024-09-21T08:28:57.368235+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:57.368235+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.450946+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00076530399999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"10"},{"start":"7f","length":"5"},{"start":"85","length":"3"},{"start":"89","length":"2"},{"start":"8c","length":"6"},{"start":"95","length":"3"},{"start":"9d","length":"6"},{"start":"a6","length":"1"},{"start":"a8","length":"1"},{"start":"aa","length":"1"}]},{"pgid":"4.d","version":"345'25","reported_seq":757,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.639428+0000","last_change":"2024-09-21T08:28:59.242015+0000","last_active":"2024-09-21T08:30:10.639428+0000","last_peered":"2024-09-21T08:30:10.639428+0000","last_clean":"2024-09-21T08:30:10.639428+0000","last_became_active":"2024-09-21T08:28:37.564492+0000","last_became_peered":"2024-09-21T08:28:37.564492+0000","last_unstale":"2024-09-21T08:30:10.639428+0000","last_undegraded":"2024-09-21T08:30:10.639428+0000","last_fullsized":"2024-09-21T08:30:10.639428+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'25","last_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_deep_scrub":"345'25","last_deep_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.241931+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:20.317306+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":266,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635416+0000","last_change":"2024-09-21T08:28:53.599763+0000","last_active":"2024-09-21T08:30:10.635416+0000","last_peered":"2024-09-21T08:30:10.635416+0000","last_clean":"2024-09-21T08:28:51.585236+0000","last_became_active":"2024-09-21T08:28:53.599763+0000","last_became_peered":"2024-09-21T08:28:53.599763+0000","last_unstale":"2024-09-21T08:30:10.635416+0000","last_undegraded":"2024-09-21T08:30:10.635416+0000","last_fullsized":"2024-09-21T08:28:53.586112+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:30.909902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.909902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":272,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634216+0000","last_change":"2024-09-21T08:28:53.601757+0000","last_active":"2024-09-21T08:30:10.634216+0000","last_peered":"2024-09-21T08:30:10.634216+0000","last_clean":"2024-09-21T08:28:51.585900+0000","last_became_active":"2024-09-21T08:28:53.601757+0000","last_became_peered":"2024-09-21T08:28:53.601757+0000","last_unstale":"2024-09-21T08:30:10.634216+0000","last_undegraded":"2024-09-21T08:30:10.634216+0000","last_fullsized":"2024-09-21T08:28:53.576584+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":271,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.635817+0000","last_change":"2024-09-21T08:30:00.276844+0000","last_active":"2024-09-21T08:30:10.635817+0000","last_peered":"2024-09-21T08:30:10.635817+0000","last_clean":"2024-09-21T08:30:10.635817+0000","last_became_active":"2024-09-21T08:28:04.726509+0000","last_became_peered":"2024-09-21T08:28:04.726509+0000","last_unstale":"2024-09-21T08:30:10.635817+0000","last_undegraded":"2024-09-21T08:30:10.635817+0000","last_fullsized":"2024-09-21T08:30:10.635817+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:00.276774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:00.276774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:26.484373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"405'73","reported_seq":1637,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.699504+0000","last_change":"2024-09-21T08:28:53.590273+0000","last_active":"2024-09-21T08:30:10.699504+0000","last_peered":"2024-09-21T08:30:10.699504+0000","last_clean":"2024-09-21T08:28:51.554220+0000","last_became_active":"2024-09-21T08:28:53.590273+0000","last_became_peered":"2024-09-21T08:28:53.590273+0000","last_unstale":"2024-09-21T08:30:10.699504+0000","last_undegraded":"2024-09-21T08:28:53.579532+0000","last_fullsized":"2024-09-21T08:28:53.579403+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:28:10.372149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.372149+0000","objects_scrubbed":1,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.019314938,"stat_sum":{"num_bytes":21987587,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":27,"num_read_kb":28273,"num_write":34,"num_write_kb":15462,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2823609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":10}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.c","version":"73'10","reported_seq":278,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.634376+0000","last_change":"2024-09-21T08:28:53.601905+0000","last_active":"2024-09-21T08:30:10.634376+0000","last_peered":"2024-09-21T08:30:10.634376+0000","last_clean":"2024-09-21T08:28:51.586041+0000","last_became_active":"2024-09-21T08:28:53.601905+0000","last_became_peered":"2024-09-21T08:28:53.601905+0000","last_unstale":"2024-09-21T08:30:10.634376+0000","last_undegraded":"2024-09-21T08:28:53.577157+0000","last_fullsized":"2024-09-21T08:28:53.576937+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:26:33.823950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.823950+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":632,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635617+0000","last_change":"2024-09-21T08:28:53.602716+0000","last_active":"2024-09-21T08:30:10.635617+0000","last_peered":"2024-09-21T08:30:10.635617+0000","last_clean":"2024-09-21T08:28:51.585127+0000","last_became_active":"2024-09-21T08:28:53.602716+0000","last_became_peered":"2024-09-21T08:28:53.602716+0000","last_unstale":"2024-09-21T08:30:10.635617+0000","last_undegraded":"2024-09-21T08:30:10.635617+0000","last_fullsized":"2024-09-21T08:28:53.576785+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:54.216528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:54.216528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"340'10","reported_seq":244,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.637400+0000","last_change":"2024-09-21T08:28:53.589819+0000","last_active":"2024-09-21T08:30:10.637400+0000","last_peered":"2024-09-21T08:30:10.637400+0000","last_clean":"2024-09-21T08:28:51.555429+0000","last_became_active":"2024-09-21T08:28:53.589819+0000","last_became_peered":"2024-09-21T08:28:53.589819+0000","last_unstale":"2024-09-21T08:30:10.637400+0000","last_undegraded":"2024-09-21T08:28:53.578694+0000","last_fullsized":"2024-09-21T08:28:53.577943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:27:51.312273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.312273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":271,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.634253+0000","last_change":"2024-09-21T08:29:40.458127+0000","last_active":"2024-09-21T08:30:10.634253+0000","last_peered":"2024-09-21T08:30:10.634253+0000","last_clean":"2024-09-21T08:30:10.634253+0000","last_became_active":"2024-09-21T08:28:04.722744+0000","last_became_peered":"2024-09-21T08:28:04.722744+0000","last_unstale":"2024-09-21T08:30:10.634253+0000","last_undegraded":"2024-09-21T08:30:10.634253+0000","last_fullsized":"2024-09-21T08:30:10.634253+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:40.458054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:40.458054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:06.465951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"413'324","reported_seq":1831,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.632727+0000","last_change":"2024-09-21T08:28:53.587761+0000","last_active":"2024-09-21T08:30:10.632727+0000","last_peered":"2024-09-21T08:30:10.632727+0000","last_clean":"2024-09-21T08:28:52.588981+0000","last_became_active":"2024-09-21T08:28:53.587761+0000","last_became_peered":"2024-09-21T08:28:53.587761+0000","last_unstale":"2024-09-21T08:30:10.632727+0000","last_undegraded":"2024-09-21T08:28:53.574111+0000","last_fullsized":"2024-09-21T08:28:53.573936+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'178","last_scrub_stamp":"2024-09-21T08:26:28.217842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.217842+0000","objects_scrubbed":4,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":38,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0347803,"stat_sum":{"num_bytes":20649664,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":104,"num_read_kb":134168,"num_write":79,"num_write_kb":30896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":23089101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":13}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"4.b","version":"73'9","reported_seq":785,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.635400+0000","last_change":"2024-09-21T08:28:53.595156+0000","last_active":"2024-09-21T08:30:10.635400+0000","last_peered":"2024-09-21T08:30:10.635400+0000","last_clean":"2024-09-21T08:28:51.585036+0000","last_became_active":"2024-09-21T08:28:53.595156+0000","last_became_peered":"2024-09-21T08:28:53.595156+0000","last_unstale":"2024-09-21T08:30:10.635400+0000","last_undegraded":"2024-09-21T08:28:53.576597+0000","last_fullsized":"2024-09-21T08:28:53.576496+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:25:22.369811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.369811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":771,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.633857+0000","last_change":"2024-09-21T08:28:59.403264+0000","last_active":"2024-09-21T08:30:10.633857+0000","last_peered":"2024-09-21T08:30:10.633857+0000","last_clean":"2024-09-21T08:30:10.633857+0000","last_became_active":"2024-09-21T08:28:29.609526+0000","last_became_peered":"2024-09-21T08:28:29.609526+0000","last_unstale":"2024-09-21T08:30:10.633857+0000","last_undegraded":"2024-09-21T08:30:10.633857+0000","last_fullsized":"2024-09-21T08:30:10.633857+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:59.403194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.403194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:20.528550+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":626,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633027+0000","last_change":"2024-09-21T08:28:53.602532+0000","last_active":"2024-09-21T08:30:10.633027+0000","last_peered":"2024-09-21T08:30:10.633027+0000","last_clean":"2024-09-21T08:28:51.584905+0000","last_became_active":"2024-09-21T08:28:53.602532+0000","last_became_peered":"2024-09-21T08:28:53.602532+0000","last_unstale":"2024-09-21T08:30:10.633027+0000","last_undegraded":"2024-09-21T08:30:10.633027+0000","last_fullsized":"2024-09-21T08:28:53.583376+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":224,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.639426+0000","last_change":"2024-09-21T08:28:53.591119+0000","last_active":"2024-09-21T08:30:10.639426+0000","last_peered":"2024-09-21T08:30:10.639426+0000","last_clean":"2024-09-21T08:28:51.552940+0000","last_became_active":"2024-09-21T08:28:53.591119+0000","last_became_peered":"2024-09-21T08:28:53.591119+0000","last_unstale":"2024-09-21T08:30:10.639426+0000","last_undegraded":"2024-09-21T08:30:10.639426+0000","last_fullsized":"2024-09-21T08:28:53.575921+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:36.308161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:36.308161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:10.649270+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"414'348","reported_seq":2058,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.639196+0000","last_change":"2024-09-21T08:28:53.606879+0000","last_active":"2024-09-21T08:30:10.639196+0000","last_peered":"2024-09-21T08:30:10.639196+0000","last_clean":"2024-09-21T08:28:52.666146+0000","last_became_active":"2024-09-21T08:28:53.606879+0000","last_became_peered":"2024-09-21T08:28:53.606879+0000","last_unstale":"2024-09-21T08:30:10.639196+0000","last_undegraded":"2024-09-21T08:28:53.584154+0000","last_fullsized":"2024-09-21T08:28:53.584033+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":348,"log_dups_size":0,"ondisk_log_size":348,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":45,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0805860919999999,"stat_sum":{"num_bytes":35514492,"num_objects":14,"num_object_clones":10,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":157,"num_read_kb":174544,"num_write":101,"num_write_kb":34472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":24504816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":14}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":286,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.635703+0000","last_change":"2024-09-21T08:28:53.600149+0000","last_active":"2024-09-21T08:30:10.635703+0000","last_peered":"2024-09-21T08:30:10.635703+0000","last_clean":"2024-09-21T08:28:51.585660+0000","last_became_active":"2024-09-21T08:28:53.600149+0000","last_became_peered":"2024-09-21T08:28:53.600149+0000","last_unstale":"2024-09-21T08:30:10.635703+0000","last_undegraded":"2024-09-21T08:28:53.586423+0000","last_fullsized":"2024-09-21T08:28:53.586235+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":633,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.636005+0000","last_change":"2024-09-21T08:28:53.589287+0000","last_active":"2024-09-21T08:30:10.636005+0000","last_peered":"2024-09-21T08:30:10.636005+0000","last_clean":"2024-09-21T08:28:51.555026+0000","last_became_active":"2024-09-21T08:28:53.589287+0000","last_became_peered":"2024-09-21T08:28:53.589287+0000","last_unstale":"2024-09-21T08:30:10.636005+0000","last_undegraded":"2024-09-21T08:30:10.636005+0000","last_fullsized":"2024-09-21T08:28:53.576540+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.901202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.901202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908949+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":721,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.640973+0000","last_change":"2024-09-21T08:28:53.592593+0000","last_active":"2024-09-21T08:30:10.640973+0000","last_peered":"2024-09-21T08:30:10.640973+0000","last_clean":"2024-09-21T08:28:51.554029+0000","last_became_active":"2024-09-21T08:28:53.592593+0000","last_became_peered":"2024-09-21T08:28:53.592593+0000","last_unstale":"2024-09-21T08:30:10.640973+0000","last_undegraded":"2024-09-21T08:30:10.640973+0000","last_fullsized":"2024-09-21T08:28:53.577826+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:15.362045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:15.362045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:13.631072+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":216,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.632429+0000","last_change":"2024-09-21T08:28:53.586960+0000","last_active":"2024-09-21T08:30:10.632429+0000","last_peered":"2024-09-21T08:30:10.632429+0000","last_clean":"2024-09-21T08:28:51.584413+0000","last_became_active":"2024-09-21T08:28:53.586960+0000","last_became_peered":"2024-09-21T08:28:53.586960+0000","last_unstale":"2024-09-21T08:30:10.632429+0000","last_undegraded":"2024-09-21T08:30:10.632429+0000","last_fullsized":"2024-09-21T08:28:53.575166+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"413'106","reported_seq":949,"reported_epoch":417,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:30:10.743688+0000","last_change":"2024-09-21T08:30:10.743755+0000","last_active":"2024-09-21T08:30:10.743688+0000","last_peered":"2024-09-21T08:30:10.743688+0000","last_clean":"2024-09-21T08:30:10.743688+0000","last_became_active":"2024-09-21T08:28:37.567400+0000","last_became_peered":"2024-09-21T08:28:37.567400+0000","last_unstale":"2024-09-21T08:30:10.743688+0000","last_undegraded":"2024-09-21T08:30:10.743688+0000","last_fullsized":"2024-09-21T08:30:10.743688+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'68","last_scrub_stamp":"2024-09-21T08:27:35.317025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:35.317025+0000","objects_scrubbed":0,"log_size":106,"log_dups_size":0,"ondisk_log_size":106,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.10990843,"stat_sum":{"num_bytes":4857421,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":1,"num_read":7,"num_read_kb":3,"num_write":19,"num_write_kb":3134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1183872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"10"},{"start":"7f","length":"5"},{"start":"85","length":"3"},{"start":"89","length":"2"},{"start":"8c","length":"5"},{"start":"95","length":"3"},{"start":"9d","length":"1"},{"start":"9f","length":"4"},{"start":"a6","length":"1"}]},{"pgid":"4.9","version":"73'12","reported_seq":296,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633024+0000","last_change":"2024-09-21T08:28:53.604531+0000","last_active":"2024-09-21T08:30:10.633024+0000","last_peered":"2024-09-21T08:30:10.633024+0000","last_clean":"2024-09-21T08:28:51.584845+0000","last_became_active":"2024-09-21T08:28:53.604531+0000","last_became_peered":"2024-09-21T08:28:53.604531+0000","last_unstale":"2024-09-21T08:30:10.633024+0000","last_undegraded":"2024-09-21T08:28:53.582724+0000","last_fullsized":"2024-09-21T08:28:53.582624+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:41.463820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.463820+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":734,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635057+0000","last_change":"2024-09-21T08:28:53.596741+0000","last_active":"2024-09-21T08:30:10.635057+0000","last_peered":"2024-09-21T08:30:10.635057+0000","last_clean":"2024-09-21T08:28:51.587101+0000","last_became_active":"2024-09-21T08:28:53.596741+0000","last_became_peered":"2024-09-21T08:28:53.596741+0000","last_unstale":"2024-09-21T08:30:10.635057+0000","last_undegraded":"2024-09-21T08:30:10.635057+0000","last_fullsized":"2024-09-21T08:28:53.580812+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:51.769044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.769044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":734,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.634735+0000","last_change":"2024-09-21T08:29:59.267735+0000","last_active":"2024-09-21T08:30:10.634735+0000","last_peered":"2024-09-21T08:30:10.634735+0000","last_clean":"2024-09-21T08:30:10.634735+0000","last_became_active":"2024-09-21T08:28:37.566192+0000","last_became_peered":"2024-09-21T08:28:37.566192+0000","last_unstale":"2024-09-21T08:30:10.634735+0000","last_undegraded":"2024-09-21T08:30:10.634735+0000","last_fullsized":"2024-09-21T08:30:10.634735+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:59.267668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:29:59.267668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.614991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":625,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635795+0000","last_change":"2024-09-21T08:28:53.595264+0000","last_active":"2024-09-21T08:30:10.635795+0000","last_peered":"2024-09-21T08:30:10.635795+0000","last_clean":"2024-09-21T08:28:51.585131+0000","last_became_active":"2024-09-21T08:28:53.595264+0000","last_became_peered":"2024-09-21T08:28:53.595264+0000","last_unstale":"2024-09-21T08:30:10.635795+0000","last_undegraded":"2024-09-21T08:30:10.635795+0000","last_fullsized":"2024-09-21T08:28:53.576894+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"417'676","reported_seq":2475,"reported_epoch":417,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:10.733312+0000","last_change":"2024-09-21T08:30:10.731812+0000","last_active":"2024-09-21T08:30:10.733312+0000","last_peered":"2024-09-21T08:30:10.733312+0000","last_clean":"2024-09-21T08:30:10.733312+0000","last_became_active":"2024-09-21T08:28:07.549709+0000","last_became_peered":"2024-09-21T08:28:07.549709+0000","last_unstale":"2024-09-21T08:30:10.733312+0000","last_undegraded":"2024-09-21T08:30:10.733312+0000","last_fullsized":"2024-09-21T08:30:10.733312+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'577","last_scrub_stamp":"2024-09-21T08:29:32.265609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.265609+0000","objects_scrubbed":10,"log_size":676,"log_dups_size":0,"ondisk_log_size":676,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:39.966572+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.089341034,"stat_sum":{"num_bytes":46024873,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":135,"num_read_kb":138014,"num_write":73,"num_write_kb":27948,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":4385271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"4"},{"start":"72","length":"b"},{"start":"7f","length":"5"},{"start":"85","length":"3"},{"start":"89","length":"2"},{"start":"8c","length":"4"},{"start":"95","length":"3"},{"start":"9d","length":"1"},{"start":"9f","length":"3"},{"start":"a6","length":"1"}]},{"pgid":"4.8","version":"73'15","reported_seq":856,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633446+0000","last_change":"2024-09-21T08:28:53.591779+0000","last_active":"2024-09-21T08:30:10.633446+0000","last_peered":"2024-09-21T08:30:10.633446+0000","last_clean":"2024-09-21T08:28:51.558045+0000","last_became_active":"2024-09-21T08:28:53.591779+0000","last_became_peered":"2024-09-21T08:28:53.591779+0000","last_unstale":"2024-09-21T08:30:10.633446+0000","last_undegraded":"2024-09-21T08:28:53.574592+0000","last_fullsized":"2024-09-21T08:28:53.574472+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908218+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":724,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.635633+0000","last_change":"2024-09-21T08:28:53.595155+0000","last_active":"2024-09-21T08:30:10.635633+0000","last_peered":"2024-09-21T08:30:10.635633+0000","last_clean":"2024-09-21T08:28:51.587656+0000","last_became_active":"2024-09-21T08:28:53.595155+0000","last_became_peered":"2024-09-21T08:28:53.595155+0000","last_unstale":"2024-09-21T08:30:10.635633+0000","last_undegraded":"2024-09-21T08:28:53.579187+0000","last_fullsized":"2024-09-21T08:28:53.579080+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"340'16","reported_seq":847,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633304+0000","last_change":"2024-09-21T08:28:53.604478+0000","last_active":"2024-09-21T08:30:10.633304+0000","last_peered":"2024-09-21T08:30:10.633304+0000","last_clean":"2024-09-21T08:28:51.584962+0000","last_became_active":"2024-09-21T08:28:53.604478+0000","last_became_peered":"2024-09-21T08:28:53.604478+0000","last_unstale":"2024-09-21T08:30:10.633304+0000","last_undegraded":"2024-09-21T08:28:53.580813+0000","last_fullsized":"2024-09-21T08:28:53.580697+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'10","last_scrub_stamp":"2024-09-21T08:25:42.708458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.708458+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":722,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.634119+0000","last_change":"2024-09-21T08:28:55.397079+0000","last_active":"2024-09-21T08:30:10.634119+0000","last_peered":"2024-09-21T08:30:10.634119+0000","last_clean":"2024-09-21T08:30:10.634119+0000","last_became_active":"2024-09-21T08:28:29.619471+0000","last_became_peered":"2024-09-21T08:28:29.619471+0000","last_unstale":"2024-09-21T08:30:10.634119+0000","last_undegraded":"2024-09-21T08:30:10.634119+0000","last_fullsized":"2024-09-21T08:30:10.634119+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:55.396944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:55.396944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:11.999892+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"180'50","reported_seq":1457,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.641367+0000","last_change":"2024-09-21T08:28:53.592066+0000","last_active":"2024-09-21T08:30:10.641367+0000","last_peered":"2024-09-21T08:30:10.641367+0000","last_clean":"2024-09-21T08:28:51.555673+0000","last_became_active":"2024-09-21T08:28:53.592066+0000","last_became_peered":"2024-09-21T08:28:53.592066+0000","last_unstale":"2024-09-21T08:30:10.641367+0000","last_undegraded":"2024-09-21T08:30:10.641367+0000","last_fullsized":"2024-09-21T08:28:53.577992+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:26:50.014780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:50.014780+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":35,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020276460000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"3.10","version":"0'0","reported_seq":520,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.633126+0000","last_change":"2024-09-21T08:29:39.177460+0000","last_active":"2024-09-21T08:30:10.633126+0000","last_peered":"2024-09-21T08:30:10.633126+0000","last_clean":"2024-09-21T08:30:10.633126+0000","last_became_active":"2024-09-21T08:28:29.610987+0000","last_became_peered":"2024-09-21T08:28:29.610987+0000","last_unstale":"2024-09-21T08:30:10.633126+0000","last_undegraded":"2024-09-21T08:30:10.633126+0000","last_fullsized":"2024-09-21T08:30:10.633126+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.177382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:59.628414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":671,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.634649+0000","last_change":"2024-09-21T08:28:53.602197+0000","last_active":"2024-09-21T08:30:10.634649+0000","last_peered":"2024-09-21T08:30:10.634649+0000","last_clean":"2024-09-21T08:28:51.587601+0000","last_became_active":"2024-09-21T08:28:53.602197+0000","last_became_peered":"2024-09-21T08:28:53.602197+0000","last_unstale":"2024-09-21T08:30:10.634649+0000","last_undegraded":"2024-09-21T08:28:53.578880+0000","last_fullsized":"2024-09-21T08:28:53.578772+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-09-21T08:27:30.806522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.806522+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":752,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.637132+0000","last_change":"2024-09-21T08:28:53.600847+0000","last_active":"2024-09-21T08:30:10.637132+0000","last_peered":"2024-09-21T08:30:10.637132+0000","last_clean":"2024-09-21T08:28:51.557075+0000","last_became_active":"2024-09-21T08:28:53.600847+0000","last_became_peered":"2024-09-21T08:28:53.600847+0000","last_unstale":"2024-09-21T08:30:10.637132+0000","last_undegraded":"2024-09-21T08:30:10.637132+0000","last_fullsized":"2024-09-21T08:28:53.578503+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.795325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908589+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":675,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635817+0000","last_change":"2024-09-21T08:28:53.600413+0000","last_active":"2024-09-21T08:30:10.635817+0000","last_peered":"2024-09-21T08:30:10.635817+0000","last_clean":"2024-09-21T08:28:27.554070+0000","last_became_active":"2024-09-21T08:28:53.600413+0000","last_became_peered":"2024-09-21T08:28:53.600413+0000","last_unstale":"2024-09-21T08:30:10.635817+0000","last_undegraded":"2024-09-21T08:30:10.635817+0000","last_fullsized":"2024-09-21T08:28:53.576255+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:10.851510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.851510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908774+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"410'440","reported_seq":2101,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.639454+0000","last_change":"2024-09-21T08:28:53.603910+0000","last_active":"2024-09-21T08:30:10.639454+0000","last_peered":"2024-09-21T08:30:10.639454+0000","last_clean":"2024-09-21T08:28:51.587746+0000","last_became_active":"2024-09-21T08:28:53.603910+0000","last_became_peered":"2024-09-21T08:28:53.603910+0000","last_unstale":"2024-09-21T08:30:10.639454+0000","last_undegraded":"2024-09-21T08:28:53.582931+0000","last_fullsized":"2024-09-21T08:28:53.582809+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'262","last_scrub_stamp":"2024-09-21T08:26:29.875830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.875830+0000","objects_scrubbed":3,"log_size":440,"log_dups_size":0,"ondisk_log_size":440,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":41,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.074129519000000005,"stat_sum":{"num_bytes":36723370,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":19,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":150,"num_read_kb":148874,"num_write":87,"num_write_kb":39971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":8623856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":19}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.16","version":"73'9","reported_seq":224,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.634479+0000","last_change":"2024-09-21T08:28:53.589261+0000","last_active":"2024-09-21T08:30:10.634479+0000","last_peered":"2024-09-21T08:30:10.634479+0000","last_clean":"2024-09-21T08:28:51.555305+0000","last_became_active":"2024-09-21T08:28:53.589261+0000","last_became_peered":"2024-09-21T08:28:53.589261+0000","last_unstale":"2024-09-21T08:30:10.634479+0000","last_undegraded":"2024-09-21T08:28:53.576380+0000","last_fullsized":"2024-09-21T08:28:53.576277+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:10.183035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.183035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908654+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":587,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.632474+0000","last_change":"2024-09-21T08:28:53.587227+0000","last_active":"2024-09-21T08:30:10.632474+0000","last_peered":"2024-09-21T08:30:10.632474+0000","last_clean":"2024-09-21T08:28:51.554027+0000","last_became_active":"2024-09-21T08:28:53.587227+0000","last_became_peered":"2024-09-21T08:28:53.587227+0000","last_unstale":"2024-09-21T08:30:10.632474+0000","last_undegraded":"2024-09-21T08:30:10.632474+0000","last_fullsized":"2024-09-21T08:28:53.573946+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:46.719545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:46.719545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908421+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":272,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.636765+0000","last_change":"2024-09-21T08:28:53.602669+0000","last_active":"2024-09-21T08:30:10.636765+0000","last_peered":"2024-09-21T08:30:10.636765+0000","last_clean":"2024-09-21T08:28:51.587829+0000","last_became_active":"2024-09-21T08:28:53.602669+0000","last_became_peered":"2024-09-21T08:28:53.602669+0000","last_unstale":"2024-09-21T08:30:10.636765+0000","last_undegraded":"2024-09-21T08:30:10.636765+0000","last_fullsized":"2024-09-21T08:28:53.580175+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:05.814753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.814753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":731,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.641695+0000","last_change":"2024-09-21T08:28:53.592444+0000","last_active":"2024-09-21T08:30:10.641695+0000","last_peered":"2024-09-21T08:30:10.641695+0000","last_clean":"2024-09-21T08:28:51.554300+0000","last_became_active":"2024-09-21T08:28:53.592444+0000","last_became_peered":"2024-09-21T08:28:53.592444+0000","last_unstale":"2024-09-21T08:30:10.641695+0000","last_undegraded":"2024-09-21T08:30:10.641695+0000","last_fullsized":"2024-09-21T08:28:53.578030+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:10.076564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.076564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:13.631148+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"412'166","reported_seq":1713,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.700456+0000","last_change":"2024-09-21T08:28:53.589835+0000","last_active":"2024-09-21T08:30:10.700456+0000","last_peered":"2024-09-21T08:30:10.700456+0000","last_clean":"2024-09-21T08:28:51.555217+0000","last_became_active":"2024-09-21T08:28:53.589835+0000","last_became_peered":"2024-09-21T08:28:53.589835+0000","last_unstale":"2024-09-21T08:30:10.700456+0000","last_undegraded":"2024-09-21T08:28:53.577195+0000","last_fullsized":"2024-09-21T08:28:53.577076+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'102","last_scrub_stamp":"2024-09-21T08:26:33.255256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.255256+0000","objects_scrubbed":1,"log_size":166,"log_dups_size":0,"ondisk_log_size":166,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011450665,"stat_sum":{"num_bytes":10084159,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":91,"num_read_kb":132942,"num_write":36,"num_write_kb":9384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4027639,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.15","version":"73'9","reported_seq":816,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.635973+0000","last_change":"2024-09-21T08:28:53.601176+0000","last_active":"2024-09-21T08:30:10.635973+0000","last_peered":"2024-09-21T08:30:10.635973+0000","last_clean":"2024-09-21T08:28:51.554870+0000","last_became_active":"2024-09-21T08:28:53.601176+0000","last_became_peered":"2024-09-21T08:28:53.601176+0000","last_unstale":"2024-09-21T08:30:10.635973+0000","last_undegraded":"2024-09-21T08:28:53.577554+0000","last_fullsized":"2024-09-21T08:28:53.576455+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:28:14.535334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.535334+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.909014+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":699,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633617+0000","last_change":"2024-09-21T08:28:53.605117+0000","last_active":"2024-09-21T08:30:10.633617+0000","last_peered":"2024-09-21T08:30:10.633617+0000","last_clean":"2024-09-21T08:28:51.584984+0000","last_became_active":"2024-09-21T08:28:53.605117+0000","last_became_peered":"2024-09-21T08:28:53.605117+0000","last_unstale":"2024-09-21T08:30:10.633617+0000","last_undegraded":"2024-09-21T08:30:10.633617+0000","last_fullsized":"2024-09-21T08:28:53.584510+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.361096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.361096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"340'16","reported_seq":716,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.634117+0000","last_change":"2024-09-21T08:28:53.600719+0000","last_active":"2024-09-21T08:30:10.634117+0000","last_peered":"2024-09-21T08:30:10.634117+0000","last_clean":"2024-09-21T08:28:51.585913+0000","last_became_active":"2024-09-21T08:28:53.600719+0000","last_became_peered":"2024-09-21T08:28:53.600719+0000","last_unstale":"2024-09-21T08:30:10.634117+0000","last_undegraded":"2024-09-21T08:28:53.576377+0000","last_fullsized":"2024-09-21T08:28:53.576220+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":702,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.643534+0000","last_change":"2024-09-21T08:28:53.597499+0000","last_active":"2024-09-21T08:30:10.643534+0000","last_peered":"2024-09-21T08:30:10.643534+0000","last_clean":"2024-09-21T08:28:51.556626+0000","last_became_active":"2024-09-21T08:28:53.597499+0000","last_became_peered":"2024-09-21T08:28:53.597499+0000","last_unstale":"2024-09-21T08:30:10.643534+0000","last_undegraded":"2024-09-21T08:30:10.643534+0000","last_fullsized":"2024-09-21T08:28:53.585175+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.364388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.364388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:11.644576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"179'63","reported_seq":1011,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635723+0000","last_change":"2024-09-21T08:28:53.590100+0000","last_active":"2024-09-21T08:30:10.635723+0000","last_peered":"2024-09-21T08:30:10.635723+0000","last_clean":"2024-09-21T08:28:51.556223+0000","last_became_active":"2024-09-21T08:28:53.590100+0000","last_became_peered":"2024-09-21T08:28:53.590100+0000","last_unstale":"2024-09-21T08:30:10.635723+0000","last_undegraded":"2024-09-21T08:30:10.635723+0000","last_fullsized":"2024-09-21T08:28:53.579666+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'63","last_scrub_stamp":"2024-09-21T08:28:13.511512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.511512+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010228017000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.14","version":"104'15","reported_seq":278,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.634765+0000","last_change":"2024-09-21T08:28:53.602117+0000","last_active":"2024-09-21T08:30:10.634765+0000","last_peered":"2024-09-21T08:30:10.634765+0000","last_clean":"2024-09-21T08:28:51.587113+0000","last_became_active":"2024-09-21T08:28:53.602117+0000","last_became_peered":"2024-09-21T08:28:53.602117+0000","last_unstale":"2024-09-21T08:30:10.634765+0000","last_undegraded":"2024-09-21T08:28:53.578695+0000","last_fullsized":"2024-09-21T08:28:53.578596+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":712,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633378+0000","last_change":"2024-09-21T08:28:53.600512+0000","last_active":"2024-09-21T08:30:10.633378+0000","last_peered":"2024-09-21T08:30:10.633378+0000","last_clean":"2024-09-21T08:28:27.552826+0000","last_became_active":"2024-09-21T08:28:53.600512+0000","last_became_peered":"2024-09-21T08:28:53.600512+0000","last_unstale":"2024-09-21T08:30:10.633378+0000","last_undegraded":"2024-09-21T08:28:53.574940+0000","last_fullsized":"2024-09-21T08:28:53.574839+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:28:16.875544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.875544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.15","version":"340'14","reported_seq":856,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.637411+0000","last_change":"2024-09-21T08:28:53.601874+0000","last_active":"2024-09-21T08:30:10.637411+0000","last_peered":"2024-09-21T08:30:10.637411+0000","last_clean":"2024-09-21T08:28:51.555323+0000","last_became_active":"2024-09-21T08:28:53.601874+0000","last_became_peered":"2024-09-21T08:28:53.601874+0000","last_unstale":"2024-09-21T08:30:10.637411+0000","last_undegraded":"2024-09-21T08:28:53.578511+0000","last_fullsized":"2024-09-21T08:28:53.577750+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908306+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":583,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635676+0000","last_change":"2024-09-21T08:28:53.600530+0000","last_active":"2024-09-21T08:30:10.635676+0000","last_peered":"2024-09-21T08:30:10.635676+0000","last_clean":"2024-09-21T08:28:51.555446+0000","last_became_active":"2024-09-21T08:28:53.600530+0000","last_became_peered":"2024-09-21T08:28:53.600530+0000","last_unstale":"2024-09-21T08:30:10.635676+0000","last_undegraded":"2024-09-21T08:30:10.635676+0000","last_fullsized":"2024-09-21T08:28:53.577413+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:48.251366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:48.251366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.10","version":"386'155","reported_seq":826,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.638590+0000","last_change":"2024-09-21T08:28:53.604012+0000","last_active":"2024-09-21T08:30:10.638590+0000","last_peered":"2024-09-21T08:30:10.638590+0000","last_clean":"2024-09-21T08:28:53.029949+0000","last_became_active":"2024-09-21T08:28:53.604012+0000","last_became_peered":"2024-09-21T08:28:53.604012+0000","last_unstale":"2024-09-21T08:30:10.638590+0000","last_undegraded":"2024-09-21T08:28:53.580392+0000","last_fullsized":"2024-09-21T08:28:53.580253+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'94","last_scrub_stamp":"2024-09-21T08:26:27.208234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.208234+0000","objects_scrubbed":1,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":41,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.075698227000000007,"stat_sum":{"num_bytes":8362248,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":13075,"num_write":33,"num_write_kb":13153,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10770304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.13","version":"73'11","reported_seq":275,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.637715+0000","last_change":"2024-09-21T08:28:04.797521+0000","last_active":"2024-09-21T08:30:10.637715+0000","last_peered":"2024-09-21T08:30:10.637715+0000","last_clean":"2024-09-21T08:30:10.637715+0000","last_became_active":"2024-09-21T08:28:04.726766+0000","last_became_peered":"2024-09-21T08:28:04.726766+0000","last_unstale":"2024-09-21T08:30:10.637715+0000","last_undegraded":"2024-09-21T08:30:10.637715+0000","last_fullsized":"2024-09-21T08:30:10.637715+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:27:11.919267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:11.919267+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":645,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633438+0000","last_change":"2024-09-21T08:28:53.587802+0000","last_active":"2024-09-21T08:30:10.633438+0000","last_peered":"2024-09-21T08:30:10.633438+0000","last_clean":"2024-09-21T08:28:27.553773+0000","last_became_active":"2024-09-21T08:28:53.587802+0000","last_became_peered":"2024-09-21T08:28:53.587802+0000","last_unstale":"2024-09-21T08:30:10.633438+0000","last_undegraded":"2024-09-21T08:30:10.633438+0000","last_fullsized":"2024-09-21T08:28:53.575420+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:06.889127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:06.889127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908613+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":266,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.637540+0000","last_change":"2024-09-21T08:28:53.600540+0000","last_active":"2024-09-21T08:30:10.637540+0000","last_peered":"2024-09-21T08:30:10.637540+0000","last_clean":"2024-09-21T08:28:51.585918+0000","last_became_active":"2024-09-21T08:28:53.600540+0000","last_became_peered":"2024-09-21T08:28:53.600540+0000","last_unstale":"2024-09-21T08:30:10.637540+0000","last_undegraded":"2024-09-21T08:30:10.637540+0000","last_fullsized":"2024-09-21T08:28:53.586917+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":462,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.633940+0000","last_change":"2024-09-21T08:28:29.611380+0000","last_active":"2024-09-21T08:30:10.633940+0000","last_peered":"2024-09-21T08:30:10.633940+0000","last_clean":"2024-09-21T08:30:10.633940+0000","last_became_active":"2024-09-21T08:28:29.611035+0000","last_became_peered":"2024-09-21T08:28:29.611035+0000","last_unstale":"2024-09-21T08:30:10.633940+0000","last_undegraded":"2024-09-21T08:30:10.633940+0000","last_fullsized":"2024-09-21T08:30:10.633940+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:16.348175+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.348175+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:11.999772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"417'176","reported_seq":1563,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.675200+0000","last_change":"2024-09-21T08:28:53.605541+0000","last_active":"2024-09-21T08:30:10.675200+0000","last_peered":"2024-09-21T08:30:10.675200+0000","last_clean":"2024-09-21T08:28:52.665793+0000","last_became_active":"2024-09-21T08:28:53.605541+0000","last_became_peered":"2024-09-21T08:28:53.605541+0000","last_unstale":"2024-09-21T08:30:10.675200+0000","last_undegraded":"2024-09-21T08:28:53.581187+0000","last_fullsized":"2024-09-21T08:28:53.581055+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'44","last_scrub_stamp":"2024-09-21T08:26:27.882963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.882963+0000","objects_scrubbed":1,"log_size":176,"log_dups_size":0,"ondisk_log_size":176,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":40,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0806039080000001,"stat_sum":{"num_bytes":38095958,"num_objects":16,"num_object_clones":14,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":72,"num_read_kb":68076,"num_write":80,"num_write_kb":37431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":11110430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":16}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"4.12","version":"73'9","reported_seq":289,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.637187+0000","last_change":"2024-09-21T08:28:29.696390+0000","last_active":"2024-09-21T08:30:10.637187+0000","last_peered":"2024-09-21T08:30:10.637187+0000","last_clean":"2024-09-21T08:30:10.637187+0000","last_became_active":"2024-09-21T08:28:29.636404+0000","last_became_peered":"2024-09-21T08:28:29.636404+0000","last_unstale":"2024-09-21T08:30:10.637187+0000","last_undegraded":"2024-09-21T08:30:10.637187+0000","last_fullsized":"2024-09-21T08:30:10.637187+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:14.398007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.398007+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":593,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634679+0000","last_change":"2024-09-21T08:28:53.587397+0000","last_active":"2024-09-21T08:30:10.634679+0000","last_peered":"2024-09-21T08:30:10.634679+0000","last_clean":"2024-09-21T08:28:51.554500+0000","last_became_active":"2024-09-21T08:28:53.587397+0000","last_became_peered":"2024-09-21T08:28:53.587397+0000","last_unstale":"2024-09-21T08:30:10.634679+0000","last_undegraded":"2024-09-21T08:30:10.634679+0000","last_fullsized":"2024-09-21T08:28:53.575265+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:49.790900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:49.790900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":578,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633828+0000","last_change":"2024-09-21T08:28:53.594733+0000","last_active":"2024-09-21T08:30:10.633828+0000","last_peered":"2024-09-21T08:30:10.633828+0000","last_clean":"2024-09-21T08:28:51.584914+0000","last_became_active":"2024-09-21T08:28:53.594733+0000","last_became_peered":"2024-09-21T08:28:53.594733+0000","last_unstale":"2024-09-21T08:30:10.633828+0000","last_undegraded":"2024-09-21T08:30:10.633828+0000","last_fullsized":"2024-09-21T08:28:53.576121+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:13.940546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:13.940546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":684,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.633719+0000","last_change":"2024-09-21T08:30:02.065158+0000","last_active":"2024-09-21T08:30:10.633719+0000","last_peered":"2024-09-21T08:30:10.633719+0000","last_clean":"2024-09-21T08:30:10.633719+0000","last_became_active":"2024-09-21T08:28:29.610998+0000","last_became_peered":"2024-09-21T08:28:29.610998+0000","last_unstale":"2024-09-21T08:30:10.633719+0000","last_undegraded":"2024-09-21T08:30:10.633719+0000","last_fullsized":"2024-09-21T08:30:10.633719+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:02.065082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:02.065082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:24.858435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.12","version":"393'392","reported_seq":2039,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.632731+0000","last_change":"2024-09-21T08:28:53.587108+0000","last_active":"2024-09-21T08:30:10.632731+0000","last_peered":"2024-09-21T08:30:10.632731+0000","last_clean":"2024-09-21T08:28:52.589762+0000","last_became_active":"2024-09-21T08:28:53.587108+0000","last_became_peered":"2024-09-21T08:28:53.587108+0000","last_unstale":"2024-09-21T08:30:10.632731+0000","last_undegraded":"2024-09-21T08:28:53.574106+0000","last_fullsized":"2024-09-21T08:28:53.573924+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'200","last_scrub_stamp":"2024-09-21T08:26:27.417028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.417028+0000","objects_scrubbed":3,"log_size":392,"log_dups_size":0,"ondisk_log_size":392,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.027231871000000001,"stat_sum":{"num_bytes":38730080,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":125,"num_read_kb":145740,"num_write":87,"num_write_kb":43667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":8981699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":18}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.11","version":"73'11","reported_seq":307,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.634084+0000","last_change":"2024-09-21T08:28:53.589901+0000","last_active":"2024-09-21T08:30:10.634084+0000","last_peered":"2024-09-21T08:30:10.634084+0000","last_clean":"2024-09-21T08:28:51.555481+0000","last_became_active":"2024-09-21T08:28:53.589901+0000","last_became_peered":"2024-09-21T08:28:53.589901+0000","last_unstale":"2024-09-21T08:30:10.634084+0000","last_undegraded":"2024-09-21T08:28:53.577098+0000","last_fullsized":"2024-09-21T08:28:53.576477+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908265+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":748,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.635922+0000","last_change":"2024-09-21T08:28:53.600628+0000","last_active":"2024-09-21T08:30:10.635922+0000","last_peered":"2024-09-21T08:30:10.635922+0000","last_clean":"2024-09-21T08:28:51.555529+0000","last_became_active":"2024-09-21T08:28:53.600628+0000","last_became_peered":"2024-09-21T08:28:53.600628+0000","last_unstale":"2024-09-21T08:30:10.635922+0000","last_undegraded":"2024-09-21T08:30:10.635922+0000","last_fullsized":"2024-09-21T08:28:53.577666+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908409+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":680,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.634625+0000","last_change":"2024-09-21T08:28:53.589537+0000","last_active":"2024-09-21T08:30:10.634625+0000","last_peered":"2024-09-21T08:30:10.634625+0000","last_clean":"2024-09-21T08:28:27.552805+0000","last_became_active":"2024-09-21T08:28:53.589537+0000","last_became_peered":"2024-09-21T08:28:53.589537+0000","last_unstale":"2024-09-21T08:30:10.634625+0000","last_undegraded":"2024-09-21T08:30:10.634625+0000","last_fullsized":"2024-09-21T08:28:53.575446+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:11.999826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":476,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.634916+0000","last_change":"2024-09-21T08:29:58.167276+0000","last_active":"2024-09-21T08:30:10.634916+0000","last_peered":"2024-09-21T08:30:10.634916+0000","last_clean":"2024-09-21T08:30:10.634916+0000","last_became_active":"2024-09-21T08:28:29.619814+0000","last_became_peered":"2024-09-21T08:28:29.619814+0000","last_unstale":"2024-09-21T08:30:10.634916+0000","last_undegraded":"2024-09-21T08:30:10.634916+0000","last_fullsized":"2024-09-21T08:30:10.634916+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:58.167184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.167184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:02.224715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"338'55","reported_seq":1547,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.632917+0000","last_change":"2024-09-21T08:28:53.588606+0000","last_active":"2024-09-21T08:30:10.632917+0000","last_peered":"2024-09-21T08:30:10.632917+0000","last_clean":"2024-09-21T08:28:52.586222+0000","last_became_active":"2024-09-21T08:28:53.588606+0000","last_became_peered":"2024-09-21T08:28:53.588606+0000","last_unstale":"2024-09-21T08:30:10.632917+0000","last_undegraded":"2024-09-21T08:28:53.574192+0000","last_fullsized":"2024-09-21T08:28:53.574061+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'22","last_scrub_stamp":"2024-09-21T08:26:29.353795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.353795+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019195647,"stat_sum":{"num_bytes":9586927,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":45,"num_read_kb":64333,"num_write":19,"num_write_kb":14200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5630444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.10","version":"73'4","reported_seq":277,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.635449+0000","last_change":"2024-09-21T08:29:39.416471+0000","last_active":"2024-09-21T08:30:10.635449+0000","last_peered":"2024-09-21T08:30:10.635449+0000","last_clean":"2024-09-21T08:30:10.635449+0000","last_became_active":"2024-09-21T08:28:04.724607+0000","last_became_peered":"2024-09-21T08:28:04.724607+0000","last_unstale":"2024-09-21T08:30:10.635449+0000","last_undegraded":"2024-09-21T08:30:10.635449+0000","last_fullsized":"2024-09-21T08:30:10.635449+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-09-21T08:29:39.416397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.416397+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:58.505501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":708,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.635633+0000","last_change":"2024-09-21T08:28:38.272400+0000","last_active":"2024-09-21T08:30:10.635633+0000","last_peered":"2024-09-21T08:30:10.635633+0000","last_clean":"2024-09-21T08:30:10.635633+0000","last_became_active":"2024-09-21T08:28:29.611592+0000","last_became_peered":"2024-09-21T08:28:29.611592+0000","last_unstale":"2024-09-21T08:30:10.635633+0000","last_undegraded":"2024-09-21T08:30:10.635633+0000","last_fullsized":"2024-09-21T08:30:10.635633+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.272319+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.272319+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:11.999797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":659,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.632704+0000","last_change":"2024-09-21T08:28:53.589633+0000","last_active":"2024-09-21T08:30:10.632704+0000","last_peered":"2024-09-21T08:30:10.632704+0000","last_clean":"2024-09-21T08:28:51.584672+0000","last_became_active":"2024-09-21T08:28:53.589633+0000","last_became_peered":"2024-09-21T08:28:53.589633+0000","last_unstale":"2024-09-21T08:30:10.632704+0000","last_undegraded":"2024-09-21T08:30:10.632704+0000","last_fullsized":"2024-09-21T08:28:53.574228+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":733,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633948+0000","last_change":"2024-09-21T08:28:53.605159+0000","last_active":"2024-09-21T08:30:10.633948+0000","last_peered":"2024-09-21T08:30:10.633948+0000","last_clean":"2024-09-21T08:28:51.585029+0000","last_became_active":"2024-09-21T08:28:53.605159+0000","last_became_peered":"2024-09-21T08:28:53.605159+0000","last_unstale":"2024-09-21T08:30:10.633948+0000","last_undegraded":"2024-09-21T08:30:10.633948+0000","last_fullsized":"2024-09-21T08:28:53.584509+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.736816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.736816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"407'212","reported_seq":1798,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.633914+0000","last_change":"2024-09-21T08:28:53.600314+0000","last_active":"2024-09-21T08:30:10.633914+0000","last_peered":"2024-09-21T08:30:10.633914+0000","last_clean":"2024-09-21T08:28:51.556402+0000","last_became_active":"2024-09-21T08:28:53.600314+0000","last_became_peered":"2024-09-21T08:28:53.600314+0000","last_unstale":"2024-09-21T08:30:10.633914+0000","last_undegraded":"2024-09-21T08:28:53.575998+0000","last_fullsized":"2024-09-21T08:28:53.575875+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'167","last_scrub_stamp":"2024-09-21T08:27:49.143452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.143452+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":39,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.015256377999999999,"stat_sum":{"num_bytes":17321215,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":73,"num_read_kb":97090,"num_write":25,"num_write_kb":14712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1129140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"6.1d","version":"0'0","reported_seq":263,"reported_epoch":417,"state":"active+clean","last_fresh":"2024-09-21T08:30:10.634232+0000","last_change":"2024-09-21T08:28:04.726882+0000","last_active":"2024-09-21T08:30:10.634232+0000","last_peered":"2024-09-21T08:30:10.634232+0000","last_clean":"2024-09-21T08:30:10.634232+0000","last_became_active":"2024-09-21T08:28:04.724461+0000","last_became_peered":"2024-09-21T08:28:04.724461+0000","last_unstale":"2024-09-21T08:30:10.634232+0000","last_undegraded":"2024-09-21T08:30:10.634232+0000","last_fullsized":"2024-09-21T08:30:10.634232+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:13.999162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:13.999162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":587,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633014+0000","last_change":"2024-09-21T08:28:53.585838+0000","last_active":"2024-09-21T08:30:10.633014+0000","last_peered":"2024-09-21T08:30:10.633014+0000","last_clean":"2024-09-21T08:28:51.584499+0000","last_became_active":"2024-09-21T08:28:53.585838+0000","last_became_peered":"2024-09-21T08:28:53.585838+0000","last_unstale":"2024-09-21T08:30:10.633014+0000","last_undegraded":"2024-09-21T08:30:10.633014+0000","last_fullsized":"2024-09-21T08:28:53.575428+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":783,"reported_epoch":417,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:10.636129+0000","last_change":"2024-09-21T08:28:53.603028+0000","last_active":"2024-09-21T08:30:10.636129+0000","last_peered":"2024-09-21T08:30:10.636129+0000","last_clean":"2024-09-21T08:28:51.556942+0000","last_became_active":"2024-09-21T08:28:53.603028+0000","last_became_peered":"2024-09-21T08:28:53.603028+0000","last_unstale":"2024-09-21T08:30:10.636129+0000","last_undegraded":"2024-09-21T08:28:53.581121+0000","last_fullsized":"2024-09-21T08:28:53.581005+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:14.908292+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":165,"reported_epoch":417,"state":"active+undersized","last_fresh":"2024-09-21T08:30:10.633887+0000","last_change":"2024-09-21T08:28:53.601908+0000","last_active":"2024-09-21T08:30:10.633887+0000","last_peered":"2024-09-21T08:30:10.633887+0000","last_clean":"2024-09-21T08:23:33.544062+0000","last_became_active":"2024-09-21T08:28:53.601908+0000","last_became_peered":"2024-09-21T08:28:53.601908+0000","last_unstale":"2024-09-21T08:30:10.633887+0000","last_undegraded":"2024-09-21T08:30:10.633887+0000","last_fullsized":"2024-09-21T08:28:53.577043+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.872291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.872291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":532357196,"num_objects":248,"num_object_clones":199,"num_object_copies":744,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":195,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":248,"num_whiteouts":13,"num_read":1633,"num_read_kb":1818375,"num_write":1125,"num_write_kb":476769,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":207157568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1220403200,"data_stored":1731011219,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7279,"ondisk_log_size":7279,"up":79,"acting":79,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":76,"acting":76,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":67,"acting":67,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":676,"num_read_kb":433,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":10760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":17316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":74,"acting":74,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4282,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":8622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":70,"acting":70,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":377,"num_read_kb":372,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":6,"acting":6,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070533,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":228312,"kb_used_data":174916,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93516840,"statfs":{"total":95995035648,"available":95761244160,"internally_reserved":0,"allocated":179113984,"data_stored":227784827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397595,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659230,"num_pgs":124,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":295040,"kb_used_data":241636,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93450112,"statfs":{"total":95995035648,"available":95692914688,"internally_reserved":0,"allocated":247435264,"data_stored":364322075,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986276,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182504,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411438,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":392828,"kb_used_data":334184,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93352324,"statfs":{"total":95995035648,"available":95592779776,"internally_reserved":0,"allocated":342204416,"data_stored":520939695,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574962,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308804,"kb_used_data":277612,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93436348,"statfs":{"total":95995035648,"available":95678820352,"internally_reserved":0,"allocated":284274688,"data_stored":420846283,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":3482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":3014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74776576,"data_stored":86348592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":283619328,"data_stored":422891788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":339664896,"data_stored":516477286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70848512,"data_stored":84468350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":247263232,"data_stored":365912478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":176136192,"data_stored":227042310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:30:14.865 INFO:tasks.thrashosds.thrasher:Setting 7.1d to [4, 3, 1] 2024-09-21T08:30:14.865 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap', '7.1d', '4', '3', '1'] 2024-09-21T08:30:14.866 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pg-upmap 7.1d 4 3 1 2024-09-21T08:30:14.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:14.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:14.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:14.963 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:14 smithi165 ceph-mon[32893]: osdmap e420: 8 total, 6 up, 4 in 2024-09-21T08:30:15.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:15.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:15.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:15.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:15.146 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:15.146 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:30:15.147 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:30:15.147 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:30:15.147 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:30:15.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:15.217 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:15.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:15.218 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:15.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:15.340 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:15.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:15.342 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:15.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:15.444 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:30:15.088+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:30:15.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:15.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:15.469 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:15.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:15.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:15.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:15.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:1666: done (1 left) 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:1667: done (0 left) 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:1668: rollback oid 12 current snap is 174 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 12 to 145 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:1669: read oid 4 snap -1 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:1669: expect (ObjNum 546 snap 163 seq_num 546) 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:1670: rollback oid 46 current snap is 174 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 46 to 170 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:1671: copy_from oid 7 from oid 18 current snap is 174 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:1672: write oid 14 current snap is 174 2024-09-21T08:30:15.598 INFO:tasks.rados.rados.0.smithi137.stdout:1672: seq_num 596 ranges {416972=723938,1743268=273704} 2024-09-21T08:30:15.608 INFO:tasks.rados.rados.0.smithi137.stdout:1672: writing smithi13779221-14 from 416972 to 1140910 tid 1 2024-09-21T08:30:15.613 INFO:tasks.rados.rados.0.smithi137.stdout:1672: writing smithi13779221-14 from 1743268 to 2016972 tid 2 2024-09-21T08:30:15.613 INFO:tasks.rados.rados.0.smithi137.stdout:1673: write oid 1 current snap is 174 2024-09-21T08:30:15.613 INFO:tasks.rados.rados.0.smithi137.stdout:1673: seq_num 597 ranges {433779=799187,1801750=232029} 2024-09-21T08:30:15.621 INFO:tasks.rados.rados.0.smithi137.stdout:1673: writing smithi13779221-1 from 433779 to 1232966 tid 1 2024-09-21T08:30:15.626 INFO:tasks.rados.rados.0.smithi137.stdout:1673: writing smithi13779221-1 from 1801750 to 2033779 tid 2 2024-09-21T08:30:15.626 INFO:tasks.rados.rados.0.smithi137.stdout:1674: snap_remove snap 152 2024-09-21T08:30:15.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:15.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:15.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:15.720 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-mon[29634]: from='client.25681 v1:172.21.15.137:0/23118493' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:30:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-mon[29634]: pgmap v536: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 508 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 29 MiB/s rd, 3.5 MiB/s wr, 33 op/s; 341/1341 objects degraded (25.429%) 2024-09-21T08:30:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-mon[29634]: osdmap e421: 8 total, 6 up, 4 in 2024-09-21T08:30:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:15.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-mon[41044]: from='client.25681 v1:172.21.15.137:0/23118493' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:30:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-mon[41044]: pgmap v536: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 508 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 29 MiB/s rd, 3.5 MiB/s wr, 33 op/s; 341/1341 objects degraded (25.429%) 2024-09-21T08:30:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-mon[41044]: osdmap e421: 8 total, 6 up, 4 in 2024-09-21T08:30:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:15.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:15 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:15.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:15.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:15.843 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:15.843 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:15.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:15.970 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:15.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:15.971 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:15 smithi165 ceph-mon[32893]: from='client.25681 v1:172.21.15.137:0/23118493' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:30:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:15 smithi165 ceph-mon[32893]: pgmap v536: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 508 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 29 MiB/s rd, 3.5 MiB/s wr, 33 op/s; 341/1341 objects degraded (25.429%) 2024-09-21T08:30:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:15 smithi165 ceph-mon[32893]: osdmap e421: 8 total, 6 up, 4 in 2024-09-21T08:30:16.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:15 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:16.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:15 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:16.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T08:30:16.071 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:16.116 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:16.116 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:16.117 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:16.218 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:16.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:16.244 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:16.245 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:16.258 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:16.258 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:16.258 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:16.274 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:16.274 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:16.295 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:16.295 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:16.312 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:16.312 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:16.328 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:16.328 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:16.345 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:16.345 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:16.345 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:16.345 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:16.371 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:16.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:16.372 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:16.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:16.495 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:16.495 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:16.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:16.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:16.611 INFO:tasks.rados.rados.0.smithi137.stdout:1674: done (6 left) 2024-09-21T08:30:16.612 INFO:tasks.rados.rados.0.smithi137.stdout:1675: delete oid 19 current snap is 174 2024-09-21T08:30:16.612 INFO:tasks.rados.rados.0.smithi137.stdout:1670: finishing rollback tid 1 to smithi13779221-46 2024-09-21T08:30:16.612 INFO:tasks.rados.rados.0.smithi137.stdout:1670: finishing rollback tid 2 to smithi13779221-46 2024-09-21T08:30:16.612 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 687 (ObjNum 499 snap 140 seq_num 499) dirty exists 2024-09-21T08:30:16.612 INFO:tasks.rados.rados.0.smithi137.stdout:1669: expect (ObjNum 546 snap 163 seq_num 546) 2024-09-21T08:30:16.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:16.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:16.624 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:16.721 INFO:tasks.rados.rados.0.smithi137.stdout:1668: finishing rollback tid 0 to smithi13779221-12 2024-09-21T08:30:16.721 INFO:tasks.rados.rados.0.smithi137.stdout:1668: finishing rollback tid 1 to smithi13779221-12 2024-09-21T08:30:16.721 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 395 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-21T08:30:16.721 INFO:tasks.rados.rados.0.smithi137.stdout:1672: finishing write tid 1 to smithi13779221-14 2024-09-21T08:30:16.721 INFO:tasks.rados.rados.0.smithi137.stdout:1668: done (6 left) 2024-09-21T08:30:16.721 INFO:tasks.rados.rados.0.smithi137.stdout:1669: done (5 left) 2024-09-21T08:30:16.721 INFO:tasks.rados.rados.0.smithi137.stdout:1670: done (4 left) 2024-09-21T08:30:16.721 INFO:tasks.rados.rados.0.smithi137.stdout:1675: done (3 left) 2024-09-21T08:30:16.722 INFO:tasks.rados.rados.0.smithi137.stdout:1676: write oid 13 current snap is 174 2024-09-21T08:30:16.722 INFO:tasks.rados.rados.0.smithi137.stdout:1672: finishing write tid 2 to smithi13779221-14 2024-09-21T08:30:16.722 INFO:tasks.rados.rados.0.smithi137.stdout:1672: finishing write tid 3 to smithi13779221-14 2024-09-21T08:30:16.722 INFO:tasks.rados.rados.0.smithi137.stdout:1676: seq_num 599 ranges {467393=549685,1802361=265032} 2024-09-21T08:30:16.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:16.726 INFO:tasks.rados.rados.0.smithi137.stdout:1676: writing smithi13779221-13 from 467393 to 1017078 tid 1 2024-09-21T08:30:16.732 INFO:tasks.rados.rados.0.smithi137.stdout:1676: writing smithi13779221-13 from 1802361 to 2067393 tid 2 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1677: delete oid 35 current snap is 174 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1672: finishing write tid 4 to smithi13779221-14 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1672: oid 14 updating version 0 to 385 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1672: oid 14 version 385 is already newer than 383 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1672: oid 14 version 385 is already newer than 384 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 385 (ObjNum 596 snap 174 seq_num 596) dirty exists 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1672: left oid 14 (ObjNum 596 snap 174 seq_num 596) 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1671: finishing copy_from racing read to smithi13779221-7 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1671: finishing copy_from to smithi13779221-7 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 229 (ObjNum 590 snap 172 seq_num 590) dirty exists 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1673: finishing write tid 1 to smithi13779221-1 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1673: finishing write tid 2 to smithi13779221-1 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1673: finishing write tid 3 to smithi13779221-1 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1673: finishing write tid 4 to smithi13779221-1 2024-09-21T08:30:16.733 INFO:tasks.rados.rados.0.smithi137.stdout:1673: oid 1 updating version 0 to 536 2024-09-21T08:30:16.734 INFO:tasks.rados.rados.0.smithi137.stdout:1673: oid 1 updating version 536 to 538 2024-09-21T08:30:16.734 INFO:tasks.rados.rados.0.smithi137.stdout:1673: oid 1 version 538 is already newer than 537 2024-09-21T08:30:16.734 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 538 (ObjNum 597 snap 174 seq_num 597) dirty exists 2024-09-21T08:30:16.734 INFO:tasks.rados.rados.0.smithi137.stdout:1673: left oid 1 (ObjNum 597 snap 174 seq_num 597) 2024-09-21T08:30:16.734 INFO:tasks.rados.rados.0.smithi137.stdout:1676: finishing write tid 1 to smithi13779221-13 2024-09-21T08:30:16.734 INFO:tasks.rados.rados.0.smithi137.stdout:1671: done (4 left) 2024-09-21T08:30:16.734 INFO:tasks.rados.rados.0.smithi137.stdout:1672: done (3 left) 2024-09-21T08:30:16.735 INFO:tasks.rados.rados.0.smithi137.stdout:1673: done (2 left) 2024-09-21T08:30:16.735 INFO:tasks.rados.rados.0.smithi137.stdout:1677: done (1 left) 2024-09-21T08:30:16.735 INFO:tasks.rados.rados.0.smithi137.stdout:1678: snap_remove snap 137 2024-09-21T08:30:16.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:16.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:16.750 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:16.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:16.872 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:16.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:16.873 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:16.974 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:16.999 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:17.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:17.000 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:16 smithi165 ceph-mon[32893]: osdmap e422: 8 total, 6 up, 4 in 2024-09-21T08:30:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:16 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/792101599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:16 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 341/1341 objects degraded (25.429%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:17.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:16 smithi137 ceph-mon[29634]: osdmap e422: 8 total, 6 up, 4 in 2024-09-21T08:30:17.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:16 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/792101599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:17.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:16 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 341/1341 objects degraded (25.429%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:17.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:16 smithi137 ceph-mon[41044]: osdmap e422: 8 total, 6 up, 4 in 2024-09-21T08:30:17.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:16 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/792101599' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:17.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:16 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 341/1341 objects degraded (25.429%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:17.099 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:17.100 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:17.101 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:17.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:17.127 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:17.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:17.228 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:17.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:17.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:17.254 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:17.355 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:17.379 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:17.380 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:17.380 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:17.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:17.381 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:17.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:17.508 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:17.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:17.509 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:17.609 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:17.611 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:30:17.617 INFO:tasks.rados.rados.0.smithi137.stdout:1678: done (1 left) 2024-09-21T08:30:17.618 INFO:tasks.rados.rados.0.smithi137.stdout:1679: read oid 17 snap -1 2024-09-21T08:30:17.618 INFO:tasks.rados.rados.0.smithi137.stdout:1679: expect (ObjNum 552 snap 166 seq_num 552) 2024-09-21T08:30:17.618 INFO:tasks.rados.rados.0.smithi137.stdout:1680: write oid 50 current snap is 174 2024-09-21T08:30:17.618 INFO:tasks.rados.rados.0.smithi137.stdout:1680: seq_num 601 ranges {501007=416536,1635678=465329} 2024-09-21T08:30:17.623 INFO:tasks.rados.rados.0.smithi137.stdout:1680: writing smithi13779221-50 from 501007 to 917543 tid 1 2024-09-21T08:30:17.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:17 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:30:17.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:17 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:30:17.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:17 smithi137 ceph-mon[29634]: pgmap v539: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 509 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 9 op/s; 344/1341 objects degraded (25.652%) 2024-09-21T08:30:17.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:17 smithi137 ceph-mon[29634]: osdmap e423: 8 total, 6 up, 4 in 2024-09-21T08:30:17.628 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:17 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1928892060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:30:17.629 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:17 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:30:17.629 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:17 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:30:17.629 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:17 smithi137 ceph-mon[41044]: pgmap v539: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 509 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 9 op/s; 344/1341 objects degraded (25.652%) 2024-09-21T08:30:17.629 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:17 smithi137 ceph-mon[41044]: osdmap e423: 8 total, 6 up, 4 in 2024-09-21T08:30:17.629 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:17 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1928892060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:30:17.631 INFO:tasks.rados.rados.0.smithi137.stdout:1680: writing smithi13779221-50 from 1635678 to 2101007 tid 2 2024-09-21T08:30:17.631 INFO:tasks.rados.rados.0.smithi137.stdout:1681: snap_remove snap 169 2024-09-21T08:30:17.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:17.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:17.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:17.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:17.764 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:17.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:17.765 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:17.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:17.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:17.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:17.890 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:17.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:17 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:30:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:17 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:30:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:17 smithi165 ceph-mon[32893]: pgmap v539: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 509 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 9 op/s; 344/1341 objects degraded (25.652%) 2024-09-21T08:30:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:17 smithi165 ceph-mon[32893]: osdmap e423: 8 total, 6 up, 4 in 2024-09-21T08:30:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:17 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1928892060' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:30:18.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:18.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:18.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:18.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:18.141 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:18.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:18.142 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:18.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:18.265 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:18.266 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:18.266 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:18.272 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:18.272 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:18.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:18.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:18.393 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:18.394 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:18.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:18.517 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:18.517 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:18.518 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:18.616 INFO:teuthology.orchestra.run.smithi137.stderr:set 7.1d pg_upmap mapping to [4,3,1] 2024-09-21T08:30:18.619 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:18.620 INFO:tasks.rados.rados.0.smithi137.stdout:1681: done (3 left) 2024-09-21T08:30:18.620 INFO:tasks.rados.rados.0.smithi137.stdout:1682: snap_create 2024-09-21T08:30:18.620 INFO:tasks.rados.rados.0.smithi137.stdout:1676: finishing write tid 2 to smithi13779221-13 2024-09-21T08:30:18.620 INFO:tasks.rados.rados.0.smithi137.stdout:1676: finishing write tid 3 to smithi13779221-13 2024-09-21T08:30:18.620 INFO:tasks.rados.rados.0.smithi137.stdout:1676: finishing write tid 4 to smithi13779221-13 2024-09-21T08:30:18.620 INFO:tasks.rados.rados.0.smithi137.stdout:1676: oid 13 updating version 0 to 691 2024-09-21T08:30:18.620 INFO:tasks.rados.rados.0.smithi137.stdout:1676: oid 13 updating version 691 to 692 2024-09-21T08:30:18.620 INFO:tasks.rados.rados.0.smithi137.stdout:1676: oid 13 updating version 692 to 693 2024-09-21T08:30:18.621 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 693 (ObjNum 599 snap 174 seq_num 599) dirty exists 2024-09-21T08:30:18.621 INFO:tasks.rados.rados.0.smithi137.stdout:1676: left oid 13 (ObjNum 599 snap 174 seq_num 599) 2024-09-21T08:30:18.621 INFO:tasks.rados.rados.0.smithi137.stdout:1679: expect (ObjNum 552 snap 166 seq_num 552) 2024-09-21T08:30:18.645 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:18.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:18.646 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:18.732 INFO:tasks.rados.rados.0.smithi137.stdout:1680: finishing write tid 1 to smithi13779221-50 2024-09-21T08:30:18.732 INFO:tasks.rados.rados.0.smithi137.stdout:1680: finishing write tid 2 to smithi13779221-50 2024-09-21T08:30:18.732 INFO:tasks.rados.rados.0.smithi137.stdout:1680: finishing write tid 3 to smithi13779221-50 2024-09-21T08:30:18.733 INFO:tasks.rados.rados.0.smithi137.stdout:1680: finishing write tid 4 to smithi13779221-50 2024-09-21T08:30:18.733 INFO:tasks.rados.rados.0.smithi137.stdout:1680: oid 50 updating version 0 to 398 2024-09-21T08:30:18.733 INFO:tasks.rados.rados.0.smithi137.stdout:1680: oid 50 version 398 is already newer than 397 2024-09-21T08:30:18.733 INFO:tasks.rados.rados.0.smithi137.stdout:1680: oid 50 updating version 398 to 399 2024-09-21T08:30:18.733 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 399 (ObjNum 601 snap 174 seq_num 601) dirty exists 2024-09-21T08:30:18.733 INFO:tasks.rados.rados.0.smithi137.stdout:1680: left oid 50 (ObjNum 601 snap 174 seq_num 601) 2024-09-21T08:30:18.747 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:18.768 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:18.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:18.769 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:18.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:18.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:18.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:18.892 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:18.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:19.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:18 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:19.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:18 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1928892060' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:30:19.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:18 smithi165 ceph-mon[32893]: osdmap e424: 8 total, 6 up, 4 in 2024-09-21T08:30:19.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:18 smithi165 ceph-mon[32893]: 7.e scrub starts 2024-09-21T08:30:19.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:18 smithi165 ceph-mon[32893]: 7.e scrub ok 2024-09-21T08:30:19.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:18 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1873665245' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.1d", "id": [4, 3, 1]}]: dispatch 2024-09-21T08:30:19.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:19.015 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:19.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:19.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:19.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1928892060' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:30:19.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[29634]: osdmap e424: 8 total, 6 up, 4 in 2024-09-21T08:30:19.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[29634]: 7.e scrub starts 2024-09-21T08:30:19.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[29634]: 7.e scrub ok 2024-09-21T08:30:19.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1873665245' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.1d", "id": [4, 3, 1]}]: dispatch 2024-09-21T08:30:19.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:19.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1928892060' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:30:19.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[41044]: osdmap e424: 8 total, 6 up, 4 in 2024-09-21T08:30:19.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[41044]: 7.e scrub starts 2024-09-21T08:30:19.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[41044]: 7.e scrub ok 2024-09-21T08:30:19.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:18 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1873665245' entity='client.admin' cmd=[{"prefix": "osd pg-upmap", "pgid": "7.1d", "id": [4, 3, 1]}]: dispatch 2024-09-21T08:30:19.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T08:30:19.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:30:19.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:19.169 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:19.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:19.170 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:19.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:19.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:19.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:19.298 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:19.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:19.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:19.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:19.420 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:19.521 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:19.542 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:19.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:19.543 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:19.626 INFO:tasks.rados.rados.0.smithi137.stdout:1676: done (3 left) 2024-09-21T08:30:19.626 INFO:tasks.rados.rados.0.smithi137.stdout:1679: done (2 left) 2024-09-21T08:30:19.626 INFO:tasks.rados.rados.0.smithi137.stdout:1680: done (1 left) 2024-09-21T08:30:19.626 INFO:tasks.rados.rados.0.smithi137.stdout:1682: done (0 left) 2024-09-21T08:30:19.626 INFO:tasks.rados.rados.0.smithi137.stdout:1683: delete oid 49 current snap is 175 2024-09-21T08:30:19.631 INFO:tasks.rados.rados.0.smithi137.stdout:1683: done (0 left) 2024-09-21T08:30:19.631 INFO:tasks.rados.rados.0.smithi137.stdout:1684: write (excl) oid 30 current snap is 175 2024-09-21T08:30:19.631 INFO:tasks.rados.rados.0.smithi137.stdout:1684: seq_num 603 ranges {534621=567034,1636289=498332} 2024-09-21T08:30:19.637 INFO:tasks.rados.rados.0.smithi137.stdout:1684: writing smithi13779221-30 from 534621 to 1101655 tid 1 2024-09-21T08:30:19.643 INFO:tasks.rados.rados.0.smithi137.stdout:1684: writing smithi13779221-30 from 1636289 to 2134621 tid 2 2024-09-21T08:30:19.643 INFO:tasks.rados.rados.0.smithi137.stdout:1685: rollback oid 23 current snap is 175 2024-09-21T08:30:19.643 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 23 to 146 2024-09-21T08:30:19.643 INFO:tasks.rados.rados.0.smithi137.stdout:1686: snap_create 2024-09-21T08:30:19.644 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:19.649 INFO:tasks.rados.rados.0.smithi137.stdout:1685: finishing rollback tid 0 to smithi13779221-23 2024-09-21T08:30:19.649 INFO:tasks.rados.rados.0.smithi137.stdout:1685: finishing rollback tid 1 to smithi13779221-23 2024-09-21T08:30:19.649 INFO:tasks.rados.rados.0.smithi137.stdout:1685: finishing rollback tid 2 to smithi13779221-23 2024-09-21T08:30:19.649 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 201 (ObjNum 428 snap 108 seq_num 428) dirty exists 2024-09-21T08:30:19.650 INFO:tasks.rados.rados.0.smithi137.stdout:1684: finishing write tid 1 to smithi13779221-30 2024-09-21T08:30:19.651 INFO:tasks.rados.rados.0.smithi137.stdout:1684: finishing write tid 2 to smithi13779221-30 2024-09-21T08:30:19.651 INFO:tasks.rados.rados.0.smithi137.stdout:1684: finishing write tid 3 to smithi13779221-30 2024-09-21T08:30:19.651 INFO:tasks.rados.rados.0.smithi137.stdout:1684: finishing write tid 4 to smithi13779221-30 2024-09-21T08:30:19.651 INFO:tasks.rados.rados.0.smithi137.stdout:1684: oid 30 updating version 0 to 682 2024-09-21T08:30:19.651 INFO:tasks.rados.rados.0.smithi137.stdout:1684: oid 30 updating version 682 to 683 2024-09-21T08:30:19.651 INFO:tasks.rados.rados.0.smithi137.stdout:1684: oid 30 version 683 is already newer than 681 2024-09-21T08:30:19.652 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 683 (ObjNum 603 snap 175 seq_num 603) dirty exists 2024-09-21T08:30:19.652 INFO:tasks.rados.rados.0.smithi137.stdout:1684: left oid 30 (ObjNum 603 snap 175 seq_num 603) 2024-09-21T08:30:19.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:19.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:19.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:19.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:19 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:30:19.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:19 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:30:19.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:19 smithi137 ceph-mon[29634]: pgmap v542: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 509 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 9 op/s; 344/1341 objects degraded (25.652%) 2024-09-21T08:30:19.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:19 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1873665245' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.1d", "id": [4, 3, 1]}]': finished 2024-09-21T08:30:19.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:19 smithi137 ceph-mon[29634]: osdmap e425: 8 total, 6 up, 4 in 2024-09-21T08:30:19.758 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:19 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:30:19.758 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:19 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:30:19.758 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:19 smithi137 ceph-mon[41044]: pgmap v542: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 509 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 9 op/s; 344/1341 objects degraded (25.652%) 2024-09-21T08:30:19.758 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:19 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1873665245' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.1d", "id": [4, 3, 1]}]': finished 2024-09-21T08:30:19.759 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:19 smithi137 ceph-mon[41044]: osdmap e425: 8 total, 6 up, 4 in 2024-09-21T08:30:19.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:19.792 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:19.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:19.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:19.894 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:19.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:19.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:19.917 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:19 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:30:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:19 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:30:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:19 smithi165 ceph-mon[32893]: pgmap v542: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 509 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 2.2 MiB/s rd, 1.6 MiB/s wr, 9 op/s; 344/1341 objects degraded (25.652%) 2024-09-21T08:30:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:19 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1873665245' entity='client.admin' cmd='[{"prefix": "osd pg-upmap", "pgid": "7.1d", "id": [4, 3, 1]}]': finished 2024-09-21T08:30:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:19 smithi165 ceph-mon[32893]: osdmap e425: 8 total, 6 up, 4 in 2024-09-21T08:30:20.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:20.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:20.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:20.045 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:20.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:20.168 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:20.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:20.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:20.272 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:20.273 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:20.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:20.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:20.297 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:20.301 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:20.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:20.424 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:20.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:20.426 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:20.526 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:20.552 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:20.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:20.554 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:20.629 INFO:tasks.rados.rados.0.smithi137.stdout:1684: done (2 left) 2024-09-21T08:30:20.629 INFO:tasks.rados.rados.0.smithi137.stdout:1685: done (1 left) 2024-09-21T08:30:20.629 INFO:tasks.rados.rados.0.smithi137.stdout:1686: done (0 left) 2024-09-21T08:30:20.629 INFO:tasks.rados.rados.0.smithi137.stdout:1687: rollback oid 44 current snap is 176 2024-09-21T08:30:20.629 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 44 to 170 2024-09-21T08:30:20.630 INFO:tasks.rados.rados.0.smithi137.stdout:1688: snap_create 2024-09-21T08:30:20.635 INFO:tasks.rados.rados.0.smithi137.stdout:1687: finishing rollback tid 0 to smithi13779221-44 2024-09-21T08:30:20.636 INFO:tasks.rados.rados.0.smithi137.stdout:1687: finishing rollback tid 1 to smithi13779221-44 2024-09-21T08:30:20.636 INFO:tasks.rados.rados.0.smithi137.stdout:1687: finishing rollback tid 2 to smithi13779221-44 2024-09-21T08:30:20.636 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 515 (ObjNum 471 snap 128 seq_num 471) dirty exists 2024-09-21T08:30:20.655 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:20.677 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:20 smithi137 ceph-mon[29634]: osdmap e426: 8 total, 6 up, 4 in 2024-09-21T08:30:20.677 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:20 smithi137 ceph-mon[41044]: osdmap e426: 8 total, 6 up, 4 in 2024-09-21T08:30:20.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:20.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:20.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:20.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:20.801 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:20.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:20.802 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:20.902 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:20.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:20.929 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:20.929 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:20.958 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:30:20 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:30:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:30:20.983 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:20.983 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:20 smithi165 ceph-mon[32893]: osdmap e426: 8 total, 6 up, 4 in 2024-09-21T08:30:21.031 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:21.053 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:21.053 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:21.053 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:21.154 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:21.156 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:21.173 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:21.173 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:21.179 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:21.180 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:21.180 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:21.189 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:21.190 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:21.205 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:21.206 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:21.223 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:21.223 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:21.240 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:21.240 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:21.257 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:21.257 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:21.257 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:21.281 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:21.306 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:21.307 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:21.307 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:21.408 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:21.433 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:21.434 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:21.434 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:21.535 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:21.558 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:21.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:21.559 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:21.577 INFO:tasks.rados.rados.0.smithi137.stdout:1687: done (1 left) 2024-09-21T08:30:21.577 INFO:tasks.rados.rados.0.smithi137.stdout:1688: done (0 left) 2024-09-21T08:30:21.578 INFO:tasks.rados.rados.0.smithi137.stdout:1689: rollback oid 49 current snap is 177 2024-09-21T08:30:21.578 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 49 to 171 2024-09-21T08:30:21.578 INFO:tasks.rados.rados.0.smithi137.stdout:1690: read oid 21 snap 175 2024-09-21T08:30:21.578 INFO:tasks.rados.rados.0.smithi137.stdout:1690: expect (ObjNum 502 snap 141 seq_num 502) 2024-09-21T08:30:21.578 INFO:tasks.rados.rados.0.smithi137.stdout:1691: delete oid 6 current snap is 177 2024-09-21T08:30:21.588 INFO:tasks.rados.rados.0.smithi137.stdout:1691: done (2 left) 2024-09-21T08:30:21.588 INFO:tasks.rados.rados.0.smithi137.stdout:1692: copy_from oid 16 from oid 3 current snap is 177 2024-09-21T08:30:21.588 INFO:tasks.rados.rados.0.smithi137.stdout:1693: read oid 24 snap -1 2024-09-21T08:30:21.588 INFO:tasks.rados.rados.0.smithi137.stdout:1693: expect deleted 2024-09-21T08:30:21.588 INFO:tasks.rados.rados.0.smithi137.stdout:1694: copy_from oid 7 from oid 37 current snap is 177 2024-09-21T08:30:21.588 INFO:tasks.rados.rados.0.smithi137.stdout:1695: copy_from oid 39 from oid 9 current snap is 177 2024-09-21T08:30:21.588 INFO:tasks.rados.rados.0.smithi137.stdout:1696: snap_create 2024-09-21T08:30:21.589 INFO:tasks.rados.rados.0.smithi137.stdout:1689: finishing rollback tid 1 to smithi13779221-49 2024-09-21T08:30:21.589 INFO:tasks.rados.rados.0.smithi137.stdout:1689: finishing rollback tid 2 to smithi13779221-49 2024-09-21T08:30:21.590 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 444 (ObjNum 560 snap 167 seq_num 560) dirty exists 2024-09-21T08:30:21.590 INFO:tasks.rados.rados.0.smithi137.stdout:1690: expect (ObjNum 502 snap 141 seq_num 502) 2024-09-21T08:30:21.660 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:21.683 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:21.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:21.684 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:21.744 INFO:tasks.rados.rados.0.smithi137.stdout:1695: finishing copy_from to smithi13779221-39 2024-09-21T08:30:21.745 INFO:tasks.rados.rados.0.smithi137.stdout:1695: got expected ENOENT (src dne) 2024-09-21T08:30:21.745 INFO:tasks.rados.rados.0.smithi137.stdout:1695: finishing copy_from racing read to smithi13779221-39 2024-09-21T08:30:21.745 INFO:tasks.rados.rados.0.smithi137.stdout:1694: finishing copy_from racing read to smithi13779221-7 2024-09-21T08:30:21.745 INFO:tasks.rados.rados.0.smithi137.stdout:1694: finishing copy_from to smithi13779221-7 2024-09-21T08:30:21.745 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 664 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:30:21.745 INFO:tasks.rados.rados.0.smithi137.stdout:1692: finishing copy_from racing read to smithi13779221-16 2024-09-21T08:30:21.745 INFO:tasks.rados.rados.0.smithi137.stdout:1692: finishing copy_from to smithi13779221-16 2024-09-21T08:30:21.745 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 498 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:30:21.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:21.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:21.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:21.809 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:21.910 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:21.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:21 smithi165 ceph-mon[32893]: pgmap v545: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 509 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 344/1341 objects degraded (25.652%) 2024-09-21T08:30:21.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:21 smithi165 ceph-mon[32893]: osdmap e427: 8 total, 6 up, 4 in 2024-09-21T08:30:21.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:21 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/751282440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:21.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:21 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 344/1341 objects degraded (25.652%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:21.913 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:21 smithi165 ceph-mon[32893]: osdmap e428: 8 total, 6 up, 4 in 2024-09-21T08:30:21.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:21.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:21.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:22.007 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:21 smithi137 ceph-mon[41044]: pgmap v545: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 509 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 344/1341 objects degraded (25.652%) 2024-09-21T08:30:22.007 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:21 smithi137 ceph-mon[41044]: osdmap e427: 8 total, 6 up, 4 in 2024-09-21T08:30:22.007 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:21 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/751282440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:22.007 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:21 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 344/1341 objects degraded (25.652%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:22.008 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:21 smithi137 ceph-mon[41044]: osdmap e428: 8 total, 6 up, 4 in 2024-09-21T08:30:22.008 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:21 smithi137 ceph-mon[29634]: pgmap v545: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 509 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 344/1341 objects degraded (25.652%) 2024-09-21T08:30:22.008 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:21 smithi137 ceph-mon[29634]: osdmap e427: 8 total, 6 up, 4 in 2024-09-21T08:30:22.008 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:21 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/751282440' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:22.008 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:21 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 344/1341 objects degraded (25.652%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:22.008 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:21 smithi137 ceph-mon[29634]: osdmap e428: 8 total, 6 up, 4 in 2024-09-21T08:30:22.033 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:22.037 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T08:30:22.059 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:22.059 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:22.059 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:22.161 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:22.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:30:22.183 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:22.184 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:22.184 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:22.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:30:21 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:30:22.285 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:22.307 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:22.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:22.308 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:22.312 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:22.409 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:22.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:22.431 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:22.431 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:22.500 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:22.500 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:22.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:22.535 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:22.557 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:22.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:22.558 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:22.589 INFO:tasks.rados.rados.0.smithi137.stdout:1689: done (6 left) 2024-09-21T08:30:22.589 INFO:tasks.rados.rados.0.smithi137.stdout:1690: done (5 left) 2024-09-21T08:30:22.590 INFO:tasks.rados.rados.0.smithi137.stdout:1692: done (4 left) 2024-09-21T08:30:22.590 INFO:tasks.rados.rados.0.smithi137.stdout:1693: done (3 left) 2024-09-21T08:30:22.590 INFO:tasks.rados.rados.0.smithi137.stdout:1694: done (2 left) 2024-09-21T08:30:22.590 INFO:tasks.rados.rados.0.smithi137.stdout:1695: done (1 left) 2024-09-21T08:30:22.590 INFO:tasks.rados.rados.0.smithi137.stdout:1696: done (0 left) 2024-09-21T08:30:22.590 INFO:tasks.rados.rados.0.smithi137.stdout:1697: rollback oid 26 current snap is 178 2024-09-21T08:30:22.590 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 26 to 171 2024-09-21T08:30:22.590 INFO:tasks.rados.rados.0.smithi137.stdout:1698: delete oid 2 current snap is 178 2024-09-21T08:30:22.597 INFO:tasks.rados.rados.0.smithi137.stdout:1698: done (1 left) 2024-09-21T08:30:22.597 INFO:tasks.rados.rados.0.smithi137.stdout:1699: snap_create 2024-09-21T08:30:22.598 INFO:tasks.rados.rados.0.smithi137.stdout:1697: finishing rollback tid 1 to smithi13779221-26 2024-09-21T08:30:22.600 INFO:tasks.rados.rados.0.smithi137.stdout:1697: finishing rollback tid 2 to smithi13779221-26 2024-09-21T08:30:22.600 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 391 (ObjNum 560 snap 167 seq_num 560) dirty exists 2024-09-21T08:30:22.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:22.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:22.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:22.680 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:22.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:22 smithi165 ceph-mon[32893]: 6.d scrub starts 2024-09-21T08:30:22.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:22 smithi165 ceph-mon[32893]: 6.d scrub ok 2024-09-21T08:30:22.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:22 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:30:22.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:22 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:30:22.751 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:22 smithi165 ceph-mon[32893]: osdmap e429: 8 total, 6 up, 4 in 2024-09-21T08:30:22.754 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:30:22.754 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:30:22.754 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:30:22.755 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:30:22.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:22 smithi137 ceph-mon[29634]: 6.d scrub starts 2024-09-21T08:30:22.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:22 smithi137 ceph-mon[29634]: 6.d scrub ok 2024-09-21T08:30:22.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:22 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:30:22.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:22 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:30:22.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:22 smithi137 ceph-mon[29634]: osdmap e429: 8 total, 6 up, 4 in 2024-09-21T08:30:22.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:22 smithi137 ceph-mon[41044]: 6.d scrub starts 2024-09-21T08:30:22.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:22 smithi137 ceph-mon[41044]: 6.d scrub ok 2024-09-21T08:30:22.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:22 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:30:22.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:22 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:30:22.770 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:22 smithi137 ceph-mon[41044]: osdmap e429: 8 total, 6 up, 4 in 2024-09-21T08:30:22.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:22.803 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:22.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:22.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:22.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:22.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:22.928 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:22.928 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:23.029 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:23.051 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:23.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:23.052 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:23.153 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:23.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:23.175 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:23.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:23.276 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:23.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:23.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:23.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:23.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:23.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:23.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:23.421 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:23.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T08:30:23.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:23.545 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:23.545 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:23.546 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:23.586 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:30:23.595 INFO:tasks.rados.rados.0.smithi137.stdout:1697: done (1 left) 2024-09-21T08:30:23.595 INFO:tasks.rados.rados.0.smithi137.stdout:1699: done (0 left) 2024-09-21T08:30:23.595 INFO:tasks.rados.rados.0.smithi137.stdout:1700: write (excl) oid 36 current snap is 179 2024-09-21T08:30:23.595 INFO:tasks.rados.rados.0.smithi137.stdout:1700: seq_num 606 ranges {585042=792781,2095382=89660} 2024-09-21T08:30:23.603 INFO:tasks.rados.rados.0.smithi137.stdout:1700: writing smithi13779221-36 from 585042 to 1377823 tid 1 2024-09-21T08:30:23.608 INFO:tasks.rados.rados.0.smithi137.stdout:1700: writing smithi13779221-36 from 2095382 to 2185042 tid 2 2024-09-21T08:30:23.608 INFO:tasks.rados.rados.0.smithi137.stdout:1701: rollback oid 5 current snap is 179 2024-09-21T08:30:23.608 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 5 to 153 2024-09-21T08:30:23.608 INFO:tasks.rados.rados.0.smithi137.stdout:1702: write oid 16 current snap is 179 2024-09-21T08:30:23.608 INFO:tasks.rados.rados.0.smithi137.stdout:1702: seq_num 607 ranges {601849=468030,1753864=447985} 2024-09-21T08:30:23.614 INFO:tasks.rados.rados.0.smithi137.stdout:1702: writing smithi13779221-16 from 601849 to 1069879 tid 1 2024-09-21T08:30:23.621 INFO:tasks.rados.rados.0.smithi137.stdout:1702: writing smithi13779221-16 from 1753864 to 2201849 tid 2 2024-09-21T08:30:23.621 INFO:tasks.rados.rados.0.smithi137.stdout:1703: snap_remove snap 167 2024-09-21T08:30:23.647 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:23.668 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:23.669 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:23.730 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:23 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:23 smithi165 ceph-mon[32893]: pgmap v548: 168 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 510 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 MiB/s rd, 675 KiB/s wr, 8 op/s; 350/1359 objects degraded (25.754%) 2024-09-21T08:30:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:23 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3542723130' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:23 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:30:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:23 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3542723130' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:23.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:23 smithi165 ceph-mon[32893]: osdmap e430: 8 total, 6 up, 4 in 2024-09-21T08:30:23.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:23.791 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:23.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:23.792 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:23.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:23.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[29634]: pgmap v548: 168 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 510 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 MiB/s rd, 675 KiB/s wr, 8 op/s; 350/1359 objects degraded (25.754%) 2024-09-21T08:30:23.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3542723130' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:23.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:30:23.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3542723130' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:23.892 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[29634]: osdmap e430: 8 total, 6 up, 4 in 2024-09-21T08:30:23.892 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:23.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[41044]: pgmap v548: 168 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 510 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 MiB/s rd, 675 KiB/s wr, 8 op/s; 350/1359 objects degraded (25.754%) 2024-09-21T08:30:23.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3542723130' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:23.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:30:23.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3542723130' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:23.893 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:23 smithi137 ceph-mon[41044]: osdmap e430: 8 total, 6 up, 4 in 2024-09-21T08:30:23.893 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:23.914 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:23.915 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:23.915 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:24.016 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:24.038 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:24.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:24.041 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:24.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:24.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:24.165 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:24.165 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:24.257 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:30:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:30:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:30:24.257 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:30:24.257 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:30:24.266 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:24.288 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:24.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:24.289 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:24.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:24.411 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:24.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:24.412 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:24.501 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:30:24.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:24.566 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:24.567 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:24.567 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:24.593 INFO:tasks.rados.rados.0.smithi137.stdout:1703: done (3 left) 2024-09-21T08:30:24.593 INFO:tasks.rados.rados.0.smithi137.stdout:1704: snap_remove snap 154 2024-09-21T08:30:24.668 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:24.690 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:24.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:24.691 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:24.754 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:24.786 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1] out_osds: [4, 6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:30:24.786 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:30:24.786 INFO:tasks.thrashosds.thrasher:Shrinking pool .mgr 2024-09-21T08:30:24.786 INFO:tasks.cephadm.ceph_manager.ceph:contract_pool .mgr by 10 min 8 2024-09-21T08:30:24.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:30:24.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:24.813 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:24.813 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:24.814 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:24.914 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:24.936 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:24.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:24.937 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:25.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:25.060 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:25.061 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:25.061 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:25.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T08:30:25.162 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:25.184 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:25.185 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:25.185 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:25.244 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:25.245 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:25.245 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:25.262 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:25.263 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:25.279 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:25.280 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:25.280 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:25.286 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:25.298 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:25.298 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:25.308 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:25.308 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:25.309 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:25.316 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:25.316 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:25.332 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:25.333 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:25.333 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:25.358 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:25.410 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:25.432 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:25.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:25.433 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:25.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:25.556 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:25.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:25.557 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:25.593 INFO:tasks.rados.rados.0.smithi137.stdout:1704: done (3 left) 2024-09-21T08:30:25.593 INFO:tasks.rados.rados.0.smithi137.stdout:1705: write (excl) oid 2 current snap is 179 2024-09-21T08:30:25.593 INFO:tasks.rados.rados.0.smithi137.stdout:1701: finishing rollback tid 1 to smithi13779221-5 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 372 (ObjNum 129 snap 0 seq_num 3760827568) dirty dne 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:1700: finishing write tid 1 to smithi13779221-36 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:1700: finishing write tid 2 to smithi13779221-36 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:1700: finishing write tid 3 to smithi13779221-36 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:1700: finishing write tid 4 to smithi13779221-36 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:1700: oid 36 updating version 0 to 691 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:1700: oid 36 version 691 is already newer than 690 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:1700: oid 36 version 691 is already newer than 689 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 691 (ObjNum 606 snap 179 seq_num 606) dirty exists 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:1700: left oid 36 (ObjNum 606 snap 179 seq_num 606) 2024-09-21T08:30:25.594 INFO:tasks.rados.rados.0.smithi137.stdout:1702: finishing write tid 1 to smithi13779221-16 2024-09-21T08:30:25.595 INFO:tasks.rados.rados.0.smithi137.stdout:1702: finishing write tid 2 to smithi13779221-16 2024-09-21T08:30:25.595 INFO:tasks.rados.rados.0.smithi137.stdout:1702: finishing write tid 3 to smithi13779221-16 2024-09-21T08:30:25.595 INFO:tasks.rados.rados.0.smithi137.stdout:1702: finishing write tid 4 to smithi13779221-16 2024-09-21T08:30:25.595 INFO:tasks.rados.rados.0.smithi137.stdout:1702: oid 16 updating version 0 to 501 2024-09-21T08:30:25.595 INFO:tasks.rados.rados.0.smithi137.stdout:1702: oid 16 version 501 is already newer than 499 2024-09-21T08:30:25.595 INFO:tasks.rados.rados.0.smithi137.stdout:1702: oid 16 version 501 is already newer than 500 2024-09-21T08:30:25.595 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 501 (ObjNum 607 snap 179 seq_num 607) dirty exists 2024-09-21T08:30:25.595 INFO:tasks.rados.rados.0.smithi137.stdout:1702: left oid 16 (ObjNum 607 snap 179 seq_num 607) 2024-09-21T08:30:25.595 INFO:tasks.rados.rados.0.smithi137.stdout:1705: seq_num 608 ranges {618656=543279,1812346=406310} 2024-09-21T08:30:25.599 INFO:tasks.rados.rados.0.smithi137.stdout:1705: writing smithi13779221-2 from 618656 to 1161935 tid 1 2024-09-21T08:30:25.606 INFO:tasks.rados.rados.0.smithi137.stdout:1705: writing smithi13779221-2 from 1812346 to 2218656 tid 2 2024-09-21T08:30:25.606 INFO:tasks.rados.rados.0.smithi137.stdout:1700: done (3 left) 2024-09-21T08:30:25.606 INFO:tasks.rados.rados.0.smithi137.stdout:1701: done (2 left) 2024-09-21T08:30:25.606 INFO:tasks.rados.rados.0.smithi137.stdout:1702: done (1 left) 2024-09-21T08:30:25.606 INFO:tasks.rados.rados.0.smithi137.stdout:1706: read oid 36 snap -1 2024-09-21T08:30:25.606 INFO:tasks.rados.rados.0.smithi137.stdout:1706: expect (ObjNum 606 snap 179 seq_num 606) 2024-09-21T08:30:25.606 INFO:tasks.rados.rados.0.smithi137.stdout:1707: snap_create 2024-09-21T08:30:25.610 INFO:tasks.rados.rados.0.smithi137.stdout:1705: finishing write tid 1 to smithi13779221-2 2024-09-21T08:30:25.610 INFO:tasks.rados.rados.0.smithi137.stdout:1705: finishing write tid 2 to smithi13779221-2 2024-09-21T08:30:25.611 INFO:tasks.rados.rados.0.smithi137.stdout:1705: finishing write tid 3 to smithi13779221-2 2024-09-21T08:30:25.611 INFO:tasks.rados.rados.0.smithi137.stdout:1705: finishing write tid 4 to smithi13779221-2 2024-09-21T08:30:25.611 INFO:tasks.rados.rados.0.smithi137.stdout:1705: oid 2 updating version 0 to 172 2024-09-21T08:30:25.611 INFO:tasks.rados.rados.0.smithi137.stdout:1705: oid 2 version 172 is already newer than 170 2024-09-21T08:30:25.611 INFO:tasks.rados.rados.0.smithi137.stdout:1705: oid 2 version 172 is already newer than 171 2024-09-21T08:30:25.611 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 172 (ObjNum 608 snap 179 seq_num 608) dirty exists 2024-09-21T08:30:25.611 INFO:tasks.rados.rados.0.smithi137.stdout:1705: left oid 2 (ObjNum 608 snap 179 seq_num 608) 2024-09-21T08:30:25.618 INFO:tasks.rados.rados.0.smithi137.stdout:1706: expect (ObjNum 606 snap 179 seq_num 606) 2024-09-21T08:30:25.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:25.680 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:25.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:25.681 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:25.756 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:25 smithi165 ceph-mon[32893]: 6.11 scrub starts 2024-09-21T08:30:25.756 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:25 smithi165 ceph-mon[32893]: 6.11 scrub ok 2024-09-21T08:30:25.756 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:25 smithi165 ceph-mon[32893]: pgmap v551: 168 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 510 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 MiB/s rd, 675 KiB/s wr, 8 op/s; 350/1359 objects degraded (25.754%) 2024-09-21T08:30:25.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:25 smithi165 ceph-mon[32893]: osdmap e431: 8 total, 6 up, 4 in 2024-09-21T08:30:25.757 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:25 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/942068740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:25 smithi137 ceph-mon[29634]: 6.11 scrub starts 2024-09-21T08:30:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:25 smithi137 ceph-mon[29634]: 6.11 scrub ok 2024-09-21T08:30:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:25 smithi137 ceph-mon[29634]: pgmap v551: 168 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 510 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 MiB/s rd, 675 KiB/s wr, 8 op/s; 350/1359 objects degraded (25.754%) 2024-09-21T08:30:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:25 smithi137 ceph-mon[29634]: osdmap e431: 8 total, 6 up, 4 in 2024-09-21T08:30:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:25 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/942068740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:25 smithi137 ceph-mon[41044]: 6.11 scrub starts 2024-09-21T08:30:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:25 smithi137 ceph-mon[41044]: 6.11 scrub ok 2024-09-21T08:30:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:25 smithi137 ceph-mon[41044]: pgmap v551: 168 pgs: 3 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 510 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.1 MiB/s rd, 675 KiB/s wr, 8 op/s; 350/1359 objects degraded (25.754%) 2024-09-21T08:30:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:25 smithi137 ceph-mon[41044]: osdmap e431: 8 total, 6 up, 4 in 2024-09-21T08:30:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:25 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/942068740' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:25.782 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:25.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:25.804 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:25.804 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:25.856 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:25.856 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:25.904 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:25.926 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:25.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:25.927 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:26.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:26.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:26.049 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:26.050 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:26.150 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:26.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:26.172 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:26.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:26.187 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:26.188 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:26.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:30:26.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:26.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:26.294 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:26.294 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:26.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:26.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:26.419 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:26.419 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:26.456 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:26.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T08:30:26.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:26.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:26.543 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:26.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:26.606 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:30:26.609 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:30:26.611 INFO:tasks.rados.rados.0.smithi137.stdout:1705: done (2 left) 2024-09-21T08:30:26.612 INFO:tasks.rados.rados.0.smithi137.stdout:1706: done (1 left) 2024-09-21T08:30:26.612 INFO:tasks.rados.rados.0.smithi137.stdout:1707: done (0 left) 2024-09-21T08:30:26.612 INFO:tasks.rados.rados.0.smithi137.stdout:1708: rollback oid 32 current snap is 180 2024-09-21T08:30:26.612 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 32 to 130 2024-09-21T08:30:26.612 INFO:tasks.rados.rados.0.smithi137.stdout:1709: delete oid 36 current snap is 180 2024-09-21T08:30:26.620 INFO:tasks.rados.rados.0.smithi137.stdout:1709: done (1 left) 2024-09-21T08:30:26.620 INFO:tasks.rados.rados.0.smithi137.stdout:1710: read oid 50 snap 172 2024-09-21T08:30:26.620 INFO:tasks.rados.rados.0.smithi137.stdout:1710: expect (ObjNum 544 snap 162 seq_num 544) 2024-09-21T08:30:26.620 INFO:tasks.rados.rados.0.smithi137.stdout:1711: write (excl) oid 45 current snap is 180 2024-09-21T08:30:26.620 INFO:tasks.rados.rados.0.smithi137.stdout:1711: seq_num 610 ranges {652270=410130,1529310=493867} 2024-09-21T08:30:26.626 INFO:tasks.rados.rados.0.smithi137.stdout:1711: writing smithi13779221-45 from 652270 to 1062400 tid 1 2024-09-21T08:30:26.632 INFO:tasks.rados.rados.0.smithi137.stdout:1711: writing smithi13779221-45 from 1529310 to 2023177 tid 2 2024-09-21T08:30:26.632 INFO:tasks.rados.rados.0.smithi137.stdout:1712: delete oid 6 current snap is 180 2024-09-21T08:30:26.632 INFO:tasks.rados.rados.0.smithi137.stdout:1708: finishing rollback tid 0 to smithi13779221-32 2024-09-21T08:30:26.632 INFO:tasks.rados.rados.0.smithi137.stdout:1708: finishing rollback tid 1 to smithi13779221-32 2024-09-21T08:30:26.632 INFO:tasks.rados.rados.0.smithi137.stdout:1708: finishing rollback tid 2 to smithi13779221-32 2024-09-21T08:30:26.632 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 525 (ObjNum 458 snap 124 seq_num 458) dirty exists 2024-09-21T08:30:26.632 INFO:tasks.rados.rados.0.smithi137.stdout:1710: expect (ObjNum 544 snap 162 seq_num 544) 2024-09-21T08:30:26.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:26.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:26.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:26.671 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:26.733 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:26.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:26 smithi165 ceph-mon[32893]: 3.17 scrub starts 2024-09-21T08:30:26.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:26 smithi165 ceph-mon[32893]: 3.17 scrub ok 2024-09-21T08:30:26.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:26 smithi165 ceph-mon[32893]: 4.13 scrub starts 2024-09-21T08:30:26.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:26 smithi165 ceph-mon[32893]: 4.13 scrub ok 2024-09-21T08:30:26.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:26 smithi165 ceph-mon[32893]: osdmap e432: 8 total, 6 up, 4 in 2024-09-21T08:30:26.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:26 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 350/1359 objects degraded (25.754%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:26.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:26.795 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:26.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:26.796 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1711: finishing write tid 1 to smithi13779221-45 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1711: finishing write tid 2 to smithi13779221-45 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1711: finishing write tid 3 to smithi13779221-45 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1711: finishing write tid 4 to smithi13779221-45 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1711: oid 45 updating version 0 to 379 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1711: oid 45 updating version 379 to 381 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1711: oid 45 version 381 is already newer than 380 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 381 (ObjNum 610 snap 180 seq_num 610) dirty exists 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1711: left oid 45 (ObjNum 610 snap 180 seq_num 610) 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1708: done (3 left) 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1710: done (2 left) 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1711: done (1 left) 2024-09-21T08:30:26.803 INFO:tasks.rados.rados.0.smithi137.stdout:1712: done (0 left) 2024-09-21T08:30:26.804 INFO:tasks.rados.rados.0.smithi137.stdout:1713: write (excl) oid 32 current snap is 180 2024-09-21T08:30:26.804 INFO:tasks.rados.rados.0.smithi137.stdout:1713: seq_num 612 ranges {685884=560628,2046274=239610} 2024-09-21T08:30:26.809 INFO:tasks.rados.rados.0.smithi137.stdout:1713: writing smithi13779221-32 from 685884 to 1246512 tid 1 2024-09-21T08:30:26.815 INFO:tasks.rados.rados.0.smithi137.stdout:1713: writing smithi13779221-32 from 2046274 to 2285884 tid 2 2024-09-21T08:30:26.815 INFO:tasks.rados.rados.0.smithi137.stdout:1714: copy_from oid 38 from oid 17 current snap is 180 2024-09-21T08:30:26.815 INFO:tasks.rados.rados.0.smithi137.stdout:1713: finishing write tid 1 to smithi13779221-32 2024-09-21T08:30:26.815 INFO:tasks.rados.rados.0.smithi137.stdout:1715: write oid 14 current snap is 180 2024-09-21T08:30:26.815 INFO:tasks.rados.rados.0.smithi137.stdout:1715: seq_num 613 ranges {702691=635877,2104756=197935} 2024-09-21T08:30:26.823 INFO:tasks.rados.rados.0.smithi137.stdout:1715: writing smithi13779221-14 from 702691 to 1338568 tid 1 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1715: writing smithi13779221-14 from 2104756 to 2302691 tid 2 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1716: delete oid 1 current snap is 180 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1713: finishing write tid 2 to smithi13779221-32 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1713: finishing write tid 3 to smithi13779221-32 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1713: finishing write tid 4 to smithi13779221-32 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1713: oid 32 updating version 0 to 526 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1713: oid 32 updating version 526 to 527 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1713: oid 32 updating version 527 to 528 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 528 (ObjNum 612 snap 180 seq_num 612) dirty exists 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1713: left oid 32 (ObjNum 612 snap 180 seq_num 612) 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1714: finishing copy_from racing read to smithi13779221-38 2024-09-21T08:30:26.829 INFO:tasks.rados.rados.0.smithi137.stdout:1715: finishing write tid 1 to smithi13779221-14 2024-09-21T08:30:26.835 INFO:tasks.rados.rados.0.smithi137.stdout:1713: done (3 left) 2024-09-21T08:30:26.835 INFO:tasks.rados.rados.0.smithi137.stdout:1716: done (2 left) 2024-09-21T08:30:26.835 INFO:tasks.rados.rados.0.smithi137.stdout:1717: write oid 4 current snap is 180 2024-09-21T08:30:26.835 INFO:tasks.rados.rados.0.smithi137.stdout:1714: finishing copy_from to smithi13779221-38 2024-09-21T08:30:26.835 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 505 (ObjNum 552 snap 166 seq_num 552) dirty exists 2024-09-21T08:30:26.835 INFO:tasks.rados.rados.0.smithi137.stdout:1717: seq_num 615 ranges {736305=786375,2105367=230938} 2024-09-21T08:30:26.843 INFO:tasks.rados.rados.0.smithi137.stdout:1717: writing smithi13779221-4 from 736305 to 1522680 tid 1 2024-09-21T08:30:26.848 INFO:tasks.rados.rados.0.smithi137.stdout:1717: writing smithi13779221-4 from 2105367 to 2336305 tid 2 2024-09-21T08:30:26.848 INFO:tasks.rados.rados.0.smithi137.stdout:1714: done (2 left) 2024-09-21T08:30:26.848 INFO:tasks.rados.rados.0.smithi137.stdout:1718: read oid 43 snap 178 2024-09-21T08:30:26.848 INFO:tasks.rados.rados.0.smithi137.stdout:1718: expect (ObjNum 575 snap 170 seq_num 575) 2024-09-21T08:30:26.848 INFO:tasks.rados.rados.0.smithi137.stdout:1719: snap_remove snap 171 2024-09-21T08:30:26.897 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:26.918 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:26.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:26.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:26.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[29634]: 3.17 scrub starts 2024-09-21T08:30:26.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[29634]: 3.17 scrub ok 2024-09-21T08:30:26.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[29634]: 4.13 scrub starts 2024-09-21T08:30:26.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[29634]: 4.13 scrub ok 2024-09-21T08:30:26.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[29634]: osdmap e432: 8 total, 6 up, 4 in 2024-09-21T08:30:26.941 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 350/1359 objects degraded (25.754%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:26.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[41044]: 3.17 scrub starts 2024-09-21T08:30:26.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[41044]: 3.17 scrub ok 2024-09-21T08:30:26.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[41044]: 4.13 scrub starts 2024-09-21T08:30:26.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[41044]: 4.13 scrub ok 2024-09-21T08:30:26.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[41044]: osdmap e432: 8 total, 6 up, 4 in 2024-09-21T08:30:26.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:26 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 350/1359 objects degraded (25.754%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:27.021 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:27.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:27.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:27.043 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:27.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:27.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:27.166 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:27.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:27.204 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:27.204 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:27.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:27.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:27.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:27.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:27.378 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2024-09-21T08:30:26.131606+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":549731140,"num_objects":457,"num_object_clones":209,"num_object_copies":1371,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":353,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":457,"num_whiteouts":14,"num_read":2985,"num_read_kb":1852703,"num_write":1796,"num_write_kb":497342,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1121,"num_bytes_recovered":210716553,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8109,"ondisk_log_size":8109,"up":374,"acting":374,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1262772,"kb_used_data":1063068,"kb_used_omap":6,"kb_used_meta":199609,"kb_avail":373717836,"statfs":{"total":383980142592,"available":382687064064,"internally_reserved":0,"allocated":1088581632,"data_stored":1578020203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6375,"internal_metadata":204400409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":14734873,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":-2,"num_read":30,"num_read_kb":19616,"num_write":17,"num_write_kb":8663,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001196"},"pg_stats":[{"pgid":"7.23","version":"179'19","reported_seq":500,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.606662+0000","last_change":"2024-09-21T08:28:53.599776+0000","last_active":"2024-09-21T08:30:25.606662+0000","last_peered":"2024-09-21T08:30:25.606662+0000","last_clean":"2024-09-21T08:28:51.585797+0000","last_became_active":"2024-09-21T08:28:53.599776+0000","last_became_peered":"2024-09-21T08:28:53.599776+0000","last_unstale":"2024-09-21T08:30:25.606662+0000","last_undegraded":"2024-09-21T08:30:25.606662+0000","last_fullsized":"2024-09-21T08:28:53.583177+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:28:10.421606+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.421606+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":48,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.072212988000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"7.22","version":"257'264","reported_seq":1885,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.602439+0000","last_change":"2024-09-21T08:28:53.599720+0000","last_active":"2024-09-21T08:30:25.602439+0000","last_peered":"2024-09-21T08:30:25.602439+0000","last_clean":"2024-09-21T08:28:51.585268+0000","last_became_active":"2024-09-21T08:28:53.599720+0000","last_became_peered":"2024-09-21T08:28:53.599720+0000","last_unstale":"2024-09-21T08:30:25.602439+0000","last_undegraded":"2024-09-21T08:30:25.602439+0000","last_fullsized":"2024-09-21T08:28:53.585695+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":6,"last_scrub":"257'264","last_scrub_stamp":"2024-09-21T08:27:48.727600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:48.727600+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.075805560999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"2"}]},{"pgid":"7.21","version":"179'43","reported_seq":1342,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605707+0000","last_change":"2024-09-21T08:28:53.594285+0000","last_active":"2024-09-21T08:30:25.605707+0000","last_peered":"2024-09-21T08:30:25.605707+0000","last_clean":"2024-09-21T08:28:51.554218+0000","last_became_active":"2024-09-21T08:28:53.594285+0000","last_became_peered":"2024-09-21T08:28:53.594285+0000","last_unstale":"2024-09-21T08:30:25.605707+0000","last_undegraded":"2024-09-21T08:30:25.605707+0000","last_fullsized":"2024-09-21T08:28:53.573996+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:36.934372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.934372+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":46,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.025027727,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"7.20","version":"432'172","reported_seq":1739,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.610818+0000","last_change":"2024-09-21T08:28:53.605282+0000","last_active":"2024-09-21T08:30:25.610818+0000","last_peered":"2024-09-21T08:30:25.610818+0000","last_clean":"2024-09-21T08:28:51.585842+0000","last_became_active":"2024-09-21T08:28:53.605282+0000","last_became_peered":"2024-09-21T08:28:53.605282+0000","last_unstale":"2024-09-21T08:30:25.610818+0000","last_undegraded":"2024-09-21T08:28:53.580783+0000","last_fullsized":"2024-09-21T08:28:53.580670+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"251'134","last_scrub_stamp":"2024-09-21T08:27:49.318143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.318143+0000","objects_scrubbed":1,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.076740219999999998,"stat_sum":{"num_bytes":13440760,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":19,"num_read_kb":11329,"num_write":29,"num_write_kb":6079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1848493,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"2"}]},{"pgid":"7.1b","version":"431'695","reported_seq":2585,"reported_epoch":432,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:25.612092+0000","last_change":"2024-09-21T08:30:24.758114+0000","last_active":"2024-09-21T08:30:25.612092+0000","last_peered":"2024-09-21T08:30:25.612092+0000","last_clean":"2024-09-21T08:30:25.612092+0000","last_became_active":"2024-09-21T08:28:37.568752+0000","last_became_peered":"2024-09-21T08:28:37.568752+0000","last_unstale":"2024-09-21T08:30:25.612092+0000","last_undegraded":"2024-09-21T08:30:25.612092+0000","last_fullsized":"2024-09-21T08:30:25.612092+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'499","last_scrub_stamp":"2024-09-21T08:28:09.328624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.328624+0000","objects_scrubbed":5,"log_size":695,"log_dups_size":0,"ondisk_log_size":695,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":13,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.122684663,"stat_sum":{"num_bytes":40245371,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":0,"num_read":119,"num_read_kb":126210,"num_write":67,"num_write_kb":26923,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":18585585,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"6"},{"start":"8c","length":"4"},{"start":"95","length":"3"},{"start":"9d","length":"1"},{"start":"9f","length":"3"}]},{"pgid":"3.1f","version":"0'0","reported_seq":246,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.608320+0000","last_change":"2024-09-21T08:28:53.591551+0000","last_active":"2024-09-21T08:30:25.608320+0000","last_peered":"2024-09-21T08:30:25.608320+0000","last_clean":"2024-09-21T08:28:51.556461+0000","last_became_active":"2024-09-21T08:28:53.591551+0000","last_became_peered":"2024-09-21T08:28:53.591551+0000","last_unstale":"2024-09-21T08:30:25.608320+0000","last_undegraded":"2024-09-21T08:30:25.608320+0000","last_fullsized":"2024-09-21T08:28:53.580526+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814448+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":319,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.605213+0000","last_change":"2024-09-21T08:28:29.692402+0000","last_active":"2024-09-21T08:30:25.605213+0000","last_peered":"2024-09-21T08:30:25.605213+0000","last_clean":"2024-09-21T08:30:25.605213+0000","last_became_active":"2024-09-21T08:28:29.635580+0000","last_became_peered":"2024-09-21T08:28:29.635580+0000","last_unstale":"2024-09-21T08:30:25.605213+0000","last_undegraded":"2024-09-21T08:30:25.605213+0000","last_fullsized":"2024-09-21T08:30:25.605213+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:56.303669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:56.303669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":304,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.601606+0000","last_change":"2024-09-21T08:28:53.594621+0000","last_active":"2024-09-21T08:30:25.601606+0000","last_peered":"2024-09-21T08:30:25.601606+0000","last_clean":"2024-09-21T08:28:51.584740+0000","last_became_active":"2024-09-21T08:28:53.594621+0000","last_became_peered":"2024-09-21T08:28:53.594621+0000","last_unstale":"2024-09-21T08:30:25.601606+0000","last_undegraded":"2024-09-21T08:30:25.601606+0000","last_fullsized":"2024-09-21T08:28:53.576425+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.424482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.424482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":730,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.607020+0000","last_change":"2024-09-21T08:28:53.601367+0000","last_active":"2024-09-21T08:30:25.607020+0000","last_peered":"2024-09-21T08:30:25.607020+0000","last_clean":"2024-09-21T08:28:51.557847+0000","last_became_active":"2024-09-21T08:28:53.601367+0000","last_became_peered":"2024-09-21T08:28:53.601367+0000","last_unstale":"2024-09-21T08:30:25.607020+0000","last_undegraded":"2024-09-21T08:30:25.607020+0000","last_fullsized":"2024-09-21T08:28:53.577149+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.464969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.464969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1a","version":"248'68","reported_seq":774,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.606297+0000","last_change":"2024-09-21T08:28:53.593471+0000","last_active":"2024-09-21T08:30:25.606297+0000","last_peered":"2024-09-21T08:30:25.606297+0000","last_clean":"2024-09-21T08:28:52.589751+0000","last_became_active":"2024-09-21T08:28:53.593471+0000","last_became_peered":"2024-09-21T08:28:53.593471+0000","last_unstale":"2024-09-21T08:30:25.606297+0000","last_undegraded":"2024-09-21T08:30:25.606297+0000","last_fullsized":"2024-09-21T08:28:53.574504+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'68","last_scrub_stamp":"2024-09-21T08:27:35.919031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:35.919031+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":46,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0328630059999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"6.1b","version":"0'0","reported_seq":661,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.606227+0000","last_change":"2024-09-21T08:28:53.606497+0000","last_active":"2024-09-21T08:30:25.606227+0000","last_peered":"2024-09-21T08:30:25.606227+0000","last_clean":"2024-09-21T08:28:51.585547+0000","last_became_active":"2024-09-21T08:28:53.606497+0000","last_became_peered":"2024-09-21T08:28:53.606497+0000","last_unstale":"2024-09-21T08:30:25.606227+0000","last_undegraded":"2024-09-21T08:30:25.606227+0000","last_fullsized":"2024-09-21T08:28:53.581183+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.804294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.804294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151079+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":588,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.603389+0000","last_change":"2024-09-21T08:28:53.602981+0000","last_active":"2024-09-21T08:30:25.603389+0000","last_peered":"2024-09-21T08:30:25.603389+0000","last_clean":"2024-09-21T08:28:51.587203+0000","last_became_active":"2024-09-21T08:28:53.602981+0000","last_became_peered":"2024-09-21T08:28:53.602981+0000","last_unstale":"2024-09-21T08:30:25.603389+0000","last_undegraded":"2024-09-21T08:30:25.603389+0000","last_fullsized":"2024-09-21T08:28:53.581112+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.146798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.146798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":585,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.605978+0000","last_change":"2024-09-21T08:29:00.355702+0000","last_active":"2024-09-21T08:30:25.605978+0000","last_peered":"2024-09-21T08:30:25.605978+0000","last_clean":"2024-09-21T08:30:25.605978+0000","last_became_active":"2024-09-21T08:28:29.620169+0000","last_became_peered":"2024-09-21T08:28:29.620169+0000","last_unstale":"2024-09-21T08:30:25.605978+0000","last_undegraded":"2024-09-21T08:30:25.605978+0000","last_fullsized":"2024-09-21T08:30:25.605978+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:00.355591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:00.355591+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":296,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605274+0000","last_change":"2024-09-21T08:28:53.601812+0000","last_active":"2024-09-21T08:30:25.605274+0000","last_peered":"2024-09-21T08:30:25.605274+0000","last_clean":"2024-09-21T08:28:51.585034+0000","last_became_active":"2024-09-21T08:28:53.601812+0000","last_became_peered":"2024-09-21T08:28:53.601812+0000","last_unstale":"2024-09-21T08:30:25.605274+0000","last_undegraded":"2024-09-21T08:30:25.605274+0000","last_fullsized":"2024-09-21T08:28:53.578966+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.340817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.340817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.19","version":"417'205","reported_seq":1457,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606281+0000","last_change":"2024-09-21T08:28:53.587606+0000","last_active":"2024-09-21T08:30:25.606281+0000","last_peered":"2024-09-21T08:30:25.606281+0000","last_clean":"2024-09-21T08:28:52.589310+0000","last_became_active":"2024-09-21T08:28:53.587606+0000","last_became_peered":"2024-09-21T08:28:53.587606+0000","last_unstale":"2024-09-21T08:30:25.606281+0000","last_undegraded":"2024-09-21T08:28:53.574680+0000","last_fullsized":"2024-09-21T08:28:53.574555+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"266'188","last_scrub_stamp":"2024-09-21T08:27:49.719124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.719124+0000","objects_scrubbed":1,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":46,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.034770314,"stat_sum":{"num_bytes":11158040,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":25354,"num_write":16,"num_write_kb":11591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"3.1d","version":"0'0","reported_seq":793,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.608363+0000","last_change":"2024-09-21T08:28:53.601545+0000","last_active":"2024-09-21T08:30:25.608363+0000","last_peered":"2024-09-21T08:30:25.608363+0000","last_clean":"2024-09-21T08:28:51.555695+0000","last_became_active":"2024-09-21T08:28:53.601545+0000","last_became_peered":"2024-09-21T08:28:53.601545+0000","last_unstale":"2024-09-21T08:30:25.608363+0000","last_undegraded":"2024-09-21T08:30:25.608363+0000","last_fullsized":"2024-09-21T08:28:53.577677+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.780545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.780545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":545,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.607814+0000","last_change":"2024-09-21T08:28:39.292016+0000","last_active":"2024-09-21T08:30:25.607814+0000","last_peered":"2024-09-21T08:30:25.607814+0000","last_clean":"2024-09-21T08:30:25.607814+0000","last_became_active":"2024-09-21T08:28:37.568659+0000","last_became_peered":"2024-09-21T08:28:37.568659+0000","last_unstale":"2024-09-21T08:30:25.607814+0000","last_undegraded":"2024-09-21T08:30:25.607814+0000","last_fullsized":"2024-09-21T08:30:25.607814+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:28:39.291909+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.291909+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":752,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605518+0000","last_change":"2024-09-21T08:28:53.593695+0000","last_active":"2024-09-21T08:30:25.605518+0000","last_peered":"2024-09-21T08:30:25.605518+0000","last_clean":"2024-09-21T08:28:51.556408+0000","last_became_active":"2024-09-21T08:28:53.593695+0000","last_became_peered":"2024-09-21T08:28:53.593695+0000","last_unstale":"2024-09-21T08:30:25.605518+0000","last_undegraded":"2024-09-21T08:30:25.605518+0000","last_fullsized":"2024-09-21T08:28:53.574067+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814413+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":707,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.601587+0000","last_change":"2024-09-21T08:28:53.586342+0000","last_active":"2024-09-21T08:30:25.601587+0000","last_peered":"2024-09-21T08:30:25.601587+0000","last_clean":"2024-09-21T08:28:51.584564+0000","last_became_active":"2024-09-21T08:28:53.586342+0000","last_became_peered":"2024-09-21T08:28:53.586342+0000","last_unstale":"2024-09-21T08:30:25.601587+0000","last_undegraded":"2024-09-21T08:30:25.601587+0000","last_fullsized":"2024-09-21T08:28:53.575506+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.291226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.291226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"413'257","reported_seq":1881,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606497+0000","last_change":"2024-09-21T08:28:53.587953+0000","last_active":"2024-09-21T08:30:25.606497+0000","last_peered":"2024-09-21T08:30:25.606497+0000","last_clean":"2024-09-21T08:28:51.550342+0000","last_became_active":"2024-09-21T08:28:53.587953+0000","last_became_peered":"2024-09-21T08:28:53.587953+0000","last_unstale":"2024-09-21T08:30:25.606497+0000","last_undegraded":"2024-09-21T08:28:53.574363+0000","last_fullsized":"2024-09-21T08:28:53.574234+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"319'218","last_scrub_stamp":"2024-09-21T08:28:32.867460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:32.867460+0000","objects_scrubbed":4,"log_size":257,"log_dups_size":0,"ondisk_log_size":257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027063920000000002,"stat_sum":{"num_bytes":24595783,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":55,"num_read_kb":68203,"num_write":55,"num_write_kb":28843,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7346792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"3.1c","version":"0'0","reported_seq":778,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.606933+0000","last_change":"2024-09-21T08:28:53.600345+0000","last_active":"2024-09-21T08:30:25.606933+0000","last_peered":"2024-09-21T08:30:25.606933+0000","last_clean":"2024-09-21T08:28:51.556880+0000","last_became_active":"2024-09-21T08:28:53.600345+0000","last_became_peered":"2024-09-21T08:28:53.600345+0000","last_unstale":"2024-09-21T08:30:25.606933+0000","last_undegraded":"2024-09-21T08:30:25.606933+0000","last_fullsized":"2024-09-21T08:28:53.576313+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814390+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":302,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606494+0000","last_change":"2024-09-21T08:28:53.605410+0000","last_active":"2024-09-21T08:30:25.606494+0000","last_peered":"2024-09-21T08:30:25.606494+0000","last_clean":"2024-09-21T08:28:51.585702+0000","last_became_active":"2024-09-21T08:28:53.605410+0000","last_became_peered":"2024-09-21T08:28:53.605410+0000","last_unstale":"2024-09-21T08:30:25.606494+0000","last_undegraded":"2024-09-21T08:28:53.583377+0000","last_fullsized":"2024-09-21T08:28:53.583262+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:27:26.356987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:26.356987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":729,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.602504+0000","last_change":"2024-09-21T08:28:53.600104+0000","last_active":"2024-09-21T08:30:25.602504+0000","last_peered":"2024-09-21T08:30:25.602504+0000","last_clean":"2024-09-21T08:28:27.553155+0000","last_became_active":"2024-09-21T08:28:53.600104+0000","last_became_peered":"2024-09-21T08:28:53.600104+0000","last_unstale":"2024-09-21T08:30:25.602504+0000","last_undegraded":"2024-09-21T08:30:25.602504+0000","last_fullsized":"2024-09-21T08:28:53.585955+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.850373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":798,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.606463+0000","last_change":"2024-09-21T08:30:04.034656+0000","last_active":"2024-09-21T08:30:25.606463+0000","last_peered":"2024-09-21T08:30:25.606463+0000","last_clean":"2024-09-21T08:30:25.606463+0000","last_became_active":"2024-09-21T08:28:04.705450+0000","last_became_peered":"2024-09-21T08:28:04.705450+0000","last_unstale":"2024-09-21T08:30:25.606463+0000","last_undegraded":"2024-09-21T08:30:25.606463+0000","last_fullsized":"2024-09-21T08:30:25.606463+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:30:04.034575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:04.034575+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:12.159925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1f","version":"247'32","reported_seq":1578,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.607672+0000","last_change":"2024-09-21T08:28:53.601095+0000","last_active":"2024-09-21T08:30:25.607672+0000","last_peered":"2024-09-21T08:30:25.607672+0000","last_clean":"2024-09-21T08:28:53.015413+0000","last_became_active":"2024-09-21T08:28:53.601095+0000","last_became_peered":"2024-09-21T08:28:53.601095+0000","last_unstale":"2024-09-21T08:30:25.607672+0000","last_undegraded":"2024-09-21T08:30:25.607672+0000","last_fullsized":"2024-09-21T08:28:53.577251+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:27:51.078531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.078531+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0098481060000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"6.1e","version":"0'0","reported_seq":729,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.609453+0000","last_change":"2024-09-21T08:28:53.591644+0000","last_active":"2024-09-21T08:30:25.609453+0000","last_peered":"2024-09-21T08:30:25.609453+0000","last_clean":"2024-09-21T08:28:51.556030+0000","last_became_active":"2024-09-21T08:28:53.591644+0000","last_became_peered":"2024-09-21T08:28:53.591644+0000","last_unstale":"2024-09-21T08:30:25.609453+0000","last_undegraded":"2024-09-21T08:30:25.609453+0000","last_fullsized":"2024-09-21T08:28:53.578796+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.855448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.855448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814441+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":617,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605221+0000","last_change":"2024-09-21T08:28:53.604972+0000","last_active":"2024-09-21T08:30:25.605221+0000","last_peered":"2024-09-21T08:30:25.605221+0000","last_clean":"2024-09-21T08:28:51.585297+0000","last_became_active":"2024-09-21T08:28:53.604972+0000","last_became_peered":"2024-09-21T08:28:53.604972+0000","last_unstale":"2024-09-21T08:30:25.605221+0000","last_undegraded":"2024-09-21T08:30:25.605221+0000","last_fullsized":"2024-09-21T08:28:53.579187+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150767+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":819,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.602393+0000","last_change":"2024-09-21T08:29:37.352960+0000","last_active":"2024-09-21T08:30:25.602393+0000","last_peered":"2024-09-21T08:30:25.602393+0000","last_clean":"2024-09-21T08:30:25.602393+0000","last_became_active":"2024-09-21T08:28:37.566225+0000","last_became_peered":"2024-09-21T08:28:37.566225+0000","last_unstale":"2024-09-21T08:30:25.602393+0000","last_undegraded":"2024-09-21T08:30:25.602393+0000","last_fullsized":"2024-09-21T08:30:25.602393+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:37.352879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:37.352879+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:41.677208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":302,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.606360+0000","last_change":"2024-09-21T08:28:53.598624+0000","last_active":"2024-09-21T08:30:25.606360+0000","last_peered":"2024-09-21T08:30:25.606360+0000","last_clean":"2024-09-21T08:28:51.587418+0000","last_became_active":"2024-09-21T08:28:53.598624+0000","last_became_peered":"2024-09-21T08:28:53.598624+0000","last_unstale":"2024-09-21T08:30:25.606360+0000","last_undegraded":"2024-09-21T08:30:25.606360+0000","last_fullsized":"2024-09-21T08:28:53.579380+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:30.310628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.310628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"429'297","reported_seq":1921,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603354+0000","last_change":"2024-09-21T08:28:53.595615+0000","last_active":"2024-09-21T08:30:25.603354+0000","last_peered":"2024-09-21T08:30:25.603354+0000","last_clean":"2024-09-21T08:28:52.666418+0000","last_became_active":"2024-09-21T08:28:53.595615+0000","last_became_peered":"2024-09-21T08:28:53.595615+0000","last_unstale":"2024-09-21T08:30:25.603354+0000","last_undegraded":"2024-09-21T08:28:53.580612+0000","last_fullsized":"2024-09-21T08:28:53.580483+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'236","last_scrub_stamp":"2024-09-21T08:28:09.464279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.464279+0000","objects_scrubbed":2,"log_size":297,"log_dups_size":0,"ondisk_log_size":297,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":49,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.0792350470000001,"stat_sum":{"num_bytes":38311949,"num_objects":16,"num_object_clones":13,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":1,"num_read":60,"num_read_kb":64280,"num_write":52,"num_write_kb":20459,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6996269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":16}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"6.1f","version":"0'0","reported_seq":646,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.610430+0000","last_change":"2024-09-21T08:29:33.233733+0000","last_active":"2024-09-21T08:30:25.610430+0000","last_peered":"2024-09-21T08:30:25.610430+0000","last_clean":"2024-09-21T08:30:25.610430+0000","last_became_active":"2024-09-21T08:28:04.725392+0000","last_became_peered":"2024-09-21T08:28:04.725392+0000","last_unstale":"2024-09-21T08:30:25.610430+0000","last_undegraded":"2024-09-21T08:30:25.610430+0000","last_fullsized":"2024-09-21T08:30:25.610430+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:33.233665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:33.233665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.550890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":302,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.604075+0000","last_change":"2024-09-21T08:28:53.605497+0000","last_active":"2024-09-21T08:30:25.604075+0000","last_peered":"2024-09-21T08:30:25.604075+0000","last_clean":"2024-09-21T08:28:51.587153+0000","last_became_active":"2024-09-21T08:28:53.605497+0000","last_became_peered":"2024-09-21T08:28:53.605497+0000","last_unstale":"2024-09-21T08:30:25.604075+0000","last_undegraded":"2024-09-21T08:30:25.604075+0000","last_fullsized":"2024-09-21T08:28:53.586766+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:33.237691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.237691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150940+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":313,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.604870+0000","last_change":"2024-09-21T08:29:15.349199+0000","last_active":"2024-09-21T08:30:25.604870+0000","last_peered":"2024-09-21T08:30:25.604870+0000","last_clean":"2024-09-21T08:30:25.604870+0000","last_became_active":"2024-09-21T08:28:04.724988+0000","last_became_peered":"2024-09-21T08:28:04.724988+0000","last_unstale":"2024-09-21T08:30:25.604870+0000","last_undegraded":"2024-09-21T08:30:25.604870+0000","last_fullsized":"2024-09-21T08:30:25.604870+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:29:15.349124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:15.349124+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":673,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.591735+0000","last_change":"2024-09-21T08:28:53.591068+0000","last_active":"2024-09-21T08:30:25.591735+0000","last_peered":"2024-09-21T08:30:25.591735+0000","last_clean":"2024-09-21T08:28:51.554175+0000","last_became_active":"2024-09-21T08:28:53.591068+0000","last_became_peered":"2024-09-21T08:28:53.591068+0000","last_unstale":"2024-09-21T08:30:25.591735+0000","last_undegraded":"2024-09-21T08:30:25.591735+0000","last_fullsized":"2024-09-21T08:28:53.580183+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:09.116412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.116412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"248'84","reported_seq":515,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605431+0000","last_change":"2024-09-21T08:28:53.595944+0000","last_active":"2024-09-21T08:30:25.605431+0000","last_peered":"2024-09-21T08:30:25.605431+0000","last_clean":"2024-09-21T08:28:51.587189+0000","last_became_active":"2024-09-21T08:28:53.595944+0000","last_became_peered":"2024-09-21T08:28:53.595944+0000","last_unstale":"2024-09-21T08:30:25.605431+0000","last_undegraded":"2024-09-21T08:30:25.605431+0000","last_fullsized":"2024-09-21T08:28:53.578716+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'84","last_scrub_stamp":"2024-09-21T08:28:12.352559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.352559+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":48,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.072813132000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"6.1c","version":"73'1","reported_seq":290,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.608276+0000","last_change":"2024-09-21T08:28:53.589802+0000","last_active":"2024-09-21T08:30:25.608276+0000","last_peered":"2024-09-21T08:30:25.608276+0000","last_clean":"2024-09-21T08:28:51.557351+0000","last_became_active":"2024-09-21T08:28:53.589802+0000","last_became_peered":"2024-09-21T08:28:53.589802+0000","last_unstale":"2024-09-21T08:30:25.608276+0000","last_undegraded":"2024-09-21T08:28:53.577740+0000","last_fullsized":"2024-09-21T08:28:53.577630+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:27:14.826047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.826047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"198'2","reported_seq":300,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.601678+0000","last_change":"2024-09-21T08:28:53.601978+0000","last_active":"2024-09-21T08:30:25.601678+0000","last_peered":"2024-09-21T08:30:25.601678+0000","last_clean":"2024-09-21T08:28:51.584776+0000","last_became_active":"2024-09-21T08:28:53.601978+0000","last_became_peered":"2024-09-21T08:28:53.601978+0000","last_unstale":"2024-09-21T08:30:25.601678+0000","last_undegraded":"2024-09-21T08:28:53.576796+0000","last_fullsized":"2024-09-21T08:28:53.576697+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'2","last_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_deep_scrub":"198'2","last_deep_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_clean_scrub_stamp":"2024-09-21T08:26:32.819313+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151034+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":801,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603646+0000","last_change":"2024-09-21T08:28:53.597832+0000","last_active":"2024-09-21T08:30:25.603646+0000","last_peered":"2024-09-21T08:30:25.603646+0000","last_clean":"2024-09-21T08:28:51.587327+0000","last_became_active":"2024-09-21T08:28:53.597832+0000","last_became_peered":"2024-09-21T08:28:53.597832+0000","last_unstale":"2024-09-21T08:30:25.603646+0000","last_undegraded":"2024-09-21T08:28:53.582409+0000","last_fullsized":"2024-09-21T08:28:53.582308+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"340'17","reported_seq":428,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.592627+0000","last_change":"2024-09-21T08:28:53.600523+0000","last_active":"2024-09-21T08:30:25.592627+0000","last_peered":"2024-09-21T08:30:25.592627+0000","last_clean":"2024-09-21T08:28:51.556529+0000","last_became_active":"2024-09-21T08:28:53.600523+0000","last_became_peered":"2024-09-21T08:28:53.600523+0000","last_unstale":"2024-09-21T08:30:25.592627+0000","last_undegraded":"2024-09-21T08:28:53.585227+0000","last_fullsized":"2024-09-21T08:28:53.585102+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'15","last_scrub_stamp":"2024-09-21T08:28:14.153130+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.153130+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"426'242","reported_seq":2008,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:30:25.610246+0000","last_change":"2024-09-21T08:30:24.758091+0000","last_active":"2024-09-21T08:30:25.610246+0000","last_peered":"2024-09-21T08:30:25.610246+0000","last_clean":"2024-09-21T08:30:25.610246+0000","last_became_active":"2024-09-21T08:28:32.557643+0000","last_became_peered":"2024-09-21T08:28:32.557643+0000","last_unstale":"2024-09-21T08:30:25.610246+0000","last_undegraded":"2024-09-21T08:30:25.610246+0000","last_fullsized":"2024-09-21T08:30:25.610246+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'160","last_scrub_stamp":"2024-09-21T08:27:34.925496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:34.925496+0000","objects_scrubbed":2,"log_size":242,"log_dups_size":0,"ondisk_log_size":242,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.022276219999999999,"stat_sum":{"num_bytes":8689351,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":50,"num_read_kb":58168,"num_write":21,"num_write_kb":9777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":6235496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"95","length":"3"},{"start":"9d","length":"1"},{"start":"9f","length":"4"},{"start":"a6","length":"1"}]},{"pgid":"4.f","version":"73'15","reported_seq":313,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.604659+0000","last_change":"2024-09-21T08:28:53.595462+0000","last_active":"2024-09-21T08:30:25.604659+0000","last_peered":"2024-09-21T08:30:25.604659+0000","last_clean":"2024-09-21T08:28:51.587748+0000","last_became_active":"2024-09-21T08:28:53.595462+0000","last_became_peered":"2024-09-21T08:28:53.595462+0000","last_unstale":"2024-09-21T08:30:25.604659+0000","last_undegraded":"2024-09-21T08:28:53.579415+0000","last_fullsized":"2024-09-21T08:28:53.579321+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:26:46.777115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.777115+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151011+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":297,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605502+0000","last_change":"2024-09-21T08:28:53.595248+0000","last_active":"2024-09-21T08:30:25.605502+0000","last_peered":"2024-09-21T08:30:25.605502+0000","last_clean":"2024-09-21T08:28:51.585248+0000","last_became_active":"2024-09-21T08:28:53.595248+0000","last_became_peered":"2024-09-21T08:28:53.595248+0000","last_unstale":"2024-09-21T08:30:25.605502+0000","last_undegraded":"2024-09-21T08:30:25.605502+0000","last_fullsized":"2024-09-21T08:28:53.579384+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:45.434055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:45.434055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"340'11","reported_seq":836,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.609469+0000","last_change":"2024-09-21T08:28:53.591354+0000","last_active":"2024-09-21T08:30:25.609469+0000","last_peered":"2024-09-21T08:30:25.609469+0000","last_clean":"2024-09-21T08:28:51.555908+0000","last_became_active":"2024-09-21T08:28:53.591354+0000","last_became_peered":"2024-09-21T08:28:53.591354+0000","last_unstale":"2024-09-21T08:30:25.609469+0000","last_undegraded":"2024-09-21T08:28:53.578763+0000","last_fullsized":"2024-09-21T08:28:53.578651+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:26:44.741275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.741275+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814536+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":774,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.609256+0000","last_change":"2024-09-21T08:30:21.800509+0000","last_active":"2024-09-21T08:30:25.609256+0000","last_peered":"2024-09-21T08:30:25.609256+0000","last_clean":"2024-09-21T08:30:25.609256+0000","last_became_active":"2024-09-21T08:28:29.619967+0000","last_became_peered":"2024-09-21T08:28:29.619967+0000","last_unstale":"2024-09-21T08:30:25.609256+0000","last_undegraded":"2024-09-21T08:30:25.609256+0000","last_fullsized":"2024-09-21T08:30:25.609256+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:21.800427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:21.800427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:50.788583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"179'19","reported_seq":1457,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.592198+0000","last_change":"2024-09-21T08:28:53.592078+0000","last_active":"2024-09-21T08:30:25.592198+0000","last_peered":"2024-09-21T08:30:25.592198+0000","last_clean":"2024-09-21T08:28:51.554873+0000","last_became_active":"2024-09-21T08:28:53.592078+0000","last_became_peered":"2024-09-21T08:28:53.592078+0000","last_unstale":"2024-09-21T08:30:25.592198+0000","last_undegraded":"2024-09-21T08:30:25.592198+0000","last_fullsized":"2024-09-21T08:28:53.580654+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:27:36.268364+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.268364+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057177200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"4.0","version":"73'18","reported_seq":730,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.604725+0000","last_change":"2024-09-21T08:28:53.593785+0000","last_active":"2024-09-21T08:30:25.604725+0000","last_peered":"2024-09-21T08:30:25.604725+0000","last_clean":"2024-09-21T08:28:51.584787+0000","last_became_active":"2024-09-21T08:28:53.593785+0000","last_became_peered":"2024-09-21T08:28:53.593785+0000","last_unstale":"2024-09-21T08:30:25.604725+0000","last_undegraded":"2024-09-21T08:28:53.574919+0000","last_fullsized":"2024-09-21T08:28:53.574229+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-09-21T08:26:54.003173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:54.003173+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151055+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":505,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.607923+0000","last_change":"2024-09-21T08:28:53.602596+0000","last_active":"2024-09-21T08:30:25.607923+0000","last_peered":"2024-09-21T08:30:25.607923+0000","last_clean":"2024-09-21T08:28:51.555756+0000","last_became_active":"2024-09-21T08:28:53.602596+0000","last_became_peered":"2024-09-21T08:28:53.602596+0000","last_unstale":"2024-09-21T08:30:25.607923+0000","last_undegraded":"2024-09-21T08:30:25.607923+0000","last_fullsized":"2024-09-21T08:28:53.579349+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814374+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":605,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605968+0000","last_change":"2024-09-21T08:28:53.587446+0000","last_active":"2024-09-21T08:30:25.605968+0000","last_peered":"2024-09-21T08:30:25.605968+0000","last_clean":"2024-09-21T08:28:27.553250+0000","last_became_active":"2024-09-21T08:28:53.587446+0000","last_became_peered":"2024-09-21T08:28:53.587446+0000","last_unstale":"2024-09-21T08:30:25.605968+0000","last_undegraded":"2024-09-21T08:30:25.605968+0000","last_fullsized":"2024-09-21T08:28:53.574310+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:32.813973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:32.813973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":453,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605935+0000","last_change":"2024-09-21T08:28:53.587280+0000","last_active":"2024-09-21T08:30:25.605935+0000","last_peered":"2024-09-21T08:30:25.605935+0000","last_clean":"2024-09-21T08:26:47.628755+0000","last_became_active":"2024-09-21T08:28:53.587280+0000","last_became_peered":"2024-09-21T08:28:53.587280+0000","last_unstale":"2024-09-21T08:30:25.605935+0000","last_undegraded":"2024-09-21T08:30:25.605935+0000","last_fullsized":"2024-09-21T08:28:53.574372+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:16.341564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.341564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"411'350","reported_seq":1965,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.604092+0000","last_change":"2024-09-21T08:28:53.605477+0000","last_active":"2024-09-21T08:30:25.604092+0000","last_peered":"2024-09-21T08:30:25.604092+0000","last_clean":"2024-09-21T08:28:52.666434+0000","last_became_active":"2024-09-21T08:28:53.605477+0000","last_became_peered":"2024-09-21T08:28:53.605477+0000","last_unstale":"2024-09-21T08:30:25.604092+0000","last_undegraded":"2024-09-21T08:28:53.586515+0000","last_fullsized":"2024-09-21T08:28:53.586400+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'273","last_scrub_stamp":"2024-09-21T08:27:50.318620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.318620+0000","objects_scrubbed":3,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":49,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.073725586999999995,"stat_sum":{"num_bytes":20243096,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":65,"num_read_kb":73718,"num_write":41,"num_write_kb":14479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":3845252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.1","version":"239'21","reported_seq":824,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606190+0000","last_change":"2024-09-21T08:28:53.588019+0000","last_active":"2024-09-21T08:30:25.606190+0000","last_peered":"2024-09-21T08:30:25.606190+0000","last_clean":"2024-09-21T08:28:51.554459+0000","last_became_active":"2024-09-21T08:28:53.588019+0000","last_became_peered":"2024-09-21T08:28:53.588019+0000","last_unstale":"2024-09-21T08:30:25.606190+0000","last_undegraded":"2024-09-21T08:28:53.575510+0000","last_fullsized":"2024-09-21T08:28:53.575377+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21","last_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_deep_scrub":"239'21","last_deep_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.969642+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814600+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":739,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603835+0000","last_change":"2024-09-21T08:28:53.598446+0000","last_active":"2024-09-21T08:30:25.603835+0000","last_peered":"2024-09-21T08:30:25.603835+0000","last_clean":"2024-09-21T08:28:51.587388+0000","last_became_active":"2024-09-21T08:28:53.598446+0000","last_became_peered":"2024-09-21T08:28:53.598446+0000","last_unstale":"2024-09-21T08:30:25.603835+0000","last_undegraded":"2024-09-21T08:28:53.582469+0000","last_fullsized":"2024-09-21T08:28:53.582373+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150790+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"317'16","reported_seq":337,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.605046+0000","last_change":"2024-09-21T08:28:29.687150+0000","last_active":"2024-09-21T08:30:25.605046+0000","last_peered":"2024-09-21T08:30:25.605046+0000","last_clean":"2024-09-21T08:30:25.605046+0000","last_became_active":"2024-09-21T08:28:29.634231+0000","last_became_peered":"2024-09-21T08:28:29.634231+0000","last_unstale":"2024-09-21T08:30:25.605046+0000","last_undegraded":"2024-09-21T08:30:25.605046+0000","last_fullsized":"2024-09-21T08:30:25.605046+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'14","last_scrub_stamp":"2024-09-21T08:27:36.998538+0000","last_deep_scrub":"263'14","last_deep_scrub_stamp":"2024-09-21T08:27:36.998538+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.998538+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":753,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.591834+0000","last_change":"2024-09-21T08:28:53.591306+0000","last_active":"2024-09-21T08:30:25.591834+0000","last_peered":"2024-09-21T08:30:25.591834+0000","last_clean":"2024-09-21T08:28:51.554109+0000","last_became_active":"2024-09-21T08:28:53.591306+0000","last_became_peered":"2024-09-21T08:28:53.591306+0000","last_unstale":"2024-09-21T08:30:25.591834+0000","last_undegraded":"2024-09-21T08:30:25.591834+0000","last_fullsized":"2024-09-21T08:28:53.580235+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.344185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.344185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"179'43","reported_seq":492,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.602925+0000","last_change":"2024-09-21T08:28:53.589976+0000","last_active":"2024-09-21T08:30:25.602925+0000","last_peered":"2024-09-21T08:30:25.602925+0000","last_clean":"2024-09-21T08:28:51.585006+0000","last_became_active":"2024-09-21T08:28:53.589976+0000","last_became_peered":"2024-09-21T08:28:53.589976+0000","last_unstale":"2024-09-21T08:30:25.602925+0000","last_undegraded":"2024-09-21T08:30:25.602925+0000","last_fullsized":"2024-09-21T08:28:53.574347+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:50.158232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.158232+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":48,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.075152813999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"4.2","version":"73'10","reported_seq":332,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.605056+0000","last_change":"2024-09-21T08:29:58.300891+0000","last_active":"2024-09-21T08:30:25.605056+0000","last_peered":"2024-09-21T08:30:25.605056+0000","last_clean":"2024-09-21T08:30:25.605056+0000","last_became_active":"2024-09-21T08:28:29.638492+0000","last_became_peered":"2024-09-21T08:28:29.638492+0000","last_unstale":"2024-09-21T08:30:25.605056+0000","last_undegraded":"2024-09-21T08:30:25.605056+0000","last_fullsized":"2024-09-21T08:30:25.605056+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:29:58.300806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.300806+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:17.610082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":801,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606944+0000","last_change":"2024-09-21T08:28:53.588911+0000","last_active":"2024-09-21T08:30:25.606944+0000","last_peered":"2024-09-21T08:30:25.606944+0000","last_clean":"2024-09-21T08:28:51.554689+0000","last_became_active":"2024-09-21T08:28:53.588911+0000","last_became_peered":"2024-09-21T08:28:53.588911+0000","last_unstale":"2024-09-21T08:30:25.606944+0000","last_undegraded":"2024-09-21T08:28:53.576074+0000","last_fullsized":"2024-09-21T08:28:53.575948+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:16.003225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.003225+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"340'16","reported_seq":560,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.605992+0000","last_change":"2024-09-21T08:28:53.594258+0000","last_active":"2024-09-21T08:30:25.605992+0000","last_peered":"2024-09-21T08:30:25.605992+0000","last_clean":"2024-09-21T08:28:51.556539+0000","last_became_active":"2024-09-21T08:28:53.594258+0000","last_became_peered":"2024-09-21T08:28:53.594258+0000","last_unstale":"2024-09-21T08:30:25.605992+0000","last_undegraded":"2024-09-21T08:28:53.574644+0000","last_fullsized":"2024-09-21T08:28:53.574523+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'13","last_scrub_stamp":"2024-09-21T08:27:27.874952+0000","last_deep_scrub":"176'11","last_deep_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_clean_scrub_stamp":"2024-09-21T08:27:27.874952+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814689+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":588,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.605485+0000","last_change":"2024-09-21T08:28:37.565930+0000","last_active":"2024-09-21T08:30:25.605485+0000","last_peered":"2024-09-21T08:30:25.605485+0000","last_clean":"2024-09-21T08:30:25.605485+0000","last_became_active":"2024-09-21T08:28:37.565624+0000","last_became_peered":"2024-09-21T08:28:37.565624+0000","last_unstale":"2024-09-21T08:30:25.605485+0000","last_undegraded":"2024-09-21T08:30:25.605485+0000","last_fullsized":"2024-09-21T08:30:25.605485+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.258570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.258570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"429'201","reported_seq":2052,"reported_epoch":432,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:30:25.608205+0000","last_change":"2024-09-21T08:30:24.757778+0000","last_active":"2024-09-21T08:30:25.608205+0000","last_peered":"2024-09-21T08:30:25.608205+0000","last_clean":"2024-09-21T08:30:25.608205+0000","last_became_active":"2024-09-21T08:28:07.549600+0000","last_became_peered":"2024-09-21T08:28:07.549600+0000","last_unstale":"2024-09-21T08:30:25.608205+0000","last_undegraded":"2024-09-21T08:30:25.608205+0000","last_fullsized":"2024-09-21T08:30:25.608205+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'166","last_scrub_stamp":"2024-09-21T08:29:32.253921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.253921+0000","objects_scrubbed":3,"log_size":201,"log_dups_size":0,"ondisk_log_size":201,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:47.173469+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.023280927,"stat_sum":{"num_bytes":11202517,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":38413,"num_write":24,"num_write_kb":8854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1484609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"95","length":"3"},{"start":"9d","length":"1"},{"start":"9f","length":"4"},{"start":"a6","length":"1"}]},{"pgid":"4.3","version":"73'19","reported_seq":318,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606288+0000","last_change":"2024-09-21T08:28:53.593536+0000","last_active":"2024-09-21T08:30:25.606288+0000","last_peered":"2024-09-21T08:30:25.606288+0000","last_clean":"2024-09-21T08:28:51.557090+0000","last_became_active":"2024-09-21T08:28:53.593536+0000","last_became_peered":"2024-09-21T08:28:53.593536+0000","last_unstale":"2024-09-21T08:30:25.606288+0000","last_undegraded":"2024-09-21T08:28:53.574981+0000","last_fullsized":"2024-09-21T08:28:53.574872+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:27:55.260127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:55.260127+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":294,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.603492+0000","last_change":"2024-09-21T08:28:04.726932+0000","last_active":"2024-09-21T08:30:25.603492+0000","last_peered":"2024-09-21T08:30:25.603492+0000","last_clean":"2024-09-21T08:30:25.603492+0000","last_became_active":"2024-09-21T08:28:04.723998+0000","last_became_peered":"2024-09-21T08:28:04.723998+0000","last_unstale":"2024-09-21T08:30:25.603492+0000","last_undegraded":"2024-09-21T08:30:25.603492+0000","last_fullsized":"2024-09-21T08:30:25.603492+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:54.289650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:54.289650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":248,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.606658+0000","last_change":"2024-09-21T08:28:53.588949+0000","last_active":"2024-09-21T08:30:25.606658+0000","last_peered":"2024-09-21T08:30:25.606658+0000","last_clean":"2024-09-21T08:28:51.557587+0000","last_became_active":"2024-09-21T08:28:53.588949+0000","last_became_peered":"2024-09-21T08:28:53.588949+0000","last_unstale":"2024-09-21T08:30:25.606658+0000","last_undegraded":"2024-09-21T08:30:25.606658+0000","last_fullsized":"2024-09-21T08:28:53.575963+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.339126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.339126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814419+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":297,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.601413+0000","last_change":"2024-09-21T08:28:53.600582+0000","last_active":"2024-09-21T08:30:25.601413+0000","last_peered":"2024-09-21T08:30:25.601413+0000","last_clean":"2024-09-21T08:28:51.584621+0000","last_became_active":"2024-09-21T08:28:53.600582+0000","last_became_peered":"2024-09-21T08:28:53.600582+0000","last_unstale":"2024-09-21T08:30:25.601413+0000","last_undegraded":"2024-09-21T08:30:25.601413+0000","last_fullsized":"2024-09-21T08:28:53.576129+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:46.263984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.263984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151026+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"413'179","reported_seq":1893,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603323+0000","last_change":"2024-09-21T08:28:53.602072+0000","last_active":"2024-09-21T08:30:25.603323+0000","last_peered":"2024-09-21T08:30:25.603323+0000","last_clean":"2024-09-21T08:28:51.587562+0000","last_became_active":"2024-09-21T08:28:53.602072+0000","last_became_peered":"2024-09-21T08:28:53.602072+0000","last_unstale":"2024-09-21T08:30:25.603323+0000","last_undegraded":"2024-09-21T08:28:53.578389+0000","last_fullsized":"2024-09-21T08:28:53.578258+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":179,"log_dups_size":0,"ondisk_log_size":179,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":52,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.074195671000000005,"stat_sum":{"num_bytes":27815826,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":66,"num_read_kb":59405,"num_write":60,"num_write_kb":34065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":25647838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"1"}]},{"pgid":"4.4","version":"340'35","reported_seq":349,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603566+0000","last_change":"2024-09-21T08:28:53.603157+0000","last_active":"2024-09-21T08:30:25.603566+0000","last_peered":"2024-09-21T08:30:25.603566+0000","last_clean":"2024-09-21T08:28:51.586037+0000","last_became_active":"2024-09-21T08:28:53.603157+0000","last_became_peered":"2024-09-21T08:28:53.603157+0000","last_unstale":"2024-09-21T08:30:25.603566+0000","last_undegraded":"2024-09-21T08:28:53.579248+0000","last_fullsized":"2024-09-21T08:28:53.579125+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'32","last_scrub_stamp":"2024-09-21T08:27:11.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:11.407384+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151048+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":689,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.605302+0000","last_change":"2024-09-21T08:28:29.625492+0000","last_active":"2024-09-21T08:30:25.605302+0000","last_peered":"2024-09-21T08:30:25.605302+0000","last_clean":"2024-09-21T08:30:25.605302+0000","last_became_active":"2024-09-21T08:28:29.625170+0000","last_became_peered":"2024-09-21T08:28:29.625170+0000","last_unstale":"2024-09-21T08:30:25.605302+0000","last_undegraded":"2024-09-21T08:30:25.605302+0000","last_fullsized":"2024-09-21T08:30:25.605302+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:14.370256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.370256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":825,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606251+0000","last_change":"2024-09-21T08:28:53.594367+0000","last_active":"2024-09-21T08:30:25.606251+0000","last_peered":"2024-09-21T08:30:25.606251+0000","last_clean":"2024-09-21T08:28:51.557303+0000","last_became_active":"2024-09-21T08:28:53.594367+0000","last_became_peered":"2024-09-21T08:28:53.594367+0000","last_unstale":"2024-09-21T08:30:25.606251+0000","last_undegraded":"2024-09-21T08:28:53.574945+0000","last_fullsized":"2024-09-21T08:28:53.574836+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814470+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":736,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605108+0000","last_change":"2024-09-21T08:28:53.603129+0000","last_active":"2024-09-21T08:30:25.605108+0000","last_peered":"2024-09-21T08:30:25.605108+0000","last_clean":"2024-09-21T08:28:51.587295+0000","last_became_active":"2024-09-21T08:28:53.603129+0000","last_became_peered":"2024-09-21T08:28:53.603129+0000","last_unstale":"2024-09-21T08:30:25.605108+0000","last_undegraded":"2024-09-21T08:30:25.605108+0000","last_fullsized":"2024-09-21T08:28:53.579036+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:45.237804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.237804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150991+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":607,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.609614+0000","last_change":"2024-09-21T08:28:53.600966+0000","last_active":"2024-09-21T08:30:25.609614+0000","last_peered":"2024-09-21T08:30:25.609614+0000","last_clean":"2024-09-21T08:28:02.651561+0000","last_became_active":"2024-09-21T08:28:53.600966+0000","last_became_peered":"2024-09-21T08:28:53.600966+0000","last_unstale":"2024-09-21T08:30:25.609614+0000","last_undegraded":"2024-09-21T08:30:25.609614+0000","last_fullsized":"2024-09-21T08:28:53.579074+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.844729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.844729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:29.824429+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"432'133","reported_seq":1868,"reported_epoch":432,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:25.602482+0000","last_change":"2024-09-21T08:30:25.602482+0000","last_active":"2024-09-21T08:30:25.602482+0000","last_peered":"2024-09-21T08:30:25.602482+0000","last_clean":"2024-09-21T08:30:25.602482+0000","last_became_active":"2024-09-21T08:28:29.626340+0000","last_became_peered":"2024-09-21T08:28:29.626340+0000","last_unstale":"2024-09-21T08:30:25.602482+0000","last_undegraded":"2024-09-21T08:30:25.602482+0000","last_fullsized":"2024-09-21T08:30:25.602482+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'93","last_scrub_stamp":"2024-09-21T08:29:13.277953+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:29:13.277953+0000","objects_scrubbed":3,"log_size":133,"log_dups_size":0,"ondisk_log_size":133,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0015864620000000001,"stat_sum":{"num_bytes":5203142,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":35599,"num_write":14,"num_write_kb":7300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":11198550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"95","length":"4"},{"start":"9a","length":"1"},{"start":"9d","length":"6"},{"start":"a6","length":"1"},{"start":"a8","length":"4"}]},{"pgid":"4.7","version":"73'13","reported_seq":335,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.604669+0000","last_change":"2024-09-21T08:28:53.595533+0000","last_active":"2024-09-21T08:30:25.604669+0000","last_peered":"2024-09-21T08:30:25.604669+0000","last_clean":"2024-09-21T08:28:51.587779+0000","last_became_active":"2024-09-21T08:28:53.595533+0000","last_became_peered":"2024-09-21T08:28:53.595533+0000","last_unstale":"2024-09-21T08:30:25.604669+0000","last_undegraded":"2024-09-21T08:28:53.580029+0000","last_fullsized":"2024-09-21T08:28:53.579372+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-09-21T08:28:16.386950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.386950+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":308,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.605414+0000","last_change":"2024-09-21T08:29:56.368212+0000","last_active":"2024-09-21T08:30:25.605414+0000","last_peered":"2024-09-21T08:30:25.605414+0000","last_clean":"2024-09-21T08:30:25.605414+0000","last_became_active":"2024-09-21T08:28:29.629320+0000","last_became_peered":"2024-09-21T08:28:29.629320+0000","last_unstale":"2024-09-21T08:30:25.605414+0000","last_undegraded":"2024-09-21T08:30:25.605414+0000","last_fullsized":"2024-09-21T08:30:25.605414+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:56.368144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:29:56.368144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:02.232771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":765,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603526+0000","last_change":"2024-09-21T08:28:53.605131+0000","last_active":"2024-09-21T08:30:25.603526+0000","last_peered":"2024-09-21T08:30:25.603526+0000","last_clean":"2024-09-21T08:28:51.585166+0000","last_became_active":"2024-09-21T08:28:53.605131+0000","last_became_peered":"2024-09-21T08:28:53.605131+0000","last_unstale":"2024-09-21T08:30:25.603526+0000","last_undegraded":"2024-09-21T08:28:53.583253+0000","last_fullsized":"2024-09-21T08:28:53.582934+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:59.262463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:27:59.262463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":754,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.593248+0000","last_change":"2024-09-21T08:28:53.598122+0000","last_active":"2024-09-21T08:30:25.593248+0000","last_peered":"2024-09-21T08:30:25.593248+0000","last_clean":"2024-09-21T08:28:51.556093+0000","last_became_active":"2024-09-21T08:28:53.598122+0000","last_became_peered":"2024-09-21T08:28:53.598122+0000","last_unstale":"2024-09-21T08:30:25.593248+0000","last_undegraded":"2024-09-21T08:30:25.593248+0000","last_fullsized":"2024-09-21T08:28:53.584774+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.374076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.374076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":652,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.606766+0000","last_change":"2024-09-21T08:28:53.587481+0000","last_active":"2024-09-21T08:30:25.606766+0000","last_peered":"2024-09-21T08:30:25.606766+0000","last_clean":"2024-09-21T08:28:51.554685+0000","last_became_active":"2024-09-21T08:28:53.587481+0000","last_became_peered":"2024-09-21T08:28:53.587481+0000","last_unstale":"2024-09-21T08:30:25.606766+0000","last_undegraded":"2024-09-21T08:30:25.606766+0000","last_fullsized":"2024-09-21T08:28:53.575943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:08.831386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:08.831386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814585+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"413'305","reported_seq":1862,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.601979+0000","last_change":"2024-09-21T08:28:53.600711+0000","last_active":"2024-09-21T08:30:25.601979+0000","last_peered":"2024-09-21T08:30:25.601979+0000","last_clean":"2024-09-21T08:28:52.661593+0000","last_became_active":"2024-09-21T08:28:53.600711+0000","last_became_peered":"2024-09-21T08:28:53.600711+0000","last_unstale":"2024-09-21T08:30:25.601979+0000","last_undegraded":"2024-09-21T08:28:53.576301+0000","last_fullsized":"2024-09-21T08:28:53.576176+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'245","last_scrub_stamp":"2024-09-21T08:26:30.864040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.864040+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.075799959,"stat_sum":{"num_bytes":6305048,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":48,"num_read_kb":51018,"num_write":34,"num_write_kb":8053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1853978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.6","version":"73'12","reported_seq":762,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.604067+0000","last_change":"2024-09-21T08:28:53.603118+0000","last_active":"2024-09-21T08:30:25.604067+0000","last_peered":"2024-09-21T08:30:25.604067+0000","last_clean":"2024-09-21T08:28:51.587443+0000","last_became_active":"2024-09-21T08:28:53.603118+0000","last_became_peered":"2024-09-21T08:28:53.603118+0000","last_unstale":"2024-09-21T08:30:25.604067+0000","last_undegraded":"2024-09-21T08:28:53.582699+0000","last_fullsized":"2024-09-21T08:28:53.582597+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:26:07.384640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:07.384640+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150899+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":623,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.606539+0000","last_change":"2024-09-21T08:28:53.599951+0000","last_active":"2024-09-21T08:30:25.606539+0000","last_peered":"2024-09-21T08:30:25.606539+0000","last_clean":"2024-09-21T08:28:51.555106+0000","last_became_active":"2024-09-21T08:28:53.599951+0000","last_became_peered":"2024-09-21T08:28:53.599951+0000","last_unstale":"2024-09-21T08:30:25.606539+0000","last_undegraded":"2024-09-21T08:30:25.606539+0000","last_fullsized":"2024-09-21T08:28:53.575796+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.917891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.917891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1195,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603751+0000","last_change":"2024-09-21T08:28:53.594845+0000","last_active":"2024-09-21T08:30:25.603751+0000","last_peered":"2024-09-21T08:30:25.603751+0000","last_clean":"2024-09-21T08:28:52.670294+0000","last_became_active":"2024-09-21T08:28:53.594845+0000","last_became_peered":"2024-09-21T08:28:53.594845+0000","last_unstale":"2024-09-21T08:30:25.603751+0000","last_undegraded":"2024-09-21T08:28:53.577170+0000","last_fullsized":"2024-09-21T08:28:53.575736+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150797+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":392,"num_read_kb":387,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":765,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.609032+0000","last_change":"2024-09-21T08:28:53.602597+0000","last_active":"2024-09-21T08:30:25.609032+0000","last_peered":"2024-09-21T08:30:25.609032+0000","last_clean":"2024-09-21T08:28:51.556542+0000","last_became_active":"2024-09-21T08:28:53.602597+0000","last_became_peered":"2024-09-21T08:28:53.602597+0000","last_unstale":"2024-09-21T08:30:25.609032+0000","last_undegraded":"2024-09-21T08:30:25.609032+0000","last_fullsized":"2024-09-21T08:28:53.580807+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.812789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.812789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814427+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":302,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.603406+0000","last_change":"2024-09-21T08:28:17.424621+0000","last_active":"2024-09-21T08:30:25.603406+0000","last_peered":"2024-09-21T08:30:25.603406+0000","last_clean":"2024-09-21T08:30:25.603406+0000","last_became_active":"2024-09-21T08:28:04.723837+0000","last_became_peered":"2024-09-21T08:28:04.723837+0000","last_unstale":"2024-09-21T08:30:25.603406+0000","last_undegraded":"2024-09-21T08:30:25.603406+0000","last_fullsized":"2024-09-21T08:30:25.603406+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:17.424515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:17.424515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"416'157","reported_seq":1768,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.608033+0000","last_change":"2024-09-21T08:28:53.589874+0000","last_active":"2024-09-21T08:30:25.608033+0000","last_peered":"2024-09-21T08:30:25.608033+0000","last_clean":"2024-09-21T08:28:51.554645+0000","last_became_active":"2024-09-21T08:28:53.589874+0000","last_became_peered":"2024-09-21T08:28:53.589874+0000","last_unstale":"2024-09-21T08:30:25.608033+0000","last_undegraded":"2024-09-21T08:28:53.577395+0000","last_fullsized":"2024-09-21T08:28:53.577254+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'138","last_scrub_stamp":"2024-09-21T08:28:41.855754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:41.855754+0000","objects_scrubbed":1,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027851600000000001,"stat_sum":{"num_bytes":11714710,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":49,"num_read_kb":63283,"num_write":24,"num_write_kb":8369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":9282889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"4.5","version":"340'22","reported_seq":610,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.608937+0000","last_change":"2024-09-21T08:28:53.600773+0000","last_active":"2024-09-21T08:30:25.608937+0000","last_peered":"2024-09-21T08:30:25.608937+0000","last_clean":"2024-09-21T08:27:13.582840+0000","last_became_active":"2024-09-21T08:28:53.600773+0000","last_became_peered":"2024-09-21T08:28:53.600773+0000","last_unstale":"2024-09-21T08:30:25.608937+0000","last_undegraded":"2024-09-21T08:28:53.577472+0000","last_fullsized":"2024-09-21T08:28:53.577362+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-21T08:28:13.956509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.956509+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814758+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":662,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.607910+0000","last_change":"2024-09-21T08:28:53.600983+0000","last_active":"2024-09-21T08:30:25.607910+0000","last_peered":"2024-09-21T08:30:25.607910+0000","last_clean":"2024-09-21T08:28:51.558067+0000","last_became_active":"2024-09-21T08:28:53.600983+0000","last_became_peered":"2024-09-21T08:28:53.600983+0000","last_unstale":"2024-09-21T08:30:25.607910+0000","last_undegraded":"2024-09-21T08:30:25.607910+0000","last_fullsized":"2024-09-21T08:28:53.577601+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.045353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.045353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814528+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":1008,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.601623+0000","last_change":"2024-09-21T08:28:53.604882+0000","last_active":"2024-09-21T08:30:25.601623+0000","last_peered":"2024-09-21T08:30:25.601623+0000","last_clean":"2024-09-21T08:28:51.584594+0000","last_became_active":"2024-09-21T08:28:53.604882+0000","last_became_peered":"2024-09-21T08:28:53.604882+0000","last_unstale":"2024-09-21T08:30:25.601623+0000","last_undegraded":"2024-09-21T08:28:53.575806+0000","last_fullsized":"2024-09-21T08:28:53.575705+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150828+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":231,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.607847+0000","last_change":"2024-09-21T08:28:53.589976+0000","last_active":"2024-09-21T08:30:25.607847+0000","last_peered":"2024-09-21T08:30:25.607847+0000","last_clean":"2024-09-21T08:28:51.557022+0000","last_became_active":"2024-09-21T08:28:53.589976+0000","last_became_peered":"2024-09-21T08:28:53.589976+0000","last_unstale":"2024-09-21T08:30:25.607847+0000","last_undegraded":"2024-09-21T08:30:25.607847+0000","last_fullsized":"2024-09-21T08:28:53.576557+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814434+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":779,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.609740+0000","last_change":"2024-09-21T08:28:53.602915+0000","last_active":"2024-09-21T08:30:25.609740+0000","last_peered":"2024-09-21T08:30:25.609740+0000","last_clean":"2024-09-21T08:28:51.556967+0000","last_became_active":"2024-09-21T08:28:53.602915+0000","last_became_peered":"2024-09-21T08:28:53.602915+0000","last_unstale":"2024-09-21T08:30:25.609740+0000","last_undegraded":"2024-09-21T08:30:25.609740+0000","last_fullsized":"2024-09-21T08:28:53.581134+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814367+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"423'142","reported_seq":610,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606362+0000","last_change":"2024-09-21T08:28:53.596348+0000","last_active":"2024-09-21T08:30:25.606362+0000","last_peered":"2024-09-21T08:30:25.606362+0000","last_clean":"2024-09-21T08:28:52.661631+0000","last_became_active":"2024-09-21T08:28:53.596348+0000","last_became_peered":"2024-09-21T08:28:53.596348+0000","last_unstale":"2024-09-21T08:30:25.606362+0000","last_undegraded":"2024-09-21T08:28:53.582490+0000","last_fullsized":"2024-09-21T08:28:53.579369+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'93","last_scrub_stamp":"2024-09-21T08:28:11.373358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:11.373358+0000","objects_scrubbed":3,"log_size":142,"log_dups_size":0,"ondisk_log_size":142,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":52,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.070749670000000001,"stat_sum":{"num_bytes":23175321,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":29,"num_read_kb":17874,"num_write":33,"num_write_kb":10379,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":6571636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"1"}]},{"pgid":"4.e","version":"73'11","reported_seq":317,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.602213+0000","last_change":"2024-09-21T08:28:53.597299+0000","last_active":"2024-09-21T08:30:25.602213+0000","last_peered":"2024-09-21T08:30:25.602213+0000","last_clean":"2024-09-21T08:28:51.585940+0000","last_became_active":"2024-09-21T08:28:53.597299+0000","last_became_peered":"2024-09-21T08:28:53.597299+0000","last_unstale":"2024-09-21T08:30:25.602213+0000","last_undegraded":"2024-09-21T08:28:53.581018+0000","last_fullsized":"2024-09-21T08:28:53.580915+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:28:14.416729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.416729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":737,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.590953+0000","last_change":"2024-09-21T08:28:53.591487+0000","last_active":"2024-09-21T08:30:25.590953+0000","last_peered":"2024-09-21T08:30:25.590953+0000","last_clean":"2024-09-21T08:28:51.552892+0000","last_became_active":"2024-09-21T08:28:53.591487+0000","last_became_peered":"2024-09-21T08:28:53.591487+0000","last_unstale":"2024-09-21T08:30:25.590953+0000","last_undegraded":"2024-09-21T08:30:25.590953+0000","last_fullsized":"2024-09-21T08:28:53.575527+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:28.347474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:28.347474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":788,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.610325+0000","last_change":"2024-09-21T08:29:55.150130+0000","last_active":"2024-09-21T08:30:25.610325+0000","last_peered":"2024-09-21T08:30:25.610325+0000","last_clean":"2024-09-21T08:30:25.610325+0000","last_became_active":"2024-09-21T08:28:04.724965+0000","last_became_peered":"2024-09-21T08:28:04.724965+0000","last_unstale":"2024-09-21T08:30:25.610325+0000","last_undegraded":"2024-09-21T08:30:25.610325+0000","last_fullsized":"2024-09-21T08:30:25.610325+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:55.150058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:32.961381+0000","last_clean_scrub_stamp":"2024-09-21T08:29:55.150058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:10.970830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":641,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.609057+0000","last_change":"2024-09-21T08:28:53.601979+0000","last_active":"2024-09-21T08:30:25.609057+0000","last_peered":"2024-09-21T08:30:25.609057+0000","last_clean":"2024-09-21T08:28:51.557371+0000","last_became_active":"2024-09-21T08:28:53.601979+0000","last_became_peered":"2024-09-21T08:28:53.601979+0000","last_unstale":"2024-09-21T08:30:25.609057+0000","last_undegraded":"2024-09-21T08:30:25.609057+0000","last_fullsized":"2024-09-21T08:28:53.577942+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.977956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"256'199","reported_seq":1948,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.593353+0000","last_change":"2024-09-21T08:30:25.593719+0000","last_active":"2024-09-21T08:30:25.593353+0000","last_peered":"2024-09-21T08:30:25.593353+0000","last_clean":"2024-09-21T08:30:25.593353+0000","last_became_active":"2024-09-21T08:28:32.558440+0000","last_became_peered":"2024-09-21T08:28:32.558440+0000","last_unstale":"2024-09-21T08:30:25.593353+0000","last_undegraded":"2024-09-21T08:30:25.593353+0000","last_fullsized":"2024-09-21T08:30:25.593353+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'199","last_scrub_stamp":"2024-09-21T08:30:17.742629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:17.742629+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:31.417666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00058208700000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"95","length":"4"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"1"},{"start":"a8","length":"4"}]},{"pgid":"4.d","version":"345'25","reported_seq":787,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.606636+0000","last_change":"2024-09-21T08:28:59.242015+0000","last_active":"2024-09-21T08:30:25.606636+0000","last_peered":"2024-09-21T08:30:25.606636+0000","last_clean":"2024-09-21T08:30:25.606636+0000","last_became_active":"2024-09-21T08:28:37.564492+0000","last_became_peered":"2024-09-21T08:28:37.564492+0000","last_unstale":"2024-09-21T08:30:25.606636+0000","last_undegraded":"2024-09-21T08:30:25.606636+0000","last_fullsized":"2024-09-21T08:30:25.606636+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'25","last_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_deep_scrub":"345'25","last_deep_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.241931+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":297,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.603586+0000","last_change":"2024-09-21T08:28:53.599763+0000","last_active":"2024-09-21T08:30:25.603586+0000","last_peered":"2024-09-21T08:30:25.603586+0000","last_clean":"2024-09-21T08:28:51.585236+0000","last_became_active":"2024-09-21T08:28:53.599763+0000","last_became_peered":"2024-09-21T08:28:53.599763+0000","last_unstale":"2024-09-21T08:30:25.603586+0000","last_undegraded":"2024-09-21T08:30:25.603586+0000","last_fullsized":"2024-09-21T08:28:53.586112+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:30.909902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.909902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":303,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.602504+0000","last_change":"2024-09-21T08:28:53.601757+0000","last_active":"2024-09-21T08:30:25.602504+0000","last_peered":"2024-09-21T08:30:25.602504+0000","last_clean":"2024-09-21T08:28:51.585900+0000","last_became_active":"2024-09-21T08:28:53.601757+0000","last_became_peered":"2024-09-21T08:28:53.601757+0000","last_unstale":"2024-09-21T08:30:25.602504+0000","last_undegraded":"2024-09-21T08:30:25.602504+0000","last_fullsized":"2024-09-21T08:28:53.576584+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":302,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.603697+0000","last_change":"2024-09-21T08:30:00.276844+0000","last_active":"2024-09-21T08:30:25.603697+0000","last_peered":"2024-09-21T08:30:25.603697+0000","last_clean":"2024-09-21T08:30:25.603697+0000","last_became_active":"2024-09-21T08:28:04.726509+0000","last_became_peered":"2024-09-21T08:28:04.726509+0000","last_unstale":"2024-09-21T08:30:25.603697+0000","last_undegraded":"2024-09-21T08:30:25.603697+0000","last_fullsized":"2024-09-21T08:30:25.603697+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:00.276774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:00.276774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:26.484373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"422'77","reported_seq":1686,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.607690+0000","last_change":"2024-09-21T08:28:53.590273+0000","last_active":"2024-09-21T08:30:25.607690+0000","last_peered":"2024-09-21T08:30:25.607690+0000","last_clean":"2024-09-21T08:28:51.554220+0000","last_became_active":"2024-09-21T08:28:53.590273+0000","last_became_peered":"2024-09-21T08:28:53.590273+0000","last_unstale":"2024-09-21T08:30:25.607690+0000","last_undegraded":"2024-09-21T08:28:53.579532+0000","last_fullsized":"2024-09-21T08:28:53.579403+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:28:10.372149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.372149+0000","objects_scrubbed":1,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.019314938,"stat_sum":{"num_bytes":23018803,"num_objects":11,"num_object_clones":10,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":28,"num_read_kb":28274,"num_write":38,"num_write_kb":16470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2823609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":11}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.c","version":"73'10","reported_seq":309,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.602695+0000","last_change":"2024-09-21T08:28:53.601905+0000","last_active":"2024-09-21T08:30:25.602695+0000","last_peered":"2024-09-21T08:30:25.602695+0000","last_clean":"2024-09-21T08:28:51.586041+0000","last_became_active":"2024-09-21T08:28:53.601905+0000","last_became_peered":"2024-09-21T08:28:53.601905+0000","last_unstale":"2024-09-21T08:30:25.602695+0000","last_undegraded":"2024-09-21T08:28:53.577157+0000","last_fullsized":"2024-09-21T08:28:53.576937+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:26:33.823950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.823950+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":662,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.603294+0000","last_change":"2024-09-21T08:28:53.602716+0000","last_active":"2024-09-21T08:30:25.603294+0000","last_peered":"2024-09-21T08:30:25.603294+0000","last_clean":"2024-09-21T08:28:51.585127+0000","last_became_active":"2024-09-21T08:28:53.602716+0000","last_became_peered":"2024-09-21T08:28:53.602716+0000","last_unstale":"2024-09-21T08:30:25.603294+0000","last_undegraded":"2024-09-21T08:30:25.603294+0000","last_fullsized":"2024-09-21T08:28:53.576785+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:54.216528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:54.216528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"340'10","reported_seq":277,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.609150+0000","last_change":"2024-09-21T08:28:53.589819+0000","last_active":"2024-09-21T08:30:25.609150+0000","last_peered":"2024-09-21T08:30:25.609150+0000","last_clean":"2024-09-21T08:28:51.555429+0000","last_became_active":"2024-09-21T08:28:53.589819+0000","last_became_peered":"2024-09-21T08:28:53.589819+0000","last_unstale":"2024-09-21T08:30:25.609150+0000","last_undegraded":"2024-09-21T08:28:53.578694+0000","last_fullsized":"2024-09-21T08:28:53.577943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:27:51.312273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.312273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":302,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.602393+0000","last_change":"2024-09-21T08:29:40.458127+0000","last_active":"2024-09-21T08:30:25.602393+0000","last_peered":"2024-09-21T08:30:25.602393+0000","last_clean":"2024-09-21T08:30:25.602393+0000","last_became_active":"2024-09-21T08:28:04.722744+0000","last_became_peered":"2024-09-21T08:28:04.722744+0000","last_unstale":"2024-09-21T08:30:25.602393+0000","last_undegraded":"2024-09-21T08:30:25.602393+0000","last_fullsized":"2024-09-21T08:30:25.602393+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:40.458054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:40.458054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:06.465951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"413'324","reported_seq":1879,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.605627+0000","last_change":"2024-09-21T08:28:53.587761+0000","last_active":"2024-09-21T08:30:25.605627+0000","last_peered":"2024-09-21T08:30:25.605627+0000","last_clean":"2024-09-21T08:28:52.588981+0000","last_became_active":"2024-09-21T08:28:53.587761+0000","last_became_peered":"2024-09-21T08:28:53.587761+0000","last_unstale":"2024-09-21T08:30:25.605627+0000","last_undegraded":"2024-09-21T08:28:53.574111+0000","last_fullsized":"2024-09-21T08:28:53.573936+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'178","last_scrub_stamp":"2024-09-21T08:26:28.217842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.217842+0000","objects_scrubbed":4,"log_size":324,"log_dups_size":0,"ondisk_log_size":324,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":46,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0347803,"stat_sum":{"num_bytes":20649664,"num_objects":13,"num_object_clones":10,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":2,"num_read":112,"num_read_kb":143318,"num_write":79,"num_write_kb":30896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":23089101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":13}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"4.b","version":"73'9","reported_seq":815,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.602389+0000","last_change":"2024-09-21T08:28:53.595156+0000","last_active":"2024-09-21T08:30:25.602389+0000","last_peered":"2024-09-21T08:30:25.602389+0000","last_clean":"2024-09-21T08:28:51.585036+0000","last_became_active":"2024-09-21T08:28:53.595156+0000","last_became_peered":"2024-09-21T08:28:53.595156+0000","last_unstale":"2024-09-21T08:30:25.602389+0000","last_undegraded":"2024-09-21T08:28:53.576597+0000","last_fullsized":"2024-09-21T08:28:53.576496+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:25:22.369811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.369811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":802,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.605882+0000","last_change":"2024-09-21T08:28:59.403264+0000","last_active":"2024-09-21T08:30:25.605882+0000","last_peered":"2024-09-21T08:30:25.605882+0000","last_clean":"2024-09-21T08:30:25.605882+0000","last_became_active":"2024-09-21T08:28:29.609526+0000","last_became_peered":"2024-09-21T08:28:29.609526+0000","last_unstale":"2024-09-21T08:30:25.605882+0000","last_undegraded":"2024-09-21T08:30:25.605882+0000","last_fullsized":"2024-09-21T08:30:25.605882+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:59.403194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.403194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":656,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.601717+0000","last_change":"2024-09-21T08:28:53.602532+0000","last_active":"2024-09-21T08:30:25.601717+0000","last_peered":"2024-09-21T08:30:25.601717+0000","last_clean":"2024-09-21T08:28:51.584905+0000","last_became_active":"2024-09-21T08:28:53.602532+0000","last_became_peered":"2024-09-21T08:28:53.602532+0000","last_unstale":"2024-09-21T08:30:25.601717+0000","last_undegraded":"2024-09-21T08:30:25.601717+0000","last_fullsized":"2024-09-21T08:28:53.583376+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150836+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":254,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.591233+0000","last_change":"2024-09-21T08:28:53.591119+0000","last_active":"2024-09-21T08:30:25.591233+0000","last_peered":"2024-09-21T08:30:25.591233+0000","last_clean":"2024-09-21T08:28:51.552940+0000","last_became_active":"2024-09-21T08:28:53.591119+0000","last_became_peered":"2024-09-21T08:28:53.591119+0000","last_unstale":"2024-09-21T08:30:25.591233+0000","last_undegraded":"2024-09-21T08:30:25.591233+0000","last_fullsized":"2024-09-21T08:28:53.575921+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:36.308161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:36.308161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"427'354","reported_seq":2110,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603993+0000","last_change":"2024-09-21T08:28:53.606879+0000","last_active":"2024-09-21T08:30:25.603993+0000","last_peered":"2024-09-21T08:30:25.603993+0000","last_clean":"2024-09-21T08:28:52.666146+0000","last_became_active":"2024-09-21T08:28:53.606879+0000","last_became_peered":"2024-09-21T08:28:53.606879+0000","last_unstale":"2024-09-21T08:30:25.603993+0000","last_undegraded":"2024-09-21T08:28:53.584154+0000","last_fullsized":"2024-09-21T08:28:53.584033+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0805860919999999,"stat_sum":{"num_bytes":35514492,"num_objects":16,"num_object_clones":12,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":2,"num_read":164,"num_read_kb":178955,"num_write":104,"num_write_kb":34472,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":24504816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":16}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":317,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603771+0000","last_change":"2024-09-21T08:28:53.600149+0000","last_active":"2024-09-21T08:30:25.603771+0000","last_peered":"2024-09-21T08:30:25.603771+0000","last_clean":"2024-09-21T08:28:51.585660+0000","last_became_active":"2024-09-21T08:28:53.600149+0000","last_became_peered":"2024-09-21T08:28:53.600149+0000","last_unstale":"2024-09-21T08:30:25.603771+0000","last_undegraded":"2024-09-21T08:28:53.586423+0000","last_fullsized":"2024-09-21T08:28:53.586235+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":663,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.607876+0000","last_change":"2024-09-21T08:28:53.589287+0000","last_active":"2024-09-21T08:30:25.607876+0000","last_peered":"2024-09-21T08:30:25.607876+0000","last_clean":"2024-09-21T08:28:51.555026+0000","last_became_active":"2024-09-21T08:28:53.589287+0000","last_became_peered":"2024-09-21T08:28:53.589287+0000","last_unstale":"2024-09-21T08:30:25.607876+0000","last_undegraded":"2024-09-21T08:30:25.607876+0000","last_fullsized":"2024-09-21T08:28:53.576540+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.901202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.901202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814765+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":752,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.591684+0000","last_change":"2024-09-21T08:28:53.592593+0000","last_active":"2024-09-21T08:30:25.591684+0000","last_peered":"2024-09-21T08:30:25.591684+0000","last_clean":"2024-09-21T08:28:51.554029+0000","last_became_active":"2024-09-21T08:28:53.592593+0000","last_became_peered":"2024-09-21T08:28:53.592593+0000","last_unstale":"2024-09-21T08:30:25.591684+0000","last_undegraded":"2024-09-21T08:30:25.591684+0000","last_fullsized":"2024-09-21T08:28:53.577826+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:15.362045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:15.362045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":246,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.601236+0000","last_change":"2024-09-21T08:28:53.586960+0000","last_active":"2024-09-21T08:30:25.601236+0000","last_peered":"2024-09-21T08:30:25.601236+0000","last_clean":"2024-09-21T08:28:51.584413+0000","last_became_active":"2024-09-21T08:28:53.586960+0000","last_became_peered":"2024-09-21T08:28:53.586960+0000","last_unstale":"2024-09-21T08:30:25.601236+0000","last_undegraded":"2024-09-21T08:30:25.601236+0000","last_fullsized":"2024-09-21T08:28:53.575166+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"430'112","reported_seq":1017,"reported_epoch":432,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:25.606574+0000","last_change":"2024-09-21T08:30:24.731326+0000","last_active":"2024-09-21T08:30:25.606574+0000","last_peered":"2024-09-21T08:30:25.606574+0000","last_clean":"2024-09-21T08:30:25.606574+0000","last_became_active":"2024-09-21T08:28:37.567400+0000","last_became_peered":"2024-09-21T08:28:37.567400+0000","last_unstale":"2024-09-21T08:30:25.606574+0000","last_undegraded":"2024-09-21T08:30:25.606574+0000","last_fullsized":"2024-09-21T08:30:25.606574+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'68","last_scrub_stamp":"2024-09-21T08:27:35.317025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:35.317025+0000","objects_scrubbed":0,"log_size":112,"log_dups_size":0,"ondisk_log_size":112,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":8,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.15774026499999999,"stat_sum":{"num_bytes":2840482,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":7,"num_read_kb":3,"num_write":19,"num_write_kb":3134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1183872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"95","length":"3"},{"start":"9d","length":"1"},{"start":"9f","length":"4"},{"start":"a6","length":"1"}]},{"pgid":"4.9","version":"73'12","reported_seq":327,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.602065+0000","last_change":"2024-09-21T08:28:53.604531+0000","last_active":"2024-09-21T08:30:25.602065+0000","last_peered":"2024-09-21T08:30:25.602065+0000","last_clean":"2024-09-21T08:28:51.584845+0000","last_became_active":"2024-09-21T08:28:53.604531+0000","last_became_peered":"2024-09-21T08:28:53.604531+0000","last_unstale":"2024-09-21T08:30:25.602065+0000","last_undegraded":"2024-09-21T08:28:53.582724+0000","last_fullsized":"2024-09-21T08:28:53.582624+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:41.463820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.463820+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":764,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.602782+0000","last_change":"2024-09-21T08:28:53.596741+0000","last_active":"2024-09-21T08:30:25.602782+0000","last_peered":"2024-09-21T08:30:25.602782+0000","last_clean":"2024-09-21T08:28:51.587101+0000","last_became_active":"2024-09-21T08:28:53.596741+0000","last_became_peered":"2024-09-21T08:28:53.596741+0000","last_unstale":"2024-09-21T08:30:25.602782+0000","last_undegraded":"2024-09-21T08:30:25.602782+0000","last_fullsized":"2024-09-21T08:28:53.580812+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:51.769044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.769044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":764,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.605164+0000","last_change":"2024-09-21T08:29:59.267735+0000","last_active":"2024-09-21T08:30:25.605164+0000","last_peered":"2024-09-21T08:30:25.605164+0000","last_clean":"2024-09-21T08:30:25.605164+0000","last_became_active":"2024-09-21T08:28:37.566192+0000","last_became_peered":"2024-09-21T08:28:37.566192+0000","last_unstale":"2024-09-21T08:30:25.605164+0000","last_undegraded":"2024-09-21T08:30:25.605164+0000","last_fullsized":"2024-09-21T08:30:25.605164+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:59.267668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:29:59.267668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.614991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":655,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.602919+0000","last_change":"2024-09-21T08:28:53.595264+0000","last_active":"2024-09-21T08:30:25.602919+0000","last_peered":"2024-09-21T08:30:25.602919+0000","last_clean":"2024-09-21T08:28:51.585131+0000","last_became_active":"2024-09-21T08:28:53.595264+0000","last_became_peered":"2024-09-21T08:28:53.595264+0000","last_unstale":"2024-09-21T08:30:25.602919+0000","last_undegraded":"2024-09-21T08:30:25.602919+0000","last_fullsized":"2024-09-21T08:28:53.576894+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"431'711","reported_seq":2585,"reported_epoch":432,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:25.606888+0000","last_change":"2024-09-21T08:30:24.747069+0000","last_active":"2024-09-21T08:30:25.606888+0000","last_peered":"2024-09-21T08:30:25.606888+0000","last_clean":"2024-09-21T08:30:25.606888+0000","last_became_active":"2024-09-21T08:28:07.549709+0000","last_became_peered":"2024-09-21T08:28:07.549709+0000","last_unstale":"2024-09-21T08:30:25.606888+0000","last_undegraded":"2024-09-21T08:30:25.606888+0000","last_fullsized":"2024-09-21T08:30:25.606888+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'577","last_scrub_stamp":"2024-09-21T08:29:32.265609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.265609+0000","objects_scrubbed":10,"log_size":711,"log_dups_size":0,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:57.795220+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.15094165200000001,"stat_sum":{"num_bytes":54985520,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":139,"num_read_kb":138016,"num_write":84,"num_write_kb":36499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":4385271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"95","length":"4"},{"start":"9a","length":"1"},{"start":"9d","length":"1"},{"start":"9f","length":"3"},{"start":"a6","length":"1"}]},{"pgid":"4.8","version":"73'15","reported_seq":886,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606875+0000","last_change":"2024-09-21T08:28:53.591779+0000","last_active":"2024-09-21T08:30:25.606875+0000","last_peered":"2024-09-21T08:30:25.606875+0000","last_clean":"2024-09-21T08:28:51.558045+0000","last_became_active":"2024-09-21T08:28:53.591779+0000","last_became_peered":"2024-09-21T08:28:53.591779+0000","last_unstale":"2024-09-21T08:30:25.606875+0000","last_undegraded":"2024-09-21T08:28:53.574592+0000","last_fullsized":"2024-09-21T08:28:53.574472+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814358+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":754,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603545+0000","last_change":"2024-09-21T08:28:53.595155+0000","last_active":"2024-09-21T08:30:25.603545+0000","last_peered":"2024-09-21T08:30:25.603545+0000","last_clean":"2024-09-21T08:28:51.587656+0000","last_became_active":"2024-09-21T08:28:53.595155+0000","last_became_peered":"2024-09-21T08:28:53.595155+0000","last_unstale":"2024-09-21T08:30:25.603545+0000","last_undegraded":"2024-09-21T08:28:53.579187+0000","last_fullsized":"2024-09-21T08:28:53.579080+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150751+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"340'16","reported_seq":881,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.667286+0000","last_change":"2024-09-21T08:28:53.604478+0000","last_active":"2024-09-21T08:30:25.667286+0000","last_peered":"2024-09-21T08:30:25.667286+0000","last_clean":"2024-09-21T08:28:51.584962+0000","last_became_active":"2024-09-21T08:28:53.604478+0000","last_became_peered":"2024-09-21T08:28:53.604478+0000","last_unstale":"2024-09-21T08:30:25.667286+0000","last_undegraded":"2024-09-21T08:28:53.580813+0000","last_fullsized":"2024-09-21T08:28:53.580697+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'10","last_scrub_stamp":"2024-09-21T08:25:42.708458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.708458+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150876+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":752,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.606768+0000","last_change":"2024-09-21T08:28:55.397079+0000","last_active":"2024-09-21T08:30:25.606768+0000","last_peered":"2024-09-21T08:30:25.606768+0000","last_clean":"2024-09-21T08:30:25.606768+0000","last_became_active":"2024-09-21T08:28:29.619471+0000","last_became_peered":"2024-09-21T08:28:29.619471+0000","last_unstale":"2024-09-21T08:30:25.606768+0000","last_undegraded":"2024-09-21T08:30:25.606768+0000","last_fullsized":"2024-09-21T08:30:25.606768+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:55.396944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:55.396944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"180'50","reported_seq":1502,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.591775+0000","last_change":"2024-09-21T08:28:53.592066+0000","last_active":"2024-09-21T08:30:25.591775+0000","last_peered":"2024-09-21T08:30:25.591775+0000","last_clean":"2024-09-21T08:28:51.555673+0000","last_became_active":"2024-09-21T08:28:53.592066+0000","last_became_peered":"2024-09-21T08:28:53.592066+0000","last_unstale":"2024-09-21T08:30:25.591775+0000","last_undegraded":"2024-09-21T08:30:25.591775+0000","last_fullsized":"2024-09-21T08:28:53.577992+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:26:50.014780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:50.014780+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":43,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020276460000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"3.10","version":"0'0","reported_seq":550,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.606774+0000","last_change":"2024-09-21T08:29:39.177460+0000","last_active":"2024-09-21T08:30:25.606774+0000","last_peered":"2024-09-21T08:30:25.606774+0000","last_clean":"2024-09-21T08:30:25.606774+0000","last_became_active":"2024-09-21T08:28:29.610987+0000","last_became_peered":"2024-09-21T08:28:29.610987+0000","last_unstale":"2024-09-21T08:30:25.606774+0000","last_undegraded":"2024-09-21T08:30:25.606774+0000","last_fullsized":"2024-09-21T08:30:25.606774+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.177382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:59.628414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":701,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603348+0000","last_change":"2024-09-21T08:28:53.602197+0000","last_active":"2024-09-21T08:30:25.603348+0000","last_peered":"2024-09-21T08:30:25.603348+0000","last_clean":"2024-09-21T08:28:51.587601+0000","last_became_active":"2024-09-21T08:28:53.602197+0000","last_became_peered":"2024-09-21T08:28:53.602197+0000","last_unstale":"2024-09-21T08:30:25.603348+0000","last_undegraded":"2024-09-21T08:28:53.578880+0000","last_fullsized":"2024-09-21T08:28:53.578772+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-09-21T08:27:30.806522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.806522+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151072+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":782,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.609283+0000","last_change":"2024-09-21T08:28:53.600847+0000","last_active":"2024-09-21T08:30:25.609283+0000","last_peered":"2024-09-21T08:30:25.609283+0000","last_clean":"2024-09-21T08:28:51.557075+0000","last_became_active":"2024-09-21T08:28:53.600847+0000","last_became_peered":"2024-09-21T08:28:53.600847+0000","last_unstale":"2024-09-21T08:30:25.609283+0000","last_undegraded":"2024-09-21T08:30:25.609283+0000","last_fullsized":"2024-09-21T08:28:53.578503+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.795325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814557+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":705,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.607157+0000","last_change":"2024-09-21T08:28:53.600413+0000","last_active":"2024-09-21T08:30:25.607157+0000","last_peered":"2024-09-21T08:30:25.607157+0000","last_clean":"2024-09-21T08:28:27.554070+0000","last_became_active":"2024-09-21T08:28:53.600413+0000","last_became_peered":"2024-09-21T08:28:53.600413+0000","last_unstale":"2024-09-21T08:30:25.607157+0000","last_undegraded":"2024-09-21T08:30:25.607157+0000","last_fullsized":"2024-09-21T08:28:53.576255+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:10.851510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.851510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814622+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"428'444","reported_seq":2149,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606761+0000","last_change":"2024-09-21T08:28:53.603910+0000","last_active":"2024-09-21T08:30:25.606761+0000","last_peered":"2024-09-21T08:30:25.606761+0000","last_clean":"2024-09-21T08:28:51.587746+0000","last_became_active":"2024-09-21T08:28:53.603910+0000","last_became_peered":"2024-09-21T08:28:53.603910+0000","last_unstale":"2024-09-21T08:30:25.606761+0000","last_undegraded":"2024-09-21T08:28:53.582931+0000","last_fullsized":"2024-09-21T08:28:53.582809+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'262","last_scrub_stamp":"2024-09-21T08:26:29.875830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.875830+0000","objects_scrubbed":3,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":49,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.074129519000000005,"stat_sum":{"num_bytes":40135290,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":2,"num_read":150,"num_read_kb":148874,"num_write":89,"num_write_kb":39971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":8623856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":20}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.16","version":"73'9","reported_seq":254,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606606+0000","last_change":"2024-09-21T08:28:53.589261+0000","last_active":"2024-09-21T08:30:25.606606+0000","last_peered":"2024-09-21T08:30:25.606606+0000","last_clean":"2024-09-21T08:28:51.555305+0000","last_became_active":"2024-09-21T08:28:53.589261+0000","last_became_peered":"2024-09-21T08:28:53.589261+0000","last_unstale":"2024-09-21T08:30:25.606606+0000","last_undegraded":"2024-09-21T08:28:53.576380+0000","last_fullsized":"2024-09-21T08:28:53.576277+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:10.183035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.183035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814592+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":617,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605488+0000","last_change":"2024-09-21T08:28:53.587227+0000","last_active":"2024-09-21T08:30:25.605488+0000","last_peered":"2024-09-21T08:30:25.605488+0000","last_clean":"2024-09-21T08:28:51.554027+0000","last_became_active":"2024-09-21T08:28:53.587227+0000","last_became_peered":"2024-09-21T08:28:53.587227+0000","last_unstale":"2024-09-21T08:30:25.605488+0000","last_undegraded":"2024-09-21T08:30:25.605488+0000","last_fullsized":"2024-09-21T08:28:53.573946+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:46.719545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:46.719545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814462+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":302,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.604871+0000","last_change":"2024-09-21T08:28:53.602669+0000","last_active":"2024-09-21T08:30:25.604871+0000","last_peered":"2024-09-21T08:30:25.604871+0000","last_clean":"2024-09-21T08:28:51.587829+0000","last_became_active":"2024-09-21T08:28:53.602669+0000","last_became_peered":"2024-09-21T08:28:53.602669+0000","last_unstale":"2024-09-21T08:30:25.604871+0000","last_undegraded":"2024-09-21T08:30:25.604871+0000","last_fullsized":"2024-09-21T08:28:53.580175+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:05.814753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.814753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150891+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":761,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.591998+0000","last_change":"2024-09-21T08:28:53.592444+0000","last_active":"2024-09-21T08:30:25.591998+0000","last_peered":"2024-09-21T08:30:25.591998+0000","last_clean":"2024-09-21T08:28:51.554300+0000","last_became_active":"2024-09-21T08:28:53.592444+0000","last_became_peered":"2024-09-21T08:28:53.592444+0000","last_unstale":"2024-09-21T08:30:25.591998+0000","last_undegraded":"2024-09-21T08:30:25.591998+0000","last_fullsized":"2024-09-21T08:28:53.578030+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:10.076564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.076564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"426'170","reported_seq":1761,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.608325+0000","last_change":"2024-09-21T08:28:53.589835+0000","last_active":"2024-09-21T08:30:25.608325+0000","last_peered":"2024-09-21T08:30:25.608325+0000","last_clean":"2024-09-21T08:28:51.555217+0000","last_became_active":"2024-09-21T08:28:53.589835+0000","last_became_peered":"2024-09-21T08:28:53.589835+0000","last_unstale":"2024-09-21T08:30:25.608325+0000","last_undegraded":"2024-09-21T08:28:53.577195+0000","last_fullsized":"2024-09-21T08:28:53.577076+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'102","last_scrub_stamp":"2024-09-21T08:26:33.255256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.255256+0000","objects_scrubbed":1,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011450665,"stat_sum":{"num_bytes":11938578,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":91,"num_read_kb":132942,"num_write":38,"num_write_kb":9384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4027639,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.15","version":"73'9","reported_seq":846,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.607682+0000","last_change":"2024-09-21T08:28:53.601176+0000","last_active":"2024-09-21T08:30:25.607682+0000","last_peered":"2024-09-21T08:30:25.607682+0000","last_clean":"2024-09-21T08:28:51.554870+0000","last_became_active":"2024-09-21T08:28:53.601176+0000","last_became_peered":"2024-09-21T08:28:53.601176+0000","last_unstale":"2024-09-21T08:30:25.607682+0000","last_undegraded":"2024-09-21T08:28:53.577554+0000","last_fullsized":"2024-09-21T08:28:53.576455+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:28:14.535334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.535334+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814772+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":729,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.601721+0000","last_change":"2024-09-21T08:28:53.605117+0000","last_active":"2024-09-21T08:30:25.601721+0000","last_peered":"2024-09-21T08:30:25.601721+0000","last_clean":"2024-09-21T08:28:51.584984+0000","last_became_active":"2024-09-21T08:28:53.605117+0000","last_became_peered":"2024-09-21T08:28:53.605117+0000","last_unstale":"2024-09-21T08:30:25.601721+0000","last_undegraded":"2024-09-21T08:30:25.601721+0000","last_fullsized":"2024-09-21T08:28:53.584510+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.361096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.361096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"340'16","reported_seq":749,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.602577+0000","last_change":"2024-09-21T08:28:53.600719+0000","last_active":"2024-09-21T08:30:25.602577+0000","last_peered":"2024-09-21T08:30:25.602577+0000","last_clean":"2024-09-21T08:28:51.585913+0000","last_became_active":"2024-09-21T08:28:53.600719+0000","last_became_peered":"2024-09-21T08:28:53.600719+0000","last_unstale":"2024-09-21T08:30:25.602577+0000","last_undegraded":"2024-09-21T08:28:53.576377+0000","last_fullsized":"2024-09-21T08:28:53.576220+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150760+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":732,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.592576+0000","last_change":"2024-09-21T08:28:53.597499+0000","last_active":"2024-09-21T08:30:25.592576+0000","last_peered":"2024-09-21T08:30:25.592576+0000","last_clean":"2024-09-21T08:28:51.556626+0000","last_became_active":"2024-09-21T08:28:53.597499+0000","last_became_peered":"2024-09-21T08:28:53.597499+0000","last_unstale":"2024-09-21T08:30:25.592576+0000","last_undegraded":"2024-09-21T08:30:25.592576+0000","last_fullsized":"2024-09-21T08:28:53.585175+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.364388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.364388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"179'63","reported_seq":1056,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.608268+0000","last_change":"2024-09-21T08:28:53.590100+0000","last_active":"2024-09-21T08:30:25.608268+0000","last_peered":"2024-09-21T08:30:25.608268+0000","last_clean":"2024-09-21T08:28:51.556223+0000","last_became_active":"2024-09-21T08:28:53.590100+0000","last_became_peered":"2024-09-21T08:28:53.590100+0000","last_unstale":"2024-09-21T08:30:25.608268+0000","last_undegraded":"2024-09-21T08:30:25.608268+0000","last_fullsized":"2024-09-21T08:28:53.579666+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'63","last_scrub_stamp":"2024-09-21T08:28:13.511512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.511512+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010228017000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.14","version":"104'15","reported_seq":309,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.603036+0000","last_change":"2024-09-21T08:28:53.602117+0000","last_active":"2024-09-21T08:30:25.603036+0000","last_peered":"2024-09-21T08:30:25.603036+0000","last_clean":"2024-09-21T08:28:51.587113+0000","last_became_active":"2024-09-21T08:28:53.602117+0000","last_became_peered":"2024-09-21T08:28:53.602117+0000","last_unstale":"2024-09-21T08:30:25.603036+0000","last_undegraded":"2024-09-21T08:28:53.578695+0000","last_fullsized":"2024-09-21T08:28:53.578596+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":742,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.601855+0000","last_change":"2024-09-21T08:28:53.600512+0000","last_active":"2024-09-21T08:30:25.601855+0000","last_peered":"2024-09-21T08:30:25.601855+0000","last_clean":"2024-09-21T08:28:27.552826+0000","last_became_active":"2024-09-21T08:28:53.600512+0000","last_became_peered":"2024-09-21T08:28:53.600512+0000","last_unstale":"2024-09-21T08:30:25.601855+0000","last_undegraded":"2024-09-21T08:28:53.574940+0000","last_fullsized":"2024-09-21T08:28:53.574839+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:28:16.875544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.875544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.15","version":"340'14","reported_seq":889,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.609260+0000","last_change":"2024-09-21T08:28:53.601874+0000","last_active":"2024-09-21T08:30:25.609260+0000","last_peered":"2024-09-21T08:30:25.609260+0000","last_clean":"2024-09-21T08:28:51.555323+0000","last_became_active":"2024-09-21T08:28:53.601874+0000","last_became_peered":"2024-09-21T08:28:53.601874+0000","last_unstale":"2024-09-21T08:30:25.609260+0000","last_undegraded":"2024-09-21T08:28:53.578511+0000","last_fullsized":"2024-09-21T08:28:53.577750+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":613,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.609057+0000","last_change":"2024-09-21T08:28:53.600530+0000","last_active":"2024-09-21T08:30:25.609057+0000","last_peered":"2024-09-21T08:30:25.609057+0000","last_clean":"2024-09-21T08:28:51.555446+0000","last_became_active":"2024-09-21T08:28:53.600530+0000","last_became_peered":"2024-09-21T08:28:53.600530+0000","last_unstale":"2024-09-21T08:30:25.609057+0000","last_undegraded":"2024-09-21T08:30:25.609057+0000","last_fullsized":"2024-09-21T08:28:53.577413+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:48.251366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:48.251366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814550+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.10","version":"430'161","reported_seq":877,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606636+0000","last_change":"2024-09-21T08:28:53.604012+0000","last_active":"2024-09-21T08:30:25.606636+0000","last_peered":"2024-09-21T08:30:25.606636+0000","last_clean":"2024-09-21T08:28:53.029949+0000","last_became_active":"2024-09-21T08:28:53.604012+0000","last_became_peered":"2024-09-21T08:28:53.604012+0000","last_unstale":"2024-09-21T08:30:25.606636+0000","last_undegraded":"2024-09-21T08:28:53.580392+0000","last_fullsized":"2024-09-21T08:28:53.580253+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'94","last_scrub_stamp":"2024-09-21T08:26:27.208234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.208234+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":49,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.075698227000000007,"stat_sum":{"num_bytes":12337203,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":13076,"num_write":38,"num_write_kb":17037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10770304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.13","version":"73'11","reported_seq":314,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.606318+0000","last_change":"2024-09-21T08:30:25.153811+0000","last_active":"2024-09-21T08:30:25.606318+0000","last_peered":"2024-09-21T08:30:25.606318+0000","last_clean":"2024-09-21T08:30:25.606318+0000","last_became_active":"2024-09-21T08:28:04.726766+0000","last_became_peered":"2024-09-21T08:28:04.726766+0000","last_unstale":"2024-09-21T08:30:25.606318+0000","last_undegraded":"2024-09-21T08:30:25.606318+0000","last_fullsized":"2024-09-21T08:30:25.606318+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:30:25.153729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:25.153729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:53.620587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":675,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605698+0000","last_change":"2024-09-21T08:28:53.587802+0000","last_active":"2024-09-21T08:30:25.605698+0000","last_peered":"2024-09-21T08:30:25.605698+0000","last_clean":"2024-09-21T08:28:27.553773+0000","last_became_active":"2024-09-21T08:28:53.587802+0000","last_became_peered":"2024-09-21T08:28:53.587802+0000","last_unstale":"2024-09-21T08:30:25.605698+0000","last_undegraded":"2024-09-21T08:30:25.605698+0000","last_fullsized":"2024-09-21T08:28:53.575420+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:06.889127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:06.889127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814570+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":297,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.605312+0000","last_change":"2024-09-21T08:28:53.600540+0000","last_active":"2024-09-21T08:30:25.605312+0000","last_peered":"2024-09-21T08:30:25.605312+0000","last_clean":"2024-09-21T08:28:51.585918+0000","last_became_active":"2024-09-21T08:28:53.600540+0000","last_became_peered":"2024-09-21T08:28:53.600540+0000","last_unstale":"2024-09-21T08:30:25.605312+0000","last_undegraded":"2024-09-21T08:30:25.605312+0000","last_fullsized":"2024-09-21T08:28:53.586917+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":500,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.606574+0000","last_change":"2024-09-21T08:30:23.816742+0000","last_active":"2024-09-21T08:30:25.606574+0000","last_peered":"2024-09-21T08:30:25.606574+0000","last_clean":"2024-09-21T08:30:25.606574+0000","last_became_active":"2024-09-21T08:28:29.611035+0000","last_became_peered":"2024-09-21T08:28:29.611035+0000","last_unstale":"2024-09-21T08:30:25.606574+0000","last_undegraded":"2024-09-21T08:30:25.606574+0000","last_fullsized":"2024-09-21T08:30:25.606574+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:23.816669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:23.816669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:53.328371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"428'182","reported_seq":1613,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.605264+0000","last_change":"2024-09-21T08:28:53.605541+0000","last_active":"2024-09-21T08:30:25.605264+0000","last_peered":"2024-09-21T08:30:25.605264+0000","last_clean":"2024-09-21T08:28:52.665793+0000","last_became_active":"2024-09-21T08:28:53.605541+0000","last_became_peered":"2024-09-21T08:28:53.605541+0000","last_unstale":"2024-09-21T08:30:25.605264+0000","last_undegraded":"2024-09-21T08:28:53.581187+0000","last_fullsized":"2024-09-21T08:28:53.581055+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'44","last_scrub_stamp":"2024-09-21T08:26:27.882963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.882963+0000","objects_scrubbed":1,"log_size":182,"log_dups_size":0,"ondisk_log_size":182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":48,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0806039080000001,"stat_sum":{"num_bytes":39093600,"num_objects":18,"num_object_clones":16,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":74,"num_read_kb":68077,"num_write":85,"num_write_kb":38406,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":11110430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":18}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"4.12","version":"73'9","reported_seq":320,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.604019+0000","last_change":"2024-09-21T08:28:29.696390+0000","last_active":"2024-09-21T08:30:25.604019+0000","last_peered":"2024-09-21T08:30:25.604019+0000","last_clean":"2024-09-21T08:30:25.604019+0000","last_became_active":"2024-09-21T08:28:29.636404+0000","last_became_peered":"2024-09-21T08:28:29.636404+0000","last_unstale":"2024-09-21T08:30:25.604019+0000","last_undegraded":"2024-09-21T08:30:25.604019+0000","last_fullsized":"2024-09-21T08:30:25.604019+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:14.398007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.398007+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":623,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.606134+0000","last_change":"2024-09-21T08:28:53.587397+0000","last_active":"2024-09-21T08:30:25.606134+0000","last_peered":"2024-09-21T08:30:25.606134+0000","last_clean":"2024-09-21T08:28:51.554500+0000","last_became_active":"2024-09-21T08:28:53.587397+0000","last_became_peered":"2024-09-21T08:28:53.587397+0000","last_unstale":"2024-09-21T08:30:25.606134+0000","last_undegraded":"2024-09-21T08:30:25.606134+0000","last_fullsized":"2024-09-21T08:28:53.575265+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:49.790900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:49.790900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814564+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":608,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.601951+0000","last_change":"2024-09-21T08:28:53.594733+0000","last_active":"2024-09-21T08:30:25.601951+0000","last_peered":"2024-09-21T08:30:25.601951+0000","last_clean":"2024-09-21T08:28:51.584914+0000","last_became_active":"2024-09-21T08:28:53.594733+0000","last_became_peered":"2024-09-21T08:28:53.594733+0000","last_unstale":"2024-09-21T08:30:25.601951+0000","last_undegraded":"2024-09-21T08:30:25.601951+0000","last_fullsized":"2024-09-21T08:28:53.576121+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:13.940546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:13.940546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151063+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":714,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.607956+0000","last_change":"2024-09-21T08:30:02.065158+0000","last_active":"2024-09-21T08:30:25.607956+0000","last_peered":"2024-09-21T08:30:25.607956+0000","last_clean":"2024-09-21T08:30:25.607956+0000","last_became_active":"2024-09-21T08:28:29.610998+0000","last_became_peered":"2024-09-21T08:28:29.610998+0000","last_unstale":"2024-09-21T08:30:25.607956+0000","last_undegraded":"2024-09-21T08:30:25.607956+0000","last_fullsized":"2024-09-21T08:30:25.607956+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:02.065082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:02.065082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:24.858435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.12","version":"430'400","reported_seq":2091,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.605845+0000","last_change":"2024-09-21T08:28:53.587108+0000","last_active":"2024-09-21T08:30:25.605845+0000","last_peered":"2024-09-21T08:30:25.605845+0000","last_clean":"2024-09-21T08:28:52.589762+0000","last_became_active":"2024-09-21T08:28:53.587108+0000","last_became_peered":"2024-09-21T08:28:53.587108+0000","last_unstale":"2024-09-21T08:30:25.605845+0000","last_undegraded":"2024-09-21T08:28:53.574106+0000","last_fullsized":"2024-09-21T08:28:53.573924+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'200","last_scrub_stamp":"2024-09-21T08:26:27.417028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.417028+0000","objects_scrubbed":3,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.027231871000000001,"stat_sum":{"num_bytes":39611945,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":3,"num_read":126,"num_read_kb":145741,"num_write":93,"num_write_kb":44529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":8981699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":20}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.11","version":"73'11","reported_seq":337,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606670+0000","last_change":"2024-09-21T08:28:53.589901+0000","last_active":"2024-09-21T08:30:25.606670+0000","last_peered":"2024-09-21T08:30:25.606670+0000","last_clean":"2024-09-21T08:28:51.555481+0000","last_became_active":"2024-09-21T08:28:53.589901+0000","last_became_peered":"2024-09-21T08:28:53.589901+0000","last_unstale":"2024-09-21T08:30:25.606670+0000","last_undegraded":"2024-09-21T08:28:53.577098+0000","last_fullsized":"2024-09-21T08:28:53.576477+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":778,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.609108+0000","last_change":"2024-09-21T08:28:53.600628+0000","last_active":"2024-09-21T08:30:25.609108+0000","last_peered":"2024-09-21T08:30:25.609108+0000","last_clean":"2024-09-21T08:28:51.555529+0000","last_became_active":"2024-09-21T08:28:53.600628+0000","last_became_peered":"2024-09-21T08:28:53.600628+0000","last_unstale":"2024-09-21T08:30:25.609108+0000","last_undegraded":"2024-09-21T08:30:25.609108+0000","last_fullsized":"2024-09-21T08:28:53.577666+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814455+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":710,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.609416+0000","last_change":"2024-09-21T08:28:53.589537+0000","last_active":"2024-09-21T08:30:25.609416+0000","last_peered":"2024-09-21T08:30:25.609416+0000","last_clean":"2024-09-21T08:28:27.552805+0000","last_became_active":"2024-09-21T08:28:53.589537+0000","last_became_peered":"2024-09-21T08:28:53.589537+0000","last_unstale":"2024-09-21T08:30:25.609416+0000","last_undegraded":"2024-09-21T08:30:25.609416+0000","last_fullsized":"2024-09-21T08:28:53.575446+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":506,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.607789+0000","last_change":"2024-09-21T08:29:58.167276+0000","last_active":"2024-09-21T08:30:25.607789+0000","last_peered":"2024-09-21T08:30:25.607789+0000","last_clean":"2024-09-21T08:30:25.607789+0000","last_became_active":"2024-09-21T08:28:29.619814+0000","last_became_peered":"2024-09-21T08:28:29.619814+0000","last_unstale":"2024-09-21T08:30:25.607789+0000","last_undegraded":"2024-09-21T08:30:25.607789+0000","last_fullsized":"2024-09-21T08:30:25.607789+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:58.167184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.167184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:02.224715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"338'55","reported_seq":1592,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.606173+0000","last_change":"2024-09-21T08:28:53.588606+0000","last_active":"2024-09-21T08:30:25.606173+0000","last_peered":"2024-09-21T08:30:25.606173+0000","last_clean":"2024-09-21T08:28:52.586222+0000","last_became_active":"2024-09-21T08:28:53.588606+0000","last_became_peered":"2024-09-21T08:28:53.588606+0000","last_unstale":"2024-09-21T08:30:25.606173+0000","last_undegraded":"2024-09-21T08:28:53.574192+0000","last_fullsized":"2024-09-21T08:28:53.574061+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'22","last_scrub_stamp":"2024-09-21T08:26:29.353795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.353795+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019195647,"stat_sum":{"num_bytes":9586927,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":45,"num_read_kb":64333,"num_write":19,"num_write_kb":14200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5630444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.10","version":"73'4","reported_seq":308,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.605159+0000","last_change":"2024-09-21T08:29:39.416471+0000","last_active":"2024-09-21T08:30:25.605159+0000","last_peered":"2024-09-21T08:30:25.605159+0000","last_clean":"2024-09-21T08:30:25.605159+0000","last_became_active":"2024-09-21T08:28:04.724607+0000","last_became_peered":"2024-09-21T08:28:04.724607+0000","last_unstale":"2024-09-21T08:30:25.605159+0000","last_undegraded":"2024-09-21T08:30:25.605159+0000","last_fullsized":"2024-09-21T08:30:25.605159+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-09-21T08:29:39.416397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.416397+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:58.505501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":746,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.608212+0000","last_change":"2024-09-21T08:30:24.826266+0000","last_active":"2024-09-21T08:30:25.608212+0000","last_peered":"2024-09-21T08:30:25.608212+0000","last_clean":"2024-09-21T08:30:25.608212+0000","last_became_active":"2024-09-21T08:28:29.611592+0000","last_became_peered":"2024-09-21T08:28:29.611592+0000","last_unstale":"2024-09-21T08:30:25.608212+0000","last_undegraded":"2024-09-21T08:30:25.608212+0000","last_fullsized":"2024-09-21T08:30:25.608212+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:24.826198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:30:24.826198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:28.562894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":689,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.601453+0000","last_change":"2024-09-21T08:28:53.589633+0000","last_active":"2024-09-21T08:30:25.601453+0000","last_peered":"2024-09-21T08:30:25.601453+0000","last_clean":"2024-09-21T08:28:51.584672+0000","last_became_active":"2024-09-21T08:28:53.589633+0000","last_became_peered":"2024-09-21T08:28:53.589633+0000","last_unstale":"2024-09-21T08:30:25.601453+0000","last_undegraded":"2024-09-21T08:30:25.601453+0000","last_fullsized":"2024-09-21T08:28:53.574228+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":763,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.601927+0000","last_change":"2024-09-21T08:28:53.605159+0000","last_active":"2024-09-21T08:30:25.601927+0000","last_peered":"2024-09-21T08:30:25.601927+0000","last_clean":"2024-09-21T08:28:51.585029+0000","last_became_active":"2024-09-21T08:28:53.605159+0000","last_became_peered":"2024-09-21T08:28:53.605159+0000","last_unstale":"2024-09-21T08:30:25.601927+0000","last_undegraded":"2024-09-21T08:30:25.601927+0000","last_fullsized":"2024-09-21T08:28:53.584509+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.736816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.736816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.151040+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"407'212","reported_seq":1843,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.605928+0000","last_change":"2024-09-21T08:28:53.600314+0000","last_active":"2024-09-21T08:30:25.605928+0000","last_peered":"2024-09-21T08:30:25.605928+0000","last_clean":"2024-09-21T08:28:51.556402+0000","last_became_active":"2024-09-21T08:28:53.600314+0000","last_became_peered":"2024-09-21T08:28:53.600314+0000","last_unstale":"2024-09-21T08:30:25.605928+0000","last_undegraded":"2024-09-21T08:28:53.575998+0000","last_fullsized":"2024-09-21T08:28:53.575875+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'167","last_scrub_stamp":"2024-09-21T08:27:49.143452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.143452+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":47,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.015256377999999999,"stat_sum":{"num_bytes":17321215,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":73,"num_read_kb":97090,"num_write":25,"num_write_kb":14712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1129140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"6.1d","version":"0'0","reported_seq":293,"reported_epoch":432,"state":"active+clean","last_fresh":"2024-09-21T08:30:25.604971+0000","last_change":"2024-09-21T08:28:04.726882+0000","last_active":"2024-09-21T08:30:25.604971+0000","last_peered":"2024-09-21T08:30:25.604971+0000","last_clean":"2024-09-21T08:30:25.604971+0000","last_became_active":"2024-09-21T08:28:04.724461+0000","last_became_peered":"2024-09-21T08:28:04.724461+0000","last_unstale":"2024-09-21T08:30:25.604971+0000","last_undegraded":"2024-09-21T08:30:25.604971+0000","last_fullsized":"2024-09-21T08:30:25.604971+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:13.999162+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:13.999162+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":617,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.601432+0000","last_change":"2024-09-21T08:28:53.585838+0000","last_active":"2024-09-21T08:30:25.601432+0000","last_peered":"2024-09-21T08:30:25.601432+0000","last_clean":"2024-09-21T08:28:51.584499+0000","last_became_active":"2024-09-21T08:28:53.585838+0000","last_became_peered":"2024-09-21T08:28:53.585838+0000","last_unstale":"2024-09-21T08:30:25.601432+0000","last_undegraded":"2024-09-21T08:30:25.601432+0000","last_fullsized":"2024-09-21T08:28:53.575428+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:30.150782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":813,"reported_epoch":432,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:25.609300+0000","last_change":"2024-09-21T08:28:53.603028+0000","last_active":"2024-09-21T08:30:25.609300+0000","last_peered":"2024-09-21T08:30:25.609300+0000","last_clean":"2024-09-21T08:28:51.556942+0000","last_became_active":"2024-09-21T08:28:53.603028+0000","last_became_peered":"2024-09-21T08:28:53.603028+0000","last_unstale":"2024-09-21T08:30:25.609300+0000","last_undegraded":"2024-09-21T08:28:53.581121+0000","last_fullsized":"2024-09-21T08:28:53.581005+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:28.814398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":195,"reported_epoch":432,"state":"active+undersized","last_fresh":"2024-09-21T08:30:25.602071+0000","last_change":"2024-09-21T08:28:53.601908+0000","last_active":"2024-09-21T08:30:25.602071+0000","last_peered":"2024-09-21T08:30:25.602071+0000","last_clean":"2024-09-21T08:23:33.544062+0000","last_became_active":"2024-09-21T08:28:53.601908+0000","last_became_peered":"2024-09-21T08:28:53.601908+0000","last_unstale":"2024-09-21T08:30:25.602071+0000","last_undegraded":"2024-09-21T08:30:25.602071+0000","last_fullsized":"2024-09-21T08:28:53.577043+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.872291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.872291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":549134633,"num_objects":258,"num_object_clones":209,"num_object_copies":774,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":207,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":258,"num_whiteouts":14,"num_read":1681,"num_read_kb":1851553,"num_write":1181,"num_write_kb":494881,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":207157568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1250725888,"data_stored":1770953675,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7421,"ondisk_log_size":7421,"up":79,"acting":79,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":76,"acting":76,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":67,"acting":67,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":676,"num_read_kb":433,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":10760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":17316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":74,"acting":74,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4282,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":8622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":70,"acting":70,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":392,"num_read_kb":387,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":6,"acting":6,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070533,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":228312,"kb_used_data":174916,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93516840,"statfs":{"total":95995035648,"available":95761244160,"internally_reserved":0,"allocated":179113984,"data_stored":227784827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397598,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659233,"num_pgs":124,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":308376,"kb_used_data":250816,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93436776,"statfs":{"total":95995035648,"available":95679258624,"internally_reserved":0,"allocated":256835584,"data_stored":375537355,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986279,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182507,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411441,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":404960,"kb_used_data":347404,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93340192,"statfs":{"total":95995035648,"available":95580356608,"internally_reserved":0,"allocated":355741696,"data_stored":536794564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574965,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":321124,"kb_used_data":289932,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93424028,"statfs":{"total":95995035648,"available":95666204672,"internally_reserved":0,"allocated":296890368,"data_stored":437903457,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":3482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":3014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74776576,"data_stored":86348592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":294436864,"data_stored":436671090,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":352391168,"data_stored":534606483,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70848512,"data_stored":84468350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":254042112,"data_stored":373946435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":176136192,"data_stored":227042310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:30:27.385 INFO:tasks.cephadm.ceph_manager.ceph:would drop below min_pgs, proj -9, currently 1 2024-09-21T08:30:27.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:27.412 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:27.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:27.413 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:27.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:27.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:27.536 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:27.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:27.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:27 smithi137 ceph-mon[29634]: pgmap v554: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 524 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 4.8 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 353/1371 objects degraded (25.748%) 2024-09-21T08:30:27.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:27 smithi137 ceph-mon[29634]: from='client.15834 v1:172.21.15.137:0/2551793760' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:30:27.605 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:27 smithi137 ceph-mon[29634]: osdmap e433: 8 total, 6 up, 4 in 2024-09-21T08:30:27.610 INFO:tasks.rados.rados.0.smithi137.stdout:1719: done (3 left) 2024-09-21T08:30:27.610 INFO:tasks.rados.rados.0.smithi137.stdout:1720: snap_create 2024-09-21T08:30:27.610 INFO:tasks.rados.rados.0.smithi137.stdout:1715: finishing write tid 2 to smithi13779221-14 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1715: finishing write tid 3 to smithi13779221-14 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1715: finishing write tid 4 to smithi13779221-14 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1715: oid 14 updating version 0 to 389 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1715: oid 14 version 389 is already newer than 387 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1715: oid 14 version 389 is already newer than 388 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 389 (ObjNum 613 snap 180 seq_num 613) dirty exists 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1715: left oid 14 (ObjNum 613 snap 180 seq_num 613) 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1717: finishing write tid 1 to smithi13779221-4 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1717: finishing write tid 2 to smithi13779221-4 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1717: finishing write tid 3 to smithi13779221-4 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1717: finishing write tid 4 to smithi13779221-4 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1717: oid 4 updating version 0 to 383 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1717: oid 4 updating version 383 to 384 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:1717: oid 4 version 384 is already newer than 382 2024-09-21T08:30:27.611 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 384 (ObjNum 615 snap 180 seq_num 615) dirty exists 2024-09-21T08:30:27.612 INFO:tasks.rados.rados.0.smithi137.stdout:1717: left oid 4 (ObjNum 615 snap 180 seq_num 615) 2024-09-21T08:30:27.612 INFO:tasks.rados.rados.0.smithi137.stdout:1718: expect (ObjNum 575 snap 170 seq_num 575) 2024-09-21T08:30:27.637 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:27.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:27.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:27.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:27.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:27 smithi165 ceph-mon[32893]: pgmap v554: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 524 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 4.8 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 353/1371 objects degraded (25.748%) 2024-09-21T08:30:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:27 smithi165 ceph-mon[32893]: from='client.15834 v1:172.21.15.137:0/2551793760' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:30:27.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:27 smithi165 ceph-mon[32893]: osdmap e433: 8 total, 6 up, 4 in 2024-09-21T08:30:27.782 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:27.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:27.783 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:27.844 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T08:30:27.884 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:27.910 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:27.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:27.911 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:28.012 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:28.017 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:27 smithi137 ceph-mon[41044]: pgmap v554: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 524 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 4.8 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 353/1371 objects degraded (25.748%) 2024-09-21T08:30:28.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:27 smithi137 ceph-mon[41044]: from='client.15834 v1:172.21.15.137:0/2551793760' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:30:28.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:27 smithi137 ceph-mon[41044]: osdmap e433: 8 total, 6 up, 4 in 2024-09-21T08:30:28.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:28.035 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:28.036 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:28.092 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:28.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:28.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:28.160 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:28.160 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:28.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:30:28.261 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:28.282 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:28.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:28.283 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:28.384 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:28.406 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:28.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:28.407 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:28.443 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:28.508 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:28.528 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:28.528 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:28.530 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:28.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:28.531 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:28.622 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:30:28.625 INFO:tasks.rados.rados.0.smithi137.stdout:1715: done (3 left) 2024-09-21T08:30:28.625 INFO:tasks.rados.rados.0.smithi137.stdout:1717: done (2 left) 2024-09-21T08:30:28.625 INFO:tasks.rados.rados.0.smithi137.stdout:1718: done (1 left) 2024-09-21T08:30:28.625 INFO:tasks.rados.rados.0.smithi137.stdout:1720: done (0 left) 2024-09-21T08:30:28.625 INFO:tasks.rados.rados.0.smithi137.stdout:1721: snap_remove snap 146 2024-09-21T08:30:28.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:28.652 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:28.653 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:28.653 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:28.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:28 smithi165 ceph-mon[32893]: 4.1a scrub starts 2024-09-21T08:30:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:28 smithi165 ceph-mon[32893]: 4.1a scrub ok 2024-09-21T08:30:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:28 smithi165 ceph-mon[32893]: osdmap e434: 8 total, 6 up, 4 in 2024-09-21T08:30:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:28 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2112619530' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:30:28.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:28 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:30:28.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:28.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:28.778 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:28.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:28.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:28.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:28.901 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:29.002 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:29.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:28 smithi137 ceph-mon[29634]: 4.1a scrub starts 2024-09-21T08:30:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:28 smithi137 ceph-mon[29634]: 4.1a scrub ok 2024-09-21T08:30:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:28 smithi137 ceph-mon[29634]: osdmap e434: 8 total, 6 up, 4 in 2024-09-21T08:30:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:28 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2112619530' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:30:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:28 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:30:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:28 smithi137 ceph-mon[41044]: 4.1a scrub starts 2024-09-21T08:30:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:28 smithi137 ceph-mon[41044]: 4.1a scrub ok 2024-09-21T08:30:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:28 smithi137 ceph-mon[41044]: osdmap e434: 8 total, 6 up, 4 in 2024-09-21T08:30:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:28 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2112619530' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:30:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:28 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:30:29.024 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:29.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:29.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:29.124 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:29.125 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:29.125 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:29.125 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:29.141 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:29.142 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:29.146 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:29.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:29.147 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:29.159 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:29.159 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:29.175 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:29.176 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:29.192 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:29.192 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:29.209 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:29.210 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:29.210 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:29.247 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:29.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:29.269 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:29.269 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:29.309 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T08:30:29.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:29.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:29.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:29.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:29.496 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:29.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:29.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:29.519 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:29.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:29.622 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:29.627 INFO:tasks.rados.rados.0.smithi137.stdout:1721: done (0 left) 2024-09-21T08:30:29.627 INFO:tasks.rados.rados.0.smithi137.stdout:1722: read oid 1 snap -1 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:1722: expect deleted 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:1723: rollback oid 14 current snap is 181 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 14 to 174 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:1724: rollback oid 40 current snap is 181 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 40 to 172 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:1725: read oid 23 snap -1 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:1725: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:1726: rollback oid 46 current snap is 181 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 46 to 180 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:1727: copy_from oid 45 from oid 34 current snap is 181 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:1728: write (excl) oid 43 current snap is 181 2024-09-21T08:30:29.628 INFO:tasks.rados.rados.0.smithi137.stdout:1728: seq_num 616 ranges {753112=577977,1880202=472910} 2024-09-21T08:30:29.635 INFO:tasks.rados.rados.0.smithi137.stdout:1728: writing smithi13779221-43 from 753112 to 1331089 tid 1 2024-09-21T08:30:29.642 INFO:tasks.rados.rados.0.smithi137.stdout:1728: writing smithi13779221-43 from 1880202 to 2353112 tid 2 2024-09-21T08:30:29.642 INFO:tasks.rados.rados.0.smithi137.stdout:1729: read oid 24 snap 162 2024-09-21T08:30:29.642 INFO:tasks.rados.rados.0.smithi137.stdout:1729: expect (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:30:29.642 INFO:tasks.rados.rados.0.smithi137.stdout:1730: snap_remove snap 170 2024-09-21T08:30:29.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:29.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:29.654 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:29.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:29.776 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:29.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:29.777 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:29.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:29 smithi165 ceph-mon[32893]: 6.1d scrub starts 2024-09-21T08:30:29.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:29 smithi165 ceph-mon[32893]: 6.1d scrub ok 2024-09-21T08:30:29.865 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:29 smithi165 ceph-mon[32893]: pgmap v557: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 524 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 4.8 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 353/1371 objects degraded (25.748%) 2024-09-21T08:30:29.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:29 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:29.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:29 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2112619530' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:30:29.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:29 smithi165 ceph-mon[32893]: osdmap e435: 8 total, 6 up, 4 in 2024-09-21T08:30:29.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:29 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1672089214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:29.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:29 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:29.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:29 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:29.877 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:29.898 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:29.899 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:29.899 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:29.973 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[29634]: 6.1d scrub starts 2024-09-21T08:30:29.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[29634]: 6.1d scrub ok 2024-09-21T08:30:29.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[29634]: pgmap v557: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 524 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 4.8 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 353/1371 objects degraded (25.748%) 2024-09-21T08:30:29.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:29.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2112619530' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:30:29.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[29634]: osdmap e435: 8 total, 6 up, 4 in 2024-09-21T08:30:29.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1672089214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:29.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:29.974 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:29.974 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[41044]: 6.1d scrub starts 2024-09-21T08:30:29.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[41044]: 6.1d scrub ok 2024-09-21T08:30:29.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[41044]: pgmap v557: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 524 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 4.8 MiB/s rd, 2.1 MiB/s wr, 11 op/s; 353/1371 objects degraded (25.748%) 2024-09-21T08:30:29.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:29.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2112619530' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:30:29.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[41044]: osdmap e435: 8 total, 6 up, 4 in 2024-09-21T08:30:29.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1672089214' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:29.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:29.975 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:29 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:30.000 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:30.021 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:30.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:30.022 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:30.088 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:30:30.088 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:30:30.088 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:30:30.088 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:30:30.088 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.088 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1083 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94837804037760 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 351.92638827399998, 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.395092834, 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1083 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.089 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159586+0000", 2024-09-21T08:30:30.090 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.48e-06 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159599+0000", 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3699e-05 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159630+0000", 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0629000000000001e-05 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.091 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159647+0000", 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7030000000000001e-05 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159767+0000", 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119571 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.092 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159781+0000", 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3844e-05 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.549248+0000", 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.38946733 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.549559+0000", 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000311057 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.093 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552031+0000", 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024720720000000001 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552119+0000", 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8261000000000001e-05 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.554673+0000", 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025538610000000002 2024-09-21T08:30:30.094 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1159 7.b 7:df2b42d9:::smithi13779221-36:head [write 642067~696188,stat] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.19584989399999, 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62427366799999995, 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.095 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1159 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.096 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890120+0000", 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6950000000000001e-06 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890849+0000", 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072891299999999998 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890861+0000", 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1899000000000001e-05 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.097 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890868+0000", 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2110000000000001e-06 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890947+0000", 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8598999999999997e-05 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890954+0000", 2024-09-21T08:30:30.098 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1180000000000003e-06 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508089+0000", 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61713446299999997 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508263+0000", 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174671 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:30:30.099 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508366+0000", 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102786 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511111+0000", 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027452710000000001 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.513304+0000", 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021930560000000001 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.100 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514317+0000", 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010129959999999999 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514355+0000", 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7404000000000003e-05 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514392+0000", 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7586000000000002e-05 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.101 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1160 7.b 7:df2b42d9:::smithi13779221-36:head [write 2025476~459397] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:30:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:30:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.18940200499998, 2024-09-21T08:30:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61820804500000004, 2024-09-21T08:30:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.102 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.103 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.103 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1160 2024-09-21T08:30:30.103 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.104 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896568+0000", 2024-09-21T08:30:30.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2610000000000001e-06 2024-09-21T08:30:30.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.105 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896958+0000", 2024-09-21T08:30:30.105 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039041199999999997 2024-09-21T08:30:30.105 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.105 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.105 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896984+0000", 2024-09-21T08:30:30.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6115e-05 2024-09-21T08:30:30.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.106 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.106 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896989+0000", 2024-09-21T08:30:30.106 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0379999999999999e-06 2024-09-21T08:30:30.106 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.106 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.897069+0000", 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9196000000000005e-05 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.897075+0000", 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.28e-06 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509576+0000", 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61250132700000004 2024-09-21T08:30:30.107 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509636+0000", 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9753999999999997e-05 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509725+0000", 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9524000000000005e-05 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511333+0000", 2024-09-21T08:30:30.108 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001607619 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.513767+0000", 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024336819999999999 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514703+0000", 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093633599999999996 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.109 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514736+0000", 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2438999999999999e-05 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514775+0000", 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9062e-05 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1161 7.b 7:df2b42d9:::smithi13779221-36:head [write 2971091~70976] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:30:30.110 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.18652307999997, 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61638660599999995, 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1161 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.111 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899446+0000", 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0800000000000004e-07 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899537+0000", 2024-09-21T08:30:30.112 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0414999999999994e-05 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899544+0000", 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8580000000000002e-06 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899547+0000", 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8970000000000001e-06 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.113 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899597+0000", 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9058999999999999e-05 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899605+0000", 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3990000000000004e-06 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510251+0000", 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61064600599999996 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.114 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510324+0000", 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2961999999999997e-05 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510438+0000", 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113793 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511337+0000", 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089906399999999998 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.115 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514615+0000", 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00327853 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515770+0000", 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001154715 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515802+0000", 2024-09-21T08:30:30.116 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1956999999999999e-05 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515832+0000", 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0142999999999999e-05 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1162 7.b 7:df2b42d9:::smithi13779221-36:head [setxattr _header (57) in=7b,truncate 3042067] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:30:30.117 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.186414012, 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61641711799999999, 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1162 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.118 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9100000000000004e-07 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.119 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899557+0000", 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3989999999999998e-06 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899561+0000", 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9349999999999996e-06 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899564+0000", 2024-09-21T08:30:30.120 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7700000000000002e-06 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899633+0000", 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9268000000000002e-05 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899637+0000", 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9650000000000002e-06 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.121 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510758+0000", 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61112027999999996 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510819+0000", 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0866999999999998e-05 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510929+0000", 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110914 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.122 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511340+0000", 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041042100000000001 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514641+0000", 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003301511 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515865+0000", 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012240319999999999 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.123 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515898+0000", 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3036e-05 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515972+0000", 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3229e-05 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.124 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1163 7.b 7:df2b42d9:::smithi13779221-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 279.18638259900001, 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61652938899999998, 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1163 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.125 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8599999999999998e-07 2024-09-21T08:30:30.126 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899589+0000", 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1299999999999999e-06 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899592+0000", 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9190000000000001e-06 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899595+0000", 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9639999999999999e-06 2024-09-21T08:30:30.127 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899654+0000", 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8958e-05 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899658+0000", 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2669999999999998e-06 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511140+0000", 2024-09-21T08:30:30.128 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61148218399999998 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511235+0000", 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4927999999999996e-05 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516004+0000", 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047691640000000002 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.129 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516060+0000", 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5726999999999997e-05 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516115+0000", 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5661999999999997e-05 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2116 7.b 7:df2b42d9:::smithi13779221-36:head [copy-from ver 318] snapc 4a=[4a,49,48,47,45,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-21T08:30:30.130 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 208.56274357300001, 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030972873000000001, 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2116 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.131 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523228+0000", 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6469999999999998e-06 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.132 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523237+0000", 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4590000000000004e-06 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523251+0000", 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3856000000000001e-05 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523260+0000", 2024-09-21T08:30:30.133 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523351+0000", 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0787999999999999e-05 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523359+0000", 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5860000000000005e-06 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.134 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.526688+0000", 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033287339999999999 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.526800+0000", 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111566 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.531948+0000", 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051486199999999996 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.135 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.538302+0000", 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063539729999999997 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.553780+0000", 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015477451 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554118+0000", 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000338549 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.136 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554160+0000", 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2218000000000001e-05 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554198+0000", 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7475999999999998e-05 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.137 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2577 7.1c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 174] snapc 59=[59,56] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 172.52321737599999, 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0145323420000001, 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2577 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.138 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562753+0000", 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.745e-06 2024-09-21T08:30:30.139 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562759+0000", 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1330000000000004e-06 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562770+0000", 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0617e-05 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.140 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562775+0000", 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4500000000000003e-06 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562831+0000", 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6233999999999997e-05 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562838+0000", 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7379999999999997e-06 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.141 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552540+0000", 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98970149799999996 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552665+0000", 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012560800000000001 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.562825+0000", 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010159916 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.142 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.569448+0000", 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066225670000000002 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577182+0000", 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077346680000000001 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577203+0000", 2024-09-21T08:30:30.143 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0619999999999999e-05 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577240+0000", 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6613999999999999e-05 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577284+0000", 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3934000000000003e-05 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.144 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2578 7.1c 7:3cae00e7:::smithi13779221-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 172.523171499, 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0041372150000001, 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2578 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.145 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.146 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562798+0000", 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5000000000000003e-07 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562800+0000", 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2460000000000002e-06 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562806+0000", 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6620000000000002e-06 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.147 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562809+0000", 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8500000000000004e-06 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562869+0000", 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9892999999999997e-05 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562879+0000", 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0054999999999999e-05 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.148 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552788+0000", 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.989908287 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552833+0000", 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5427999999999999e-05 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566793+0000", 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013959875 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.149 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566858+0000", 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4870999999999996e-05 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566934+0000", 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6497999999999998e-05 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.150 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3197 7.0 7:00f4c0d4:::smithi13779221-18:head [delete] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.553696+0000", 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 113.532272956, 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.022024709999999999, 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3197 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.151 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553696+0000", 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553696+0000", 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553699+0000", 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9639999999999999e-06 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.152 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553706+0000", 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0140000000000001e-06 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553718+0000", 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2369e-05 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553729+0000", 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1454000000000001e-05 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.153 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553819+0000", 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9420999999999996e-05 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.553826+0000", 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2679999999999997e-06 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.556946+0000", 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003119798 2024-09-21T08:30:30.154 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.557067+0000", 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121544 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.557495+0000", 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042718500000000001 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.560546+0000", 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030512069999999998 2024-09-21T08:30:30.155 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.561411+0000", 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00086473699999999997 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.575597+0000", 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014186025 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.575668+0000", 2024-09-21T08:30:30.156 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1898999999999997e-05 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.575720+0000", 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1824999999999999e-05 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3198 7.8 7:15ddb887:::smithi13779221-8:head [append 0~0] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.713165+0000", 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 113.372803858, 2024-09-21T08:30:30.157 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85552278699999995, 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3198 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713165+0000", 2024-09-21T08:30:30.158 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.159 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713165+0000", 2024-09-21T08:30:30.159 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.159 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.159 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.159 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.159 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:30.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713169+0000", 2024-09-21T08:30:30.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.177e-06 2024-09-21T08:30:30.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.161 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.161 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713182+0000", 2024-09-21T08:30:30.161 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3593e-05 2024-09-21T08:30:30.161 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.161 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713212+0000", 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9292999999999999e-05 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713227+0000", 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5630000000000001e-05 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713409+0000", 2024-09-21T08:30:30.162 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018163700000000001 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713424+0000", 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5366000000000002e-05 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.562792+0000", 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.84936760099999997 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.163 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.563075+0000", 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000282945 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.563308+0000", 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023297899999999999 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566597+0000", 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003289541 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.164 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.567690+0000", 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001092906 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568570+0000", 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087925999999999998 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568631+0000", 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1511000000000001e-05 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.165 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568687+0000", 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6348000000000003e-05 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3199 7.8 7:15ddb887:::smithi13779221-8:head [rollback 69] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.713255+0000", 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 113.372713425, 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85558638099999995, 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.166 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3199 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713255+0000", 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.167 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713255+0000", 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713257+0000", 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5260000000000001e-06 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713264+0000", 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2599999999999999e-06 2024-09-21T08:30:30.168 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713287+0000", 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3470000000000001e-05 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713302+0000", 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4195999999999999e-05 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713470+0000", 2024-09-21T08:30:30.169 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016798400000000001 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713478+0000", 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3059999999999998e-06 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.564578+0000", 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85110020600000003 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.170 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.564669+0000", 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1321000000000002e-05 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.564879+0000", 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020947899999999999 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566604+0000", 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017247409999999999 2024-09-21T08:30:30.171 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.172 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.172 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.567857+0000", 2024-09-21T08:30:30.172 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001253852 2024-09-21T08:30:30.172 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.172 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.173 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.173 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568733+0000", 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087562999999999996 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568793+0000", 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9429000000000003e-05 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568842+0000", 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8980999999999997e-05 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.174 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3200 7.8 7:15ddb887:::smithi13779221-8:head [append 0~0] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.713346+0000", 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 113.372622716, 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85569378200000001, 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.175 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3200 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713346+0000", 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713346+0000", 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.176 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713347+0000", 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.626e-06 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713354+0000", 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.567e-06 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713369+0000", 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5216e-05 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.177 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713386+0000", 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6286e-05 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713509+0000", 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012365299999999999 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713520+0000", 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0536e-05 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.178 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566084+0000", 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85256443900000001 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566173+0000", 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8936000000000004e-05 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566334+0000", 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160941 2024-09-21T08:30:30.179 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.567127+0000", 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079256800000000001 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568004+0000", 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087773799999999998 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568895+0000", 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089024700000000004 2024-09-21T08:30:30.180 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568975+0000", 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0527999999999999e-05 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.569040+0000", 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4500999999999998e-05 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.181 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3391 7.12 7:4b20cb2f:::smithi13779221-12:head [copy-from ver 108] snapc 81=[81,80,7f,7e,7d,7c,7a,79,78,77,76,72,71,69] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 100.49976487799999, 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026028484000000001, 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3391 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.182 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499999999999998e-07 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.183 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586207+0000", 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.588e-06 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586216+0000", 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1079999999999999e-06 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586220+0000", 2024-09-21T08:30:30.184 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5319999999999998e-06 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586265+0000", 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5469999999999997e-05 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586369+0000", 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103698 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.185 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.597860+0000", 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011490887999999999 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.607396+0000", 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095365120000000005 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.607414+0000", 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8102e-05 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.186 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.612155+0000", 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047404739999999997 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.612196+0000", 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1007999999999998e-05 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.612232+0000", 2024-09-21T08:30:30.187 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6449000000000003e-05 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4034 7.8 7:144c2ec5:::smithi13779221-45:head [copy-from ver 374] snapc a8=[a8,a7,a6,a5,a4,a3,a2,a1,9e,9c,9b,9a,99,98,94,93,92,91,90,8c,8b,88,86,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:29:53.687463+0000", 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.398505569000001, 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025036505000000001, 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.188 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4034 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687463+0000", 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687463+0000", 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.189 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687464+0000", 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.046e-06 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687468+0000", 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3490000000000002e-06 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.190 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687475+0000", 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5250000000000002e-06 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687481+0000", 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9800000000000003e-06 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687529+0000", 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8270999999999999e-05 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.191 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687635+0000", 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010632999999999999 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.704060+0000", 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016425006999999998 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.192 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.708268+0000", 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042073040000000003 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.712346+0000", 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040779060000000001 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.712418+0000", 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2045000000000006e-05 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.193 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.712499+0000", 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1742000000000004e-05 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4035 7.8 7:144c2ec5:::smithi13779221-45:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e400)", 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:29:53.687492+0000", 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 36.398476183, 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020076963999999999, 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.194 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4035 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687492+0000", 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.195 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687492+0000", 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687493+0000", 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6599999999999996e-07 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687505+0000", 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.181e-05 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.196 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687511+0000", 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0469999999999999e-06 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687517+0000", 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6100000000000002e-06 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687813+0000", 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029564199999999999 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.197 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687862+0000", 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9150000000000002e-05 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.707246+0000", 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019383507000000001 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.707368+0000", 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012196 2024-09-21T08:30:30.198 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.707569+0000", 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020167199999999999 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4049 7.6 7:611b1c4b:::smithi13779221-3:head [copy-from ver 495] snapc a8=[a8,a7,a5,a4,a3,a2,a1,9e,9c,9b,9a,99,98,94,93,92,91,90,8c,8b,88,86,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:29:54.710877+0000", 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.375091759999997, 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.019350467999999999, 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.199 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4049 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.710877+0000", 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.200 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.710877+0000", 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.710879+0000", 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9489999999999999e-06 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.710888+0000", 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9120000000000004e-06 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.201 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.710899+0000", 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1082e-05 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.710904+0000", 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8280000000000002e-06 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.710976+0000", 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2587000000000001e-05 2024-09-21T08:30:30.202 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711079+0000", 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102501 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.725073+0000", 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013994685 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.728799+0000", 2024-09-21T08:30:30.203 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037258650000000001 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.730096+0000", 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012972420000000001 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.730148+0000", 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1106999999999997e-05 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.204 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.730227+0000", 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9709999999999994e-05 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4057 7.1c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 336] snapc a8=[a8,a7,a5,a4,a3,a2,a1,9e,9c,9b,9a,99,98,94,93,92,91,90,8c,8b,88,86,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:29:54.711023+0000", 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.374945818999997, 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027355181999999999, 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.205 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4057 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711023+0000", 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711023+0000", 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.206 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711023+0000", 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1500000000000004e-07 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711026+0000", 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9450000000000002e-06 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711033+0000", 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7769999999999997e-06 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.207 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711036+0000", 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1219999999999999e-06 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711605+0000", 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056895899999999996 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711739+0000", 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013337599999999999 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.208 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1", 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.729672+0000", 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017933359999999999 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.735954+0000", 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062820030000000004 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.738163+0000", 2024-09-21T08:30:30.209 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022090489999999998 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.738243+0000", 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9549999999999996e-05 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.738378+0000", 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013532600000000001 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.210 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4058 7.1c 7:3cae00e7:::smithi13779221-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e401)", 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:29:54.711041+0000", 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.374927479999997, 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021728573000000001, 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4058 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.211 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711041+0000", 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711041+0000", 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711041+0000", 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.39e-07 2024-09-21T08:30:30.212 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711043+0000", 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3710000000000001e-06 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711047+0000", 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7639999999999999e-06 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711049+0000", 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.553e-06 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.213 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711915+0000", 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00086636600000000003 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711980+0000", 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4951999999999997e-05 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.732569+0000", 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020588805000000002 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.214 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.732647+0000", 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7997000000000006e-05 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.732770+0000", 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122426 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.215 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4471 7.8 7:144c2ec5:::smithi13779221-45:head [copy-from ver 432] snapc b6=[b6,b5,b4,b3,b2,b1,b0,af,ae,ac,a7,a5,a4,a3,9b,99,93,92,84,7e] ondisk+write+known_if_redirected+supports_pool_eio e436)", 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:30:29.628086+0000", 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.457882077, 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.021641858, 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4471 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.216 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.628086+0000", 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.628086+0000", 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.628087+0000", 2024-09-21T08:30:30.217 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5000000000000003e-07 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.628091+0000", 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0720000000000001e-06 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.628097+0000", 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9549999999999997e-06 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:30:30.218 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.628102+0000", 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2429999999999996e-06 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.628144+0000", 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1350999999999998e-05 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.628150+0000", 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7039999999999997e-06 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:30:30.219 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.630666+0000", 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002515429 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.630791+0000", 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000125084 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.639717+0000", 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089261519999999997 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.220 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.647238+0000", 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0075208200000000001 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.649478+0000", 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022396909999999998 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.649567+0000", 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9693999999999999e-05 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:30:30.221 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:30:30.222 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:30:30.222 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:29.649728+0000", 2024-09-21T08:30:30.222 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000161113 2024-09-21T08:30:30.222 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.222 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.222 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.222 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:30:30.222 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:30:30.222 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:30:30.223 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:30.224 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:30.224 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:30.267 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:30:30.099+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:30:30.315 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:30.316 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:30.324 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:30.346 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:30.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:30.347 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:30.448 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:30.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:30.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:30.470 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:30.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:30.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:30.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:30.593 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:30.631 INFO:tasks.rados.rados.0.smithi137.stdout:1730: done (8 left) 2024-09-21T08:30:30.631 INFO:tasks.rados.rados.0.smithi137.stdout:1731: snap_remove snap 165 2024-09-21T08:30:30.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:30.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:30.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:30.719 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:30.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:30.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:30.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:30.844 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:30.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:30.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:30.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:30.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:30 smithi165 ceph-mon[32893]: osdmap e436: 8 total, 6 up, 4 in 2024-09-21T08:30:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:30:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:30:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:31.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-mon[29634]: osdmap e436: 8 total, 6 up, 4 in 2024-09-21T08:30:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:30:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:30:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:31.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:30:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:30:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-mon[41044]: osdmap e436: 8 total, 6 up, 4 in 2024-09-21T08:30:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:30:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:30:31.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:30:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:31.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:31.093 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:31.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:31.094 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:31.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:31.217 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:31.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:31.218 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:31.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T08:30:31.319 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:31.341 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:31.341 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:31.342 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:31.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:31.463 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:31.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:31.464 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:31.530 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:31.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:31.573 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:30:31.583 INFO:tasks.rados.rados.0.smithi137.stdout:1731: done (8 left) 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:1732: rollback oid 26 current snap is 181 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 26 to 176 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:1723: finishing rollback tid 0 to smithi13779221-14 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:1723: finishing rollback tid 1 to smithi13779221-14 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:1723: finishing rollback tid 2 to smithi13779221-14 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 392 (ObjNum 596 snap 174 seq_num 596) dirty exists 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:1726: finishing rollback tid 0 to smithi13779221-46 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:1726: finishing rollback tid 1 to smithi13779221-46 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:1726: finishing rollback tid 2 to smithi13779221-46 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 721 (ObjNum 499 snap 140 seq_num 499) dirty exists 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:1728: finishing write tid 1 to smithi13779221-43 2024-09-21T08:30:31.584 INFO:tasks.rados.rados.0.smithi137.stdout:1729: expect (ObjNum 433 snap 110 seq_num 433) 2024-09-21T08:30:31.591 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:31.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:31.592 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:31.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:31.714 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:31.714 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:31.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:31.741 INFO:tasks.rados.rados.0.smithi137.stdout:1728: finishing write tid 2 to smithi13779221-43 2024-09-21T08:30:31.742 INFO:tasks.rados.rados.0.smithi137.stdout:1728: finishing write tid 3 to smithi13779221-43 2024-09-21T08:30:31.742 INFO:tasks.rados.rados.0.smithi137.stdout:1728: finishing write tid 4 to smithi13779221-43 2024-09-21T08:30:31.742 INFO:tasks.rados.rados.0.smithi137.stdout:1728: oid 43 updating version 0 to 394 2024-09-21T08:30:31.742 INFO:tasks.rados.rados.0.smithi137.stdout:1728: oid 43 version 394 is already newer than 393 2024-09-21T08:30:31.742 INFO:tasks.rados.rados.0.smithi137.stdout:1728: oid 43 version 394 is already newer than 392 2024-09-21T08:30:31.742 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 394 (ObjNum 616 snap 181 seq_num 616) dirty exists 2024-09-21T08:30:31.742 INFO:tasks.rados.rados.0.smithi137.stdout:1728: left oid 43 (ObjNum 616 snap 181 seq_num 616) 2024-09-21T08:30:31.742 INFO:tasks.rados.rados.0.smithi137.stdout:1725: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:30:31.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:31.838 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:31.838 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:31.839 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:31.840 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:31.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:31 smithi165 ceph-mon[32893]: pgmap v560: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 524 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 353/1371 objects degraded (25.748%) 2024-09-21T08:30:31.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:31 smithi165 ceph-mon[32893]: osdmap e437: 8 total, 6 up, 4 in 2024-09-21T08:30:31.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:31 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 353/1371 objects degraded (25.748%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:31.924 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:31 smithi165 ceph-mon[32893]: osdmap e438: 8 total, 6 up, 4 in 2024-09-21T08:30:31.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:31.961 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:31 smithi137 ceph-mon[29634]: pgmap v560: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 524 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 353/1371 objects degraded (25.748%) 2024-09-21T08:30:31.961 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:31 smithi137 ceph-mon[29634]: osdmap e437: 8 total, 6 up, 4 in 2024-09-21T08:30:31.961 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:31 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 353/1371 objects degraded (25.748%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:31.961 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:31 smithi137 ceph-mon[29634]: osdmap e438: 8 total, 6 up, 4 in 2024-09-21T08:30:31.961 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:31 smithi137 ceph-mon[41044]: pgmap v560: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 524 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 353/1371 objects degraded (25.748%) 2024-09-21T08:30:31.961 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:31 smithi137 ceph-mon[41044]: osdmap e437: 8 total, 6 up, 4 in 2024-09-21T08:30:31.962 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:31 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 353/1371 objects degraded (25.748%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:31.962 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:31 smithi137 ceph-mon[41044]: osdmap e438: 8 total, 6 up, 4 in 2024-09-21T08:30:31.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:31.962 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:31.963 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:31.971 INFO:tasks.rados.rados.0.smithi137.stdout:1724: finishing rollback tid 0 to smithi13779221-40 2024-09-21T08:30:31.971 INFO:tasks.rados.rados.0.smithi137.stdout:1724: finishing rollback tid 1 to smithi13779221-40 2024-09-21T08:30:31.971 INFO:tasks.rados.rados.0.smithi137.stdout:1724: finishing rollback tid 2 to smithi13779221-40 2024-09-21T08:30:31.971 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 328 (ObjNum 522 snap 149 seq_num 522) dirty exists 2024-09-21T08:30:31.971 INFO:tasks.rados.rados.0.smithi137.stdout:1727: finishing copy_from racing read to smithi13779221-45 2024-09-21T08:30:31.971 INFO:tasks.rados.rados.0.smithi137.stdout:1727: finishing copy_from to smithi13779221-45 2024-09-21T08:30:31.972 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 432 (ObjNum 545 snap 162 seq_num 545) dirty exists 2024-09-21T08:30:31.972 INFO:tasks.rados.rados.0.smithi137.stdout:1722: done (8 left) 2024-09-21T08:30:31.972 INFO:tasks.rados.rados.0.smithi137.stdout:1723: done (7 left) 2024-09-21T08:30:31.972 INFO:tasks.rados.rados.0.smithi137.stdout:1724: done (6 left) 2024-09-21T08:30:31.972 INFO:tasks.rados.rados.0.smithi137.stdout:1725: done (5 left) 2024-09-21T08:30:31.972 INFO:tasks.rados.rados.0.smithi137.stdout:1726: done (4 left) 2024-09-21T08:30:31.972 INFO:tasks.rados.rados.0.smithi137.stdout:1727: done (3 left) 2024-09-21T08:30:31.972 INFO:tasks.rados.rados.0.smithi137.stdout:1728: done (2 left) 2024-09-21T08:30:31.972 INFO:tasks.rados.rados.0.smithi137.stdout:1729: done (1 left) 2024-09-21T08:30:31.972 INFO:tasks.rados.rados.0.smithi137.stdout:1733: snap_remove snap 145 2024-09-21T08:30:32.030 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:30:32.030 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:30:32.030 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:30:32.030 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:30:32.063 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:32.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:32.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:32.086 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:32.187 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:32.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:32.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:32.210 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:32.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:30:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:30:32.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:32.316 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:30:32.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:32.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:32.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:32.386 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1] out_osds: [4, 6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:30:32.386 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:30:32.387 INFO:tasks.thrashosds.thrasher:inject_pause on osd.5 2024-09-21T08:30:32.387 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T08:30:32.387 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T08:30:32.387 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 config set filestore_inject_stall 3 2024-09-21T08:30:32.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:32.455 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:32.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:32.456 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:32.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:32.573 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:32.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:32.579 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:32.579 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:32.638 INFO:tasks.rados.rados.0.smithi137.stdout:1733: done (1 left) 2024-09-21T08:30:32.638 INFO:tasks.rados.rados.0.smithi137.stdout:1734: write (excl) oid 31 current snap is 181 2024-09-21T08:30:32.638 INFO:tasks.rados.rados.0.smithi137.stdout:1734: seq_num 617 ranges {769919=653226,1938684=431235} 2024-09-21T08:30:32.638 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:30:32.645 INFO:tasks.rados.rados.0.smithi137.stdout:1734: writing smithi13779221-31 from 769919 to 1423145 tid 1 2024-09-21T08:30:32.650 INFO:tasks.rados.rados.0.smithi137.stdout:1734: writing smithi13779221-31 from 1938684 to 2369919 tid 2 2024-09-21T08:30:32.650 INFO:tasks.rados.rados.0.smithi137.stdout:1735: snap_create 2024-09-21T08:30:32.650 INFO:tasks.rados.rados.0.smithi137.stdout:1732: finishing rollback tid 0 to smithi13779221-26 2024-09-21T08:30:32.651 INFO:tasks.rados.rados.0.smithi137.stdout:1732: finishing rollback tid 1 to smithi13779221-26 2024-09-21T08:30:32.651 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 396 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-21T08:30:32.655 INFO:tasks.rados.rados.0.smithi137.stdout:1734: finishing write tid 1 to smithi13779221-31 2024-09-21T08:30:32.655 INFO:tasks.rados.rados.0.smithi137.stdout:1734: finishing write tid 2 to smithi13779221-31 2024-09-21T08:30:32.655 INFO:tasks.rados.rados.0.smithi137.stdout:1734: finishing write tid 3 to smithi13779221-31 2024-09-21T08:30:32.656 INFO:tasks.rados.rados.0.smithi137.stdout:1734: finishing write tid 4 to smithi13779221-31 2024-09-21T08:30:32.656 INFO:tasks.rados.rados.0.smithi137.stdout:1734: oid 31 updating version 0 to 355 2024-09-21T08:30:32.656 INFO:tasks.rados.rados.0.smithi137.stdout:1734: oid 31 updating version 355 to 356 2024-09-21T08:30:32.656 INFO:tasks.rados.rados.0.smithi137.stdout:1734: oid 31 version 356 is already newer than 354 2024-09-21T08:30:32.656 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 356 (ObjNum 617 snap 181 seq_num 617) dirty exists 2024-09-21T08:30:32.656 INFO:tasks.rados.rados.0.smithi137.stdout:1734: left oid 31 (ObjNum 617 snap 181 seq_num 617) 2024-09-21T08:30:32.678 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T08:30:32.680 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:32.702 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:32.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:32.703 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:32.720 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:32.804 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:32.825 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:32.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:32.826 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:32.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:32 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1666355035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:32.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:32 smithi137 ceph-mon[29634]: from='client.45743 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:32.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:32 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1666355035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:32.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:32 smithi137 ceph-mon[41044]: from='client.45743 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:32.927 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:32.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:32.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:32.951 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:32.957 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:32 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1666355035' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:32.957 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:32 smithi165 ceph-mon[32893]: from='client.45743 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:33.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:33.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:33.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:33.074 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:33.175 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:33.196 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:33.197 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:33.236 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:30:33.236 INFO:teuthology.orchestra.run.smithi165.stdout: "success": "" 2024-09-21T08:30:33.236 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:30:33.298 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:33.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:33.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:33.321 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:33.412 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:33.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:33.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:33.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:33.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:33.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:33.571 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:33.572 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:33.572 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:33.641 INFO:tasks.rados.rados.0.smithi137.stdout:1732: done (2 left) 2024-09-21T08:30:33.641 INFO:tasks.rados.rados.0.smithi137.stdout:1734: done (1 left) 2024-09-21T08:30:33.641 INFO:tasks.rados.rados.0.smithi137.stdout:1735: done (0 left) 2024-09-21T08:30:33.641 INFO:tasks.rados.rados.0.smithi137.stdout:1736: write oid 19 current snap is 182 2024-09-21T08:30:33.641 INFO:tasks.rados.rados.0.smithi137.stdout:1736: seq_num 618 ranges {786726=728475,2280813=105913} 2024-09-21T08:30:33.649 INFO:tasks.rados.rados.0.smithi137.stdout:1736: writing smithi13779221-19 from 786726 to 1515201 tid 1 2024-09-21T08:30:33.654 INFO:tasks.rados.rados.0.smithi137.stdout:1736: writing smithi13779221-19 from 2280813 to 2386726 tid 2 2024-09-21T08:30:33.654 INFO:tasks.rados.rados.0.smithi137.stdout:1737: write oid 14 current snap is 182 2024-09-21T08:30:33.654 INFO:tasks.rados.rados.0.smithi137.stdout:1737: seq_num 619 ranges {403533=403724,1539295=652201} 2024-09-21T08:30:33.658 INFO:tasks.rados.rados.0.smithi137.stdout:1737: writing smithi13779221-14 from 403533 to 807257 tid 1 2024-09-21T08:30:33.666 INFO:tasks.rados.rados.0.smithi137.stdout:1737: writing smithi13779221-14 from 1539295 to 2191496 tid 2 2024-09-21T08:30:33.666 INFO:tasks.rados.rados.0.smithi137.stdout:1738: write oid 26 current snap is 182 2024-09-21T08:30:33.667 INFO:tasks.rados.rados.0.smithi137.stdout:1736: finishing write tid 1 to smithi13779221-19 2024-09-21T08:30:33.667 INFO:tasks.rados.rados.0.smithi137.stdout:1738: seq_num 620 ranges {420340=478973,1597777=512212} 2024-09-21T08:30:33.671 INFO:tasks.rados.rados.0.smithi137.stdout:1738: writing smithi13779221-26 from 420340 to 899313 tid 1 2024-09-21T08:30:33.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:33.679 INFO:tasks.rados.rados.0.smithi137.stdout:1738: writing smithi13779221-26 from 1597777 to 2109989 tid 2 2024-09-21T08:30:33.679 INFO:tasks.rados.rados.0.smithi137.stdout:1739: copy_from oid 17 from oid 46 current snap is 182 2024-09-21T08:30:33.679 INFO:tasks.rados.rados.0.smithi137.stdout:1736: finishing write tid 2 to smithi13779221-19 2024-09-21T08:30:33.679 INFO:tasks.rados.rados.0.smithi137.stdout:1736: finishing write tid 3 to smithi13779221-19 2024-09-21T08:30:33.679 INFO:tasks.rados.rados.0.smithi137.stdout:1736: finishing write tid 4 to smithi13779221-19 2024-09-21T08:30:33.679 INFO:tasks.rados.rados.0.smithi137.stdout:1736: oid 19 updating version 0 to 508 2024-09-21T08:30:33.679 INFO:tasks.rados.rados.0.smithi137.stdout:1736: oid 19 updating version 508 to 510 2024-09-21T08:30:33.679 INFO:tasks.rados.rados.0.smithi137.stdout:1736: oid 19 version 510 is already newer than 509 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 510 (ObjNum 618 snap 182 seq_num 618) dirty exists 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1736: left oid 19 (ObjNum 618 snap 182 seq_num 618) 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1737: finishing write tid 1 to smithi13779221-14 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1736: done (3 left) 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1740: read oid 8 snap -1 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1740: expect deleted 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1741: read oid 9 snap -1 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1741: expect deleted 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1742: snap_create 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1737: finishing write tid 2 to smithi13779221-14 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1737: finishing write tid 3 to smithi13779221-14 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1737: finishing write tid 4 to smithi13779221-14 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1737: oid 14 updating version 0 to 394 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1737: oid 14 version 394 is already newer than 393 2024-09-21T08:30:33.680 INFO:tasks.rados.rados.0.smithi137.stdout:1737: oid 14 updating version 394 to 395 2024-09-21T08:30:33.681 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 395 (ObjNum 619 snap 182 seq_num 619) dirty exists 2024-09-21T08:30:33.681 INFO:tasks.rados.rados.0.smithi137.stdout:1737: left oid 14 (ObjNum 619 snap 182 seq_num 619) 2024-09-21T08:30:33.681 INFO:tasks.rados.rados.0.smithi137.stdout:1738: finishing write tid 1 to smithi13779221-26 2024-09-21T08:30:33.681 INFO:tasks.rados.rados.0.smithi137.stdout:1738: finishing write tid 2 to smithi13779221-26 2024-09-21T08:30:33.681 INFO:tasks.rados.rados.0.smithi137.stdout:1738: finishing write tid 3 to smithi13779221-26 2024-09-21T08:30:33.682 INFO:tasks.rados.rados.0.smithi137.stdout:1738: finishing write tid 4 to smithi13779221-26 2024-09-21T08:30:33.682 INFO:tasks.rados.rados.0.smithi137.stdout:1738: oid 26 updating version 0 to 399 2024-09-21T08:30:33.682 INFO:tasks.rados.rados.0.smithi137.stdout:1738: oid 26 version 399 is already newer than 398 2024-09-21T08:30:33.682 INFO:tasks.rados.rados.0.smithi137.stdout:1738: oid 26 version 399 is already newer than 397 2024-09-21T08:30:33.683 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 399 (ObjNum 620 snap 182 seq_num 620) dirty exists 2024-09-21T08:30:33.683 INFO:tasks.rados.rados.0.smithi137.stdout:1738: left oid 26 (ObjNum 620 snap 182 seq_num 620) 2024-09-21T08:30:33.686 INFO:tasks.rados.rados.0.smithi137.stdout:1739: finishing copy_from racing read to smithi13779221-17 2024-09-21T08:30:33.689 INFO:tasks.rados.rados.0.smithi137.stdout:1739: finishing copy_from to smithi13779221-17 2024-09-21T08:30:33.689 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 721 (ObjNum 499 snap 140 seq_num 499) dirty exists 2024-09-21T08:30:33.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:33.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:33.695 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:33.786 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:33 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:33.786 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:33 smithi137 ceph-mon[29634]: pgmap v563: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 538 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 19 op/s; 360/1404 objects degraded (25.641%) 2024-09-21T08:30:33.787 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:33 smithi137 ceph-mon[29634]: from='client.45743 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:33.787 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:33 smithi137 ceph-mon[29634]: osdmap e439: 8 total, 6 up, 4 in 2024-09-21T08:30:33.787 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:33 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:33.787 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:33 smithi137 ceph-mon[41044]: pgmap v563: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 538 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 19 op/s; 360/1404 objects degraded (25.641%) 2024-09-21T08:30:33.787 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:33 smithi137 ceph-mon[41044]: from='client.45743 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:33.787 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:33 smithi137 ceph-mon[41044]: osdmap e439: 8 total, 6 up, 4 in 2024-09-21T08:30:33.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:33.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:33 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:33.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:33 smithi165 ceph-mon[32893]: pgmap v563: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 538 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 19 op/s; 360/1404 objects degraded (25.641%) 2024-09-21T08:30:33.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:33 smithi165 ceph-mon[32893]: from='client.45743 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:33.806 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:33 smithi165 ceph-mon[32893]: osdmap e439: 8 total, 6 up, 4 in 2024-09-21T08:30:33.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:33.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:33.819 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:33.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:33.941 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:33.942 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:33.942 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:34.043 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:34.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:34.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:34.066 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:34.119 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:34.119 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:34.119 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:34.120 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:34.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:34.188 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:34.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:34.189 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:34.290 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:34.311 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:34.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:34.312 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:34.351 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:34.351 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:34.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:34.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:34.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:34.438 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:34.538 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:34.560 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:34.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:34.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:34.648 INFO:tasks.rados.rados.0.smithi137.stdout:1737: done (5 left) 2024-09-21T08:30:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:1738: done (4 left) 2024-09-21T08:30:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:1739: done (3 left) 2024-09-21T08:30:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:1740: done (2 left) 2024-09-21T08:30:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:1741: done (1 left) 2024-09-21T08:30:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:1742: done (0 left) 2024-09-21T08:30:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:1743: snap_remove snap 175 2024-09-21T08:30:34.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:34.682 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:34.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:34.683 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:34.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:34.806 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:34.807 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:34.807 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:34.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:34 smithi137 ceph-mon[29634]: osdmap e440: 8 total, 6 up, 4 in 2024-09-21T08:30:34.890 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/616329349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:34.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:34 smithi137 ceph-mon[41044]: osdmap e440: 8 total, 6 up, 4 in 2024-09-21T08:30:34.891 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/616329349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:34.908 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:34.918 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:34.919 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:34.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:34.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:34.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:34.963 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:34 smithi165 ceph-mon[32893]: osdmap e440: 8 total, 6 up, 4 in 2024-09-21T08:30:34.963 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/616329349' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:34.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T08:30:35.032 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:35.054 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:35.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:35.055 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:35.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:35.178 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:35.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:35.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:35.237 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:30:35.279 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:35.301 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:35.301 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:35.302 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:35.402 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:35.423 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:35.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:35.424 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:35.525 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:35.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:30:35.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:35.547 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:35.547 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:35.648 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:35.654 INFO:tasks.rados.rados.0.smithi137.stdout:1743: done (0 left) 2024-09-21T08:30:35.654 INFO:tasks.rados.rados.0.smithi137.stdout:1744: read oid 20 snap -1 2024-09-21T08:30:35.654 INFO:tasks.rados.rados.0.smithi137.stdout:1744: expect deleted 2024-09-21T08:30:35.654 INFO:tasks.rados.rados.0.smithi137.stdout:1745: write oid 17 current snap is 183 2024-09-21T08:30:35.654 INFO:tasks.rados.rados.0.smithi137.stdout:1745: seq_num 621 ranges {437147=554222,1656259=655870} 2024-09-21T08:30:35.660 INFO:tasks.rados.rados.0.smithi137.stdout:1745: writing smithi13779221-17 from 437147 to 991369 tid 1 2024-09-21T08:30:35.668 INFO:tasks.rados.rados.0.smithi137.stdout:1745: writing smithi13779221-17 from 1656259 to 2312129 tid 2 2024-09-21T08:30:35.670 INFO:tasks.rados.rados.0.smithi137.stdout:1745: finishing write tid 1 to smithi13779221-17 2024-09-21T08:30:35.670 INFO:tasks.rados.rados.0.smithi137.stdout:1744: done (1 left) 2024-09-21T08:30:35.670 INFO:tasks.rados.rados.0.smithi137.stdout:1746: read oid 2 snap -1 2024-09-21T08:30:35.670 INFO:tasks.rados.rados.0.smithi137.stdout:1746: expect (ObjNum 608 snap 179 seq_num 608) 2024-09-21T08:30:35.670 INFO:tasks.rados.rados.0.smithi137.stdout:1747: snap_create 2024-09-21T08:30:35.672 INFO:tasks.rados.rados.0.smithi137.stdout:1745: finishing write tid 2 to smithi13779221-17 2024-09-21T08:30:35.672 INFO:tasks.rados.rados.0.smithi137.stdout:1745: finishing write tid 3 to smithi13779221-17 2024-09-21T08:30:35.673 INFO:tasks.rados.rados.0.smithi137.stdout:1745: finishing write tid 4 to smithi13779221-17 2024-09-21T08:30:35.674 INFO:tasks.rados.rados.0.smithi137.stdout:1745: oid 17 updating version 0 to 723 2024-09-21T08:30:35.674 INFO:tasks.rados.rados.0.smithi137.stdout:1745: oid 17 version 723 is already newer than 722 2024-09-21T08:30:35.674 INFO:tasks.rados.rados.0.smithi137.stdout:1745: oid 17 updating version 723 to 724 2024-09-21T08:30:35.674 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 724 (ObjNum 621 snap 183 seq_num 621) dirty exists 2024-09-21T08:30:35.674 INFO:tasks.rados.rados.0.smithi137.stdout:1745: left oid 17 (ObjNum 621 snap 183 seq_num 621) 2024-09-21T08:30:35.676 INFO:tasks.rados.rados.0.smithi137.stdout:1746: expect (ObjNum 608 snap 179 seq_num 608) 2024-09-21T08:30:35.679 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:35.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:35.680 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:35.781 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:35.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:35.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:35.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:35.817 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:35.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:35.927 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:35.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:35.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:35.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:35 smithi137 ceph-mon[29634]: pgmap v566: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 538 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 19 op/s; 360/1404 objects degraded (25.641%) 2024-09-21T08:30:35.938 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:35 smithi137 ceph-mon[29634]: osdmap e441: 8 total, 6 up, 4 in 2024-09-21T08:30:35.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:35 smithi137 ceph-mon[41044]: pgmap v566: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 538 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 19 op/s; 360/1404 objects degraded (25.641%) 2024-09-21T08:30:35.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:35 smithi137 ceph-mon[41044]: osdmap e441: 8 total, 6 up, 4 in 2024-09-21T08:30:35.966 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:35.967 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:35.967 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:35 smithi165 ceph-mon[32893]: pgmap v566: 168 pgs: 4 active+clean+snaptrim, 2 active+clean+snaptrim_wait, 72 active+undersized, 58 active+undersized+degraded, 32 active+clean; 538 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 11 MiB/s rd, 2.6 MiB/s wr, 19 op/s; 360/1404 objects degraded (25.641%) 2024-09-21T08:30:35.967 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:35 smithi165 ceph-mon[32893]: osdmap e441: 8 total, 6 up, 4 in 2024-09-21T08:30:36.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:36.049 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:36.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:36.050 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:36.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:36.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:36.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:36.173 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:36.274 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:36.297 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:36.298 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:36.298 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:36.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:36.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:36.422 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:36.422 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:36.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:36.548 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:36.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:36.549 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:36.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:36.658 INFO:tasks.rados.rados.0.smithi137.stdout:1745: done (2 left) 2024-09-21T08:30:36.659 INFO:tasks.rados.rados.0.smithi137.stdout:1746: done (1 left) 2024-09-21T08:30:36.659 INFO:tasks.rados.rados.0.smithi137.stdout:1747: done (0 left) 2024-09-21T08:30:36.659 INFO:tasks.rados.rados.0.smithi137.stdout:1748: read oid 30 snap -1 2024-09-21T08:30:36.659 INFO:tasks.rados.rados.0.smithi137.stdout:1748: expect (ObjNum 603 snap 175 seq_num 603) 2024-09-21T08:30:36.659 INFO:tasks.rados.rados.0.smithi137.stdout:1749: rollback oid 12 current snap is 184 2024-09-21T08:30:36.659 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 12 to 179 2024-09-21T08:30:36.659 INFO:tasks.rados.rados.0.smithi137.stdout:1750: read oid 45 snap -1 2024-09-21T08:30:36.659 INFO:tasks.rados.rados.0.smithi137.stdout:1750: expect (ObjNum 545 snap 162 seq_num 545) 2024-09-21T08:30:36.659 INFO:tasks.rados.rados.0.smithi137.stdout:1751: delete oid 41 current snap is 184 2024-09-21T08:30:36.659 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:30:36.670 INFO:tasks.rados.rados.0.smithi137.stdout:1751: done (3 left) 2024-09-21T08:30:36.670 INFO:tasks.rados.rados.0.smithi137.stdout:1752: write (excl) oid 11 current snap is 184 2024-09-21T08:30:36.670 INFO:tasks.rados.rados.0.smithi137.stdout:1752: seq_num 623 ranges {470761=704720,1656870=659539} 2024-09-21T08:30:36.676 INFO:tasks.rados.rados.0.smithi137.stdout:1752: writing smithi13779221-11 from 470761 to 1175481 tid 1 2024-09-21T08:30:36.678 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:36.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:36.679 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:36.684 INFO:tasks.rados.rados.0.smithi137.stdout:1752: writing smithi13779221-11 from 1656870 to 2316409 tid 2 2024-09-21T08:30:36.684 INFO:tasks.rados.rados.0.smithi137.stdout:1753: snap_remove snap 172 2024-09-21T08:30:36.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T08:30:36.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:36.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:36.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:36.832 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:36.919 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:30:36.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:36.942 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:36 smithi137 ceph-mon[29634]: osdmap e442: 8 total, 6 up, 4 in 2024-09-21T08:30:36.942 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:36 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/224967713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:30:36.942 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:36 smithi137 ceph-mon[29634]: from='client.45755 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:30:36.942 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:36 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:36.942 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:36 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 360/1404 objects degraded (25.641%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:36.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:36 smithi137 ceph-mon[41044]: osdmap e442: 8 total, 6 up, 4 in 2024-09-21T08:30:36.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:36 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/224967713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:30:36.942 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:36 smithi137 ceph-mon[41044]: from='client.45755 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:30:36.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:36 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:36.943 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:36 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 360/1404 objects degraded (25.641%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:37.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:37.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:37.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:36 smithi165 ceph-mon[32893]: osdmap e442: 8 total, 6 up, 4 in 2024-09-21T08:30:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:36 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/224967713' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:30:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:36 smithi165 ceph-mon[32893]: from='client.45755 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:30:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:36 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:36 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 360/1404 objects degraded (25.641%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:37.022 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:37.109 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:37.135 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:37.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:37.135 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:37.236 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:37.258 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:37.259 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:37.259 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:37.360 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:37.382 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:37.383 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:37.383 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:37.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:37.507 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:37.508 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:37.508 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:37.609 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:37.630 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:37.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:37.631 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:37.665 INFO:tasks.rados.rados.0.smithi137.stdout:1753: done (4 left) 2024-09-21T08:30:37.666 INFO:tasks.rados.rados.0.smithi137.stdout:1754: read oid 18 snap 153 2024-09-21T08:30:37.666 INFO:tasks.rados.rados.0.smithi137.stdout:1754: expect (ObjNum 406 snap 99 seq_num 406) 2024-09-21T08:30:37.666 INFO:tasks.rados.rados.0.smithi137.stdout:1755: read oid 43 snap -1 2024-09-21T08:30:37.666 INFO:tasks.rados.rados.0.smithi137.stdout:1755: expect (ObjNum 616 snap 181 seq_num 616) 2024-09-21T08:30:37.666 INFO:tasks.rados.rados.0.smithi137.stdout:1756: snap_remove snap 178 2024-09-21T08:30:37.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:37.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:37.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:37.754 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:37.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:37 smithi137 ceph-mon[41044]: pgmap v569: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 10 op/s; 365/1419 objects degraded (25.722%) 2024-09-21T08:30:37.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:37 smithi137 ceph-mon[41044]: from='client.45755 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:30:37.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:37 smithi137 ceph-mon[41044]: osdmap e443: 8 total, 6 up, 4 in 2024-09-21T08:30:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:37 smithi137 ceph-mon[29634]: pgmap v569: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 10 op/s; 365/1419 objects degraded (25.722%) 2024-09-21T08:30:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:37 smithi137 ceph-mon[29634]: from='client.45755 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:30:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:37 smithi137 ceph-mon[29634]: osdmap e443: 8 total, 6 up, 4 in 2024-09-21T08:30:37.855 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:37.877 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:37.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:37.878 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:37.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:38.002 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:38.003 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:37 smithi165 ceph-mon[32893]: pgmap v569: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 10 op/s; 365/1419 objects degraded (25.722%) 2024-09-21T08:30:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:37 smithi165 ceph-mon[32893]: from='client.45755 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:30:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:37 smithi165 ceph-mon[32893]: osdmap e443: 8 total, 6 up, 4 in 2024-09-21T08:30:38.035 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:38.064 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:38.064 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:38.104 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:38.125 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:38.126 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:38.227 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:38.248 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:38.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:38.249 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:38.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:38.386 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:38.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:38.387 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:38.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:38.510 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:38.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:38.511 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:38.612 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:38.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:38.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:38.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:38.678 INFO:tasks.rados.rados.0.smithi137.stdout:1756: done (6 left) 2024-09-21T08:30:38.678 INFO:tasks.rados.rados.0.smithi137.stdout:1757: rollback oid 6 current snap is 184 2024-09-21T08:30:38.678 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 6 to 153 2024-09-21T08:30:38.678 INFO:tasks.rados.rados.0.smithi137.stdout:1758: read oid 41 snap -1 2024-09-21T08:30:38.678 INFO:tasks.rados.rados.0.smithi137.stdout:1758: expect deleted 2024-09-21T08:30:38.678 INFO:tasks.rados.rados.0.smithi137.stdout:1759: rollback oid 44 current snap is 184 2024-09-21T08:30:38.679 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 44 to 130 2024-09-21T08:30:38.679 INFO:tasks.rados.rados.0.smithi137.stdout:1760: read oid 25 snap -1 2024-09-21T08:30:38.679 INFO:tasks.rados.rados.0.smithi137.stdout:1760: expect (ObjNum 459 snap 124 seq_num 459) 2024-09-21T08:30:38.679 INFO:tasks.rados.rados.0.smithi137.stdout:1761: read oid 47 snap 176 2024-09-21T08:30:38.679 INFO:tasks.rados.rados.0.smithi137.stdout:1761: expect deleted 2024-09-21T08:30:38.679 INFO:tasks.rados.rados.0.smithi137.stdout:1762: read oid 50 snap 162 2024-09-21T08:30:38.679 INFO:tasks.rados.rados.0.smithi137.stdout:1762: expect (ObjNum 544 snap 162 seq_num 544) 2024-09-21T08:30:38.679 INFO:tasks.rados.rados.0.smithi137.stdout:1763: copy_from oid 37 from oid 19 current snap is 184 2024-09-21T08:30:38.679 INFO:tasks.rados.rados.0.smithi137.stdout:1764: write oid 33 current snap is 184 2024-09-21T08:30:38.679 INFO:tasks.rados.rados.0.smithi137.stdout:1764: seq_num 624 ranges {487568=496322,1431705=519550} 2024-09-21T08:30:38.683 INFO:tasks.rados.rados.0.smithi137.stdout:1764: writing smithi13779221-33 from 487568 to 983890 tid 1 2024-09-21T08:30:38.691 INFO:tasks.rados.rados.0.smithi137.stdout:1764: writing smithi13779221-33 from 1431705 to 1951255 tid 2 2024-09-21T08:30:38.691 INFO:tasks.rados.rados.0.smithi137.stdout:1765: delete oid 46 current snap is 184 2024-09-21T08:30:38.691 INFO:tasks.rados.rados.0.smithi137.stdout:1749: finishing rollback tid 1 to smithi13779221-12 2024-09-21T08:30:38.691 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 395 (ObjNum 129 snap 0 seq_num 3760587920) dirty dne 2024-09-21T08:30:38.691 INFO:tasks.rados.rados.0.smithi137.stdout:1750: expect (ObjNum 545 snap 162 seq_num 545) 2024-09-21T08:30:38.714 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:38.729 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:38.729 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:38.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:38.751 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:38.751 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:38.763 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:38.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:38.764 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:38.773 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:38.773 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:38.790 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:38.790 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:38.806 INFO:tasks.rados.rados.0.smithi137.stdout:1748: expect (ObjNum 603 snap 175 seq_num 603) 2024-09-21T08:30:38.806 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:38.807 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:38.823 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:38.824 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:38.824 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:38.835 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1] out_osds: [4, 6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:30:38.835 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:30:38.835 INFO:tasks.thrashosds.thrasher:Growing pool .mgr 2024-09-21T08:30:38.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:30:38.864 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:38.889 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:38.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:38.890 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:38.943 INFO:tasks.rados.rados.0.smithi137.stdout:1752: finishing write tid 1 to smithi13779221-11 2024-09-21T08:30:38.943 INFO:tasks.rados.rados.0.smithi137.stdout:1752: finishing write tid 2 to smithi13779221-11 2024-09-21T08:30:38.943 INFO:tasks.rados.rados.0.smithi137.stdout:1748: done (14 left) 2024-09-21T08:30:38.943 INFO:tasks.rados.rados.0.smithi137.stdout:1749: done (13 left) 2024-09-21T08:30:38.943 INFO:tasks.rados.rados.0.smithi137.stdout:1750: done (12 left) 2024-09-21T08:30:38.943 INFO:tasks.rados.rados.0.smithi137.stdout:1765: done (11 left) 2024-09-21T08:30:38.943 INFO:tasks.rados.rados.0.smithi137.stdout:1766: write oid 30 current snap is 184 2024-09-21T08:30:38.943 INFO:tasks.rados.rados.0.smithi137.stdout:1752: finishing write tid 3 to smithi13779221-11 2024-09-21T08:30:38.943 INFO:tasks.rados.rados.0.smithi137.stdout:1752: finishing write tid 4 to smithi13779221-11 2024-09-21T08:30:38.944 INFO:tasks.rados.rados.0.smithi137.stdout:1752: oid 11 updating version 0 to 530 2024-09-21T08:30:38.944 INFO:tasks.rados.rados.0.smithi137.stdout:1752: oid 11 updating version 530 to 531 2024-09-21T08:30:38.944 INFO:tasks.rados.rados.0.smithi137.stdout:1752: oid 11 version 531 is already newer than 529 2024-09-21T08:30:38.944 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 531 (ObjNum 623 snap 184 seq_num 623) dirty exists 2024-09-21T08:30:38.944 INFO:tasks.rados.rados.0.smithi137.stdout:1752: left oid 11 (ObjNum 623 snap 184 seq_num 623) 2024-09-21T08:30:38.944 INFO:tasks.rados.rados.0.smithi137.stdout:1754: expect (ObjNum 406 snap 99 seq_num 406) 2024-09-21T08:30:38.990 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:38 smithi165 ceph-mon[32893]: osdmap e444: 8 total, 6 up, 4 in 2024-09-21T08:30:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:38 smithi165 ceph-mon[32893]: 6.1f scrub starts 2024-09-21T08:30:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:38 smithi165 ceph-mon[32893]: 6.1f scrub ok 2024-09-21T08:30:39.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:39.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:39.016 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:39.017 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:38 smithi137 ceph-mon[41044]: osdmap e444: 8 total, 6 up, 4 in 2024-09-21T08:30:39.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:38 smithi137 ceph-mon[41044]: 6.1f scrub starts 2024-09-21T08:30:39.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:38 smithi137 ceph-mon[41044]: 6.1f scrub ok 2024-09-21T08:30:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:38 smithi137 ceph-mon[29634]: osdmap e444: 8 total, 6 up, 4 in 2024-09-21T08:30:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:38 smithi137 ceph-mon[29634]: 6.1f scrub starts 2024-09-21T08:30:39.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:38 smithi137 ceph-mon[29634]: 6.1f scrub ok 2024-09-21T08:30:39.033 INFO:tasks.rados.rados.0.smithi137.stdout:1755: expect (ObjNum 616 snap 181 seq_num 616) 2024-09-21T08:30:39.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:39.137 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:39.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:39.138 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:39.170 INFO:tasks.rados.rados.0.smithi137.stdout:1757: finishing rollback tid 1 to smithi13779221-6 2024-09-21T08:30:39.171 INFO:tasks.rados.rados.0.smithi137.stdout:1757: finishing rollback tid 2 to smithi13779221-6 2024-09-21T08:30:39.171 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 397 (ObjNum 503 snap 141 seq_num 503) dirty exists 2024-09-21T08:30:39.171 INFO:tasks.rados.rados.0.smithi137.stdout:1759: finishing rollback tid 0 to smithi13779221-44 2024-09-21T08:30:39.171 INFO:tasks.rados.rados.0.smithi137.stdout:1759: finishing rollback tid 1 to smithi13779221-44 2024-09-21T08:30:39.171 INFO:tasks.rados.rados.0.smithi137.stdout:1764: finishing write tid 1 to smithi13779221-33 2024-09-21T08:30:39.171 INFO:tasks.rados.rados.0.smithi137.stdout:1766: seq_num 626 ranges {521182=646820,1948669=523219} 2024-09-21T08:30:39.176 INFO:tasks.rados.rados.0.smithi137.stdout:1766: writing smithi13779221-30 from 521182 to 1168002 tid 1 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1766: writing smithi13779221-30 from 1948669 to 2471888 tid 2 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1752: done (11 left) 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1754: done (10 left) 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1755: done (9 left) 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1757: done (8 left) 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1758: done (7 left) 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1761: done (6 left) 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1767: snap_create 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1759: finishing rollback tid 2 to smithi13779221-44 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 727 (ObjNum 471 snap 128 seq_num 471) dirty exists 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1764: finishing write tid 2 to smithi13779221-33 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1764: finishing write tid 3 to smithi13779221-33 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1764: finishing write tid 4 to smithi13779221-33 2024-09-21T08:30:39.185 INFO:tasks.rados.rados.0.smithi137.stdout:1764: oid 33 updating version 0 to 290 2024-09-21T08:30:39.186 INFO:tasks.rados.rados.0.smithi137.stdout:1764: oid 33 updating version 290 to 292 2024-09-21T08:30:39.186 INFO:tasks.rados.rados.0.smithi137.stdout:1764: oid 33 version 292 is already newer than 291 2024-09-21T08:30:39.186 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 292 (ObjNum 624 snap 184 seq_num 624) dirty exists 2024-09-21T08:30:39.186 INFO:tasks.rados.rados.0.smithi137.stdout:1764: left oid 33 (ObjNum 624 snap 184 seq_num 624) 2024-09-21T08:30:39.186 INFO:tasks.rados.rados.0.smithi137.stdout:1760: expect (ObjNum 459 snap 124 seq_num 459) 2024-09-21T08:30:39.228 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T08:30:39.239 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:39.263 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:39.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:39.264 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:39.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:39.386 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:39.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:39.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:39.404 INFO:tasks.rados.rados.0.smithi137.stdout:1762: expect (ObjNum 544 snap 162 seq_num 544) 2024-09-21T08:30:39.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:39.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:39.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:39.510 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:39.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:30:39.574 INFO:tasks.rados.rados.0.smithi137.stdout:1763: finishing copy_from racing read to smithi13779221-37 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:1763: finishing copy_from to smithi13779221-37 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 510 (ObjNum 618 snap 182 seq_num 618) dirty exists 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:1766: finishing write tid 1 to smithi13779221-30 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:1766: finishing write tid 2 to smithi13779221-30 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:1766: finishing write tid 3 to smithi13779221-30 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:1766: finishing write tid 4 to smithi13779221-30 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:1766: oid 30 updating version 0 to 720 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:1766: oid 30 version 720 is already newer than 719 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:1766: oid 30 updating version 720 to 721 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 721 (ObjNum 626 snap 184 seq_num 626) dirty exists 2024-09-21T08:30:39.575 INFO:tasks.rados.rados.0.smithi137.stdout:1766: left oid 30 (ObjNum 626 snap 184 seq_num 626) 2024-09-21T08:30:39.611 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:39.636 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:39.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:39.637 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:39.680 INFO:tasks.rados.rados.0.smithi137.stdout:1759: done (6 left) 2024-09-21T08:30:39.680 INFO:tasks.rados.rados.0.smithi137.stdout:1760: done (5 left) 2024-09-21T08:30:39.680 INFO:tasks.rados.rados.0.smithi137.stdout:1762: done (4 left) 2024-09-21T08:30:39.680 INFO:tasks.rados.rados.0.smithi137.stdout:1763: done (3 left) 2024-09-21T08:30:39.680 INFO:tasks.rados.rados.0.smithi137.stdout:1764: done (2 left) 2024-09-21T08:30:39.680 INFO:tasks.rados.rados.0.smithi137.stdout:1766: done (1 left) 2024-09-21T08:30:39.680 INFO:tasks.rados.rados.0.smithi137.stdout:1767: done (0 left) 2024-09-21T08:30:39.680 INFO:tasks.rados.rados.0.smithi137.stdout:1768: snap_create 2024-09-21T08:30:39.738 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:39.763 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:39.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:39.764 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:39.865 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:39.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:39.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:39.891 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:39.992 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:39 smithi165 ceph-mon[32893]: pgmap v572: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 10 op/s; 365/1419 objects degraded (25.722%) 2024-09-21T08:30:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:39 smithi165 ceph-mon[32893]: osdmap e445: 8 total, 6 up, 4 in 2024-09-21T08:30:40.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:39 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2838766589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:40.014 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:40.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:40.015 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:40.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:39 smithi137 ceph-mon[29634]: pgmap v572: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 10 op/s; 365/1419 objects degraded (25.722%) 2024-09-21T08:30:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:39 smithi137 ceph-mon[29634]: osdmap e445: 8 total, 6 up, 4 in 2024-09-21T08:30:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:39 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2838766589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:40.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:39 smithi137 ceph-mon[41044]: pgmap v572: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 1.5 MiB/s rd, 1.8 MiB/s wr, 10 op/s; 365/1419 objects degraded (25.722%) 2024-09-21T08:30:40.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:39 smithi137 ceph-mon[41044]: osdmap e445: 8 total, 6 up, 4 in 2024-09-21T08:30:40.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:39 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2838766589' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:40.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:40.135 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:40.135 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:40.135 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:40.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:40.142 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:40.143 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:40.151 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:40.243 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:40.268 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:40.269 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:40.269 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:40.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:40.392 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:40.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:40.393 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:40.493 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:40.515 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:40.515 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:40.516 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:40.616 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:40.637 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:40.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:40.638 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:40.685 INFO:tasks.rados.rados.0.smithi137.stdout:1768: done (0 left) 2024-09-21T08:30:40.685 INFO:tasks.rados.rados.0.smithi137.stdout:1769: snap_create 2024-09-21T08:30:40.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:40.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:40.767 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:40.767 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:40.787 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:30:40.787 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:30:40.787 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:30:40.787 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:30:40.787 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:30:40.868 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:40.891 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:40.891 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:40.892 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:40.993 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:40 smithi165 ceph-mon[32893]: osdmap e446: 8 total, 6 up, 4 in 2024-09-21T08:30:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:40 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:30:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:40 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:30:41.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:41.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:41.017 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:40 smithi137 ceph-mon[29634]: osdmap e446: 8 total, 6 up, 4 in 2024-09-21T08:30:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:40 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:30:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:40 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:30:41.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:30:40 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:30:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:30:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:40 smithi137 ceph-mon[41044]: osdmap e446: 8 total, 6 up, 4 in 2024-09-21T08:30:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:40 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:30:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:40 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:30:41.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:41.140 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:41.140 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:41.141 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:41.241 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:41.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:41.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:41.265 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:41.366 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:41.387 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:41.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:41.388 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:41.400 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:30:41.402 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:30:41.489 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:41.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:41.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:41.514 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:41.583 INFO:tasks.rados.rados.0.smithi137.stdout:1769: done (0 left) 2024-09-21T08:30:41.583 INFO:tasks.rados.rados.0.smithi137.stdout:1770: write (excl) oid 4 current snap is 187 2024-09-21T08:30:41.583 INFO:tasks.rados.rados.0.smithi137.stdout:1770: seq_num 627 ranges {537989=722069,1890798=647191} 2024-09-21T08:30:41.591 INFO:tasks.rados.rados.0.smithi137.stdout:1770: writing smithi13779221-4 from 537989 to 1260058 tid 1 2024-09-21T08:30:41.598 INFO:tasks.rados.rados.0.smithi137.stdout:1770: writing smithi13779221-4 from 1890798 to 2537989 tid 2 2024-09-21T08:30:41.598 INFO:tasks.rados.rados.0.smithi137.stdout:1771: delete oid 25 current snap is 187 2024-09-21T08:30:41.601 INFO:tasks.rados.rados.0.smithi137.stdout:1770: finishing write tid 1 to smithi13779221-4 2024-09-21T08:30:41.602 INFO:tasks.rados.rados.0.smithi137.stdout:1770: finishing write tid 2 to smithi13779221-4 2024-09-21T08:30:41.602 INFO:tasks.rados.rados.0.smithi137.stdout:1770: finishing write tid 3 to smithi13779221-4 2024-09-21T08:30:41.602 INFO:tasks.rados.rados.0.smithi137.stdout:1770: finishing write tid 4 to smithi13779221-4 2024-09-21T08:30:41.602 INFO:tasks.rados.rados.0.smithi137.stdout:1770: oid 4 updating version 0 to 434 2024-09-21T08:30:41.602 INFO:tasks.rados.rados.0.smithi137.stdout:1770: oid 4 version 434 is already newer than 433 2024-09-21T08:30:41.602 INFO:tasks.rados.rados.0.smithi137.stdout:1770: oid 4 updating version 434 to 435 2024-09-21T08:30:41.602 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 435 (ObjNum 627 snap 187 seq_num 627) dirty exists 2024-09-21T08:30:41.602 INFO:tasks.rados.rados.0.smithi137.stdout:1770: left oid 4 (ObjNum 627 snap 187 seq_num 627) 2024-09-21T08:30:41.603 INFO:tasks.rados.rados.0.smithi137.stdout:1770: done (1 left) 2024-09-21T08:30:41.603 INFO:tasks.rados.rados.0.smithi137.stdout:1771: done (0 left) 2024-09-21T08:30:41.603 INFO:tasks.rados.rados.0.smithi137.stdout:1772: read oid 49 snap -1 2024-09-21T08:30:41.603 INFO:tasks.rados.rados.0.smithi137.stdout:1772: expect (ObjNum 560 snap 167 seq_num 560) 2024-09-21T08:30:41.603 INFO:tasks.rados.rados.0.smithi137.stdout:1773: snap_create 2024-09-21T08:30:41.608 INFO:tasks.rados.rados.0.smithi137.stdout:1772: expect (ObjNum 560 snap 167 seq_num 560) 2024-09-21T08:30:41.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:41.638 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:41.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:41.639 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:41.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T08:30:41.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:41.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:41.763 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:41.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:41.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:41.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:41.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:41.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:41.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:41 smithi165 ceph-mon[32893]: 7.4 scrub starts 2024-09-21T08:30:41.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:41 smithi165 ceph-mon[32893]: 7.4 scrub ok 2024-09-21T08:30:41.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:41 smithi165 ceph-mon[32893]: pgmap v575: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 365/1419 objects degraded (25.722%) 2024-09-21T08:30:41.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:41 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:30:41.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:41 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:30:41.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:41 smithi165 ceph-mon[32893]: osdmap e447: 8 total, 6 up, 4 in 2024-09-21T08:30:41.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:41 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:41.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:41 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 365/1419 objects degraded (25.722%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:41.936 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:41 smithi165 ceph-mon[32893]: osdmap e448: 8 total, 6 up, 4 in 2024-09-21T08:30:41.987 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:42.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:42.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:42.009 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[29634]: 7.4 scrub starts 2024-09-21T08:30:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[29634]: 7.4 scrub ok 2024-09-21T08:30:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[29634]: pgmap v575: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 365/1419 objects degraded (25.722%) 2024-09-21T08:30:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:30:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:30:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[29634]: osdmap e447: 8 total, 6 up, 4 in 2024-09-21T08:30:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 365/1419 objects degraded (25.722%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[29634]: osdmap e448: 8 total, 6 up, 4 in 2024-09-21T08:30:42.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[41044]: 7.4 scrub starts 2024-09-21T08:30:42.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[41044]: 7.4 scrub ok 2024-09-21T08:30:42.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[41044]: pgmap v575: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 365/1419 objects degraded (25.722%) 2024-09-21T08:30:42.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:30:42.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:30:42.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[41044]: osdmap e447: 8 total, 6 up, 4 in 2024-09-21T08:30:42.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:42.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 365/1419 objects degraded (25.722%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:42.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:41 smithi137 ceph-mon[41044]: osdmap e448: 8 total, 6 up, 4 in 2024-09-21T08:30:42.110 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:42.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:42.132 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:42.132 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:42.136 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:30:42.233 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:42.255 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:42.255 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:42.256 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:42.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:30:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:30:42.356 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:42.377 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:42.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:42.378 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:42.478 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:42.483 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":576,"stamp":"2024-09-21T08:30:40.679601+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":575138974,"num_objects":473,"num_object_clones":225,"num_object_copies":1419,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":365,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":473,"num_whiteouts":14,"num_read":3053,"num_read_kb":1903794,"num_write":1851,"num_write_kb":515522,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1121,"num_bytes_recovered":210716553,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8219,"ondisk_log_size":8219,"up":374,"acting":374,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":504,"num_osds":4,"num_per_pool_osds":4,"num_per_pool_omap_osds":4,"kb":374980608,"kb_used":1301392,"kb_used_data":1101688,"kb_used_omap":6,"kb_used_meta":199609,"kb_avail":373679216,"statfs":{"total":383980142592,"available":382647517184,"internally_reserved":0,"allocated":1128128512,"data_stored":1642172461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":6375,"internal_metadata":204400409},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.024689"},"pg_stats":[{"pgid":"7.23","version":"179'19","reported_seq":530,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.662905+0000","last_change":"2024-09-21T08:28:53.599776+0000","last_active":"2024-09-21T08:30:35.662905+0000","last_peered":"2024-09-21T08:30:35.662905+0000","last_clean":"2024-09-21T08:28:51.585797+0000","last_became_active":"2024-09-21T08:28:53.599776+0000","last_became_peered":"2024-09-21T08:28:53.599776+0000","last_unstale":"2024-09-21T08:30:35.662905+0000","last_undegraded":"2024-09-21T08:30:35.662905+0000","last_fullsized":"2024-09-21T08:28:53.583177+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:28:10.421606+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.421606+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":54,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.072212988000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"7.22","version":"257'264","reported_seq":1915,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.657414+0000","last_change":"2024-09-21T08:28:53.599720+0000","last_active":"2024-09-21T08:30:35.657414+0000","last_peered":"2024-09-21T08:30:35.657414+0000","last_clean":"2024-09-21T08:28:51.585268+0000","last_became_active":"2024-09-21T08:28:53.599720+0000","last_became_peered":"2024-09-21T08:28:53.599720+0000","last_unstale":"2024-09-21T08:30:35.657414+0000","last_undegraded":"2024-09-21T08:30:35.657414+0000","last_fullsized":"2024-09-21T08:28:53.585695+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":6,"last_scrub":"257'264","last_scrub_stamp":"2024-09-21T08:27:48.727600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:48.727600+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.075805560999999994,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"2"}]},{"pgid":"7.21","version":"179'43","reported_seq":1372,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.671430+0000","last_change":"2024-09-21T08:28:53.594285+0000","last_active":"2024-09-21T08:30:35.671430+0000","last_peered":"2024-09-21T08:30:35.671430+0000","last_clean":"2024-09-21T08:28:51.554218+0000","last_became_active":"2024-09-21T08:28:53.594285+0000","last_became_peered":"2024-09-21T08:28:53.594285+0000","last_unstale":"2024-09-21T08:30:35.671430+0000","last_undegraded":"2024-09-21T08:30:35.671430+0000","last_fullsized":"2024-09-21T08:28:53.573996+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:36.934372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.934372+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":52,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.025027727,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"7.20","version":"432'172","reported_seq":1772,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.674214+0000","last_change":"2024-09-21T08:28:53.605282+0000","last_active":"2024-09-21T08:30:35.674214+0000","last_peered":"2024-09-21T08:30:35.674214+0000","last_clean":"2024-09-21T08:28:51.585842+0000","last_became_active":"2024-09-21T08:28:53.605282+0000","last_became_peered":"2024-09-21T08:28:53.605282+0000","last_unstale":"2024-09-21T08:30:35.674214+0000","last_undegraded":"2024-09-21T08:28:53.580783+0000","last_fullsized":"2024-09-21T08:28:53.580670+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"251'134","last_scrub_stamp":"2024-09-21T08:27:49.318143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.318143+0000","objects_scrubbed":1,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.076740219999999998,"stat_sum":{"num_bytes":13440760,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":27,"num_read_kb":17523,"num_write":29,"num_write_kb":6079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1848493,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"2"}]},{"pgid":"7.1b","version":"440'711","reported_seq":2644,"reported_epoch":442,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:35.673368+0000","last_change":"2024-09-21T08:30:32.739920+0000","last_active":"2024-09-21T08:30:35.673368+0000","last_peered":"2024-09-21T08:30:35.673368+0000","last_clean":"2024-09-21T08:30:35.673368+0000","last_became_active":"2024-09-21T08:28:37.568752+0000","last_became_peered":"2024-09-21T08:28:37.568752+0000","last_unstale":"2024-09-21T08:30:35.673368+0000","last_undegraded":"2024-09-21T08:30:35.673368+0000","last_fullsized":"2024-09-21T08:30:35.673368+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'499","last_scrub_stamp":"2024-09-21T08:28:09.328624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.328624+0000","objects_scrubbed":5,"log_size":711,"log_dups_size":0,"ondisk_log_size":711,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":15,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.024821855,"stat_sum":{"num_bytes":43522835,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":1,"num_read":120,"num_read_kb":126210,"num_write":69,"num_write_kb":30124,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":18585585,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"95","length":"4"},{"start":"9d","length":"1"},{"start":"9f","length":"3"}]},{"pgid":"3.1f","version":"0'0","reported_seq":267,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.674834+0000","last_change":"2024-09-21T08:28:53.591551+0000","last_active":"2024-09-21T08:30:35.674834+0000","last_peered":"2024-09-21T08:30:35.674834+0000","last_clean":"2024-09-21T08:28:51.556461+0000","last_became_active":"2024-09-21T08:28:53.591551+0000","last_became_peered":"2024-09-21T08:28:53.591551+0000","last_unstale":"2024-09-21T08:30:35.674834+0000","last_undegraded":"2024-09-21T08:30:35.674834+0000","last_fullsized":"2024-09-21T08:28:53.580526+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":340,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.662157+0000","last_change":"2024-09-21T08:28:29.692402+0000","last_active":"2024-09-21T08:30:35.662157+0000","last_peered":"2024-09-21T08:30:35.662157+0000","last_clean":"2024-09-21T08:30:35.662157+0000","last_became_active":"2024-09-21T08:28:29.635580+0000","last_became_peered":"2024-09-21T08:28:29.635580+0000","last_unstale":"2024-09-21T08:30:35.662157+0000","last_undegraded":"2024-09-21T08:30:35.662157+0000","last_fullsized":"2024-09-21T08:30:35.662157+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:56.303669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:56.303669+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":325,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.656473+0000","last_change":"2024-09-21T08:28:53.594621+0000","last_active":"2024-09-21T08:30:35.656473+0000","last_peered":"2024-09-21T08:30:35.656473+0000","last_clean":"2024-09-21T08:28:51.584740+0000","last_became_active":"2024-09-21T08:28:53.594621+0000","last_became_peered":"2024-09-21T08:28:53.594621+0000","last_unstale":"2024-09-21T08:30:35.656473+0000","last_undegraded":"2024-09-21T08:30:35.656473+0000","last_fullsized":"2024-09-21T08:28:53.576425+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.424482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.424482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.079005+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":750,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.674822+0000","last_change":"2024-09-21T08:28:53.601367+0000","last_active":"2024-09-21T08:30:35.674822+0000","last_peered":"2024-09-21T08:30:35.674822+0000","last_clean":"2024-09-21T08:28:51.557847+0000","last_became_active":"2024-09-21T08:28:53.601367+0000","last_became_peered":"2024-09-21T08:28:53.601367+0000","last_unstale":"2024-09-21T08:30:35.674822+0000","last_undegraded":"2024-09-21T08:30:35.674822+0000","last_fullsized":"2024-09-21T08:28:53.577149+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.464969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.464969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704559+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1a","version":"248'68","reported_seq":804,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.674617+0000","last_change":"2024-09-21T08:28:53.593471+0000","last_active":"2024-09-21T08:30:35.674617+0000","last_peered":"2024-09-21T08:30:35.674617+0000","last_clean":"2024-09-21T08:28:52.589751+0000","last_became_active":"2024-09-21T08:28:53.593471+0000","last_became_peered":"2024-09-21T08:28:53.593471+0000","last_unstale":"2024-09-21T08:30:35.674617+0000","last_undegraded":"2024-09-21T08:30:35.674617+0000","last_fullsized":"2024-09-21T08:28:53.574504+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'68","last_scrub_stamp":"2024-09-21T08:27:35.919031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:35.919031+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":52,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0328630059999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"6.1b","version":"0'0","reported_seq":681,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.662217+0000","last_change":"2024-09-21T08:28:53.606497+0000","last_active":"2024-09-21T08:30:35.662217+0000","last_peered":"2024-09-21T08:30:35.662217+0000","last_clean":"2024-09-21T08:28:51.585547+0000","last_became_active":"2024-09-21T08:28:53.606497+0000","last_became_peered":"2024-09-21T08:28:53.606497+0000","last_unstale":"2024-09-21T08:30:35.662217+0000","last_undegraded":"2024-09-21T08:30:35.662217+0000","last_fullsized":"2024-09-21T08:28:53.581183+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.804294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.804294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078670+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":608,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.660413+0000","last_change":"2024-09-21T08:28:53.602981+0000","last_active":"2024-09-21T08:30:35.660413+0000","last_peered":"2024-09-21T08:30:35.660413+0000","last_clean":"2024-09-21T08:28:51.587203+0000","last_became_active":"2024-09-21T08:28:53.602981+0000","last_became_peered":"2024-09-21T08:28:53.602981+0000","last_unstale":"2024-09-21T08:30:35.660413+0000","last_undegraded":"2024-09-21T08:30:35.660413+0000","last_fullsized":"2024-09-21T08:28:53.581112+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.146798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.146798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078895+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":605,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.671670+0000","last_change":"2024-09-21T08:29:00.355702+0000","last_active":"2024-09-21T08:30:35.671670+0000","last_peered":"2024-09-21T08:30:35.671670+0000","last_clean":"2024-09-21T08:30:35.671670+0000","last_became_active":"2024-09-21T08:28:29.620169+0000","last_became_peered":"2024-09-21T08:28:29.620169+0000","last_unstale":"2024-09-21T08:30:35.671670+0000","last_undegraded":"2024-09-21T08:30:35.671670+0000","last_fullsized":"2024-09-21T08:30:35.671670+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:00.355591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:00.355591+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.670185+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":317,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.662240+0000","last_change":"2024-09-21T08:28:53.601812+0000","last_active":"2024-09-21T08:30:35.662240+0000","last_peered":"2024-09-21T08:30:35.662240+0000","last_clean":"2024-09-21T08:28:51.585034+0000","last_became_active":"2024-09-21T08:28:53.601812+0000","last_became_peered":"2024-09-21T08:28:53.601812+0000","last_unstale":"2024-09-21T08:30:35.662240+0000","last_undegraded":"2024-09-21T08:30:35.662240+0000","last_fullsized":"2024-09-21T08:28:53.578966+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.340817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.340817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078704+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.19","version":"417'205","reported_seq":1487,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.671906+0000","last_change":"2024-09-21T08:28:53.587606+0000","last_active":"2024-09-21T08:30:35.671906+0000","last_peered":"2024-09-21T08:30:35.671906+0000","last_clean":"2024-09-21T08:28:52.589310+0000","last_became_active":"2024-09-21T08:28:53.587606+0000","last_became_peered":"2024-09-21T08:28:53.587606+0000","last_unstale":"2024-09-21T08:30:35.671906+0000","last_undegraded":"2024-09-21T08:28:53.574680+0000","last_fullsized":"2024-09-21T08:28:53.574555+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"266'188","last_scrub_stamp":"2024-09-21T08:27:49.719124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.719124+0000","objects_scrubbed":1,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":52,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.034770314,"stat_sum":{"num_bytes":11158040,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":25354,"num_write":16,"num_write_kb":11591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"3.1d","version":"0'0","reported_seq":813,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.675173+0000","last_change":"2024-09-21T08:28:53.601545+0000","last_active":"2024-09-21T08:30:35.675173+0000","last_peered":"2024-09-21T08:30:35.675173+0000","last_clean":"2024-09-21T08:28:51.555695+0000","last_became_active":"2024-09-21T08:28:53.601545+0000","last_became_peered":"2024-09-21T08:28:53.601545+0000","last_unstale":"2024-09-21T08:30:35.675173+0000","last_undegraded":"2024-09-21T08:30:35.675173+0000","last_fullsized":"2024-09-21T08:28:53.577677+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.780545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.780545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704067+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":573,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.674931+0000","last_change":"2024-09-21T08:30:26.781618+0000","last_active":"2024-09-21T08:30:35.674931+0000","last_peered":"2024-09-21T08:30:35.674931+0000","last_clean":"2024-09-21T08:30:35.674931+0000","last_became_active":"2024-09-21T08:28:37.568659+0000","last_became_peered":"2024-09-21T08:28:37.568659+0000","last_unstale":"2024-09-21T08:30:35.674931+0000","last_undegraded":"2024-09-21T08:30:35.674931+0000","last_fullsized":"2024-09-21T08:30:35.674931+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:26.781541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:26.781541+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:42.771929+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":772,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.671908+0000","last_change":"2024-09-21T08:28:53.593695+0000","last_active":"2024-09-21T08:30:35.671908+0000","last_peered":"2024-09-21T08:30:35.671908+0000","last_clean":"2024-09-21T08:28:51.556408+0000","last_became_active":"2024-09-21T08:28:53.593695+0000","last_became_peered":"2024-09-21T08:28:53.593695+0000","last_unstale":"2024-09-21T08:30:35.671908+0000","last_undegraded":"2024-09-21T08:30:35.671908+0000","last_fullsized":"2024-09-21T08:28:53.574067+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703815+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":727,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.661183+0000","last_change":"2024-09-21T08:28:53.586342+0000","last_active":"2024-09-21T08:30:35.661183+0000","last_peered":"2024-09-21T08:30:35.661183+0000","last_clean":"2024-09-21T08:28:51.584564+0000","last_became_active":"2024-09-21T08:28:53.586342+0000","last_became_peered":"2024-09-21T08:28:53.586342+0000","last_unstale":"2024-09-21T08:30:35.661183+0000","last_undegraded":"2024-09-21T08:30:35.661183+0000","last_fullsized":"2024-09-21T08:28:53.575506+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.291226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.291226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078886+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"439'261","reported_seq":1915,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.671481+0000","last_change":"2024-09-21T08:28:53.587953+0000","last_active":"2024-09-21T08:30:35.671481+0000","last_peered":"2024-09-21T08:30:35.671481+0000","last_clean":"2024-09-21T08:28:51.550342+0000","last_became_active":"2024-09-21T08:28:53.587953+0000","last_became_peered":"2024-09-21T08:28:53.587953+0000","last_unstale":"2024-09-21T08:30:35.671481+0000","last_undegraded":"2024-09-21T08:28:53.574363+0000","last_fullsized":"2024-09-21T08:28:53.574234+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"319'218","last_scrub_stamp":"2024-09-21T08:28:32.867460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:32.867460+0000","objects_scrubbed":4,"log_size":261,"log_dups_size":0,"ondisk_log_size":261,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":49,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027063920000000002,"stat_sum":{"num_bytes":25680244,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":57,"num_read_kb":68204,"num_write":59,"num_write_kb":29903,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7346792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":10}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"3.1c","version":"0'0","reported_seq":798,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.672740+0000","last_change":"2024-09-21T08:28:53.600345+0000","last_active":"2024-09-21T08:30:35.672740+0000","last_peered":"2024-09-21T08:30:35.672740+0000","last_clean":"2024-09-21T08:28:51.556880+0000","last_became_active":"2024-09-21T08:28:53.600345+0000","last_became_peered":"2024-09-21T08:28:53.600345+0000","last_unstale":"2024-09-21T08:30:35.672740+0000","last_undegraded":"2024-09-21T08:30:35.672740+0000","last_fullsized":"2024-09-21T08:28:53.576313+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":323,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.664604+0000","last_change":"2024-09-21T08:28:53.605410+0000","last_active":"2024-09-21T08:30:35.664604+0000","last_peered":"2024-09-21T08:30:35.664604+0000","last_clean":"2024-09-21T08:28:51.585702+0000","last_became_active":"2024-09-21T08:28:53.605410+0000","last_became_peered":"2024-09-21T08:28:53.605410+0000","last_unstale":"2024-09-21T08:30:35.664604+0000","last_undegraded":"2024-09-21T08:28:53.583377+0000","last_fullsized":"2024-09-21T08:28:53.583262+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:27:26.356987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:26.356987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":749,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.657276+0000","last_change":"2024-09-21T08:28:53.600104+0000","last_active":"2024-09-21T08:30:35.657276+0000","last_peered":"2024-09-21T08:30:35.657276+0000","last_clean":"2024-09-21T08:28:27.553155+0000","last_became_active":"2024-09-21T08:28:53.600104+0000","last_became_peered":"2024-09-21T08:28:53.600104+0000","last_unstale":"2024-09-21T08:30:35.657276+0000","last_undegraded":"2024-09-21T08:30:35.657276+0000","last_fullsized":"2024-09-21T08:28:53.585955+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.850373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078998+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":818,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.672090+0000","last_change":"2024-09-21T08:30:04.034656+0000","last_active":"2024-09-21T08:30:35.672090+0000","last_peered":"2024-09-21T08:30:35.672090+0000","last_clean":"2024-09-21T08:30:35.672090+0000","last_became_active":"2024-09-21T08:28:04.705450+0000","last_became_peered":"2024-09-21T08:28:04.705450+0000","last_unstale":"2024-09-21T08:30:35.672090+0000","last_undegraded":"2024-09-21T08:30:35.672090+0000","last_fullsized":"2024-09-21T08:30:35.672090+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:30:04.034575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:04.034575+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:12.159925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1f","version":"247'32","reported_seq":1608,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.672781+0000","last_change":"2024-09-21T08:28:53.601095+0000","last_active":"2024-09-21T08:30:35.672781+0000","last_peered":"2024-09-21T08:30:35.672781+0000","last_clean":"2024-09-21T08:28:53.015413+0000","last_became_active":"2024-09-21T08:28:53.601095+0000","last_became_peered":"2024-09-21T08:28:53.601095+0000","last_unstale":"2024-09-21T08:30:35.672781+0000","last_undegraded":"2024-09-21T08:30:35.672781+0000","last_fullsized":"2024-09-21T08:28:53.577251+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:27:51.078531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.078531+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0098481060000000006,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"6.1e","version":"0'0","reported_seq":749,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.675056+0000","last_change":"2024-09-21T08:28:53.591644+0000","last_active":"2024-09-21T08:30:35.675056+0000","last_peered":"2024-09-21T08:30:35.675056+0000","last_clean":"2024-09-21T08:28:51.556030+0000","last_became_active":"2024-09-21T08:28:53.591644+0000","last_became_peered":"2024-09-21T08:28:53.591644+0000","last_unstale":"2024-09-21T08:30:35.675056+0000","last_undegraded":"2024-09-21T08:30:35.675056+0000","last_fullsized":"2024-09-21T08:28:53.578796+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.855448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.855448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703868+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":637,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.662075+0000","last_change":"2024-09-21T08:28:53.604972+0000","last_active":"2024-09-21T08:30:35.662075+0000","last_peered":"2024-09-21T08:30:35.662075+0000","last_clean":"2024-09-21T08:28:51.585297+0000","last_became_active":"2024-09-21T08:28:53.604972+0000","last_became_peered":"2024-09-21T08:28:53.604972+0000","last_unstale":"2024-09-21T08:30:35.662075+0000","last_undegraded":"2024-09-21T08:30:35.662075+0000","last_fullsized":"2024-09-21T08:28:53.579187+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078364+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":839,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.657599+0000","last_change":"2024-09-21T08:29:37.352960+0000","last_active":"2024-09-21T08:30:35.657599+0000","last_peered":"2024-09-21T08:30:35.657599+0000","last_clean":"2024-09-21T08:30:35.657599+0000","last_became_active":"2024-09-21T08:28:37.566225+0000","last_became_peered":"2024-09-21T08:28:37.566225+0000","last_unstale":"2024-09-21T08:30:35.657599+0000","last_undegraded":"2024-09-21T08:30:35.657599+0000","last_fullsized":"2024-09-21T08:30:35.657599+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:37.352879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:37.352879+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:41.677208+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":323,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.659540+0000","last_change":"2024-09-21T08:28:53.598624+0000","last_active":"2024-09-21T08:30:35.659540+0000","last_peered":"2024-09-21T08:30:35.659540+0000","last_clean":"2024-09-21T08:28:51.587418+0000","last_became_active":"2024-09-21T08:28:53.598624+0000","last_became_peered":"2024-09-21T08:28:53.598624+0000","last_unstale":"2024-09-21T08:30:35.659540+0000","last_undegraded":"2024-09-21T08:30:35.659540+0000","last_fullsized":"2024-09-21T08:28:53.579380+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:30.310628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.310628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"440'307","reported_seq":1964,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.659871+0000","last_change":"2024-09-21T08:28:53.595615+0000","last_active":"2024-09-21T08:30:35.659871+0000","last_peered":"2024-09-21T08:30:35.659871+0000","last_clean":"2024-09-21T08:28:52.666418+0000","last_became_active":"2024-09-21T08:28:53.595615+0000","last_became_peered":"2024-09-21T08:28:53.595615+0000","last_unstale":"2024-09-21T08:30:35.659871+0000","last_undegraded":"2024-09-21T08:28:53.580612+0000","last_fullsized":"2024-09-21T08:28:53.580483+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'236","last_scrub_stamp":"2024-09-21T08:28:09.464279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.464279+0000","objects_scrubbed":2,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":55,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.0792350470000001,"stat_sum":{"num_bytes":41783176,"num_objects":18,"num_object_clones":15,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":71,"num_read_kb":76028,"num_write":62,"num_write_kb":22455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":6996269,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":18}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"6.1f","version":"0'0","reported_seq":666,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.675953+0000","last_change":"2024-09-21T08:29:33.233733+0000","last_active":"2024-09-21T08:30:35.675953+0000","last_peered":"2024-09-21T08:30:35.675953+0000","last_clean":"2024-09-21T08:30:35.675953+0000","last_became_active":"2024-09-21T08:28:04.725392+0000","last_became_peered":"2024-09-21T08:28:04.725392+0000","last_unstale":"2024-09-21T08:30:35.675953+0000","last_undegraded":"2024-09-21T08:30:35.675953+0000","last_fullsized":"2024-09-21T08:30:35.675953+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:33.233665+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:33.233665+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.550890+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":323,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.658655+0000","last_change":"2024-09-21T08:28:53.605497+0000","last_active":"2024-09-21T08:30:35.658655+0000","last_peered":"2024-09-21T08:30:35.658655+0000","last_clean":"2024-09-21T08:28:51.587153+0000","last_became_active":"2024-09-21T08:28:53.605497+0000","last_became_peered":"2024-09-21T08:28:53.605497+0000","last_unstale":"2024-09-21T08:30:35.658655+0000","last_undegraded":"2024-09-21T08:30:35.658655+0000","last_fullsized":"2024-09-21T08:28:53.586766+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:33.237691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.237691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":334,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.658245+0000","last_change":"2024-09-21T08:29:15.349199+0000","last_active":"2024-09-21T08:30:35.658245+0000","last_peered":"2024-09-21T08:30:35.658245+0000","last_clean":"2024-09-21T08:30:35.658245+0000","last_became_active":"2024-09-21T08:28:04.724988+0000","last_became_peered":"2024-09-21T08:28:04.724988+0000","last_unstale":"2024-09-21T08:30:35.658245+0000","last_undegraded":"2024-09-21T08:30:35.658245+0000","last_fullsized":"2024-09-21T08:30:35.658245+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:29:15.349124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:15.349124+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.079039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":693,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.668908+0000","last_change":"2024-09-21T08:28:53.591068+0000","last_active":"2024-09-21T08:30:35.668908+0000","last_peered":"2024-09-21T08:30:35.668908+0000","last_clean":"2024-09-21T08:28:51.554175+0000","last_became_active":"2024-09-21T08:28:53.591068+0000","last_became_peered":"2024-09-21T08:28:53.591068+0000","last_unstale":"2024-09-21T08:30:35.668908+0000","last_undegraded":"2024-09-21T08:30:35.668908+0000","last_fullsized":"2024-09-21T08:28:53.580183+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:09.116412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.116412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:36.882808+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1d","version":"248'84","reported_seq":546,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.658694+0000","last_change":"2024-09-21T08:28:53.595944+0000","last_active":"2024-09-21T08:30:35.658694+0000","last_peered":"2024-09-21T08:30:35.658694+0000","last_clean":"2024-09-21T08:28:51.587189+0000","last_became_active":"2024-09-21T08:28:53.595944+0000","last_became_peered":"2024-09-21T08:28:53.595944+0000","last_unstale":"2024-09-21T08:30:35.658694+0000","last_undegraded":"2024-09-21T08:30:35.658694+0000","last_fullsized":"2024-09-21T08:28:53.578716+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'84","last_scrub_stamp":"2024-09-21T08:28:12.352559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.352559+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":54,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.072813132000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"6.1c","version":"73'1","reported_seq":310,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.673191+0000","last_change":"2024-09-21T08:28:53.589802+0000","last_active":"2024-09-21T08:30:35.673191+0000","last_peered":"2024-09-21T08:30:35.673191+0000","last_clean":"2024-09-21T08:28:51.557351+0000","last_became_active":"2024-09-21T08:28:53.589802+0000","last_became_peered":"2024-09-21T08:28:53.589802+0000","last_unstale":"2024-09-21T08:30:35.673191+0000","last_undegraded":"2024-09-21T08:28:53.577740+0000","last_fullsized":"2024-09-21T08:28:53.577630+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:27:14.826047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.826047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"198'2","reported_seq":321,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.656651+0000","last_change":"2024-09-21T08:28:53.601978+0000","last_active":"2024-09-21T08:30:35.656651+0000","last_peered":"2024-09-21T08:30:35.656651+0000","last_clean":"2024-09-21T08:28:51.584776+0000","last_became_active":"2024-09-21T08:28:53.601978+0000","last_became_peered":"2024-09-21T08:28:53.601978+0000","last_unstale":"2024-09-21T08:30:35.656651+0000","last_undegraded":"2024-09-21T08:28:53.576796+0000","last_fullsized":"2024-09-21T08:28:53.576697+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'2","last_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_deep_scrub":"198'2","last_deep_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_clean_scrub_stamp":"2024-09-21T08:26:32.819313+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078613+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":822,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.660585+0000","last_change":"2024-09-21T08:28:53.597832+0000","last_active":"2024-09-21T08:30:35.660585+0000","last_peered":"2024-09-21T08:30:35.660585+0000","last_clean":"2024-09-21T08:28:51.587327+0000","last_became_active":"2024-09-21T08:28:53.597832+0000","last_became_peered":"2024-09-21T08:28:53.597832+0000","last_unstale":"2024-09-21T08:30:35.660585+0000","last_undegraded":"2024-09-21T08:28:53.582409+0000","last_fullsized":"2024-09-21T08:28:53.582308+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078340+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"340'17","reported_seq":450,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.671349+0000","last_change":"2024-09-21T08:28:53.600523+0000","last_active":"2024-09-21T08:30:35.671349+0000","last_peered":"2024-09-21T08:30:35.671349+0000","last_clean":"2024-09-21T08:28:51.556529+0000","last_became_active":"2024-09-21T08:28:53.600523+0000","last_became_peered":"2024-09-21T08:28:53.600523+0000","last_unstale":"2024-09-21T08:30:35.671349+0000","last_undegraded":"2024-09-21T08:28:53.585227+0000","last_fullsized":"2024-09-21T08:28:53.585102+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'15","last_scrub_stamp":"2024-09-21T08:28:14.153130+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.153130+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:36.882794+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.c","version":"439'250","reported_seq":2073,"reported_epoch":442,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:30:35.674491+0000","last_change":"2024-09-21T08:30:34.770400+0000","last_active":"2024-09-21T08:30:35.674491+0000","last_peered":"2024-09-21T08:30:35.674491+0000","last_clean":"2024-09-21T08:30:35.674491+0000","last_became_active":"2024-09-21T08:28:32.557643+0000","last_became_peered":"2024-09-21T08:28:32.557643+0000","last_unstale":"2024-09-21T08:30:35.674491+0000","last_undegraded":"2024-09-21T08:30:35.674491+0000","last_fullsized":"2024-09-21T08:30:35.674491+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"260'160","last_scrub_stamp":"2024-09-21T08:27:34.925496+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:34.925496+0000","objects_scrubbed":2,"log_size":250,"log_dups_size":0,"ondisk_log_size":250,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":7,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.023408228,"stat_sum":{"num_bytes":8689351,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":50,"num_read_kb":58168,"num_write":23,"num_write_kb":9777,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":6235496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"1"},{"start":"a8","length":"1"}]},{"pgid":"4.f","version":"73'15","reported_seq":333,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.658772+0000","last_change":"2024-09-21T08:28:53.595462+0000","last_active":"2024-09-21T08:30:35.658772+0000","last_peered":"2024-09-21T08:30:35.658772+0000","last_clean":"2024-09-21T08:28:51.587748+0000","last_became_active":"2024-09-21T08:28:53.595462+0000","last_became_peered":"2024-09-21T08:28:53.595462+0000","last_unstale":"2024-09-21T08:30:35.658772+0000","last_undegraded":"2024-09-21T08:28:53.579415+0000","last_fullsized":"2024-09-21T08:28:53.579321+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:26:46.777115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.777115+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078591+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":317,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.662757+0000","last_change":"2024-09-21T08:28:53.595248+0000","last_active":"2024-09-21T08:30:35.662757+0000","last_peered":"2024-09-21T08:30:35.662757+0000","last_clean":"2024-09-21T08:28:51.585248+0000","last_became_active":"2024-09-21T08:28:53.595248+0000","last_became_peered":"2024-09-21T08:28:53.595248+0000","last_unstale":"2024-09-21T08:30:35.662757+0000","last_undegraded":"2024-09-21T08:30:35.662757+0000","last_fullsized":"2024-09-21T08:28:53.579384+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:45.434055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:45.434055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078476+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"340'11","reported_seq":858,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.674834+0000","last_change":"2024-09-21T08:28:53.591354+0000","last_active":"2024-09-21T08:30:35.674834+0000","last_peered":"2024-09-21T08:30:35.674834+0000","last_clean":"2024-09-21T08:28:51.555908+0000","last_became_active":"2024-09-21T08:28:53.591354+0000","last_became_peered":"2024-09-21T08:28:53.591354+0000","last_unstale":"2024-09-21T08:30:35.674834+0000","last_undegraded":"2024-09-21T08:28:53.578763+0000","last_fullsized":"2024-09-21T08:28:53.578651+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:26:44.741275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.741275+0000","objects_scrubbed":1,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704054+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":794,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.676046+0000","last_change":"2024-09-21T08:30:21.800509+0000","last_active":"2024-09-21T08:30:35.676046+0000","last_peered":"2024-09-21T08:30:35.676046+0000","last_clean":"2024-09-21T08:30:35.676046+0000","last_became_active":"2024-09-21T08:28:29.619967+0000","last_became_peered":"2024-09-21T08:28:29.619967+0000","last_unstale":"2024-09-21T08:30:35.676046+0000","last_undegraded":"2024-09-21T08:30:35.676046+0000","last_fullsized":"2024-09-21T08:30:35.676046+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:21.800427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:21.800427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:50.788583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.3","version":"179'19","reported_seq":1487,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.668675+0000","last_change":"2024-09-21T08:28:53.592078+0000","last_active":"2024-09-21T08:30:35.668675+0000","last_peered":"2024-09-21T08:30:35.668675+0000","last_clean":"2024-09-21T08:28:51.554873+0000","last_became_active":"2024-09-21T08:28:53.592078+0000","last_became_peered":"2024-09-21T08:28:53.592078+0000","last_unstale":"2024-09-21T08:30:35.668675+0000","last_undegraded":"2024-09-21T08:30:35.668675+0000","last_fullsized":"2024-09-21T08:28:53.580654+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:27:36.268364+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.268364+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":49,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00057177200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"4.0","version":"73'18","reported_seq":751,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.661778+0000","last_change":"2024-09-21T08:28:53.593785+0000","last_active":"2024-09-21T08:30:35.661778+0000","last_peered":"2024-09-21T08:30:35.661778+0000","last_clean":"2024-09-21T08:28:51.584787+0000","last_became_active":"2024-09-21T08:28:53.593785+0000","last_became_peered":"2024-09-21T08:28:53.593785+0000","last_unstale":"2024-09-21T08:30:35.661778+0000","last_undegraded":"2024-09-21T08:28:53.574919+0000","last_fullsized":"2024-09-21T08:28:53.574229+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-09-21T08:26:54.003173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:54.003173+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078647+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":525,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.673476+0000","last_change":"2024-09-21T08:28:53.602596+0000","last_active":"2024-09-21T08:30:35.673476+0000","last_peered":"2024-09-21T08:30:35.673476+0000","last_clean":"2024-09-21T08:28:51.555756+0000","last_became_active":"2024-09-21T08:28:53.602596+0000","last_became_peered":"2024-09-21T08:28:53.602596+0000","last_unstale":"2024-09-21T08:30:35.673476+0000","last_undegraded":"2024-09-21T08:30:35.673476+0000","last_fullsized":"2024-09-21T08:28:53.579349+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":625,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.671445+0000","last_change":"2024-09-21T08:28:53.587446+0000","last_active":"2024-09-21T08:30:35.671445+0000","last_peered":"2024-09-21T08:30:35.671445+0000","last_clean":"2024-09-21T08:28:27.553250+0000","last_became_active":"2024-09-21T08:28:53.587446+0000","last_became_peered":"2024-09-21T08:28:53.587446+0000","last_unstale":"2024-09-21T08:30:35.671445+0000","last_undegraded":"2024-09-21T08:30:35.671445+0000","last_fullsized":"2024-09-21T08:28:53.574310+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:32.813973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:32.813973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":473,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.671991+0000","last_change":"2024-09-21T08:28:53.587280+0000","last_active":"2024-09-21T08:30:35.671991+0000","last_peered":"2024-09-21T08:30:35.671991+0000","last_clean":"2024-09-21T08:26:47.628755+0000","last_became_active":"2024-09-21T08:28:53.587280+0000","last_became_peered":"2024-09-21T08:28:53.587280+0000","last_unstale":"2024-09-21T08:30:35.671991+0000","last_undegraded":"2024-09-21T08:30:35.671991+0000","last_fullsized":"2024-09-21T08:28:53.574372+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:16.341564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.341564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704243+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"411'350","reported_seq":1995,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.658248+0000","last_change":"2024-09-21T08:28:53.605477+0000","last_active":"2024-09-21T08:30:35.658248+0000","last_peered":"2024-09-21T08:30:35.658248+0000","last_clean":"2024-09-21T08:28:52.666434+0000","last_became_active":"2024-09-21T08:28:53.605477+0000","last_became_peered":"2024-09-21T08:28:53.605477+0000","last_unstale":"2024-09-21T08:30:35.658248+0000","last_undegraded":"2024-09-21T08:28:53.586515+0000","last_fullsized":"2024-09-21T08:28:53.586400+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'273","last_scrub_stamp":"2024-09-21T08:27:50.318620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.318620+0000","objects_scrubbed":3,"log_size":350,"log_dups_size":0,"ondisk_log_size":350,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":55,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.073725586999999995,"stat_sum":{"num_bytes":20243096,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":65,"num_read_kb":73718,"num_write":41,"num_write_kb":14479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":3845252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.1","version":"239'21","reported_seq":844,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.671776+0000","last_change":"2024-09-21T08:28:53.588019+0000","last_active":"2024-09-21T08:30:35.671776+0000","last_peered":"2024-09-21T08:30:35.671776+0000","last_clean":"2024-09-21T08:28:51.554459+0000","last_became_active":"2024-09-21T08:28:53.588019+0000","last_became_peered":"2024-09-21T08:28:53.588019+0000","last_unstale":"2024-09-21T08:30:35.671776+0000","last_undegraded":"2024-09-21T08:28:53.575510+0000","last_fullsized":"2024-09-21T08:28:53.575377+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21","last_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_deep_scrub":"239'21","last_deep_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.969642+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704186+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":759,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.661096+0000","last_change":"2024-09-21T08:28:53.598446+0000","last_active":"2024-09-21T08:30:35.661096+0000","last_peered":"2024-09-21T08:30:35.661096+0000","last_clean":"2024-09-21T08:28:51.587388+0000","last_became_active":"2024-09-21T08:28:53.598446+0000","last_became_peered":"2024-09-21T08:28:53.598446+0000","last_unstale":"2024-09-21T08:30:35.661096+0000","last_undegraded":"2024-09-21T08:28:53.582469+0000","last_fullsized":"2024-09-21T08:28:53.582373+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078388+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":138,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"317'16","reported_seq":359,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.662001+0000","last_change":"2024-09-21T08:28:29.687150+0000","last_active":"2024-09-21T08:30:35.662001+0000","last_peered":"2024-09-21T08:30:35.662001+0000","last_clean":"2024-09-21T08:30:35.662001+0000","last_became_active":"2024-09-21T08:28:29.634231+0000","last_became_peered":"2024-09-21T08:28:29.634231+0000","last_unstale":"2024-09-21T08:30:35.662001+0000","last_undegraded":"2024-09-21T08:30:35.662001+0000","last_fullsized":"2024-09-21T08:30:35.662001+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"263'14","last_scrub_stamp":"2024-09-21T08:27:36.998538+0000","last_deep_scrub":"263'14","last_deep_scrub_stamp":"2024-09-21T08:27:36.998538+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.998538+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078732+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":773,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.668265+0000","last_change":"2024-09-21T08:28:53.591306+0000","last_active":"2024-09-21T08:30:35.668265+0000","last_peered":"2024-09-21T08:30:35.668265+0000","last_clean":"2024-09-21T08:28:51.554109+0000","last_became_active":"2024-09-21T08:28:53.591306+0000","last_became_peered":"2024-09-21T08:28:53.591306+0000","last_unstale":"2024-09-21T08:30:35.668265+0000","last_undegraded":"2024-09-21T08:30:35.668265+0000","last_fullsized":"2024-09-21T08:28:53.580235+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.344185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.344185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:36.882832+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.1","version":"179'43","reported_seq":522,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.657224+0000","last_change":"2024-09-21T08:28:53.589976+0000","last_active":"2024-09-21T08:30:35.657224+0000","last_peered":"2024-09-21T08:30:35.657224+0000","last_clean":"2024-09-21T08:28:51.585006+0000","last_became_active":"2024-09-21T08:28:53.589976+0000","last_became_peered":"2024-09-21T08:28:53.589976+0000","last_unstale":"2024-09-21T08:30:35.657224+0000","last_undegraded":"2024-09-21T08:30:35.657224+0000","last_fullsized":"2024-09-21T08:28:53.574347+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:50.158232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.158232+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":54,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.075152813999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"4.2","version":"73'10","reported_seq":352,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.658737+0000","last_change":"2024-09-21T08:29:58.300891+0000","last_active":"2024-09-21T08:30:35.658737+0000","last_peered":"2024-09-21T08:30:35.658737+0000","last_clean":"2024-09-21T08:30:35.658737+0000","last_became_active":"2024-09-21T08:28:29.638492+0000","last_became_peered":"2024-09-21T08:28:29.638492+0000","last_unstale":"2024-09-21T08:30:35.658737+0000","last_undegraded":"2024-09-21T08:30:35.658737+0000","last_fullsized":"2024-09-21T08:30:35.658737+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:29:58.300806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.300806+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:17.610082+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":821,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.672998+0000","last_change":"2024-09-21T08:28:53.588911+0000","last_active":"2024-09-21T08:30:35.672998+0000","last_peered":"2024-09-21T08:30:35.672998+0000","last_clean":"2024-09-21T08:28:51.554689+0000","last_became_active":"2024-09-21T08:28:53.588911+0000","last_became_peered":"2024-09-21T08:28:53.588911+0000","last_unstale":"2024-09-21T08:30:35.672998+0000","last_undegraded":"2024-09-21T08:28:53.576074+0000","last_fullsized":"2024-09-21T08:28:53.575948+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:16.003225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.003225+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704216+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"340'16","reported_seq":583,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.672229+0000","last_change":"2024-09-21T08:28:53.594258+0000","last_active":"2024-09-21T08:30:35.672229+0000","last_peered":"2024-09-21T08:30:35.672229+0000","last_clean":"2024-09-21T08:28:51.556539+0000","last_became_active":"2024-09-21T08:28:53.594258+0000","last_became_peered":"2024-09-21T08:28:53.594258+0000","last_unstale":"2024-09-21T08:30:35.672229+0000","last_undegraded":"2024-09-21T08:28:53.574644+0000","last_fullsized":"2024-09-21T08:28:53.574523+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'13","last_scrub_stamp":"2024-09-21T08:27:27.874952+0000","last_deep_scrub":"176'11","last_deep_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_clean_scrub_stamp":"2024-09-21T08:27:27.874952+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704401+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":608,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.671183+0000","last_change":"2024-09-21T08:28:37.565930+0000","last_active":"2024-09-21T08:30:35.671183+0000","last_peered":"2024-09-21T08:30:35.671183+0000","last_clean":"2024-09-21T08:30:35.671183+0000","last_became_active":"2024-09-21T08:28:37.565624+0000","last_became_peered":"2024-09-21T08:28:37.565624+0000","last_unstale":"2024-09-21T08:30:35.671183+0000","last_undegraded":"2024-09-21T08:30:35.671183+0000","last_fullsized":"2024-09-21T08:30:35.671183+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.258570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.258570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.670128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.0","version":"441'207","reported_seq":2106,"reported_epoch":442,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:35.673084+0000","last_change":"2024-09-21T08:30:34.770422+0000","last_active":"2024-09-21T08:30:35.673084+0000","last_peered":"2024-09-21T08:30:35.673084+0000","last_clean":"2024-09-21T08:30:35.673084+0000","last_became_active":"2024-09-21T08:28:07.549600+0000","last_became_peered":"2024-09-21T08:28:07.549600+0000","last_unstale":"2024-09-21T08:30:35.673084+0000","last_undegraded":"2024-09-21T08:30:35.673084+0000","last_fullsized":"2024-09-21T08:30:35.673084+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'166","last_scrub_stamp":"2024-09-21T08:29:32.253921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.253921+0000","objects_scrubbed":3,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:46.168730+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.11946623200000001,"stat_sum":{"num_bytes":11202517,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":38,"num_read_kb":38413,"num_write":24,"num_write_kb":8854,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1484609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"94","length":"5"},{"start":"9a","length":"1"},{"start":"9c","length":"2"},{"start":"9f","length":"4"},{"start":"a6","length":"1"}]},{"pgid":"4.3","version":"73'19","reported_seq":339,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.672516+0000","last_change":"2024-09-21T08:28:53.593536+0000","last_active":"2024-09-21T08:30:35.672516+0000","last_peered":"2024-09-21T08:30:35.672516+0000","last_clean":"2024-09-21T08:28:51.557090+0000","last_became_active":"2024-09-21T08:28:53.593536+0000","last_became_peered":"2024-09-21T08:28:53.593536+0000","last_unstale":"2024-09-21T08:30:35.672516+0000","last_undegraded":"2024-09-21T08:28:53.574981+0000","last_fullsized":"2024-09-21T08:28:53.574872+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:27:55.260127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:55.260127+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704417+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":314,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.658274+0000","last_change":"2024-09-21T08:28:04.726932+0000","last_active":"2024-09-21T08:30:35.658274+0000","last_peered":"2024-09-21T08:30:35.658274+0000","last_clean":"2024-09-21T08:30:35.658274+0000","last_became_active":"2024-09-21T08:28:04.723998+0000","last_became_peered":"2024-09-21T08:28:04.723998+0000","last_unstale":"2024-09-21T08:30:35.658274+0000","last_undegraded":"2024-09-21T08:30:35.658274+0000","last_fullsized":"2024-09-21T08:30:35.658274+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:54.289650+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:54.289650+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078781+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":269,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.674557+0000","last_change":"2024-09-21T08:28:53.588949+0000","last_active":"2024-09-21T08:30:35.674557+0000","last_peered":"2024-09-21T08:30:35.674557+0000","last_clean":"2024-09-21T08:28:51.557587+0000","last_became_active":"2024-09-21T08:28:53.588949+0000","last_became_peered":"2024-09-21T08:28:53.588949+0000","last_unstale":"2024-09-21T08:30:35.674557+0000","last_undegraded":"2024-09-21T08:30:35.674557+0000","last_fullsized":"2024-09-21T08:28:53.575963+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.339126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.339126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703829+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":317,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.656103+0000","last_change":"2024-09-21T08:28:53.600582+0000","last_active":"2024-09-21T08:30:35.656103+0000","last_peered":"2024-09-21T08:30:35.656103+0000","last_clean":"2024-09-21T08:28:51.584621+0000","last_became_active":"2024-09-21T08:28:53.600582+0000","last_became_peered":"2024-09-21T08:28:53.600582+0000","last_unstale":"2024-09-21T08:30:35.656103+0000","last_undegraded":"2024-09-21T08:30:35.656103+0000","last_fullsized":"2024-09-21T08:28:53.576129+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:46.263984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.263984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078605+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.7","version":"433'186","reported_seq":1930,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.657596+0000","last_change":"2024-09-21T08:28:53.602072+0000","last_active":"2024-09-21T08:30:35.657596+0000","last_peered":"2024-09-21T08:30:35.657596+0000","last_clean":"2024-09-21T08:28:51.587562+0000","last_became_active":"2024-09-21T08:28:53.602072+0000","last_became_peered":"2024-09-21T08:28:53.602072+0000","last_unstale":"2024-09-21T08:30:35.657596+0000","last_undegraded":"2024-09-21T08:28:53.578389+0000","last_fullsized":"2024-09-21T08:28:53.578258+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":186,"log_dups_size":0,"ondisk_log_size":186,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":58,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.074195671000000005,"stat_sum":{"num_bytes":30101710,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":68,"num_read_kb":59406,"num_write":66,"num_write_kb":34847,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":25647838,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":13}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"1"}]},{"pgid":"4.4","version":"340'35","reported_seq":371,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.657994+0000","last_change":"2024-09-21T08:28:53.603157+0000","last_active":"2024-09-21T08:30:35.657994+0000","last_peered":"2024-09-21T08:30:35.657994+0000","last_clean":"2024-09-21T08:28:51.586037+0000","last_became_active":"2024-09-21T08:28:53.603157+0000","last_became_peered":"2024-09-21T08:28:53.603157+0000","last_unstale":"2024-09-21T08:30:35.657994+0000","last_undegraded":"2024-09-21T08:28:53.579248+0000","last_fullsized":"2024-09-21T08:28:53.579125+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'32","last_scrub_stamp":"2024-09-21T08:27:11.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:11.407384+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078640+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":710,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.662146+0000","last_change":"2024-09-21T08:28:29.625492+0000","last_active":"2024-09-21T08:30:35.662146+0000","last_peered":"2024-09-21T08:30:35.662146+0000","last_clean":"2024-09-21T08:30:35.662146+0000","last_became_active":"2024-09-21T08:28:29.625170+0000","last_became_peered":"2024-09-21T08:28:29.625170+0000","last_unstale":"2024-09-21T08:30:35.662146+0000","last_undegraded":"2024-09-21T08:30:35.662146+0000","last_fullsized":"2024-09-21T08:30:35.662146+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:14.370256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.370256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":845,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.673076+0000","last_change":"2024-09-21T08:28:53.594367+0000","last_active":"2024-09-21T08:30:35.673076+0000","last_peered":"2024-09-21T08:30:35.673076+0000","last_clean":"2024-09-21T08:28:51.557303+0000","last_became_active":"2024-09-21T08:28:53.594367+0000","last_became_peered":"2024-09-21T08:28:53.594367+0000","last_unstale":"2024-09-21T08:30:35.673076+0000","last_undegraded":"2024-09-21T08:28:53.574945+0000","last_fullsized":"2024-09-21T08:28:53.574836+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703929+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":757,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.658942+0000","last_change":"2024-09-21T08:28:53.603129+0000","last_active":"2024-09-21T08:30:35.658942+0000","last_peered":"2024-09-21T08:30:35.658942+0000","last_clean":"2024-09-21T08:28:51.587295+0000","last_became_active":"2024-09-21T08:28:53.603129+0000","last_became_peered":"2024-09-21T08:28:53.603129+0000","last_unstale":"2024-09-21T08:30:35.658942+0000","last_undegraded":"2024-09-21T08:30:35.658942+0000","last_fullsized":"2024-09-21T08:28:53.579036+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:45.237804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.237804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":627,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.675163+0000","last_change":"2024-09-21T08:28:53.600966+0000","last_active":"2024-09-21T08:30:35.675163+0000","last_peered":"2024-09-21T08:30:35.675163+0000","last_clean":"2024-09-21T08:28:02.651561+0000","last_became_active":"2024-09-21T08:28:53.600966+0000","last_became_peered":"2024-09-21T08:28:53.600966+0000","last_unstale":"2024-09-21T08:30:35.675163+0000","last_undegraded":"2024-09-21T08:30:35.675163+0000","last_fullsized":"2024-09-21T08:28:53.579074+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.844729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.844729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704617+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.4","version":"439'137","reported_seq":1929,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.658762+0000","last_change":"2024-09-21T08:30:35.658762+0000","last_active":"2024-09-21T08:30:35.658762+0000","last_peered":"2024-09-21T08:30:35.658762+0000","last_clean":"2024-09-21T08:30:35.658762+0000","last_became_active":"2024-09-21T08:28:29.626340+0000","last_became_peered":"2024-09-21T08:28:29.626340+0000","last_unstale":"2024-09-21T08:30:35.658762+0000","last_undegraded":"2024-09-21T08:30:35.658762+0000","last_fullsized":"2024-09-21T08:30:35.658762+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"350'93","last_scrub_stamp":"2024-09-21T08:29:13.277953+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:29:13.277953+0000","objects_scrubbed":3,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:41.372665+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0023152250000000002,"stat_sum":{"num_bytes":5203142,"num_objects":3,"num_object_clones":2,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":28,"num_read_kb":35599,"num_write":14,"num_write_kb":7300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":11198550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"8"},{"start":"b1","length":"1"}]},{"pgid":"4.7","version":"73'13","reported_seq":355,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.659546+0000","last_change":"2024-09-21T08:28:53.595533+0000","last_active":"2024-09-21T08:30:35.659546+0000","last_peered":"2024-09-21T08:30:35.659546+0000","last_clean":"2024-09-21T08:28:51.587779+0000","last_became_active":"2024-09-21T08:28:53.595533+0000","last_became_peered":"2024-09-21T08:28:53.595533+0000","last_unstale":"2024-09-21T08:30:35.659546+0000","last_undegraded":"2024-09-21T08:28:53.580029+0000","last_fullsized":"2024-09-21T08:28:53.579372+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-09-21T08:28:16.386950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.386950+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078969+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":328,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.662697+0000","last_change":"2024-09-21T08:29:56.368212+0000","last_active":"2024-09-21T08:30:35.662697+0000","last_peered":"2024-09-21T08:30:35.662697+0000","last_clean":"2024-09-21T08:30:35.662697+0000","last_became_active":"2024-09-21T08:28:29.629320+0000","last_became_peered":"2024-09-21T08:28:29.629320+0000","last_unstale":"2024-09-21T08:30:35.662697+0000","last_undegraded":"2024-09-21T08:30:35.662697+0000","last_fullsized":"2024-09-21T08:30:35.662697+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:56.368144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:29:56.368144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:02.232771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":785,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.660646+0000","last_change":"2024-09-21T08:28:53.605131+0000","last_active":"2024-09-21T08:30:35.660646+0000","last_peered":"2024-09-21T08:30:35.660646+0000","last_clean":"2024-09-21T08:28:51.585166+0000","last_became_active":"2024-09-21T08:28:53.605131+0000","last_became_peered":"2024-09-21T08:28:53.605131+0000","last_unstale":"2024-09-21T08:30:35.660646+0000","last_undegraded":"2024-09-21T08:28:53.583253+0000","last_fullsized":"2024-09-21T08:28:53.582934+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:59.262463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:27:59.262463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078903+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":774,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.671368+0000","last_change":"2024-09-21T08:28:53.598122+0000","last_active":"2024-09-21T08:30:35.671368+0000","last_peered":"2024-09-21T08:30:35.671368+0000","last_clean":"2024-09-21T08:28:51.556093+0000","last_became_active":"2024-09-21T08:28:53.598122+0000","last_became_peered":"2024-09-21T08:28:53.598122+0000","last_unstale":"2024-09-21T08:30:35.671368+0000","last_undegraded":"2024-09-21T08:30:35.671368+0000","last_fullsized":"2024-09-21T08:28:53.584774+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.374076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.374076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:36.882860+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":672,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.672743+0000","last_change":"2024-09-21T08:28:53.587481+0000","last_active":"2024-09-21T08:30:35.672743+0000","last_peered":"2024-09-21T08:30:35.672743+0000","last_clean":"2024-09-21T08:28:51.554685+0000","last_became_active":"2024-09-21T08:28:53.587481+0000","last_became_peered":"2024-09-21T08:28:53.587481+0000","last_unstale":"2024-09-21T08:30:35.672743+0000","last_undegraded":"2024-09-21T08:30:35.672743+0000","last_fullsized":"2024-09-21T08:28:53.575943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:08.831386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:08.831386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.5","version":"413'305","reported_seq":1892,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.657531+0000","last_change":"2024-09-21T08:28:53.600711+0000","last_active":"2024-09-21T08:30:35.657531+0000","last_peered":"2024-09-21T08:30:35.657531+0000","last_clean":"2024-09-21T08:28:52.661593+0000","last_became_active":"2024-09-21T08:28:53.600711+0000","last_became_peered":"2024-09-21T08:28:53.600711+0000","last_unstale":"2024-09-21T08:30:35.657531+0000","last_undegraded":"2024-09-21T08:28:53.576301+0000","last_fullsized":"2024-09-21T08:28:53.576176+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'245","last_scrub_stamp":"2024-09-21T08:26:30.864040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.864040+0000","objects_scrubbed":0,"log_size":305,"log_dups_size":0,"ondisk_log_size":305,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.075799959,"stat_sum":{"num_bytes":6305048,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":48,"num_read_kb":51018,"num_write":34,"num_write_kb":8053,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1853978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.6","version":"73'12","reported_seq":782,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.663341+0000","last_change":"2024-09-21T08:28:53.603118+0000","last_active":"2024-09-21T08:30:35.663341+0000","last_peered":"2024-09-21T08:30:35.663341+0000","last_clean":"2024-09-21T08:28:51.587443+0000","last_became_active":"2024-09-21T08:28:53.603118+0000","last_became_peered":"2024-09-21T08:28:53.603118+0000","last_unstale":"2024-09-21T08:30:35.663341+0000","last_undegraded":"2024-09-21T08:28:53.582699+0000","last_fullsized":"2024-09-21T08:28:53.582597+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:26:07.384640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:07.384640+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078490+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":644,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.672802+0000","last_change":"2024-09-21T08:28:53.599951+0000","last_active":"2024-09-21T08:30:35.672802+0000","last_peered":"2024-09-21T08:30:35.672802+0000","last_clean":"2024-09-21T08:28:51.555106+0000","last_became_active":"2024-09-21T08:28:53.599951+0000","last_became_peered":"2024-09-21T08:28:53.599951+0000","last_unstale":"2024-09-21T08:30:35.672802+0000","last_undegraded":"2024-09-21T08:30:35.672802+0000","last_fullsized":"2024-09-21T08:28:53.575796+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.917891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.917891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1225,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.661528+0000","last_change":"2024-09-21T08:28:53.594845+0000","last_active":"2024-09-21T08:30:35.661528+0000","last_peered":"2024-09-21T08:30:35.661528+0000","last_clean":"2024-09-21T08:28:52.670294+0000","last_became_active":"2024-09-21T08:28:53.594845+0000","last_became_peered":"2024-09-21T08:28:53.594845+0000","last_unstale":"2024-09-21T08:30:35.661528+0000","last_undegraded":"2024-09-21T08:28:53.577170+0000","last_fullsized":"2024-09-21T08:28:53.575736+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078396+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":402,"num_read_kb":397,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":785,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.674716+0000","last_change":"2024-09-21T08:28:53.602597+0000","last_active":"2024-09-21T08:30:35.674716+0000","last_peered":"2024-09-21T08:30:35.674716+0000","last_clean":"2024-09-21T08:28:51.556542+0000","last_became_active":"2024-09-21T08:28:53.602597+0000","last_became_peered":"2024-09-21T08:28:53.602597+0000","last_unstale":"2024-09-21T08:30:35.674716+0000","last_undegraded":"2024-09-21T08:30:35.674716+0000","last_fullsized":"2024-09-21T08:28:53.580807+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.812789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.812789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703841+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":322,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.658187+0000","last_change":"2024-09-21T08:28:17.424621+0000","last_active":"2024-09-21T08:30:35.658187+0000","last_peered":"2024-09-21T08:30:35.658187+0000","last_clean":"2024-09-21T08:30:35.658187+0000","last_became_active":"2024-09-21T08:28:04.723837+0000","last_became_peered":"2024-09-21T08:28:04.723837+0000","last_unstale":"2024-09-21T08:30:35.658187+0000","last_undegraded":"2024-09-21T08:30:35.658187+0000","last_fullsized":"2024-09-21T08:30:35.658187+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:17.424515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:17.424515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078845+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"416'157","reported_seq":1798,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.673912+0000","last_change":"2024-09-21T08:28:53.589874+0000","last_active":"2024-09-21T08:30:35.673912+0000","last_peered":"2024-09-21T08:30:35.673912+0000","last_clean":"2024-09-21T08:28:51.554645+0000","last_became_active":"2024-09-21T08:28:53.589874+0000","last_became_peered":"2024-09-21T08:28:53.589874+0000","last_unstale":"2024-09-21T08:30:35.673912+0000","last_undegraded":"2024-09-21T08:28:53.577395+0000","last_fullsized":"2024-09-21T08:28:53.577254+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'138","last_scrub_stamp":"2024-09-21T08:28:41.855754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:41.855754+0000","objects_scrubbed":1,"log_size":157,"log_dups_size":0,"ondisk_log_size":157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":49,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027851600000000001,"stat_sum":{"num_bytes":11714710,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":49,"num_read_kb":63283,"num_write":24,"num_write_kb":8369,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":9282889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"4.5","version":"340'22","reported_seq":632,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.675033+0000","last_change":"2024-09-21T08:28:53.600773+0000","last_active":"2024-09-21T08:30:35.675033+0000","last_peered":"2024-09-21T08:30:35.675033+0000","last_clean":"2024-09-21T08:27:13.582840+0000","last_became_active":"2024-09-21T08:28:53.600773+0000","last_became_peered":"2024-09-21T08:28:53.600773+0000","last_unstale":"2024-09-21T08:30:35.675033+0000","last_undegraded":"2024-09-21T08:28:53.577472+0000","last_fullsized":"2024-09-21T08:28:53.577362+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-21T08:28:13.956509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.956509+0000","objects_scrubbed":0,"log_size":22,"log_dups_size":0,"ondisk_log_size":22,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":308,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":682,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.675024+0000","last_change":"2024-09-21T08:28:53.600983+0000","last_active":"2024-09-21T08:30:35.675024+0000","last_peered":"2024-09-21T08:30:35.675024+0000","last_clean":"2024-09-21T08:28:51.558067+0000","last_became_active":"2024-09-21T08:28:53.600983+0000","last_became_peered":"2024-09-21T08:28:53.600983+0000","last_unstale":"2024-09-21T08:30:35.675024+0000","last_undegraded":"2024-09-21T08:30:35.675024+0000","last_fullsized":"2024-09-21T08:28:53.577601+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.045353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.045353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704039+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":1029,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.661358+0000","last_change":"2024-09-21T08:28:53.604882+0000","last_active":"2024-09-21T08:30:35.661358+0000","last_peered":"2024-09-21T08:30:35.661358+0000","last_clean":"2024-09-21T08:28:51.584594+0000","last_became_active":"2024-09-21T08:28:53.604882+0000","last_became_peered":"2024-09-21T08:28:53.604882+0000","last_unstale":"2024-09-21T08:30:35.661358+0000","last_undegraded":"2024-09-21T08:28:53.575806+0000","last_fullsized":"2024-09-21T08:28:53.575705+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078428+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":251,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.673006+0000","last_change":"2024-09-21T08:28:53.589976+0000","last_active":"2024-09-21T08:30:35.673006+0000","last_peered":"2024-09-21T08:30:35.673006+0000","last_clean":"2024-09-21T08:28:51.557022+0000","last_became_active":"2024-09-21T08:28:53.589976+0000","last_became_peered":"2024-09-21T08:28:53.589976+0000","last_unstale":"2024-09-21T08:30:35.673006+0000","last_undegraded":"2024-09-21T08:30:35.673006+0000","last_fullsized":"2024-09-21T08:28:53.576557+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":799,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.675670+0000","last_change":"2024-09-21T08:28:53.602915+0000","last_active":"2024-09-21T08:30:35.675670+0000","last_peered":"2024-09-21T08:30:35.675670+0000","last_clean":"2024-09-21T08:28:51.556967+0000","last_became_active":"2024-09-21T08:28:53.602915+0000","last_became_peered":"2024-09-21T08:28:53.602915+0000","last_unstale":"2024-09-21T08:30:35.675670+0000","last_undegraded":"2024-09-21T08:30:35.675670+0000","last_fullsized":"2024-09-21T08:28:53.581134+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703729+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"440'145","reported_seq":644,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.659606+0000","last_change":"2024-09-21T08:28:53.596348+0000","last_active":"2024-09-21T08:30:35.659606+0000","last_peered":"2024-09-21T08:30:35.659606+0000","last_clean":"2024-09-21T08:28:52.661631+0000","last_became_active":"2024-09-21T08:28:53.596348+0000","last_became_peered":"2024-09-21T08:28:53.596348+0000","last_unstale":"2024-09-21T08:30:35.659606+0000","last_undegraded":"2024-09-21T08:28:53.582490+0000","last_fullsized":"2024-09-21T08:28:53.579369+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'93","last_scrub_stamp":"2024-09-21T08:28:11.373358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:11.373358+0000","objects_scrubbed":3,"log_size":145,"log_dups_size":0,"ondisk_log_size":145,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":58,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.070749670000000001,"stat_sum":{"num_bytes":25562047,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":30,"num_read_kb":17875,"num_write":37,"num_write_kb":11195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":6571636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":11}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"1"}]},{"pgid":"4.e","version":"73'11","reported_seq":337,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.658721+0000","last_change":"2024-09-21T08:28:53.597299+0000","last_active":"2024-09-21T08:30:35.658721+0000","last_peered":"2024-09-21T08:30:35.658721+0000","last_clean":"2024-09-21T08:28:51.585940+0000","last_became_active":"2024-09-21T08:28:53.597299+0000","last_became_peered":"2024-09-21T08:28:53.597299+0000","last_unstale":"2024-09-21T08:30:35.658721+0000","last_undegraded":"2024-09-21T08:28:53.581018+0000","last_fullsized":"2024-09-21T08:28:53.580915+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:28:14.416729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.416729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078983+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":757,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.667254+0000","last_change":"2024-09-21T08:28:53.591487+0000","last_active":"2024-09-21T08:30:35.667254+0000","last_peered":"2024-09-21T08:30:35.667254+0000","last_clean":"2024-09-21T08:28:51.552892+0000","last_became_active":"2024-09-21T08:28:53.591487+0000","last_became_peered":"2024-09-21T08:28:53.591487+0000","last_unstale":"2024-09-21T08:30:35.667254+0000","last_undegraded":"2024-09-21T08:30:35.667254+0000","last_fullsized":"2024-09-21T08:28:53.575527+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:28.347474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:28.347474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:36.882780+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":808,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.675739+0000","last_change":"2024-09-21T08:29:55.150130+0000","last_active":"2024-09-21T08:30:35.675739+0000","last_peered":"2024-09-21T08:30:35.675739+0000","last_clean":"2024-09-21T08:30:35.675739+0000","last_became_active":"2024-09-21T08:28:04.724965+0000","last_became_peered":"2024-09-21T08:28:04.724965+0000","last_unstale":"2024-09-21T08:30:35.675739+0000","last_undegraded":"2024-09-21T08:30:35.675739+0000","last_fullsized":"2024-09-21T08:30:35.675739+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:55.150058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:32.961381+0000","last_clean_scrub_stamp":"2024-09-21T08:29:55.150058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:10.970830+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":661,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.673636+0000","last_change":"2024-09-21T08:28:53.601979+0000","last_active":"2024-09-21T08:30:35.673636+0000","last_peered":"2024-09-21T08:30:35.673636+0000","last_clean":"2024-09-21T08:28:51.557371+0000","last_became_active":"2024-09-21T08:28:53.601979+0000","last_became_peered":"2024-09-21T08:28:53.601979+0000","last_unstale":"2024-09-21T08:30:35.673636+0000","last_undegraded":"2024-09-21T08:30:35.673636+0000","last_fullsized":"2024-09-21T08:28:53.577942+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.977956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"256'199","reported_seq":2002,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.671641+0000","last_change":"2024-09-21T08:30:35.672189+0000","last_active":"2024-09-21T08:30:35.671641+0000","last_peered":"2024-09-21T08:30:35.671641+0000","last_clean":"2024-09-21T08:30:35.671641+0000","last_became_active":"2024-09-21T08:28:32.558440+0000","last_became_peered":"2024-09-21T08:28:32.558440+0000","last_unstale":"2024-09-21T08:30:35.671641+0000","last_undegraded":"2024-09-21T08:30:35.671641+0000","last_fullsized":"2024-09-21T08:30:35.671641+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'199","last_scrub_stamp":"2024-09-21T08:30:17.742629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:17.742629+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:27.804718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00097294000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"8"},{"start":"b1","length":"1"}]},{"pgid":"4.d","version":"345'25","reported_seq":807,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.659878+0000","last_change":"2024-09-21T08:28:59.242015+0000","last_active":"2024-09-21T08:30:35.659878+0000","last_peered":"2024-09-21T08:30:35.659878+0000","last_clean":"2024-09-21T08:30:35.659878+0000","last_became_active":"2024-09-21T08:28:37.564492+0000","last_became_peered":"2024-09-21T08:28:37.564492+0000","last_unstale":"2024-09-21T08:30:35.659878+0000","last_undegraded":"2024-09-21T08:30:35.659878+0000","last_fullsized":"2024-09-21T08:30:35.659878+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'25","last_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_deep_scrub":"345'25","last_deep_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.241931+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.079023+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":317,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.657448+0000","last_change":"2024-09-21T08:28:53.599763+0000","last_active":"2024-09-21T08:30:35.657448+0000","last_peered":"2024-09-21T08:30:35.657448+0000","last_clean":"2024-09-21T08:28:51.585236+0000","last_became_active":"2024-09-21T08:28:53.599763+0000","last_became_peered":"2024-09-21T08:28:53.599763+0000","last_unstale":"2024-09-21T08:30:35.657448+0000","last_undegraded":"2024-09-21T08:30:35.657448+0000","last_fullsized":"2024-09-21T08:28:53.586112+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:30.909902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.909902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":323,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.657702+0000","last_change":"2024-09-21T08:28:53.601757+0000","last_active":"2024-09-21T08:30:35.657702+0000","last_peered":"2024-09-21T08:30:35.657702+0000","last_clean":"2024-09-21T08:28:51.585900+0000","last_became_active":"2024-09-21T08:28:53.601757+0000","last_became_peered":"2024-09-21T08:28:53.601757+0000","last_unstale":"2024-09-21T08:30:35.657702+0000","last_undegraded":"2024-09-21T08:30:35.657702+0000","last_fullsized":"2024-09-21T08:28:53.576584+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":322,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.659441+0000","last_change":"2024-09-21T08:30:00.276844+0000","last_active":"2024-09-21T08:30:35.659441+0000","last_peered":"2024-09-21T08:30:35.659441+0000","last_clean":"2024-09-21T08:30:35.659441+0000","last_became_active":"2024-09-21T08:28:04.726509+0000","last_became_peered":"2024-09-21T08:28:04.726509+0000","last_unstale":"2024-09-21T08:30:35.659441+0000","last_undegraded":"2024-09-21T08:30:35.659441+0000","last_fullsized":"2024-09-21T08:30:35.659441+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:00.276774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:00.276774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:26.484373+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.f","version":"433'79","reported_seq":1717,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.673571+0000","last_change":"2024-09-21T08:28:53.590273+0000","last_active":"2024-09-21T08:30:35.673571+0000","last_peered":"2024-09-21T08:30:35.673571+0000","last_clean":"2024-09-21T08:28:51.554220+0000","last_became_active":"2024-09-21T08:28:53.590273+0000","last_became_peered":"2024-09-21T08:28:53.590273+0000","last_unstale":"2024-09-21T08:30:35.673571+0000","last_undegraded":"2024-09-21T08:28:53.579532+0000","last_fullsized":"2024-09-21T08:28:53.579403+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:28:10.372149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.372149+0000","objects_scrubbed":1,"log_size":79,"log_dups_size":0,"ondisk_log_size":79,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.019314938,"stat_sum":{"num_bytes":23018803,"num_objects":12,"num_object_clones":11,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":1,"num_read":29,"num_read_kb":28274,"num_write":39,"num_write_kb":16470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":2823609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":12}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.c","version":"73'10","reported_seq":329,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.657898+0000","last_change":"2024-09-21T08:28:53.601905+0000","last_active":"2024-09-21T08:30:35.657898+0000","last_peered":"2024-09-21T08:30:35.657898+0000","last_clean":"2024-09-21T08:28:51.586041+0000","last_became_active":"2024-09-21T08:28:53.601905+0000","last_became_peered":"2024-09-21T08:28:53.601905+0000","last_unstale":"2024-09-21T08:30:35.657898+0000","last_undegraded":"2024-09-21T08:28:53.577157+0000","last_fullsized":"2024-09-21T08:28:53.576937+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:26:33.823950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.823950+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078576+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":682,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.661600+0000","last_change":"2024-09-21T08:28:53.602716+0000","last_active":"2024-09-21T08:30:35.661600+0000","last_peered":"2024-09-21T08:30:35.661600+0000","last_clean":"2024-09-21T08:28:51.585127+0000","last_became_active":"2024-09-21T08:28:53.602716+0000","last_became_peered":"2024-09-21T08:28:53.602716+0000","last_unstale":"2024-09-21T08:30:35.661600+0000","last_undegraded":"2024-09-21T08:30:35.661600+0000","last_fullsized":"2024-09-21T08:28:53.576785+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:54.216528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:54.216528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078751+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.d","version":"340'10","reported_seq":299,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.675803+0000","last_change":"2024-09-21T08:28:53.589819+0000","last_active":"2024-09-21T08:30:35.675803+0000","last_peered":"2024-09-21T08:30:35.675803+0000","last_clean":"2024-09-21T08:28:51.555429+0000","last_became_active":"2024-09-21T08:28:53.589819+0000","last_became_peered":"2024-09-21T08:28:53.589819+0000","last_unstale":"2024-09-21T08:30:35.675803+0000","last_undegraded":"2024-09-21T08:28:53.578694+0000","last_fullsized":"2024-09-21T08:28:53.577943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:27:51.312273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.312273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704432+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":322,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.658824+0000","last_change":"2024-09-21T08:29:40.458127+0000","last_active":"2024-09-21T08:30:35.658824+0000","last_peered":"2024-09-21T08:30:35.658824+0000","last_clean":"2024-09-21T08:30:35.658824+0000","last_became_active":"2024-09-21T08:28:04.722744+0000","last_became_peered":"2024-09-21T08:28:04.722744+0000","last_unstale":"2024-09-21T08:30:35.658824+0000","last_undegraded":"2024-09-21T08:30:35.658824+0000","last_fullsized":"2024-09-21T08:30:35.658824+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:40.458054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:40.458054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:06.465951+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.8","version":"436'333","reported_seq":1919,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.671329+0000","last_change":"2024-09-21T08:28:53.587761+0000","last_active":"2024-09-21T08:30:35.671329+0000","last_peered":"2024-09-21T08:30:35.671329+0000","last_clean":"2024-09-21T08:28:52.588981+0000","last_became_active":"2024-09-21T08:28:53.587761+0000","last_became_peered":"2024-09-21T08:28:53.587761+0000","last_unstale":"2024-09-21T08:30:35.671329+0000","last_undegraded":"2024-09-21T08:28:53.574111+0000","last_fullsized":"2024-09-21T08:28:53.573936+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'178","last_scrub_stamp":"2024-09-21T08:26:28.217842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.217842+0000","objects_scrubbed":4,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":52,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0347803,"stat_sum":{"num_bytes":27079062,"num_objects":15,"num_object_clones":12,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":1,"num_read":116,"num_read_kb":143320,"num_write":88,"num_write_kb":35860,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":23089101,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":15}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"4.b","version":"73'9","reported_seq":835,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.661730+0000","last_change":"2024-09-21T08:28:53.595156+0000","last_active":"2024-09-21T08:30:35.661730+0000","last_peered":"2024-09-21T08:30:35.661730+0000","last_clean":"2024-09-21T08:28:51.585036+0000","last_became_active":"2024-09-21T08:28:53.595156+0000","last_became_peered":"2024-09-21T08:28:53.595156+0000","last_unstale":"2024-09-21T08:30:35.661730+0000","last_undegraded":"2024-09-21T08:28:53.576597+0000","last_fullsized":"2024-09-21T08:28:53.576496+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:25:22.369811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.369811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078419+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":4}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":822,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.672597+0000","last_change":"2024-09-21T08:28:59.403264+0000","last_active":"2024-09-21T08:30:35.672597+0000","last_peered":"2024-09-21T08:30:35.672597+0000","last_clean":"2024-09-21T08:30:35.672597+0000","last_became_active":"2024-09-21T08:28:29.609526+0000","last_became_peered":"2024-09-21T08:28:29.609526+0000","last_unstale":"2024-09-21T08:30:35.672597+0000","last_undegraded":"2024-09-21T08:30:35.672597+0000","last_fullsized":"2024-09-21T08:30:35.672597+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:59.403194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.403194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.670199+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":676,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.656385+0000","last_change":"2024-09-21T08:28:53.602532+0000","last_active":"2024-09-21T08:30:35.656385+0000","last_peered":"2024-09-21T08:30:35.656385+0000","last_clean":"2024-09-21T08:28:51.584905+0000","last_became_active":"2024-09-21T08:28:53.602532+0000","last_became_peered":"2024-09-21T08:28:53.602532+0000","last_unstale":"2024-09-21T08:30:35.656385+0000","last_undegraded":"2024-09-21T08:30:35.656385+0000","last_fullsized":"2024-09-21T08:28:53.583376+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":274,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.667318+0000","last_change":"2024-09-21T08:28:53.591119+0000","last_active":"2024-09-21T08:30:35.667318+0000","last_peered":"2024-09-21T08:30:35.667318+0000","last_clean":"2024-09-21T08:28:51.552940+0000","last_became_active":"2024-09-21T08:28:53.591119+0000","last_became_peered":"2024-09-21T08:28:53.591119+0000","last_unstale":"2024-09-21T08:30:35.667318+0000","last_undegraded":"2024-09-21T08:30:35.667318+0000","last_fullsized":"2024-09-21T08:28:53.575921+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:36.308161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:36.308161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:36.882843+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"442'360","reported_seq":2150,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.672780+0000","last_change":"2024-09-21T08:28:53.606879+0000","last_active":"2024-09-21T08:30:35.672780+0000","last_peered":"2024-09-21T08:30:35.672780+0000","last_clean":"2024-09-21T08:28:52.666146+0000","last_became_active":"2024-09-21T08:28:53.606879+0000","last_became_peered":"2024-09-21T08:28:53.606879+0000","last_unstale":"2024-09-21T08:30:35.672780+0000","last_undegraded":"2024-09-21T08:28:53.584154+0000","last_fullsized":"2024-09-21T08:28:53.584033+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":360,"log_dups_size":0,"ondisk_log_size":360,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":59,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0805860919999999,"stat_sum":{"num_bytes":39161731,"num_objects":18,"num_object_clones":14,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":18,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":2,"num_read":174,"num_read_kb":188458,"num_write":109,"num_write_kb":37986,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":67,"num_bytes_recovered":24504816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":18}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":337,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.657874+0000","last_change":"2024-09-21T08:28:53.600149+0000","last_active":"2024-09-21T08:30:35.657874+0000","last_peered":"2024-09-21T08:30:35.657874+0000","last_clean":"2024-09-21T08:28:51.585660+0000","last_became_active":"2024-09-21T08:28:53.600149+0000","last_became_peered":"2024-09-21T08:28:53.600149+0000","last_unstale":"2024-09-21T08:30:35.657874+0000","last_undegraded":"2024-09-21T08:28:53.586423+0000","last_fullsized":"2024-09-21T08:28:53.586235+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078450+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":684,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.674712+0000","last_change":"2024-09-21T08:28:53.589287+0000","last_active":"2024-09-21T08:30:35.674712+0000","last_peered":"2024-09-21T08:30:35.674712+0000","last_clean":"2024-09-21T08:28:51.555026+0000","last_became_active":"2024-09-21T08:28:53.589287+0000","last_became_peered":"2024-09-21T08:28:53.589287+0000","last_unstale":"2024-09-21T08:30:35.674712+0000","last_undegraded":"2024-09-21T08:30:35.674712+0000","last_fullsized":"2024-09-21T08:28:53.576540+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.901202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.901202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704461+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":772,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.668219+0000","last_change":"2024-09-21T08:28:53.592593+0000","last_active":"2024-09-21T08:30:35.668219+0000","last_peered":"2024-09-21T08:30:35.668219+0000","last_clean":"2024-09-21T08:28:51.554029+0000","last_became_active":"2024-09-21T08:28:53.592593+0000","last_became_peered":"2024-09-21T08:28:53.592593+0000","last_unstale":"2024-09-21T08:30:35.668219+0000","last_undegraded":"2024-09-21T08:30:35.668219+0000","last_fullsized":"2024-09-21T08:28:53.577826+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:15.362045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:15.362045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:36.882746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":266,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.655825+0000","last_change":"2024-09-21T08:28:53.586960+0000","last_active":"2024-09-21T08:30:35.655825+0000","last_peered":"2024-09-21T08:30:35.655825+0000","last_clean":"2024-09-21T08:28:51.584413+0000","last_became_active":"2024-09-21T08:28:53.586960+0000","last_became_peered":"2024-09-21T08:28:53.586960+0000","last_unstale":"2024-09-21T08:30:35.655825+0000","last_undegraded":"2024-09-21T08:30:35.655825+0000","last_fullsized":"2024-09-21T08:28:53.575166+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"433'114","reported_seq":1080,"reported_epoch":442,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:30:35.673764+0000","last_change":"2024-09-21T08:30:32.768685+0000","last_active":"2024-09-21T08:30:35.673764+0000","last_peered":"2024-09-21T08:30:35.673764+0000","last_clean":"2024-09-21T08:30:35.673764+0000","last_became_active":"2024-09-21T08:28:37.567400+0000","last_became_peered":"2024-09-21T08:28:37.567400+0000","last_unstale":"2024-09-21T08:30:35.673764+0000","last_undegraded":"2024-09-21T08:30:35.673764+0000","last_fullsized":"2024-09-21T08:30:35.673764+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"248'68","last_scrub_stamp":"2024-09-21T08:27:35.317025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:35.317025+0000","objects_scrubbed":0,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.11912101999999999,"stat_sum":{"num_bytes":2840482,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":7,"num_read_kb":3,"num_write":19,"num_write_kb":3134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1183872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"1"},{"start":"a8","length":"2"}]},{"pgid":"4.9","version":"73'12","reported_seq":347,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.656744+0000","last_change":"2024-09-21T08:28:53.604531+0000","last_active":"2024-09-21T08:30:35.656744+0000","last_peered":"2024-09-21T08:30:35.656744+0000","last_clean":"2024-09-21T08:28:51.584845+0000","last_became_active":"2024-09-21T08:28:53.604531+0000","last_became_peered":"2024-09-21T08:28:53.604531+0000","last_unstale":"2024-09-21T08:30:35.656744+0000","last_undegraded":"2024-09-21T08:28:53.582724+0000","last_fullsized":"2024-09-21T08:28:53.582624+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:41.463820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.463820+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078459+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1100,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":785,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.660262+0000","last_change":"2024-09-21T08:28:53.596741+0000","last_active":"2024-09-21T08:30:35.660262+0000","last_peered":"2024-09-21T08:30:35.660262+0000","last_clean":"2024-09-21T08:28:51.587101+0000","last_became_active":"2024-09-21T08:28:53.596741+0000","last_became_peered":"2024-09-21T08:28:53.596741+0000","last_unstale":"2024-09-21T08:30:35.660262+0000","last_undegraded":"2024-09-21T08:30:35.660262+0000","last_fullsized":"2024-09-21T08:28:53.580812+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:51.769044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.769044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078598+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":784,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.661694+0000","last_change":"2024-09-21T08:29:59.267735+0000","last_active":"2024-09-21T08:30:35.661694+0000","last_peered":"2024-09-21T08:30:35.661694+0000","last_clean":"2024-09-21T08:30:35.661694+0000","last_became_active":"2024-09-21T08:28:37.566192+0000","last_became_peered":"2024-09-21T08:28:37.566192+0000","last_unstale":"2024-09-21T08:30:35.661694+0000","last_undegraded":"2024-09-21T08:30:35.661694+0000","last_fullsized":"2024-09-21T08:30:35.661694+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:59.267668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:29:59.267668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.614991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":676,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.661766+0000","last_change":"2024-09-21T08:28:53.595264+0000","last_active":"2024-09-21T08:30:35.661766+0000","last_peered":"2024-09-21T08:30:35.661766+0000","last_clean":"2024-09-21T08:28:51.585131+0000","last_became_active":"2024-09-21T08:28:53.595264+0000","last_became_peered":"2024-09-21T08:28:53.595264+0000","last_unstale":"2024-09-21T08:30:35.661766+0000","last_undegraded":"2024-09-21T08:30:35.661766+0000","last_fullsized":"2024-09-21T08:28:53.576894+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078442+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"441'731","reported_seq":2644,"reported_epoch":442,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:30:35.663581+0000","last_change":"2024-09-21T08:30:34.756426+0000","last_active":"2024-09-21T08:30:35.663581+0000","last_peered":"2024-09-21T08:30:35.663581+0000","last_clean":"2024-09-21T08:30:35.663581+0000","last_became_active":"2024-09-21T08:28:07.549709+0000","last_became_peered":"2024-09-21T08:28:07.549709+0000","last_unstale":"2024-09-21T08:30:35.663581+0000","last_undegraded":"2024-09-21T08:30:35.663581+0000","last_fullsized":"2024-09-21T08:30:35.663581+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'577","last_scrub_stamp":"2024-09-21T08:29:32.265609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.265609+0000","objects_scrubbed":10,"log_size":731,"log_dups_size":0,"ondisk_log_size":731,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:48.979152+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.104670624,"stat_sum":{"num_bytes":54985520,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":139,"num_read_kb":138016,"num_write":86,"num_write_kb":36499,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":4385271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"11"},{"start":"7f","length":"5"},{"start":"85","length":"d"},{"start":"94","length":"5"},{"start":"9a","length":"1"},{"start":"9c","length":"6"},{"start":"a6","length":"1"}]},{"pgid":"4.8","version":"73'15","reported_seq":906,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.671697+0000","last_change":"2024-09-21T08:28:53.591779+0000","last_active":"2024-09-21T08:30:35.671697+0000","last_peered":"2024-09-21T08:30:35.671697+0000","last_clean":"2024-09-21T08:28:51.558045+0000","last_became_active":"2024-09-21T08:28:53.591779+0000","last_became_peered":"2024-09-21T08:28:53.591779+0000","last_unstale":"2024-09-21T08:30:35.671697+0000","last_undegraded":"2024-09-21T08:28:53.574592+0000","last_fullsized":"2024-09-21T08:28:53.574472+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703711+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":774,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.658419+0000","last_change":"2024-09-21T08:28:53.595155+0000","last_active":"2024-09-21T08:30:35.658419+0000","last_peered":"2024-09-21T08:30:35.658419+0000","last_clean":"2024-09-21T08:28:51.587656+0000","last_became_active":"2024-09-21T08:28:53.595155+0000","last_became_peered":"2024-09-21T08:28:53.595155+0000","last_unstale":"2024-09-21T08:30:35.658419+0000","last_undegraded":"2024-09-21T08:28:53.579187+0000","last_fullsized":"2024-09-21T08:28:53.579080+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078348+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":276,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.9","version":"340'16","reported_seq":903,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.665331+0000","last_change":"2024-09-21T08:28:53.604478+0000","last_active":"2024-09-21T08:30:35.665331+0000","last_peered":"2024-09-21T08:30:35.665331+0000","last_clean":"2024-09-21T08:28:51.584962+0000","last_became_active":"2024-09-21T08:28:53.604478+0000","last_became_peered":"2024-09-21T08:28:53.604478+0000","last_unstale":"2024-09-21T08:30:35.665331+0000","last_undegraded":"2024-09-21T08:28:53.580813+0000","last_fullsized":"2024-09-21T08:28:53.580697+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'10","last_scrub_stamp":"2024-09-21T08:25:42.708458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.708458+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078469+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":772,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.674727+0000","last_change":"2024-09-21T08:28:55.397079+0000","last_active":"2024-09-21T08:30:35.674727+0000","last_peered":"2024-09-21T08:30:35.674727+0000","last_clean":"2024-09-21T08:30:35.674727+0000","last_became_active":"2024-09-21T08:28:29.619471+0000","last_became_peered":"2024-09-21T08:28:29.619471+0000","last_unstale":"2024-09-21T08:30:35.674727+0000","last_undegraded":"2024-09-21T08:30:35.674727+0000","last_fullsized":"2024-09-21T08:30:35.674727+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:55.396944+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:55.396944+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.670169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"180'50","reported_seq":1532,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.668601+0000","last_change":"2024-09-21T08:28:53.592066+0000","last_active":"2024-09-21T08:30:35.668601+0000","last_peered":"2024-09-21T08:30:35.668601+0000","last_clean":"2024-09-21T08:28:51.555673+0000","last_became_active":"2024-09-21T08:28:53.592066+0000","last_became_peered":"2024-09-21T08:28:53.592066+0000","last_unstale":"2024-09-21T08:30:35.668601+0000","last_undegraded":"2024-09-21T08:30:35.668601+0000","last_fullsized":"2024-09-21T08:28:53.577992+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:26:50.014780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:50.014780+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":49,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020276460000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"3.10","version":"0'0","reported_seq":570,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.671604+0000","last_change":"2024-09-21T08:29:39.177460+0000","last_active":"2024-09-21T08:30:35.671604+0000","last_peered":"2024-09-21T08:30:35.671604+0000","last_clean":"2024-09-21T08:30:35.671604+0000","last_became_active":"2024-09-21T08:28:29.610987+0000","last_became_peered":"2024-09-21T08:28:29.610987+0000","last_unstale":"2024-09-21T08:30:35.671604+0000","last_undegraded":"2024-09-21T08:30:35.671604+0000","last_fullsized":"2024-09-21T08:30:35.671604+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.177382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:59.628414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":721,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.658224+0000","last_change":"2024-09-21T08:28:53.602197+0000","last_active":"2024-09-21T08:30:35.658224+0000","last_peered":"2024-09-21T08:30:35.658224+0000","last_clean":"2024-09-21T08:28:51.587601+0000","last_became_active":"2024-09-21T08:28:53.602197+0000","last_became_peered":"2024-09-21T08:28:53.602197+0000","last_unstale":"2024-09-21T08:30:35.658224+0000","last_undegraded":"2024-09-21T08:28:53.578880+0000","last_fullsized":"2024-09-21T08:28:53.578772+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-09-21T08:27:30.806522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.806522+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078662+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":802,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.674589+0000","last_change":"2024-09-21T08:28:53.600847+0000","last_active":"2024-09-21T08:30:35.674589+0000","last_peered":"2024-09-21T08:30:35.674589+0000","last_clean":"2024-09-21T08:28:51.557075+0000","last_became_active":"2024-09-21T08:28:53.600847+0000","last_became_peered":"2024-09-21T08:28:53.600847+0000","last_unstale":"2024-09-21T08:30:35.674589+0000","last_undegraded":"2024-09-21T08:30:35.674589+0000","last_fullsized":"2024-09-21T08:28:53.578503+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.795325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704094+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":725,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.674342+0000","last_change":"2024-09-21T08:28:53.600413+0000","last_active":"2024-09-21T08:30:35.674342+0000","last_peered":"2024-09-21T08:30:35.674342+0000","last_clean":"2024-09-21T08:28:27.554070+0000","last_became_active":"2024-09-21T08:28:53.600413+0000","last_became_peered":"2024-09-21T08:28:53.600413+0000","last_unstale":"2024-09-21T08:30:35.674342+0000","last_undegraded":"2024-09-21T08:30:35.674342+0000","last_fullsized":"2024-09-21T08:28:53.576255+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:10.851510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.851510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704230+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.15","version":"428'444","reported_seq":2180,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.659886+0000","last_change":"2024-09-21T08:28:53.603910+0000","last_active":"2024-09-21T08:30:35.659886+0000","last_peered":"2024-09-21T08:30:35.659886+0000","last_clean":"2024-09-21T08:28:51.587746+0000","last_became_active":"2024-09-21T08:28:53.603910+0000","last_became_peered":"2024-09-21T08:28:53.603910+0000","last_unstale":"2024-09-21T08:30:35.659886+0000","last_undegraded":"2024-09-21T08:28:53.582931+0000","last_fullsized":"2024-09-21T08:28:53.582809+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'262","last_scrub_stamp":"2024-09-21T08:26:29.875830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.875830+0000","objects_scrubbed":3,"log_size":444,"log_dups_size":0,"ondisk_log_size":444,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":55,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.074129519000000005,"stat_sum":{"num_bytes":40135290,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":2,"num_read":150,"num_read_kb":148874,"num_write":89,"num_write_kb":39971,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":8623856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":20}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.16","version":"73'9","reported_seq":274,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.672891+0000","last_change":"2024-09-21T08:28:53.589261+0000","last_active":"2024-09-21T08:30:35.672891+0000","last_peered":"2024-09-21T08:30:35.672891+0000","last_clean":"2024-09-21T08:28:51.555305+0000","last_became_active":"2024-09-21T08:28:53.589261+0000","last_became_peered":"2024-09-21T08:28:53.589261+0000","last_unstale":"2024-09-21T08:30:35.672891+0000","last_undegraded":"2024-09-21T08:28:53.576380+0000","last_fullsized":"2024-09-21T08:28:53.576277+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:10.183035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.183035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704169+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":637,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.671200+0000","last_change":"2024-09-21T08:28:53.587227+0000","last_active":"2024-09-21T08:30:35.671200+0000","last_peered":"2024-09-21T08:30:35.671200+0000","last_clean":"2024-09-21T08:28:51.554027+0000","last_became_active":"2024-09-21T08:28:53.587227+0000","last_became_peered":"2024-09-21T08:28:53.587227+0000","last_unstale":"2024-09-21T08:30:35.671200+0000","last_undegraded":"2024-09-21T08:30:35.671200+0000","last_fullsized":"2024-09-21T08:28:53.573946+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:46.719545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:46.719545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703911+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":323,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.659754+0000","last_change":"2024-09-21T08:28:53.602669+0000","last_active":"2024-09-21T08:30:35.659754+0000","last_peered":"2024-09-21T08:30:35.659754+0000","last_clean":"2024-09-21T08:28:51.587829+0000","last_became_active":"2024-09-21T08:28:53.602669+0000","last_became_peered":"2024-09-21T08:28:53.602669+0000","last_unstale":"2024-09-21T08:30:35.659754+0000","last_undegraded":"2024-09-21T08:30:35.659754+0000","last_fullsized":"2024-09-21T08:28:53.580175+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:05.814753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.814753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078483+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":781,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.669020+0000","last_change":"2024-09-21T08:28:53.592444+0000","last_active":"2024-09-21T08:30:35.669020+0000","last_peered":"2024-09-21T08:30:35.669020+0000","last_clean":"2024-09-21T08:28:51.554300+0000","last_became_active":"2024-09-21T08:28:53.592444+0000","last_became_peered":"2024-09-21T08:28:53.592444+0000","last_unstale":"2024-09-21T08:30:35.669020+0000","last_undegraded":"2024-09-21T08:30:35.669020+0000","last_fullsized":"2024-09-21T08:28:53.578030+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:10.076564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.076564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:36.882820+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"426'170","reported_seq":1794,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.673514+0000","last_change":"2024-09-21T08:28:53.589835+0000","last_active":"2024-09-21T08:30:35.673514+0000","last_peered":"2024-09-21T08:30:35.673514+0000","last_clean":"2024-09-21T08:28:51.555217+0000","last_became_active":"2024-09-21T08:28:53.589835+0000","last_became_peered":"2024-09-21T08:28:53.589835+0000","last_unstale":"2024-09-21T08:30:35.673514+0000","last_undegraded":"2024-09-21T08:28:53.577195+0000","last_fullsized":"2024-09-21T08:28:53.577076+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'102","last_scrub_stamp":"2024-09-21T08:26:33.255256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.255256+0000","objects_scrubbed":1,"log_size":170,"log_dups_size":0,"ondisk_log_size":170,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011450665,"stat_sum":{"num_bytes":11938578,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":98,"num_read_kb":141433,"num_write":38,"num_write_kb":9384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11,"num_bytes_recovered":4027639,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.15","version":"73'9","reported_seq":866,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.674495+0000","last_change":"2024-09-21T08:28:53.601176+0000","last_active":"2024-09-21T08:30:35.674495+0000","last_peered":"2024-09-21T08:30:35.674495+0000","last_clean":"2024-09-21T08:28:51.554870+0000","last_became_active":"2024-09-21T08:28:53.601176+0000","last_became_peered":"2024-09-21T08:28:53.601176+0000","last_unstale":"2024-09-21T08:30:35.674495+0000","last_undegraded":"2024-09-21T08:28:53.577554+0000","last_fullsized":"2024-09-21T08:28:53.576455+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:28:14.535334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.535334+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704545+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":749,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.656573+0000","last_change":"2024-09-21T08:28:53.605117+0000","last_active":"2024-09-21T08:30:35.656573+0000","last_peered":"2024-09-21T08:30:35.656573+0000","last_clean":"2024-09-21T08:28:51.584984+0000","last_became_active":"2024-09-21T08:28:53.605117+0000","last_became_peered":"2024-09-21T08:28:53.605117+0000","last_unstale":"2024-09-21T08:30:35.656573+0000","last_undegraded":"2024-09-21T08:30:35.656573+0000","last_fullsized":"2024-09-21T08:28:53.584510+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.361096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.361096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078976+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"340'16","reported_seq":772,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.669511+0000","last_change":"2024-09-21T08:28:53.600719+0000","last_active":"2024-09-21T08:30:35.669511+0000","last_peered":"2024-09-21T08:30:35.669511+0000","last_clean":"2024-09-21T08:28:51.585913+0000","last_became_active":"2024-09-21T08:28:53.600719+0000","last_became_peered":"2024-09-21T08:28:53.600719+0000","last_unstale":"2024-09-21T08:30:35.669511+0000","last_undegraded":"2024-09-21T08:28:53.576377+0000","last_fullsized":"2024-09-21T08:28:53.576220+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078357+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":752,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.671571+0000","last_change":"2024-09-21T08:28:53.597499+0000","last_active":"2024-09-21T08:30:35.671571+0000","last_peered":"2024-09-21T08:30:35.671571+0000","last_clean":"2024-09-21T08:28:51.556626+0000","last_became_active":"2024-09-21T08:28:53.597499+0000","last_became_peered":"2024-09-21T08:28:53.597499+0000","last_unstale":"2024-09-21T08:30:35.671571+0000","last_undegraded":"2024-09-21T08:30:35.671571+0000","last_fullsized":"2024-09-21T08:28:53.585175+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.364388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.364388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:36.882874+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.17","version":"179'63","reported_seq":1086,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.674453+0000","last_change":"2024-09-21T08:28:53.590100+0000","last_active":"2024-09-21T08:30:35.674453+0000","last_peered":"2024-09-21T08:30:35.674453+0000","last_clean":"2024-09-21T08:28:51.556223+0000","last_became_active":"2024-09-21T08:28:53.590100+0000","last_became_peered":"2024-09-21T08:28:53.590100+0000","last_unstale":"2024-09-21T08:30:35.674453+0000","last_undegraded":"2024-09-21T08:30:35.674453+0000","last_fullsized":"2024-09-21T08:28:53.579666+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'63","last_scrub_stamp":"2024-09-21T08:28:13.511512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.511512+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010228017000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.14","version":"104'15","reported_seq":329,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.658129+0000","last_change":"2024-09-21T08:28:53.602117+0000","last_active":"2024-09-21T08:30:35.658129+0000","last_peered":"2024-09-21T08:30:35.658129+0000","last_clean":"2024-09-21T08:28:51.587113+0000","last_became_active":"2024-09-21T08:28:53.602117+0000","last_became_peered":"2024-09-21T08:28:53.602117+0000","last_unstale":"2024-09-21T08:30:35.658129+0000","last_undegraded":"2024-09-21T08:28:53.578695+0000","last_fullsized":"2024-09-21T08:28:53.578596+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078404+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":762,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.661314+0000","last_change":"2024-09-21T08:28:53.600512+0000","last_active":"2024-09-21T08:30:35.661314+0000","last_peered":"2024-09-21T08:30:35.661314+0000","last_clean":"2024-09-21T08:28:27.552826+0000","last_became_active":"2024-09-21T08:28:53.600512+0000","last_became_peered":"2024-09-21T08:28:53.600512+0000","last_unstale":"2024-09-21T08:30:35.661314+0000","last_undegraded":"2024-09-21T08:28:53.574940+0000","last_fullsized":"2024-09-21T08:28:53.574839+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:28:16.875544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.875544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078926+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.15","version":"340'14","reported_seq":911,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.675750+0000","last_change":"2024-09-21T08:28:53.601874+0000","last_active":"2024-09-21T08:30:35.675750+0000","last_peered":"2024-09-21T08:30:35.675750+0000","last_clean":"2024-09-21T08:28:51.555323+0000","last_became_active":"2024-09-21T08:28:53.601874+0000","last_became_peered":"2024-09-21T08:28:53.601874+0000","last_unstale":"2024-09-21T08:30:35.675750+0000","last_undegraded":"2024-09-21T08:28:53.578511+0000","last_fullsized":"2024-09-21T08:28:53.577750+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":633,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.673760+0000","last_change":"2024-09-21T08:28:53.600530+0000","last_active":"2024-09-21T08:30:35.673760+0000","last_peered":"2024-09-21T08:30:35.673760+0000","last_clean":"2024-09-21T08:28:51.555446+0000","last_became_active":"2024-09-21T08:28:53.600530+0000","last_became_peered":"2024-09-21T08:28:53.600530+0000","last_unstale":"2024-09-21T08:30:35.673760+0000","last_undegraded":"2024-09-21T08:30:35.673760+0000","last_fullsized":"2024-09-21T08:28:53.577413+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:48.251366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:48.251366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704081+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.10","version":"430'161","reported_seq":908,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.662965+0000","last_change":"2024-09-21T08:28:53.604012+0000","last_active":"2024-09-21T08:30:35.662965+0000","last_peered":"2024-09-21T08:30:35.662965+0000","last_clean":"2024-09-21T08:28:53.029949+0000","last_became_active":"2024-09-21T08:28:53.604012+0000","last_became_peered":"2024-09-21T08:28:53.604012+0000","last_unstale":"2024-09-21T08:30:35.662965+0000","last_undegraded":"2024-09-21T08:28:53.580392+0000","last_fullsized":"2024-09-21T08:28:53.580253+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'94","last_scrub_stamp":"2024-09-21T08:26:27.208234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.208234+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":55,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.075698227000000007,"stat_sum":{"num_bytes":12337203,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":21,"num_read_kb":13076,"num_write":38,"num_write_kb":17037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10770304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.13","version":"73'11","reported_seq":334,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.662277+0000","last_change":"2024-09-21T08:30:25.153811+0000","last_active":"2024-09-21T08:30:35.662277+0000","last_peered":"2024-09-21T08:30:35.662277+0000","last_clean":"2024-09-21T08:30:35.662277+0000","last_became_active":"2024-09-21T08:28:04.726766+0000","last_became_peered":"2024-09-21T08:28:04.726766+0000","last_unstale":"2024-09-21T08:30:35.662277+0000","last_undegraded":"2024-09-21T08:30:35.662277+0000","last_fullsized":"2024-09-21T08:30:35.662277+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:30:25.153729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:25.153729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:53.620587+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":695,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.672264+0000","last_change":"2024-09-21T08:28:53.587802+0000","last_active":"2024-09-21T08:30:35.672264+0000","last_peered":"2024-09-21T08:30:35.672264+0000","last_clean":"2024-09-21T08:28:27.553773+0000","last_became_active":"2024-09-21T08:28:53.587802+0000","last_became_peered":"2024-09-21T08:28:53.587802+0000","last_unstale":"2024-09-21T08:30:35.672264+0000","last_undegraded":"2024-09-21T08:30:35.672264+0000","last_fullsized":"2024-09-21T08:28:53.575420+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:06.889127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:06.889127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704122+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":317,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.658828+0000","last_change":"2024-09-21T08:28:53.600540+0000","last_active":"2024-09-21T08:30:35.658828+0000","last_peered":"2024-09-21T08:30:35.658828+0000","last_clean":"2024-09-21T08:28:51.585918+0000","last_became_active":"2024-09-21T08:28:53.600540+0000","last_became_peered":"2024-09-21T08:28:53.600540+0000","last_unstale":"2024-09-21T08:30:35.658828+0000","last_undegraded":"2024-09-21T08:30:35.658828+0000","last_fullsized":"2024-09-21T08:28:53.586917+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":520,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.672203+0000","last_change":"2024-09-21T08:30:23.816742+0000","last_active":"2024-09-21T08:30:35.672203+0000","last_peered":"2024-09-21T08:30:35.672203+0000","last_clean":"2024-09-21T08:30:35.672203+0000","last_became_active":"2024-09-21T08:28:29.611035+0000","last_became_peered":"2024-09-21T08:28:29.611035+0000","last_unstale":"2024-09-21T08:30:35.672203+0000","last_undegraded":"2024-09-21T08:30:35.672203+0000","last_fullsized":"2024-09-21T08:30:35.672203+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:23.816669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:23.816669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:53.328371+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"440'195","reported_seq":1656,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.660507+0000","last_change":"2024-09-21T08:28:53.605541+0000","last_active":"2024-09-21T08:30:35.660507+0000","last_peered":"2024-09-21T08:30:35.660507+0000","last_clean":"2024-09-21T08:28:52.665793+0000","last_became_active":"2024-09-21T08:28:53.605541+0000","last_became_peered":"2024-09-21T08:28:53.605541+0000","last_unstale":"2024-09-21T08:30:35.660507+0000","last_undegraded":"2024-09-21T08:28:53.581187+0000","last_fullsized":"2024-09-21T08:28:53.581055+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'44","last_scrub_stamp":"2024-09-21T08:26:27.882963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.882963+0000","objects_scrubbed":1,"log_size":195,"log_dups_size":0,"ondisk_log_size":195,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":54,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0806039080000001,"stat_sum":{"num_bytes":41919035,"num_objects":21,"num_object_clones":19,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":1,"num_read":76,"num_read_kb":68079,"num_write":95,"num_write_kb":40253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":11110430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"4.12","version":"73'9","reported_seq":340,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.658480+0000","last_change":"2024-09-21T08:28:29.696390+0000","last_active":"2024-09-21T08:30:35.658480+0000","last_peered":"2024-09-21T08:30:35.658480+0000","last_clean":"2024-09-21T08:30:35.658480+0000","last_became_active":"2024-09-21T08:28:29.636404+0000","last_became_peered":"2024-09-21T08:28:29.636404+0000","last_unstale":"2024-09-21T08:30:35.658480+0000","last_undegraded":"2024-09-21T08:30:35.658480+0000","last_fullsized":"2024-09-21T08:30:35.658480+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:14.398007+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.398007+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":643,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.672351+0000","last_change":"2024-09-21T08:28:53.587397+0000","last_active":"2024-09-21T08:30:35.672351+0000","last_peered":"2024-09-21T08:30:35.672351+0000","last_clean":"2024-09-21T08:28:51.554500+0000","last_became_active":"2024-09-21T08:28:53.587397+0000","last_became_peered":"2024-09-21T08:28:53.587397+0000","last_unstale":"2024-09-21T08:30:35.672351+0000","last_undegraded":"2024-09-21T08:30:35.672351+0000","last_fullsized":"2024-09-21T08:28:53.575265+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:49.790900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:49.790900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.704107+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":628,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.661357+0000","last_change":"2024-09-21T08:28:53.594733+0000","last_active":"2024-09-21T08:30:35.661357+0000","last_peered":"2024-09-21T08:30:35.661357+0000","last_clean":"2024-09-21T08:28:51.584914+0000","last_became_active":"2024-09-21T08:28:53.594733+0000","last_became_peered":"2024-09-21T08:28:53.594733+0000","last_unstale":"2024-09-21T08:30:35.661357+0000","last_undegraded":"2024-09-21T08:30:35.661357+0000","last_fullsized":"2024-09-21T08:28:53.576121+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:13.940546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:13.940546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078655+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":734,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.672184+0000","last_change":"2024-09-21T08:30:02.065158+0000","last_active":"2024-09-21T08:30:35.672184+0000","last_peered":"2024-09-21T08:30:35.672184+0000","last_clean":"2024-09-21T08:30:35.672184+0000","last_became_active":"2024-09-21T08:28:29.610998+0000","last_became_peered":"2024-09-21T08:28:29.610998+0000","last_unstale":"2024-09-21T08:30:35.672184+0000","last_undegraded":"2024-09-21T08:30:35.672184+0000","last_fullsized":"2024-09-21T08:30:35.672184+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:02.065082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:02.065082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:24.858435+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.12","version":"430'400","reported_seq":2124,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.671503+0000","last_change":"2024-09-21T08:28:53.587108+0000","last_active":"2024-09-21T08:30:35.671503+0000","last_peered":"2024-09-21T08:30:35.671503+0000","last_clean":"2024-09-21T08:28:52.589762+0000","last_became_active":"2024-09-21T08:28:53.587108+0000","last_became_peered":"2024-09-21T08:28:53.587108+0000","last_unstale":"2024-09-21T08:30:35.671503+0000","last_undegraded":"2024-09-21T08:28:53.574106+0000","last_fullsized":"2024-09-21T08:28:53.573924+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'200","last_scrub_stamp":"2024-09-21T08:26:27.417028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.417028+0000","objects_scrubbed":3,"log_size":400,"log_dups_size":0,"ondisk_log_size":400,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.027231871000000001,"stat_sum":{"num_bytes":39611945,"num_objects":20,"num_object_clones":16,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":20,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":3,"num_read":135,"num_read_kb":160879,"num_write":93,"num_write_kb":44529,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":8981699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":20}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.11","version":"73'11","reported_seq":358,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.672427+0000","last_change":"2024-09-21T08:28:53.589901+0000","last_active":"2024-09-21T08:30:35.672427+0000","last_peered":"2024-09-21T08:30:35.672427+0000","last_clean":"2024-09-21T08:28:51.555481+0000","last_became_active":"2024-09-21T08:28:53.589901+0000","last_became_peered":"2024-09-21T08:28:53.589901+0000","last_unstale":"2024-09-21T08:30:35.672427+0000","last_undegraded":"2024-09-21T08:28:53.577098+0000","last_fullsized":"2024-09-21T08:28:53.576477+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703757+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":798,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.673938+0000","last_change":"2024-09-21T08:28:53.600628+0000","last_active":"2024-09-21T08:30:35.673938+0000","last_peered":"2024-09-21T08:30:35.673938+0000","last_clean":"2024-09-21T08:28:51.555529+0000","last_became_active":"2024-09-21T08:28:53.600628+0000","last_became_peered":"2024-09-21T08:28:53.600628+0000","last_unstale":"2024-09-21T08:30:35.673938+0000","last_undegraded":"2024-09-21T08:30:35.673938+0000","last_fullsized":"2024-09-21T08:28:53.577666+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703897+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":730,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.672887+0000","last_change":"2024-09-21T08:28:53.589537+0000","last_active":"2024-09-21T08:30:35.672887+0000","last_peered":"2024-09-21T08:30:35.672887+0000","last_clean":"2024-09-21T08:28:27.552805+0000","last_became_active":"2024-09-21T08:28:53.589537+0000","last_became_peered":"2024-09-21T08:28:53.589537+0000","last_unstale":"2024-09-21T08:30:35.672887+0000","last_undegraded":"2024-09-21T08:30:35.672887+0000","last_fullsized":"2024-09-21T08:28:53.575446+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.670110+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":526,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.673401+0000","last_change":"2024-09-21T08:29:58.167276+0000","last_active":"2024-09-21T08:30:35.673401+0000","last_peered":"2024-09-21T08:30:35.673401+0000","last_clean":"2024-09-21T08:30:35.673401+0000","last_became_active":"2024-09-21T08:28:29.619814+0000","last_became_peered":"2024-09-21T08:28:29.619814+0000","last_unstale":"2024-09-21T08:30:35.673401+0000","last_undegraded":"2024-09-21T08:30:35.673401+0000","last_fullsized":"2024-09-21T08:30:35.673401+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:58.167184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.167184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:02.224715+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.13","version":"338'55","reported_seq":1622,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.671896+0000","last_change":"2024-09-21T08:28:53.588606+0000","last_active":"2024-09-21T08:30:35.671896+0000","last_peered":"2024-09-21T08:30:35.671896+0000","last_clean":"2024-09-21T08:28:52.586222+0000","last_became_active":"2024-09-21T08:28:53.588606+0000","last_became_peered":"2024-09-21T08:28:53.588606+0000","last_unstale":"2024-09-21T08:30:35.671896+0000","last_undegraded":"2024-09-21T08:28:53.574192+0000","last_fullsized":"2024-09-21T08:28:53.574061+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'22","last_scrub_stamp":"2024-09-21T08:26:29.353795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.353795+0000","objects_scrubbed":0,"log_size":55,"log_dups_size":0,"ondisk_log_size":55,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019195647,"stat_sum":{"num_bytes":9586927,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":45,"num_read_kb":64333,"num_write":19,"num_write_kb":14200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5630444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.10","version":"73'4","reported_seq":328,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.662054+0000","last_change":"2024-09-21T08:29:39.416471+0000","last_active":"2024-09-21T08:30:35.662054+0000","last_peered":"2024-09-21T08:30:35.662054+0000","last_clean":"2024-09-21T08:30:35.662054+0000","last_became_active":"2024-09-21T08:28:04.724607+0000","last_became_peered":"2024-09-21T08:28:04.724607+0000","last_unstale":"2024-09-21T08:30:35.662054+0000","last_undegraded":"2024-09-21T08:30:35.662054+0000","last_fullsized":"2024-09-21T08:30:35.662054+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-09-21T08:29:39.416397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.416397+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:58.505501+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":766,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.673836+0000","last_change":"2024-09-21T08:30:24.826266+0000","last_active":"2024-09-21T08:30:35.673836+0000","last_peered":"2024-09-21T08:30:35.673836+0000","last_clean":"2024-09-21T08:30:35.673836+0000","last_became_active":"2024-09-21T08:28:29.611592+0000","last_became_peered":"2024-09-21T08:28:29.611592+0000","last_unstale":"2024-09-21T08:30:35.673836+0000","last_undegraded":"2024-09-21T08:30:35.673836+0000","last_fullsized":"2024-09-21T08:30:35.673836+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:24.826198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:30:24.826198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:28.562894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":709,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.656298+0000","last_change":"2024-09-21T08:28:53.589633+0000","last_active":"2024-09-21T08:30:35.656298+0000","last_peered":"2024-09-21T08:30:35.656298+0000","last_clean":"2024-09-21T08:28:51.584672+0000","last_became_active":"2024-09-21T08:28:53.589633+0000","last_became_peered":"2024-09-21T08:28:53.589633+0000","last_unstale":"2024-09-21T08:30:35.656298+0000","last_undegraded":"2024-09-21T08:30:35.656298+0000","last_fullsized":"2024-09-21T08:28:53.574228+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":784,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.656833+0000","last_change":"2024-09-21T08:28:53.605159+0000","last_active":"2024-09-21T08:30:35.656833+0000","last_peered":"2024-09-21T08:30:35.656833+0000","last_clean":"2024-09-21T08:28:51.585029+0000","last_became_active":"2024-09-21T08:28:53.605159+0000","last_became_peered":"2024-09-21T08:28:53.605159+0000","last_unstale":"2024-09-21T08:30:35.656833+0000","last_undegraded":"2024-09-21T08:30:35.656833+0000","last_fullsized":"2024-09-21T08:28:53.584509+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.736816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.736816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078633+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1c","version":"407'212","reported_seq":1873,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.672619+0000","last_change":"2024-09-21T08:28:53.600314+0000","last_active":"2024-09-21T08:30:35.672619+0000","last_peered":"2024-09-21T08:30:35.672619+0000","last_clean":"2024-09-21T08:28:51.556402+0000","last_became_active":"2024-09-21T08:28:53.600314+0000","last_became_peered":"2024-09-21T08:28:53.600314+0000","last_unstale":"2024-09-21T08:30:35.672619+0000","last_undegraded":"2024-09-21T08:28:53.575998+0000","last_fullsized":"2024-09-21T08:28:53.575875+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'167","last_scrub_stamp":"2024-09-21T08:27:49.143452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.143452+0000","objects_scrubbed":0,"log_size":212,"log_dups_size":0,"ondisk_log_size":212,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":53,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.015256377999999999,"stat_sum":{"num_bytes":17321215,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":73,"num_read_kb":97090,"num_write":25,"num_write_kb":14712,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1129140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"6.1d","version":"0'0","reported_seq":322,"reported_epoch":442,"state":"active+clean","last_fresh":"2024-09-21T08:30:35.658341+0000","last_change":"2024-09-21T08:30:28.069522+0000","last_active":"2024-09-21T08:30:35.658341+0000","last_peered":"2024-09-21T08:30:35.658341+0000","last_clean":"2024-09-21T08:30:35.658341+0000","last_became_active":"2024-09-21T08:28:04.724461+0000","last_became_peered":"2024-09-21T08:28:04.724461+0000","last_unstale":"2024-09-21T08:30:35.658341+0000","last_undegraded":"2024-09-21T08:30:35.658341+0000","last_fullsized":"2024-09-21T08:30:35.658341+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:28.069450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:28.069450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:42.395335+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":637,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.656981+0000","last_change":"2024-09-21T08:28:53.585838+0000","last_active":"2024-09-21T08:30:35.656981+0000","last_peered":"2024-09-21T08:30:35.656981+0000","last_clean":"2024-09-21T08:28:51.584499+0000","last_became_active":"2024-09-21T08:28:53.585838+0000","last_became_peered":"2024-09-21T08:28:53.585838+0000","last_unstale":"2024-09-21T08:30:35.656981+0000","last_undegraded":"2024-09-21T08:30:35.656981+0000","last_fullsized":"2024-09-21T08:28:53.575428+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":833,"reported_epoch":442,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:30:35.675027+0000","last_change":"2024-09-21T08:28:53.603028+0000","last_active":"2024-09-21T08:30:35.675027+0000","last_peered":"2024-09-21T08:30:35.675027+0000","last_clean":"2024-09-21T08:28:51.556942+0000","last_became_active":"2024-09-21T08:28:53.603028+0000","last_became_peered":"2024-09-21T08:28:53.603028+0000","last_unstale":"2024-09-21T08:30:35.675027+0000","last_undegraded":"2024-09-21T08:28:53.581121+0000","last_fullsized":"2024-09-21T08:28:53.581005+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:37.703785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":215,"reported_epoch":442,"state":"active+undersized","last_fresh":"2024-09-21T08:30:35.658540+0000","last_change":"2024-09-21T08:28:53.601908+0000","last_active":"2024-09-21T08:30:35.658540+0000","last_peered":"2024-09-21T08:30:35.658540+0000","last_clean":"2024-09-21T08:23:33.544062+0000","last_became_active":"2024-09-21T08:28:53.601908+0000","last_became_peered":"2024-09-21T08:28:53.601908+0000","last_unstale":"2024-09-21T08:30:35.658540+0000","last_undegraded":"2024-09-21T08:30:35.658540+0000","last_fullsized":"2024-09-21T08:28:53.577043+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.872291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.872291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:30:40.078991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":574542467,"num_objects":274,"num_object_clones":225,"num_object_copies":822,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":219,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":274,"num_whiteouts":14,"num_read":1739,"num_read_kb":1902634,"num_write":1236,"num_write_kb":513061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":539,"num_bytes_recovered":207157568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1291403264,"data_stored":1838715386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7531,"ondisk_log_size":7531,"up":79,"acting":79,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":76,"acting":76,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":116,"ondisk_log_size":116,"up":67,"acting":67,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":127,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":676,"num_read_kb":433,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":509,"num_bytes_recovered":10760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":647168,"data_stored":17316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":426,"ondisk_log_size":426,"up":74,"acting":74,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":4282,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":8622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":70,"acting":70,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":402,"num_read_kb":397,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":1670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":6,"acting":6,"num_store_stats":7},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":3542208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":129,"ondisk_log_size":129,"up":2,"acting":2,"num_store_stats":5}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070533,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":228312,"kb_used_data":174916,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93516840,"statfs":{"total":95995035648,"available":95761244160,"internally_reserved":0,"allocated":179113984,"data_stored":227784827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397600,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659235,"num_pgs":124,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":320568,"kb_used_data":263012,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93424584,"statfs":{"total":95995035648,"available":95666774016,"internally_reserved":0,"allocated":269324288,"data_stored":393186743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":0,"seq":133143986281,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182509,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411443,"num_pgs":115,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":419308,"kb_used_data":361752,"kb_used_omap":1,"kb_used_meta":57534,"kb_avail":93325844,"statfs":{"total":95995035648,"available":95565664256,"internally_reserved":0,"allocated":370434048,"data_stored":561300095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":58915269},"hb_peers":[1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574967,"num_pgs":135,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":333204,"kb_used_data":302008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93411948,"statfs":{"total":95995035648,"available":95653834752,"internally_reserved":0,"allocated":309256192,"data_stored":459900796,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":3482,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":1898,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1474,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":3014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74776576,"data_stored":86348592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":306946048,"data_stored":460031613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":368193536,"data_stored":561432203,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70848512,"data_stored":84468350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":266407936,"data_stored":391521903,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":176136192,"data_stored":227042310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:30:42.492 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T08:30:42.492 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pool set .mgr pg_num 11 2024-09-21T08:30:42.493 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:42.505 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:42.505 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:42.506 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:42.538 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:42.585 INFO:tasks.rados.rados.0.smithi137.stdout:1772: done (1 left) 2024-09-21T08:30:42.585 INFO:tasks.rados.rados.0.smithi137.stdout:1773: done (0 left) 2024-09-21T08:30:42.585 INFO:tasks.rados.rados.0.smithi137.stdout:1774: rollback oid 38 current snap is 188 2024-09-21T08:30:42.585 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 38 to 183 2024-09-21T08:30:42.585 INFO:tasks.rados.rados.0.smithi137.stdout:1775: delete oid 36 current snap is 188 2024-09-21T08:30:42.595 INFO:tasks.rados.rados.0.smithi137.stdout:1774: finishing rollback tid 0 to smithi13779221-38 2024-09-21T08:30:42.596 INFO:tasks.rados.rados.0.smithi137.stdout:1774: finishing rollback tid 1 to smithi13779221-38 2024-09-21T08:30:42.600 INFO:tasks.rados.rados.0.smithi137.stdout:1774: finishing rollback tid 2 to smithi13779221-38 2024-09-21T08:30:42.600 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 729 (ObjNum 552 snap 166 seq_num 552) dirty exists 2024-09-21T08:30:42.600 INFO:tasks.rados.rados.0.smithi137.stdout:1774: done (1 left) 2024-09-21T08:30:42.600 INFO:tasks.rados.rados.0.smithi137.stdout:1775: done (0 left) 2024-09-21T08:30:42.600 INFO:tasks.rados.rados.0.smithi137.stdout:1776: delete oid 19 current snap is 188 2024-09-21T08:30:42.602 INFO:tasks.rados.rados.0.smithi137.stdout:1776: done (0 left) 2024-09-21T08:30:42.602 INFO:tasks.rados.rados.0.smithi137.stdout:1777: rollback oid 19 current snap is 188 2024-09-21T08:30:42.602 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 19 to 151 2024-09-21T08:30:42.602 INFO:tasks.rados.rados.0.smithi137.stdout:1778: read oid 9 snap -1 2024-09-21T08:30:42.602 INFO:tasks.rados.rados.0.smithi137.stdout:1778: expect deleted 2024-09-21T08:30:42.602 INFO:tasks.rados.rados.0.smithi137.stdout:1779: snap_create 2024-09-21T08:30:42.603 INFO:tasks.rados.rados.0.smithi137.stdout:1777: finishing rollback tid 1 to smithi13779221-19 2024-09-21T08:30:42.603 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 511 (ObjNum 129 snap 0 seq_num 3760546176) dirty dne 2024-09-21T08:30:42.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:42.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:42.631 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:42.631 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:42.732 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:42.744 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:42 smithi137 ceph-mon[29634]: from='client.45767 v1:172.21.15.137:0/1029368649' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:30:42.744 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:42 smithi137 ceph-mon[29634]: 7.a scrub starts 2024-09-21T08:30:42.744 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:42 smithi137 ceph-mon[29634]: 7.a scrub ok 2024-09-21T08:30:42.744 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:42 smithi137 ceph-mon[29634]: osdmap e449: 8 total, 6 up, 4 in 2024-09-21T08:30:42.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:42 smithi137 ceph-mon[41044]: from='client.45767 v1:172.21.15.137:0/1029368649' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:30:42.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:42 smithi137 ceph-mon[41044]: 7.a scrub starts 2024-09-21T08:30:42.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:42 smithi137 ceph-mon[41044]: 7.a scrub ok 2024-09-21T08:30:42.745 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:42 smithi137 ceph-mon[41044]: osdmap e449: 8 total, 6 up, 4 in 2024-09-21T08:30:42.756 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:42.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:42.756 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:42.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:42.880 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:42.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:42.881 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:42.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:43.003 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:43.004 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:43.004 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:42 smithi165 ceph-mon[32893]: from='client.45767 v1:172.21.15.137:0/1029368649' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:30:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:42 smithi165 ceph-mon[32893]: 7.a scrub starts 2024-09-21T08:30:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:42 smithi165 ceph-mon[32893]: 7.a scrub ok 2024-09-21T08:30:43.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:42 smithi165 ceph-mon[32893]: osdmap e449: 8 total, 6 up, 4 in 2024-09-21T08:30:43.105 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:43.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:43.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:43.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:43.175 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:43.188 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:30:43.188 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:30:43.188 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14154.0:1 1.0 1:30306672:devicehealth::main.db.0000000000000000:head [create,setxattr striper.version (16) in=15b,setxattr striper.excl in=12b,setxattr striper.size (16) in=12b,setxattr striper.allocated (16) in=17b,setxattr striper.layout.stripe_unit (16) in=26b,setxattr striper.layout.stripe_count (16) in=27b,setxattr striper.layout.object_size (16) in=26b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e20)", 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 566.71193825099999, 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27973036899999998, 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14154", 2024-09-21T08:30:43.189 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/1718301516", 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473699+0000", 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.190 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473701+0000", 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1320000000000001e-06 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473716+0000", 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5622999999999999e-05 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.191 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473730+0000", 2024-09-21T08:30:43.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3259e-05 2024-09-21T08:30:43.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.192 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.192 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473740+0000", 2024-09-21T08:30:43.192 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0746e-05 2024-09-21T08:30:43.192 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.192 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473802+0000", 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2043000000000002e-05 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.473810+0000", 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1500000000000002e-06 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.193 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752339+0000", 2024-09-21T08:30:43.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.27852951199999998 2024-09-21T08:30:43.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.194 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752485+0000", 2024-09-21T08:30:43.194 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014628699999999999 2024-09-21T08:30:43.194 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.194 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.194 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:30:43.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.752609+0000", 2024-09-21T08:30:43.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012376499999999999 2024-09-21T08:30:43.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.195 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753145+0000", 2024-09-21T08:30:43.195 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00053571000000000001 2024-09-21T08:30:43.195 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.195 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.195 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753326+0000", 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018102799999999999 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753367+0000", 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1253e-05 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.196 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753400+0000", 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.3065999999999998e-05 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:21:16.753429+0000", 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.8795000000000001e-05 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.197 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:422 7.d 7:b31d99ce:::smithi13779221-21:head [write 443718~615603] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 346.68066126899998, 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034885076000000001, 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 422 2024-09-21T08:30:43.198 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.199 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.199 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:30:43.199 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.199 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.199 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.199 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:30:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504978+0000", 2024-09-21T08:30:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0090000000000002e-06 2024-09-21T08:30:43.200 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506268+0000", 2024-09-21T08:30:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001290444 2024-09-21T08:30:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.201 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506279+0000", 2024-09-21T08:30:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0526999999999999e-05 2024-09-21T08:30:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506288+0000", 2024-09-21T08:30:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5470000000000002e-06 2024-09-21T08:30:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.202 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506343+0000", 2024-09-21T08:30:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5287000000000001e-05 2024-09-21T08:30:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506354+0000", 2024-09-21T08:30:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0798999999999999e-05 2024-09-21T08:30:43.203 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.525308+0000", 2024-09-21T08:30:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018953616 2024-09-21T08:30:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:43.204 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529390+0000", 2024-09-21T08:30:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040819029999999996 2024-09-21T08:30:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533318+0000", 2024-09-21T08:30:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039282190000000002 2024-09-21T08:30:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.205 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533380+0000", 2024-09-21T08:30:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1890000000000005e-05 2024-09-21T08:30:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533709+0000", 2024-09-21T08:30:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032948400000000001 2024-09-21T08:30:43.206 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536916+0000", 2024-09-21T08:30:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032063399999999998 2024-09-21T08:30:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539553+0000", 2024-09-21T08:30:43.207 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026370909999999998 2024-09-21T08:30:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539658+0000", 2024-09-21T08:30:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105585 2024-09-21T08:30:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.208 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539723+0000", 2024-09-21T08:30:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4343000000000004e-05 2024-09-21T08:30:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539861+0000", 2024-09-21T08:30:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137992 2024-09-21T08:30:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.209 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:423 7.d 7:b31d99ce:::smithi13779221-21:head [write 1564138~470341] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:30:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 346.67927160099998, 2024-09-21T08:30:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033782404000000002, 2024-09-21T08:30:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.210 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 423 2024-09-21T08:30:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.211 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:30:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.212 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:30:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506366+0000", 2024-09-21T08:30:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0449999999999999e-06 2024-09-21T08:30:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.213 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507012+0000", 2024-09-21T08:30:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064568899999999996 2024-09-21T08:30:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507021+0000", 2024-09-21T08:30:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2550000000000003e-06 2024-09-21T08:30:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.214 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507029+0000", 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2559999999999996e-06 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507079+0000", 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0957e-05 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507088+0000", 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3650000000000004e-06 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.215 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529457+0000", 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022369185999999999 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529526+0000", 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8683999999999995e-05 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535157+0000", 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056315849999999997 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.216 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535244+0000", 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6840000000000002e-05 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535398+0000", 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015404399999999999 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539569+0000", 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004170361 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.217 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539613+0000", 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4101999999999999e-05 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539985+0000", 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037279300000000001 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540037+0000", 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1345999999999999e-05 2024-09-21T08:30:43.218 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540148+0000", 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110896 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:424 7.d 7:b31d99ce:::smithi13779221-21:head [write 2633066~610652] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:43.219 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 346.67228258699998, 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027015847999999999, 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 424 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.220 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.221 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513356+0000", 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8789999999999999e-06 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513753+0000", 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039670999999999999 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513764+0000", 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.154e-05 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.222 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513772+0000", 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9149999999999996e-06 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513833+0000", 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1072000000000001e-05 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:43.223 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513843+0000", 2024-09-21T08:30:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8349999999999999e-06 2024-09-21T08:30:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529644+0000", 2024-09-21T08:30:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015801084 2024-09-21T08:30:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.224 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529688+0000", 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3568e-05 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536031+0000", 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063434659999999999 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536135+0000", 2024-09-21T08:30:43.226 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103599 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536291+0000", 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155668 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539575+0000", 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032840450000000002 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539925+0000", 2024-09-21T08:30:43.227 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034988899999999998 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540200+0000", 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027585000000000001 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540250+0000", 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9413000000000003e-05 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540370+0000", 2024-09-21T08:30:43.228 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120315 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:425 7.d 7:b31d99ce:::smithi13779221-21:head [setxattr _header (56) in=7b,truncate 3243718] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 346.671779947, 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030130239, 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.229 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 425 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.230 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513858+0000", 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9199999999999999e-07 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513863+0000", 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7650000000000001e-06 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.231 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513869+0000", 2024-09-21T08:30:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9880000000000001e-06 2024-09-21T08:30:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513874+0000", 2024-09-21T08:30:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.643e-06 2024-09-21T08:30:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.232 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515915+0000", 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020407279999999999 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515925+0000", 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0051000000000001e-05 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529722+0000", 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013796780999999999 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:43.233 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529804+0000", 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1663999999999996e-05 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540423+0000", 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010619635000000001 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540671+0000", 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024766600000000001 2024-09-21T08:30:43.234 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540795+0000", 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012383899999999999 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541324+0000", 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052910200000000004 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.542654+0000", 2024-09-21T08:30:43.235 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013304370000000001 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543785+0000", 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001130603 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543841+0000", 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5939000000000001e-05 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.236 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543987+0000", 2024-09-21T08:30:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014640599999999999 2024-09-21T08:30:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.237 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:426 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:30:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 346.66982100199999, 2024-09-21T08:30:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028482668999999999, 2024-09-21T08:30:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:43.238 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 426 2024-09-21T08:30:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.239 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:30:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:30:43.240 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515818+0000", 2024-09-21T08:30:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7999999999999999e-06 2024-09-21T08:30:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.241 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515825+0000", 2024-09-21T08:30:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0790000000000003e-06 2024-09-21T08:30:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.242 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515840+0000", 2024-09-21T08:30:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4765e-05 2024-09-21T08:30:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.243 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515857+0000", 2024-09-21T08:30:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7414999999999998e-05 2024-09-21T08:30:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515968+0000", 2024-09-21T08:30:43.244 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111161 2024-09-21T08:30:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515975+0000", 2024-09-21T08:30:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7499999999999997e-06 2024-09-21T08:30:43.245 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529841+0000", 2024-09-21T08:30:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013866387000000001 2024-09-21T08:30:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.246 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529951+0000", 2024-09-21T08:30:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010953100000000001 2024-09-21T08:30:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.247 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541110+0000", 2024-09-21T08:30:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011159591 2024-09-21T08:30:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:30:43.248 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541190+0000", 2024-09-21T08:30:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9243e-05 2024-09-21T08:30:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.249 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544032+0000", 2024-09-21T08:30:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028424050000000001 2024-09-21T08:30:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544118+0000", 2024-09-21T08:30:43.250 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5798999999999996e-05 2024-09-21T08:30:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544299+0000", 2024-09-21T08:30:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180743 2024-09-21T08:30:43.251 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:430 7.d 7:b0b32695:::smithi13779221-19:head [delete] snapc a=[9] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:30:43.252 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 346.66965032399997, 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028508957000000001, 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 430 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.253 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:30:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:30:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.254 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:30:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1900000000000001e-07 2024-09-21T08:30:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515991+0000", 2024-09-21T08:30:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6899999999999998e-06 2024-09-21T08:30:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.255 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515997+0000", 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2999999999999998e-06 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516003+0000", 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0220000000000001e-06 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.256 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516045+0000", 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1069000000000003e-05 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516052+0000", 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9890000000000003e-06 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529980+0000", 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013927999999999999 2024-09-21T08:30:43.257 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.532305+0000", 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023250580000000001 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541672+0000", 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093675080000000001 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541749+0000", 2024-09-21T08:30:43.258 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6829000000000006e-05 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541934+0000", 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018531 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543560+0000", 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00162525 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543661+0000", 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101604 2024-09-21T08:30:43.259 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544339+0000", 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067745299999999995 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544394+0000", 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5515e-05 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544496+0000", 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010144100000000001 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.260 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:571 7.1 7:88bf4ad2:::smithi13779221-6:head [copy-from ver 13] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 332.63858367, 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037746406000000003, 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 571 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.261 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547056+0000", 2024-09-21T08:30:43.262 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.146e-06 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547067+0000", 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0609e-05 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547090+0000", 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2714999999999999e-05 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.263 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547100+0000", 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9920000000000002e-06 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547203+0000", 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102828 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547210+0000", 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3629999999999996e-06 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.264 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553577+0000", 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063672829999999996 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553669+0000", 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1856999999999998e-05 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573941+0000", 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020272168 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.265 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.583019+0000", 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090777630000000008 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584643+0000", 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016245319999999999 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584686+0000", 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2259999999999999e-05 2024-09-21T08:30:43.266 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584748+0000", 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2049000000000002e-05 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584800+0000", 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1841000000000001e-05 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.267 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:572 7.1 7:88bf4ad2:::smithi13779221-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 332.63851464099997, 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032210599999999999, 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 572 2024-09-21T08:30:43.268 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.269 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547123+0000", 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5300000000000003e-07 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547138+0000", 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4718999999999999e-05 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.270 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547145+0000", 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2520000000000002e-06 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547151+0000", 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7649999999999999e-06 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547263+0000", 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112001 2024-09-21T08:30:43.271 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547269+0000", 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8919999999999997e-06 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553824+0000", 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065557499999999999 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.272 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553873+0000", 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8727999999999998e-05 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579133+0000", 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025259964999999999 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579227+0000", 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3517999999999995e-05 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.273 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579333+0000", 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010615699999999999 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3155 7.9 7:9247b5b1:::smithi13779221-17:head [copy-from ver 183] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 130.613140827, 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029797267999999998, 2024-09-21T08:30:43.274 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3155 2024-09-21T08:30:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.275 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.277 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572497+0000", 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4899999999999995e-07 2024-09-21T08:30:43.278 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572500+0000", 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.05e-06 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572505+0000", 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2820000000000004e-06 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572510+0000", 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8969999999999999e-06 2024-09-21T08:30:43.279 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573522+0000", 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001012481 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573635+0000", 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112253 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:30:43.280 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.583897+0000", 2024-09-21T08:30:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010262288 2024-09-21T08:30:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594011+0000", 2024-09-21T08:30:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010113641 2024-09-21T08:30:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.281 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594202+0000", 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000191418 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602174+0000", 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079717079999999992 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602233+0000", 2024-09-21T08:30:43.282 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9565999999999999e-05 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602293+0000", 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0134999999999999e-05 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3441 7.15 7:a9dd6c7d:::smithi13779221-47:head [write 783325~557161,stat] snapc 84=[84,83,82,81,7f,7e,7d,7c,78,77,72,71,69] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 106.62229099300001, 2024-09-21T08:30:43.283 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028447395, 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3441 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.284 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563352+0000", 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9039999999999997e-06 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563878+0000", 2024-09-21T08:30:43.285 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052587200000000004 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563908+0000", 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0685999999999997e-05 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563922+0000", 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3738e-05 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.286 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.564007+0000", 2024-09-21T08:30:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5283000000000003e-05 2024-09-21T08:30:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.564017+0000", 2024-09-21T08:30:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4909999999999994e-06 2024-09-21T08:30:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.287 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571275+0000", 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072579860000000001 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571419+0000", 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014457700000000001 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571666+0000", 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024687599999999999 2024-09-21T08:30:43.288 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591189+0000", 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019522923000000001 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591633+0000", 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044407499999999999 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591717+0000", 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4010000000000004e-05 2024-09-21T08:30:43.289 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591793+0000", 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5974000000000001e-05 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4000 7.b 7:d6c5a045:::smithi13779221-46:head [copy-from ver 529] snapc a7=[a7,a6,a5,a4,a3,a2,a1,9e,9d,9c,9b,9a,99,98,94,93,92,91,90,8c,8b,88,86,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 51.617312646999999, 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037495776000000001, 2024-09-21T08:30:43.290 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4000 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.291 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568328+0000", 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.349e-06 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568346+0000", 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8839e-05 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.292 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568362+0000", 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5432999999999999e-05 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568371+0000", 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3009999999999995e-06 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.293 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568435+0000", 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4199e-05 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568441+0000", 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0050000000000001e-06 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.575820+0000", 2024-09-21T08:30:43.294 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073784970000000004 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.575934+0000", 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011455699999999999 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587076+0000", 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011141287 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.295 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.595892+0000", 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088164750000000007 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605259+0000", 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093663830000000007 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605662+0000", 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040333900000000001 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.296 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605744+0000", 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2560999999999998e-05 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605820+0000", 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5550999999999995e-05 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.297 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4168 7.b 7:d7506421:::smithi13779221-13:head [write 447218~719401,stat] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.581231+0000", 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 40.604405757999999, 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.085401058000000002, 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4168 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.298 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581231+0000", 2024-09-21T08:30:43.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.299 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.299 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.299 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.299 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581231+0000", 2024-09-21T08:30:43.299 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581234+0000", 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9909999999999998e-06 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581576+0000", 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000342109 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.300 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581675+0000", 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8947e-05 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581686+0000", 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1039999999999999e-05 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582031+0000", 2024-09-21T08:30:43.301 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000344333 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582110+0000", 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9197999999999996e-05 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582254+0000", 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000143881 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.302 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.584348+0000", 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020943440000000002 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.585410+0000", 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001061782 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.666486+0000", 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081075990000000001 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.303 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.666565+0000", 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9076e-05 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.666632+0000", 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7366999999999994e-05 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.304 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4169 7.b 7:d7506421:::smithi13779221-13:head [write 1889370~660531] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.588798+0000", 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 40.596839291000002, 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081088987000000001, 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4169 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.305 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588798+0000", 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588798+0000", 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588800+0000", 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.903e-06 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.306 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588978+0000", 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017865599999999999 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588995+0000", 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6696000000000001e-05 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.307 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589003+0000", 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8520000000000004e-06 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589056+0000", 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3665000000000002e-05 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589145+0000", 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8455000000000006e-05 2024-09-21T08:30:43.308 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589276+0000", 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000131032 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.590553+0000", 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001277572 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.591751+0000", 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001197137 2024-09-21T08:30:43.309 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669765+0000", 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.078014230000000004 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669824+0000", 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9651999999999997e-05 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.310 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669887+0000", 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2137000000000005e-05 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4170 7.b 7:d7506421:::smithi13779221-13:head [write 2963386~614253] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.595002+0000", 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 40.590635177999999, 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.075006211000000003, 2024-09-21T08:30:43.311 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4170 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595002+0000", 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.312 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595002+0000", 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595004+0000", 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4559999999999999e-06 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595379+0000", 2024-09-21T08:30:43.313 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037446400000000003 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595396+0000", 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7303999999999999e-05 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595403+0000", 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7689999999999999e-06 2024-09-21T08:30:43.314 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595459+0000", 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6373999999999999e-05 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595547+0000", 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7869999999999997e-05 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:30:43.315 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595681+0000", 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133857 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599798+0000", 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041171790000000003 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599868+0000", 2024-09-21T08:30:43.316 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9839000000000007e-05 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669931+0000", 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070063149000000005 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669973+0000", 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1838999999999997e-05 2024-09-21T08:30:43.317 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670008+0000", 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5111000000000003e-05 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4171 7.b 7:d7506421:::smithi13779221-13:head [setxattr _header (58) in=7b,truncate 3647218] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.595420+0000", 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 40.590217162999998, 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074711124000000004, 2024-09-21T08:30:43.318 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4171 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595420+0000", 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.319 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595420+0000", 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595421+0000", 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3099999999999996e-07 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595425+0000", 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3109999999999999e-06 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.320 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595431+0000", 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3600000000000001e-06 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595435+0000", 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2780000000000002e-06 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596387+0000", 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00095243599999999997 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.321 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596459+0000", 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2077999999999997e-05 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596640+0000", 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018089500000000001 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599908+0000", 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032677769999999999 2024-09-21T08:30:43.322 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599918+0000", 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0059e-05 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670040+0000", 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070122219 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670079+0000", 2024-09-21T08:30:43.323 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8436000000000003e-05 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670131+0000", 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2343999999999999e-05 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4172 7.b 7:d7506421:::smithi13779221-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.595449+0000", 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 40.590187444000001, 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074831616000000004, 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.324 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4172 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595449+0000", 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595449+0000", 2024-09-21T08:30:43.325 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595450+0000", 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.85e-07 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595453+0000", 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7879999999999998e-06 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.326 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595460+0000", 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.131e-06 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595466+0000", 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7910000000000001e-06 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596937+0000", 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001470823 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.597004+0000", 2024-09-21T08:30:43.327 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7550000000000002e-05 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670154+0000", 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.073150017999999997 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670214+0000", 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9565999999999999e-05 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670281+0000", 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7163999999999996e-05 2024-09-21T08:30:43.328 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4344 7.b 7:d4aadaae:::smithi13779221-7:head [copy-from ver 229] snapc af=[af,ae,ad,ac,ab,a9,a7,a5,a4,a3,9c,9b,9a,99,94,93,92,8b,84,7e] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:15.597787+0000", 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 27.587849572, 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033249305, 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.329 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4344 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597787+0000", 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597787+0000", 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.330 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597788+0000", 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3000000000000001e-07 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597792+0000", 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1749999999999998e-06 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597796+0000", 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9219999999999999e-06 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.331 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597800+0000", 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2069999999999997e-06 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598500+0000", 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069988899999999998 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598508+0000", 2024-09-21T08:30:43.332 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8310000000000001e-06 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610088+0000", 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011580653 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610205+0000", 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116882 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.618623+0000", 2024-09-21T08:30:43.333 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0084175770000000007 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.629957+0000", 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011334222999999999 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.630882+0000", 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00092455300000000005 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.334 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.630902+0000", 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0522000000000001e-05 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.630982+0000", 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9491000000000002e-05 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.335 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.631037+0000", 2024-09-21T08:30:43.336 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4950000000000001e-05 2024-09-21T08:30:43.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.336 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.336 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.336 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.336 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.336 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:43.337 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4345 7.b 7:d4aadaae:::smithi13779221-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e422)", 2024-09-21T08:30:43.337 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:15.597855+0000", 2024-09-21T08:30:43.337 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 27.587781441000001, 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027086114000000001, 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4345 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.338 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597855+0000", 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597855+0000", 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597856+0000", 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6199999999999996e-07 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.339 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597860+0000", 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6270000000000002e-06 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598442+0000", 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058186999999999996 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598450+0000", 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0420000000000003e-06 2024-09-21T08:30:43.340 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.599133+0000", 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00068317699999999998 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.599141+0000", 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240000000000007e-06 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.341 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610453+0000", 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011312546999999999 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610511+0000", 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7732999999999998e-05 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.624495+0000", 2024-09-21T08:30:43.342 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013983647 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.624674+0000", 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017928099999999999 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.624942+0000", 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000267404 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.343 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4390 7.10 7:0979c720:::smithi13779221-16:head [copy-from ver 498] snapc b2=[b2,b1,b0,af,ae,ad,ac,a9,a7,a5,a4,a3,9c,9b,99,94,93,92,84,7e] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:21.587905+0000", 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 21.597732335, 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028087958, 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4390 2024-09-21T08:30:43.344 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587905+0000", 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587905+0000", 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.345 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587906+0000", 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.404e-06 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587911+0000", 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.305e-06 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587921+0000", 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3780000000000007e-06 2024-09-21T08:30:43.346 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587926+0000", 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0889999999999999e-06 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587979+0000", 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3261999999999999e-05 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:43.347 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.588091+0000", 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112184 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.608584+0000", 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020492429999999999 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615479+0000", 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00689502 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.348 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615773+0000", 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029482400000000001 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615854+0000", 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0430000000000001e-05 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615993+0000", 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000138632 2024-09-21T08:30:43.349 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.350 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.350 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.350 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:43.350 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:43.350 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:30:43.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:43.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:43.363 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:43.464 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:43.488 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:43.489 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:43.490 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:43.591 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:43.593 INFO:tasks.rados.rados.0.smithi137.stdout:1777: done (2 left) 2024-09-21T08:30:43.593 INFO:tasks.rados.rados.0.smithi137.stdout:1778: done (1 left) 2024-09-21T08:30:43.593 INFO:tasks.rados.rados.0.smithi137.stdout:1779: done (0 left) 2024-09-21T08:30:43.593 INFO:tasks.rados.rados.0.smithi137.stdout:1780: delete oid 43 current snap is 189 2024-09-21T08:30:43.597 INFO:tasks.rados.rados.0.smithi137.stdout:1780: done (0 left) 2024-09-21T08:30:43.597 INFO:tasks.rados.rados.0.smithi137.stdout:1781: read oid 5 snap 161 2024-09-21T08:30:43.597 INFO:tasks.rados.rados.0.smithi137.stdout:1781: expect deleted 2024-09-21T08:30:43.597 INFO:tasks.rados.rados.0.smithi137.stdout:1782: copy_from oid 33 from oid 31 current snap is 189 2024-09-21T08:30:43.597 INFO:tasks.rados.rados.0.smithi137.stdout:1783: rollback oid 36 current snap is 189 2024-09-21T08:30:43.597 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 36 to 177 2024-09-21T08:30:43.597 INFO:tasks.rados.rados.0.smithi137.stdout:1784: write oid 35 current snap is 189 2024-09-21T08:30:43.598 INFO:tasks.rados.rados.0.smithi137.stdout:1784: seq_num 632 ranges {622024=414667,1783208=417873} 2024-09-21T08:30:43.602 INFO:tasks.rados.rados.0.smithi137.stdout:1784: writing smithi13779221-35 from 622024 to 1036691 tid 1 2024-09-21T08:30:43.609 INFO:tasks.rados.rados.0.smithi137.stdout:1784: writing smithi13779221-35 from 1783208 to 2201081 tid 2 2024-09-21T08:30:43.610 INFO:tasks.rados.rados.0.smithi137.stdout:1785: snap_create 2024-09-21T08:30:43.610 INFO:tasks.rados.rados.0.smithi137.stdout:1783: finishing rollback tid 1 to smithi13779221-36 2024-09-21T08:30:43.610 INFO:tasks.rados.rados.0.smithi137.stdout:1783: finishing rollback tid 2 to smithi13779221-36 2024-09-21T08:30:43.610 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 734 (ObjNum 566 snap 168 seq_num 566) dirty exists 2024-09-21T08:30:43.610 INFO:tasks.rados.rados.0.smithi137.stdout:1784: finishing write tid 1 to smithi13779221-35 2024-09-21T08:30:43.611 INFO:tasks.rados.rados.0.smithi137.stdout:1782: finishing copy_from racing read to smithi13779221-33 2024-09-21T08:30:43.614 INFO:tasks.rados.rados.0.smithi137.stdout:1782: finishing copy_from to smithi13779221-33 2024-09-21T08:30:43.615 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 356 (ObjNum 617 snap 181 seq_num 617) dirty exists 2024-09-21T08:30:43.615 INFO:tasks.rados.rados.0.smithi137.stdout:1784: finishing write tid 2 to smithi13779221-35 2024-09-21T08:30:43.615 INFO:tasks.rados.rados.0.smithi137.stdout:1784: finishing write tid 3 to smithi13779221-35 2024-09-21T08:30:43.615 INFO:tasks.rados.rados.0.smithi137.stdout:1784: finishing write tid 4 to smithi13779221-35 2024-09-21T08:30:43.616 INFO:tasks.rados.rados.0.smithi137.stdout:1784: oid 35 updating version 0 to 728 2024-09-21T08:30:43.616 INFO:tasks.rados.rados.0.smithi137.stdout:1784: oid 35 updating version 728 to 730 2024-09-21T08:30:43.616 INFO:tasks.rados.rados.0.smithi137.stdout:1784: oid 35 version 730 is already newer than 729 2024-09-21T08:30:43.616 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 730 (ObjNum 632 snap 189 seq_num 632) dirty exists 2024-09-21T08:30:43.616 INFO:tasks.rados.rados.0.smithi137.stdout:1784: left oid 35 (ObjNum 632 snap 189 seq_num 632) 2024-09-21T08:30:43.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:43.621 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:43.622 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:43.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:43.747 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:43.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:43.748 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:43.848 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:43.871 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:43.871 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:43.871 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:43.972 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:43.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:43.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:43.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:43 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:43 smithi165 ceph-mon[32893]: 4.12 scrub starts 2024-09-21T08:30:44.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:43 smithi165 ceph-mon[32893]: 4.12 scrub ok 2024-09-21T08:30:44.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:43 smithi165 ceph-mon[32893]: pgmap v578: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 17 MiB/s rd, 1.4 MiB/s wr, 19 op/s; 367/1425 objects degraded (25.754%) 2024-09-21T08:30:44.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:43 smithi165 ceph-mon[32893]: osdmap e450: 8 total, 6 up, 4 in 2024-09-21T08:30:44.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:43 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:44.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:43 smithi137 ceph-mon[29634]: 4.12 scrub starts 2024-09-21T08:30:44.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:43 smithi137 ceph-mon[29634]: 4.12 scrub ok 2024-09-21T08:30:44.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:43 smithi137 ceph-mon[29634]: pgmap v578: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 17 MiB/s rd, 1.4 MiB/s wr, 19 op/s; 367/1425 objects degraded (25.754%) 2024-09-21T08:30:44.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:43 smithi137 ceph-mon[29634]: osdmap e450: 8 total, 6 up, 4 in 2024-09-21T08:30:44.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:43 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:44.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:43 smithi137 ceph-mon[41044]: 4.12 scrub starts 2024-09-21T08:30:44.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:43 smithi137 ceph-mon[41044]: 4.12 scrub ok 2024-09-21T08:30:44.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:43 smithi137 ceph-mon[41044]: pgmap v578: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 17 MiB/s rd, 1.4 MiB/s wr, 19 op/s; 367/1425 objects degraded (25.754%) 2024-09-21T08:30:44.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:43 smithi137 ceph-mon[41044]: osdmap e450: 8 total, 6 up, 4 in 2024-09-21T08:30:44.096 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:44.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:44.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:44.119 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:44.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:44.240 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:44.241 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:44.241 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:44.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:44.364 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:44.365 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:44.365 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:44.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T08:30:44.466 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:44.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:44.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:44.488 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:44.589 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:44.591 INFO:tasks.rados.rados.0.smithi137.stdout:1781: done (4 left) 2024-09-21T08:30:44.591 INFO:tasks.rados.rados.0.smithi137.stdout:1782: done (3 left) 2024-09-21T08:30:44.591 INFO:tasks.rados.rados.0.smithi137.stdout:1783: done (2 left) 2024-09-21T08:30:44.591 INFO:tasks.rados.rados.0.smithi137.stdout:1784: done (1 left) 2024-09-21T08:30:44.592 INFO:tasks.rados.rados.0.smithi137.stdout:1785: done (0 left) 2024-09-21T08:30:44.592 INFO:tasks.rados.rados.0.smithi137.stdout:1786: read oid 17 snap 173 2024-09-21T08:30:44.592 INFO:tasks.rados.rados.0.smithi137.stdout:1786: expect (ObjNum 552 snap 166 seq_num 552) 2024-09-21T08:30:44.592 INFO:tasks.rados.rados.0.smithi137.stdout:1787: rollback oid 39 current snap is 190 2024-09-21T08:30:44.592 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 39 to 153 2024-09-21T08:30:44.592 INFO:tasks.rados.rados.0.smithi137.stdout:1788: delete oid 20 current snap is 190 2024-09-21T08:30:44.599 INFO:tasks.rados.rados.0.smithi137.stdout:1787: finishing rollback tid 1 to smithi13779221-39 2024-09-21T08:30:44.599 INFO:tasks.rados.rados.0.smithi137.stdout:1787: finishing rollback tid 2 to smithi13779221-39 2024-09-21T08:30:44.599 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 238 (ObjNum 526 snap 149 seq_num 526) dirty exists 2024-09-21T08:30:44.599 INFO:tasks.rados.rados.0.smithi137.stdout:1787: done (2 left) 2024-09-21T08:30:44.599 INFO:tasks.rados.rados.0.smithi137.stdout:1788: done (1 left) 2024-09-21T08:30:44.600 INFO:tasks.rados.rados.0.smithi137.stdout:1789: delete oid 33 current snap is 190 2024-09-21T08:30:44.601 INFO:tasks.rados.rados.0.smithi137.stdout:1789: done (1 left) 2024-09-21T08:30:44.601 INFO:tasks.rados.rados.0.smithi137.stdout:1790: rollback oid 14 current snap is 190 2024-09-21T08:30:44.601 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 14 to 186 2024-09-21T08:30:44.602 INFO:tasks.rados.rados.0.smithi137.stdout:1791: snap_remove snap 185 2024-09-21T08:30:44.615 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:44.615 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:44.616 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:44.643 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:44.669 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:44.670 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:44.686 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:44.686 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:44.703 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:44.703 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:44.717 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:44.720 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:44.721 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:44.738 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:44.738 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:44.739 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:44.739 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:44.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:44.755 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:44.755 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:44.755 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:44.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:44.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:44.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:44.866 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:44.908 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:44 smithi137 ceph-mon[29634]: 5.0 scrub starts 2024-09-21T08:30:44.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:44 smithi137 ceph-mon[29634]: 5.0 scrub ok 2024-09-21T08:30:44.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:44 smithi137 ceph-mon[29634]: pgmap v581: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 17 MiB/s rd, 1.4 MiB/s wr, 19 op/s; 367/1425 objects degraded (25.754%) 2024-09-21T08:30:44.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:44 smithi137 ceph-mon[29634]: osdmap e451: 8 total, 6 up, 4 in 2024-09-21T08:30:44.910 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:44 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2489751200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:44.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:44 smithi137 ceph-mon[41044]: 5.0 scrub starts 2024-09-21T08:30:44.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:44 smithi137 ceph-mon[41044]: 5.0 scrub ok 2024-09-21T08:30:44.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:44 smithi137 ceph-mon[41044]: pgmap v581: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 17 MiB/s rd, 1.4 MiB/s wr, 19 op/s; 367/1425 objects degraded (25.754%) 2024-09-21T08:30:44.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:44 smithi137 ceph-mon[41044]: osdmap e451: 8 total, 6 up, 4 in 2024-09-21T08:30:44.910 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:44 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2489751200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:44.967 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:44.991 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:44.992 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:44 smithi165 ceph-mon[32893]: 5.0 scrub starts 2024-09-21T08:30:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:44 smithi165 ceph-mon[32893]: 5.0 scrub ok 2024-09-21T08:30:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:44 smithi165 ceph-mon[32893]: pgmap v581: 168 pgs: 2 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 33 active+clean; 548 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 17 MiB/s rd, 1.4 MiB/s wr, 19 op/s; 367/1425 objects degraded (25.754%) 2024-09-21T08:30:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:44 smithi165 ceph-mon[32893]: osdmap e451: 8 total, 6 up, 4 in 2024-09-21T08:30:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:44 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2489751200' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:45.093 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:45.115 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:45.115 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:45.116 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:45.120 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:45.158 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:45.216 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:45.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:45.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:45.238 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:45.251 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:30:45.109+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:30:45.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:45.365 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:45.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:45.366 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:45.467 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:45.488 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:45.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:45.489 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:45.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:45.595 INFO:tasks.rados.rados.0.smithi137.stdout:1791: done (2 left) 2024-09-21T08:30:45.595 INFO:tasks.rados.rados.0.smithi137.stdout:1792: snap_create 2024-09-21T08:30:45.595 INFO:tasks.rados.rados.0.smithi137.stdout:1786: expect (ObjNum 552 snap 166 seq_num 552) 2024-09-21T08:30:45.595 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:30:45.624 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:45.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:45.625 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:45.719 INFO:tasks.rados.rados.0.smithi137.stdout:1790: finishing rollback tid 0 to smithi13779221-14 2024-09-21T08:30:45.719 INFO:tasks.rados.rados.0.smithi137.stdout:1790: finishing rollback tid 1 to smithi13779221-14 2024-09-21T08:30:45.719 INFO:tasks.rados.rados.0.smithi137.stdout:1790: finishing rollback tid 2 to smithi13779221-14 2024-09-21T08:30:45.719 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 400 (ObjNum 619 snap 182 seq_num 619) dirty exists 2024-09-21T08:30:45.726 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:45.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:45.748 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:45.748 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:45.849 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:45.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:45.870 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:45.870 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:45.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[29634]: 4.18 scrub starts 2024-09-21T08:30:45.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[29634]: 4.18 scrub ok 2024-09-21T08:30:45.936 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4023007727' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[29634]: from='client.45779 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[29634]: from='client.45779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[29634]: osdmap e452: 8 total, 6 up, 4 in 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[41044]: 4.18 scrub starts 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[41044]: 4.18 scrub ok 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4023007727' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[41044]: from='client.45779 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:30:45.937 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[41044]: from='client.45779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:45.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:45 smithi137 ceph-mon[41044]: osdmap e452: 8 total, 6 up, 4 in 2024-09-21T08:30:45.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:45.997 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:45.998 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:45.998 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:45 smithi165 ceph-mon[32893]: 4.18 scrub starts 2024-09-21T08:30:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:45 smithi165 ceph-mon[32893]: 4.18 scrub ok 2024-09-21T08:30:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:45 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4023007727' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:45 smithi165 ceph-mon[32893]: from='client.45779 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:45 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:30:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:45 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:30:46.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:45 smithi165 ceph-mon[32893]: from='client.45779 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:30:46.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:45 smithi165 ceph-mon[32893]: osdmap e452: 8 total, 6 up, 4 in 2024-09-21T08:30:46.043 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:46.043 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:46.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:46.120 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:46.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:46.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:46.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:46.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:46.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:46.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:46.348 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:46.373 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:46.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:46.374 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:46.475 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:46.496 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:46.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:46.497 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1786: done (2 left) 2024-09-21T08:30:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1790: done (1 left) 2024-09-21T08:30:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1792: done (0 left) 2024-09-21T08:30:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1793: rollback oid 23 current snap is 191 2024-09-21T08:30:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 23 to 173 2024-09-21T08:30:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1794: write (excl) oid 3 current snap is 191 2024-09-21T08:30:46.592 INFO:tasks.rados.rados.0.smithi137.stdout:1794: seq_num 635 ranges {672445=640414,1842301=681553} 2024-09-21T08:30:46.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:46.600 INFO:tasks.rados.rados.0.smithi137.stdout:1794: writing smithi13779221-3 from 672445 to 1312859 tid 1 2024-09-21T08:30:46.607 INFO:tasks.rados.rados.0.smithi137.stdout:1794: writing smithi13779221-3 from 1842301 to 2523854 tid 2 2024-09-21T08:30:46.607 INFO:tasks.rados.rados.0.smithi137.stdout:1795: rollback oid 45 current snap is 191 2024-09-21T08:30:46.607 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 45 to 181 2024-09-21T08:30:46.607 INFO:tasks.rados.rados.0.smithi137.stdout:1796: read oid 20 snap -1 2024-09-21T08:30:46.607 INFO:tasks.rados.rados.0.smithi137.stdout:1796: expect deleted 2024-09-21T08:30:46.607 INFO:tasks.rados.rados.0.smithi137.stdout:1797: rollback oid 19 current snap is 191 2024-09-21T08:30:46.607 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 19 to 182 2024-09-21T08:30:46.607 INFO:tasks.rados.rados.0.smithi137.stdout:1798: copy_from oid 34 from oid 26 current snap is 191 2024-09-21T08:30:46.608 INFO:tasks.rados.rados.0.smithi137.stdout:1799: read oid 8 snap 179 2024-09-21T08:30:46.608 INFO:tasks.rados.rados.0.smithi137.stdout:1799: expect deleted 2024-09-21T08:30:46.608 INFO:tasks.rados.rados.0.smithi137.stdout:1800: write oid 31 current snap is 191 2024-09-21T08:30:46.608 INFO:tasks.rados.rados.0.smithi137.stdout:1800: seq_num 636 ranges {689252=715663,1900783=425211} 2024-09-21T08:30:46.615 INFO:tasks.rados.rados.0.smithi137.stdout:1800: writing smithi13779221-31 from 689252 to 1404915 tid 1 2024-09-21T08:30:46.621 INFO:tasks.rados.rados.0.smithi137.stdout:1800: writing smithi13779221-31 from 1900783 to 2325994 tid 2 2024-09-21T08:30:46.621 INFO:tasks.rados.rados.0.smithi137.stdout:1801: delete oid 9 current snap is 191 2024-09-21T08:30:46.621 INFO:tasks.rados.rados.0.smithi137.stdout:1797: finishing rollback tid 1 to smithi13779221-19 2024-09-21T08:30:46.621 INFO:tasks.rados.rados.0.smithi137.stdout:1797: finishing rollback tid 2 to smithi13779221-19 2024-09-21T08:30:46.621 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 19 v 513 (ObjNum 618 snap 182 seq_num 618) dirty exists 2024-09-21T08:30:46.621 INFO:tasks.rados.rados.0.smithi137.stdout:1793: finishing rollback tid 0 to smithi13779221-23 2024-09-21T08:30:46.621 INFO:tasks.rados.rados.0.smithi137.stdout:1793: finishing rollback tid 1 to smithi13779221-23 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1793: finishing rollback tid 2 to smithi13779221-23 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 204 (ObjNum 581 snap 171 seq_num 581) dirty exists 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1794: finishing write tid 1 to smithi13779221-3 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1794: finishing write tid 2 to smithi13779221-3 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1794: finishing write tid 3 to smithi13779221-3 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1794: finishing write tid 4 to smithi13779221-3 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1794: oid 3 updating version 0 to 501 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1794: oid 3 version 501 is already newer than 500 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1794: oid 3 version 501 is already newer than 499 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 501 (ObjNum 635 snap 191 seq_num 635) dirty exists 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1794: left oid 3 (ObjNum 635 snap 191 seq_num 635) 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1795: finishing rollback tid 0 to smithi13779221-45 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1795: finishing rollback tid 1 to smithi13779221-45 2024-09-21T08:30:46.622 INFO:tasks.rados.rados.0.smithi137.stdout:1800: finishing write tid 1 to smithi13779221-31 2024-09-21T08:30:46.623 INFO:tasks.rados.rados.0.smithi137.stdout:1795: finishing rollback tid 2 to smithi13779221-45 2024-09-21T08:30:46.623 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 438 (ObjNum 545 snap 162 seq_num 545) dirty exists 2024-09-21T08:30:46.623 INFO:tasks.rados.rados.0.smithi137.stdout:1793: done (8 left) 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1794: done (7 left) 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1795: done (6 left) 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1796: done (5 left) 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1797: done (4 left) 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1799: done (3 left) 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1801: done (2 left) 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1802: read oid 33 snap -1 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1802: expect deleted 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1803: write oid 28 current snap is 191 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1798: finishing copy_from racing read to smithi13779221-34 2024-09-21T08:30:46.624 INFO:tasks.rados.rados.0.smithi137.stdout:1803: seq_num 638 ranges {722866=466161,1617747=428880,2587634=135232} 2024-09-21T08:30:46.625 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:46.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:46.626 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:46.630 INFO:tasks.rados.rados.0.smithi137.stdout:1803: writing smithi13779221-28 from 722866 to 1189027 tid 1 2024-09-21T08:30:46.635 INFO:tasks.rados.rados.0.smithi137.stdout:1803: writing smithi13779221-28 from 1617747 to 2046627 tid 2 2024-09-21T08:30:46.639 INFO:tasks.rados.rados.0.smithi137.stdout:1803: writing smithi13779221-28 from 2587634 to 2722866 tid 3 2024-09-21T08:30:46.639 INFO:tasks.rados.rados.0.smithi137.stdout:1804: read oid 47 snap -1 2024-09-21T08:30:46.639 INFO:tasks.rados.rados.0.smithi137.stdout:1804: expect deleted 2024-09-21T08:30:46.639 INFO:tasks.rados.rados.0.smithi137.stdout:1805: copy_from oid 5 from oid 48 current snap is 191 2024-09-21T08:30:46.639 INFO:tasks.rados.rados.0.smithi137.stdout:1798: finishing copy_from to smithi13779221-34 2024-09-21T08:30:46.639 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 399 (ObjNum 620 snap 182 seq_num 620) dirty exists 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1798: done (5 left) 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1802: done (4 left) 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1806: read oid 39 snap -1 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1806: expect (ObjNum 526 snap 149 seq_num 526) 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1807: delete oid 21 current snap is 191 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1800: finishing write tid 2 to smithi13779221-31 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1800: finishing write tid 3 to smithi13779221-31 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1800: finishing write tid 4 to smithi13779221-31 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1800: oid 31 updating version 0 to 359 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1800: oid 31 version 359 is already newer than 357 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1800: oid 31 version 359 is already newer than 358 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 359 (ObjNum 636 snap 191 seq_num 636) dirty exists 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1800: left oid 31 (ObjNum 636 snap 191 seq_num 636) 2024-09-21T08:30:46.640 INFO:tasks.rados.rados.0.smithi137.stdout:1803: finishing write tid 1 to smithi13779221-28 2024-09-21T08:30:46.641 INFO:tasks.rados.rados.0.smithi137.stdout:1803: finishing write tid 2 to smithi13779221-28 2024-09-21T08:30:46.641 INFO:tasks.rados.rados.0.smithi137.stdout:1800: done (5 left) 2024-09-21T08:30:46.641 INFO:tasks.rados.rados.0.smithi137.stdout:1804: done (4 left) 2024-09-21T08:30:46.641 INFO:tasks.rados.rados.0.smithi137.stdout:1807: done (3 left) 2024-09-21T08:30:46.641 INFO:tasks.rados.rados.0.smithi137.stdout:1808: snap_remove snap 124 2024-09-21T08:30:46.726 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:46.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T08:30:46.749 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:46.750 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:46.750 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:46.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:46.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:46.873 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:46.874 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:46.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:46.996 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:46.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:46.997 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:46 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1790219144' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-21T08:30:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:46 smithi165 ceph-mon[32893]: from='client.45785 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-21T08:30:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:46 smithi165 ceph-mon[32893]: 3.4 deep-scrub starts 2024-09-21T08:30:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:46 smithi165 ceph-mon[32893]: 3.4 deep-scrub ok 2024-09-21T08:30:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:46 smithi165 ceph-mon[32893]: pgmap v584: 168 pgs: 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 35 active+clean; 552 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 6.0 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 372/1440 objects degraded (25.833%) 2024-09-21T08:30:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:46 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 367/1425 objects degraded (25.754%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:47.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:46 smithi165 ceph-mon[32893]: from='client.45785 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T08:30:47.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:46 smithi165 ceph-mon[32893]: osdmap e453: 8 total, 6 up, 4 in 2024-09-21T08:30:47.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1790219144' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-21T08:30:47.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[29634]: from='client.45785 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-21T08:30:47.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[29634]: 3.4 deep-scrub starts 2024-09-21T08:30:47.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[29634]: 3.4 deep-scrub ok 2024-09-21T08:30:47.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[29634]: pgmap v584: 168 pgs: 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 35 active+clean; 552 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 6.0 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 372/1440 objects degraded (25.833%) 2024-09-21T08:30:47.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 367/1425 objects degraded (25.754%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:47.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[29634]: from='client.45785 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T08:30:47.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[29634]: osdmap e453: 8 total, 6 up, 4 in 2024-09-21T08:30:47.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1790219144' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-21T08:30:47.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[41044]: from='client.45785 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]: dispatch 2024-09-21T08:30:47.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[41044]: 3.4 deep-scrub starts 2024-09-21T08:30:47.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[41044]: 3.4 deep-scrub ok 2024-09-21T08:30:47.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[41044]: pgmap v584: 168 pgs: 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 35 active+clean; 552 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 6.0 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 372/1440 objects degraded (25.833%) 2024-09-21T08:30:47.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 367/1425 objects degraded (25.754%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:47.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[41044]: from='client.45785 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num", "val": "11"}]': finished 2024-09-21T08:30:47.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:46 smithi137 ceph-mon[41044]: osdmap e453: 8 total, 6 up, 4 in 2024-09-21T08:30:47.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:47.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:47.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:47.120 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:47.147 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:47.147 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:47.221 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:47.242 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:47.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:47.243 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:47.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:47.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:47.370 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:47.371 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:47.471 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:47.492 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:47.493 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:47.493 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:47.591 INFO:tasks.rados.rados.0.smithi137.stdout:1808: done (3 left) 2024-09-21T08:30:47.591 INFO:tasks.rados.rados.0.smithi137.stdout:1809: write (excl) oid 29 current snap is 191 2024-09-21T08:30:47.591 INFO:tasks.rados.rados.0.smithi137.stdout:1803: finishing write tid 3 to smithi13779221-28 2024-09-21T08:30:47.591 INFO:tasks.rados.rados.0.smithi137.stdout:1809: seq_num 640 ranges {756480=733012,2134711=432549} 2024-09-21T08:30:47.594 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:47.599 INFO:tasks.rados.rados.0.smithi137.stdout:1809: writing smithi13779221-29 from 756480 to 1489492 tid 1 2024-09-21T08:30:47.604 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 1 pg_num to 11 2024-09-21T08:30:47.605 INFO:tasks.rados.rados.0.smithi137.stdout:1809: writing smithi13779221-29 from 2134711 to 2567260 tid 2 2024-09-21T08:30:47.605 INFO:tasks.rados.rados.0.smithi137.stdout:1810: copy_from oid 16 from oid 43 current snap is 191 2024-09-21T08:30:47.605 INFO:tasks.rados.rados.0.smithi137.stdout:1811: write oid 25 current snap is 191 2024-09-21T08:30:47.606 INFO:tasks.rados.rados.0.smithi137.stdout:1811: seq_num 641 ranges {773287=408261,1793193=576207} 2024-09-21T08:30:47.612 INFO:tasks.rados.rados.0.smithi137.stdout:1811: writing smithi13779221-25 from 773287 to 1181548 tid 1 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:1811: writing smithi13779221-25 from 1793193 to 2369400 tid 2 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:1812: write oid 17 current snap is 191 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:1803: finishing write tid 4 to smithi13779221-28 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:1803: finishing write tid 5 to smithi13779221-28 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:1803: oid 28 updating version 0 to 377 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:1803: oid 28 version 377 is already newer than 375 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:1803: oid 28 version 377 is already newer than 374 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:1803: oid 28 version 377 is already newer than 376 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 377 (ObjNum 638 snap 191 seq_num 638) dirty exists 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:1803: left oid 28 (ObjNum 638 snap 191 seq_num 638) 2024-09-21T08:30:47.620 INFO:tasks.rados.rados.0.smithi137.stdout:1812: seq_num 642 ranges {790094=483510,1851675=436218} 2024-09-21T08:30:47.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:47.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:47.623 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:47.626 INFO:tasks.rados.rados.0.smithi137.stdout:1812: writing smithi13779221-17 from 790094 to 1273604 tid 1 2024-09-21T08:30:47.632 INFO:tasks.rados.rados.0.smithi137.stdout:1812: writing smithi13779221-17 from 1851675 to 2287893 tid 2 2024-09-21T08:30:47.632 INFO:tasks.rados.rados.0.smithi137.stdout:1803: done (6 left) 2024-09-21T08:30:47.632 INFO:tasks.rados.rados.0.smithi137.stdout:1813: copy_from oid 46 from oid 41 current snap is 191 2024-09-21T08:30:47.632 INFO:tasks.rados.rados.0.smithi137.stdout:1806: expect (ObjNum 526 snap 149 seq_num 526) 2024-09-21T08:30:47.724 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:47.748 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:47.749 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:47.749 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:47.750 INFO:tasks.rados.rados.0.smithi137.stdout:1805: finishing copy_from racing read to smithi13779221-5 2024-09-21T08:30:47.751 INFO:tasks.rados.rados.0.smithi137.stdout:1805: finishing copy_from to smithi13779221-5 2024-09-21T08:30:47.751 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 518 (ObjNum 497 snap 137 seq_num 497) dirty exists 2024-09-21T08:30:47.751 INFO:tasks.rados.rados.0.smithi137.stdout:1810: finishing copy_from to smithi13779221-16 2024-09-21T08:30:47.751 INFO:tasks.rados.rados.0.smithi137.stdout:1810: got expected ENOENT (src dne) 2024-09-21T08:30:47.751 INFO:tasks.rados.rados.0.smithi137.stdout:1810: finishing copy_from racing read to smithi13779221-16 2024-09-21T08:30:47.751 INFO:tasks.rados.rados.0.smithi137.stdout:1809: finishing write tid 1 to smithi13779221-29 2024-09-21T08:30:47.751 INFO:tasks.rados.rados.0.smithi137.stdout:1809: finishing write tid 2 to smithi13779221-29 2024-09-21T08:30:47.751 INFO:tasks.rados.rados.0.smithi137.stdout:1809: finishing write tid 3 to smithi13779221-29 2024-09-21T08:30:47.751 INFO:tasks.rados.rados.0.smithi137.stdout:1805: done (6 left) 2024-09-21T08:30:47.752 INFO:tasks.rados.rados.0.smithi137.stdout:1806: done (5 left) 2024-09-21T08:30:47.752 INFO:tasks.rados.rados.0.smithi137.stdout:1810: done (4 left) 2024-09-21T08:30:47.752 INFO:tasks.rados.rados.0.smithi137.stdout:1814: write oid 38 current snap is 191 2024-09-21T08:30:47.752 INFO:tasks.rados.rados.0.smithi137.stdout:1811: finishing write tid 1 to smithi13779221-25 2024-09-21T08:30:47.752 INFO:tasks.rados.rados.0.smithi137.stdout:1814: seq_num 643 ranges {406901=558759,1393804=579876,2608396=198505} 2024-09-21T08:30:47.755 INFO:tasks.rados.rados.0.smithi137.stdout:1814: writing smithi13779221-38 from 406901 to 965660 tid 1 2024-09-21T08:30:47.761 INFO:tasks.rados.rados.0.smithi137.stdout:1814: writing smithi13779221-38 from 1393804 to 1973680 tid 2 2024-09-21T08:30:47.766 INFO:tasks.rados.rados.0.smithi137.stdout:1814: writing smithi13779221-38 from 2608396 to 2806901 tid 3 2024-09-21T08:30:47.766 INFO:tasks.rados.rados.0.smithi137.stdout:1815: delete oid 42 current snap is 191 2024-09-21T08:30:47.766 INFO:tasks.rados.rados.0.smithi137.stdout:1809: finishing write tid 4 to smithi13779221-29 2024-09-21T08:30:47.766 INFO:tasks.rados.rados.0.smithi137.stdout:1809: oid 29 updating version 0 to 752 2024-09-21T08:30:47.766 INFO:tasks.rados.rados.0.smithi137.stdout:1809: oid 29 version 752 is already newer than 751 2024-09-21T08:30:47.766 INFO:tasks.rados.rados.0.smithi137.stdout:1809: oid 29 updating version 752 to 753 2024-09-21T08:30:47.766 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 753 (ObjNum 640 snap 191 seq_num 640) dirty exists 2024-09-21T08:30:47.766 INFO:tasks.rados.rados.0.smithi137.stdout:1809: left oid 29 (ObjNum 640 snap 191 seq_num 640) 2024-09-21T08:30:47.766 INFO:tasks.rados.rados.0.smithi137.stdout:1811: finishing write tid 2 to smithi13779221-25 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1811: finishing write tid 3 to smithi13779221-25 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1811: finishing write tid 4 to smithi13779221-25 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1811: oid 25 updating version 0 to 342 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1811: oid 25 version 342 is already newer than 341 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1811: oid 25 updating version 342 to 343 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 343 (ObjNum 641 snap 191 seq_num 641) dirty exists 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1811: left oid 25 (ObjNum 641 snap 191 seq_num 641) 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1812: finishing write tid 1 to smithi13779221-17 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1813: finishing copy_from to smithi13779221-46 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1813: got expected ENOENT (src dne) 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1813: finishing copy_from racing read to smithi13779221-46 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1812: finishing write tid 2 to smithi13779221-17 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1812: finishing write tid 3 to smithi13779221-17 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1812: finishing write tid 4 to smithi13779221-17 2024-09-21T08:30:47.767 INFO:tasks.rados.rados.0.smithi137.stdout:1812: oid 17 updating version 0 to 732 2024-09-21T08:30:47.768 INFO:tasks.rados.rados.0.smithi137.stdout:1812: oid 17 updating version 732 to 733 2024-09-21T08:30:47.768 INFO:tasks.rados.rados.0.smithi137.stdout:1812: oid 17 version 733 is already newer than 731 2024-09-21T08:30:47.768 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 733 (ObjNum 642 snap 191 seq_num 642) dirty exists 2024-09-21T08:30:47.768 INFO:tasks.rados.rados.0.smithi137.stdout:1812: left oid 17 (ObjNum 642 snap 191 seq_num 642) 2024-09-21T08:30:47.768 INFO:tasks.rados.rados.0.smithi137.stdout:1814: finishing write tid 1 to smithi13779221-38 2024-09-21T08:30:47.768 INFO:tasks.rados.rados.0.smithi137.stdout:1814: finishing write tid 2 to smithi13779221-38 2024-09-21T08:30:47.769 INFO:tasks.rados.rados.0.smithi137.stdout:1809: done (5 left) 2024-09-21T08:30:47.769 INFO:tasks.rados.rados.0.smithi137.stdout:1811: done (4 left) 2024-09-21T08:30:47.769 INFO:tasks.rados.rados.0.smithi137.stdout:1812: done (3 left) 2024-09-21T08:30:47.769 INFO:tasks.rados.rados.0.smithi137.stdout:1813: done (2 left) 2024-09-21T08:30:47.769 INFO:tasks.rados.rados.0.smithi137.stdout:1815: done (1 left) 2024-09-21T08:30:47.769 INFO:tasks.rados.rados.0.smithi137.stdout:1816: delete oid 1 current snap is 191 2024-09-21T08:30:47.769 INFO:tasks.rados.rados.0.smithi137.stdout:1814: finishing write tid 3 to smithi13779221-38 2024-09-21T08:30:47.769 INFO:tasks.rados.rados.0.smithi137.stdout:1814: finishing write tid 4 to smithi13779221-38 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1816: done (1 left) 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1817: rollback oid 25 current snap is 191 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 25 to 186 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1818: read oid 7 snap -1 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1818: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1819: snap_create 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1814: finishing write tid 5 to smithi13779221-38 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1814: oid 38 updating version 0 to 742 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1814: oid 38 updating version 742 to 745 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1814: oid 38 version 745 is already newer than 743 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1814: oid 38 version 745 is already newer than 744 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 745 (ObjNum 643 snap 191 seq_num 643) dirty exists 2024-09-21T08:30:47.770 INFO:tasks.rados.rados.0.smithi137.stdout:1814: left oid 38 (ObjNum 643 snap 191 seq_num 643) 2024-09-21T08:30:47.771 INFO:tasks.rados.rados.0.smithi137.stdout:1817: finishing rollback tid 0 to smithi13779221-25 2024-09-21T08:30:47.771 INFO:tasks.rados.rados.0.smithi137.stdout:1817: finishing rollback tid 1 to smithi13779221-25 2024-09-21T08:30:47.771 INFO:tasks.rados.rados.0.smithi137.stdout:1817: finishing rollback tid 2 to smithi13779221-25 2024-09-21T08:30:47.771 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 346 (ObjNum 459 snap 124 seq_num 459) dirty exists 2024-09-21T08:30:47.774 INFO:tasks.rados.rados.0.smithi137.stdout:1818: expect (ObjNum 420 snap 103 seq_num 420) 2024-09-21T08:30:47.850 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:47.875 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:47.876 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:47.876 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:47.977 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:47.998 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:47.999 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:47.999 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:48.100 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:48.121 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:48.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:48.122 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:48.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:48.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:48.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:48.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:48.351 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:48.372 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:48.377 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:48.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:48.378 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:48.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:48.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:48.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:48.501 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:48.600 INFO:tasks.rados.rados.0.smithi137.stdout:1814: done (3 left) 2024-09-21T08:30:48.600 INFO:tasks.rados.rados.0.smithi137.stdout:1817: done (2 left) 2024-09-21T08:30:48.600 INFO:tasks.rados.rados.0.smithi137.stdout:1818: done (1 left) 2024-09-21T08:30:48.600 INFO:tasks.rados.rados.0.smithi137.stdout:1819: done (0 left) 2024-09-21T08:30:48.600 INFO:tasks.rados.rados.0.smithi137.stdout:1820: read oid 49 snap 173 2024-09-21T08:30:48.600 INFO:tasks.rados.rados.0.smithi137.stdout:1820: expect (ObjNum 560 snap 167 seq_num 560) 2024-09-21T08:30:48.601 INFO:tasks.rados.rados.0.smithi137.stdout:1821: delete oid 6 current snap is 192 2024-09-21T08:30:48.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:48.609 INFO:tasks.rados.rados.0.smithi137.stdout:1821: done (1 left) 2024-09-21T08:30:48.609 INFO:tasks.rados.rados.0.smithi137.stdout:1822: delete oid 10 current snap is 192 2024-09-21T08:30:48.611 INFO:tasks.rados.rados.0.smithi137.stdout:1822: done (1 left) 2024-09-21T08:30:48.612 INFO:tasks.rados.rados.0.smithi137.stdout:1823: rollback oid 4 current snap is 192 2024-09-21T08:30:48.612 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 4 to 161 2024-09-21T08:30:48.612 INFO:tasks.rados.rados.0.smithi137.stdout:1824: write oid 25 current snap is 192 2024-09-21T08:30:48.612 INFO:tasks.rados.rados.0.smithi137.stdout:1824: seq_num 648 ranges {490936=651357,1686214=447225,2861864=29072} 2024-09-21T08:30:48.618 INFO:tasks.rados.rados.0.smithi137.stdout:1824: writing smithi13779221-25 from 490936 to 1142293 tid 1 2024-09-21T08:30:48.624 INFO:tasks.rados.rados.0.smithi137.stdout:1824: writing smithi13779221-25 from 1686214 to 2133439 tid 2 2024-09-21T08:30:48.628 INFO:tasks.rados.rados.0.smithi137.stdout:1824: writing smithi13779221-25 from 2861864 to 2890936 tid 3 2024-09-21T08:30:48.629 INFO:tasks.rados.rados.0.smithi137.stdout:1825: read oid 36 snap -1 2024-09-21T08:30:48.629 INFO:tasks.rados.rados.0.smithi137.stdout:1825: expect (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:30:48.629 INFO:tasks.rados.rados.0.smithi137.stdout:1826: write oid 30 current snap is 192 2024-09-21T08:30:48.629 INFO:tasks.rados.rados.0.smithi137.stdout:1826: seq_num 649 ranges {507743=726606,1744696=590883,2772934=134809} 2024-09-21T08:30:48.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:48.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:48.630 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:48.636 INFO:tasks.rados.rados.0.smithi137.stdout:1826: writing smithi13779221-30 from 507743 to 1234349 tid 1 2024-09-21T08:30:48.641 INFO:tasks.rados.rados.0.smithi137.stdout:1826: writing smithi13779221-30 from 1744696 to 2335579 tid 2 2024-09-21T08:30:48.645 INFO:tasks.rados.rados.0.smithi137.stdout:1826: writing smithi13779221-30 from 2772934 to 2907743 tid 3 2024-09-21T08:30:48.645 INFO:tasks.rados.rados.0.smithi137.stdout:1827: delete oid 27 current snap is 192 2024-09-21T08:30:48.645 INFO:tasks.rados.rados.0.smithi137.stdout:1820: expect (ObjNum 560 snap 167 seq_num 560) 2024-09-21T08:30:48.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:48.751 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:48.752 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:48.752 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:48.754 INFO:tasks.rados.rados.0.smithi137.stdout:1823: finishing rollback tid 0 to smithi13779221-4 2024-09-21T08:30:48.754 INFO:tasks.rados.rados.0.smithi137.stdout:1823: finishing rollback tid 1 to smithi13779221-4 2024-09-21T08:30:48.754 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 440 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-21T08:30:48.754 INFO:tasks.rados.rados.0.smithi137.stdout:1824: finishing write tid 1 to smithi13779221-25 2024-09-21T08:30:48.754 INFO:tasks.rados.rados.0.smithi137.stdout:1824: finishing write tid 2 to smithi13779221-25 2024-09-21T08:30:48.754 INFO:tasks.rados.rados.0.smithi137.stdout:1820: done (5 left) 2024-09-21T08:30:48.754 INFO:tasks.rados.rados.0.smithi137.stdout:1823: done (4 left) 2024-09-21T08:30:48.755 INFO:tasks.rados.rados.0.smithi137.stdout:1827: done (3 left) 2024-09-21T08:30:48.755 INFO:tasks.rados.rados.0.smithi137.stdout:1828: write oid 15 current snap is 192 2024-09-21T08:30:48.755 INFO:tasks.rados.rados.0.smithi137.stdout:1824: finishing write tid 3 to smithi13779221-25 2024-09-21T08:30:48.755 INFO:tasks.rados.rados.0.smithi137.stdout:1824: finishing write tid 4 to smithi13779221-25 2024-09-21T08:30:48.755 INFO:tasks.rados.rados.0.smithi137.stdout:1825: expect (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:30:48.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:48 smithi137 ceph-mon[29634]: 7.c scrub starts 2024-09-21T08:30:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:48 smithi137 ceph-mon[29634]: 7.c scrub ok 2024-09-21T08:30:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:48 smithi137 ceph-mon[29634]: osdmap e454: 8 total, 6 up, 4 in 2024-09-21T08:30:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:48 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:48.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:48 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:48 smithi137 ceph-mon[41044]: 7.c scrub starts 2024-09-21T08:30:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:48 smithi137 ceph-mon[41044]: 7.c scrub ok 2024-09-21T08:30:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:48 smithi137 ceph-mon[41044]: osdmap e454: 8 total, 6 up, 4 in 2024-09-21T08:30:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:48 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:48.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:48 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:48.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:48.880 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:48.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:48.881 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:48.890 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:48.891 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:48.982 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:48.986 INFO:tasks.rados.rados.0.smithi137.stdout:1824: finishing write tid 5 to smithi13779221-25 2024-09-21T08:30:48.986 INFO:tasks.rados.rados.0.smithi137.stdout:1824: oid 25 updating version 0 to 348 2024-09-21T08:30:48.986 INFO:tasks.rados.rados.0.smithi137.stdout:1824: oid 25 version 348 is already newer than 347 2024-09-21T08:30:48.986 INFO:tasks.rados.rados.0.smithi137.stdout:1824: oid 25 updating version 348 to 350 2024-09-21T08:30:48.986 INFO:tasks.rados.rados.0.smithi137.stdout:1824: oid 25 version 350 is already newer than 349 2024-09-21T08:30:48.986 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 350 (ObjNum 648 snap 192 seq_num 648) dirty exists 2024-09-21T08:30:48.986 INFO:tasks.rados.rados.0.smithi137.stdout:1824: left oid 25 (ObjNum 648 snap 192 seq_num 648) 2024-09-21T08:30:48.987 INFO:tasks.rados.rados.0.smithi137.stdout:1828: seq_num 651 ranges {541357=477104,1745307=594552} 2024-09-21T08:30:48.991 INFO:tasks.rados.rados.0.smithi137.stdout:1828: writing smithi13779221-15 from 541357 to 1018461 tid 1 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1828: writing smithi13779221-15 from 1745307 to 2339859 tid 2 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1824: done (3 left) 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1825: done (2 left) 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1829: delete oid 19 current snap is 192 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1826: finishing write tid 1 to smithi13779221-30 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1826: finishing write tid 2 to smithi13779221-30 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1826: finishing write tid 3 to smithi13779221-30 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1826: finishing write tid 4 to smithi13779221-30 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1826: finishing write tid 5 to smithi13779221-30 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1826: oid 30 updating version 0 to 747 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1826: oid 30 updating version 747 to 748 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1826: oid 30 updating version 748 to 749 2024-09-21T08:30:49.000 INFO:tasks.rados.rados.0.smithi137.stdout:1826: oid 30 updating version 749 to 750 2024-09-21T08:30:49.001 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 750 (ObjNum 649 snap 192 seq_num 649) dirty exists 2024-09-21T08:30:49.001 INFO:tasks.rados.rados.0.smithi137.stdout:1826: left oid 30 (ObjNum 649 snap 192 seq_num 649) 2024-09-21T08:30:49.001 INFO:tasks.rados.rados.0.smithi137.stdout:1828: finishing write tid 1 to smithi13779221-15 2024-09-21T08:30:49.002 INFO:tasks.rados.rados.0.smithi137.stdout:1826: done (2 left) 2024-09-21T08:30:49.002 INFO:tasks.rados.rados.0.smithi137.stdout:1829: done (1 left) 2024-09-21T08:30:49.002 INFO:tasks.rados.rados.0.smithi137.stdout:1830: snap_create 2024-09-21T08:30:49.003 INFO:tasks.rados.rados.0.smithi137.stdout:1828: finishing write tid 2 to smithi13779221-15 2024-09-21T08:30:49.003 INFO:tasks.rados.rados.0.smithi137.stdout:1828: finishing write tid 3 to smithi13779221-15 2024-09-21T08:30:49.003 INFO:tasks.rados.rados.0.smithi137.stdout:1828: finishing write tid 4 to smithi13779221-15 2024-09-21T08:30:49.003 INFO:tasks.rados.rados.0.smithi137.stdout:1828: oid 15 updating version 0 to 331 2024-09-21T08:30:49.003 INFO:tasks.rados.rados.0.smithi137.stdout:1828: oid 15 version 331 is already newer than 329 2024-09-21T08:30:49.004 INFO:tasks.rados.rados.0.smithi137.stdout:1828: oid 15 version 331 is already newer than 330 2024-09-21T08:30:49.004 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 331 (ObjNum 651 snap 192 seq_num 651) dirty exists 2024-09-21T08:30:49.004 INFO:tasks.rados.rados.0.smithi137.stdout:1828: left oid 15 (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:30:49.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:49.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:49.005 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:48 smithi165 ceph-mon[32893]: 7.c scrub starts 2024-09-21T08:30:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:48 smithi165 ceph-mon[32893]: 7.c scrub ok 2024-09-21T08:30:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:48 smithi165 ceph-mon[32893]: osdmap e454: 8 total, 6 up, 4 in 2024-09-21T08:30:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:48 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:49.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:48 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:49.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:49.132 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:49.133 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:49.133 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:49.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:30:49.234 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:49.256 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:49.256 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:49.257 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:49.357 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:49.379 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:49.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:49.380 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:49.481 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:49.506 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:49.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:49.507 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:49.606 INFO:tasks.rados.rados.0.smithi137.stdout:1828: done (1 left) 2024-09-21T08:30:49.606 INFO:tasks.rados.rados.0.smithi137.stdout:1830: done (0 left) 2024-09-21T08:30:49.606 INFO:tasks.rados.rados.0.smithi137.stdout:1831: write (excl) oid 5 current snap is 193 2024-09-21T08:30:49.606 INFO:tasks.rados.rados.0.smithi137.stdout:1831: seq_num 653 ranges {574971=627602,1862271=598221} 2024-09-21T08:30:49.608 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:49.614 INFO:tasks.rados.rados.0.smithi137.stdout:1831: writing smithi13779221-5 from 574971 to 1202573 tid 1 2024-09-21T08:30:49.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T08:30:49.622 INFO:tasks.rados.rados.0.smithi137.stdout:1831: writing smithi13779221-5 from 1862271 to 2460492 tid 2 2024-09-21T08:30:49.622 INFO:tasks.rados.rados.0.smithi137.stdout:1832: read oid 43 snap -1 2024-09-21T08:30:49.622 INFO:tasks.rados.rados.0.smithi137.stdout:1832: expect deleted 2024-09-21T08:30:49.622 INFO:tasks.rados.rados.0.smithi137.stdout:1833: copy_from oid 21 from oid 24 current snap is 193 2024-09-21T08:30:49.622 INFO:tasks.rados.rados.0.smithi137.stdout:1834: snap_create 2024-09-21T08:30:49.623 INFO:tasks.rados.rados.0.smithi137.stdout:1833: finishing copy_from to smithi13779221-21 2024-09-21T08:30:49.623 INFO:tasks.rados.rados.0.smithi137.stdout:1833: got expected ENOENT (src dne) 2024-09-21T08:30:49.624 INFO:tasks.rados.rados.0.smithi137.stdout:1833: finishing copy_from racing read to smithi13779221-21 2024-09-21T08:30:49.624 INFO:tasks.rados.rados.0.smithi137.stdout:1831: finishing write tid 1 to smithi13779221-5 2024-09-21T08:30:49.625 INFO:tasks.rados.rados.0.smithi137.stdout:1831: finishing write tid 2 to smithi13779221-5 2024-09-21T08:30:49.625 INFO:tasks.rados.rados.0.smithi137.stdout:1831: finishing write tid 3 to smithi13779221-5 2024-09-21T08:30:49.625 INFO:tasks.rados.rados.0.smithi137.stdout:1831: finishing write tid 4 to smithi13779221-5 2024-09-21T08:30:49.625 INFO:tasks.rados.rados.0.smithi137.stdout:1831: oid 5 updating version 0 to 519 2024-09-21T08:30:49.625 INFO:tasks.rados.rados.0.smithi137.stdout:1831: oid 5 updating version 519 to 521 2024-09-21T08:30:49.625 INFO:tasks.rados.rados.0.smithi137.stdout:1831: oid 5 version 521 is already newer than 520 2024-09-21T08:30:49.626 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 521 (ObjNum 653 snap 193 seq_num 653) dirty exists 2024-09-21T08:30:49.626 INFO:tasks.rados.rados.0.smithi137.stdout:1831: left oid 5 (ObjNum 653 snap 193 seq_num 653) 2024-09-21T08:30:49.636 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:49.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:49.637 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:49.728 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:49.738 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:49.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:49.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:49.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:49.850 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:49 smithi137 ceph-mon[29634]: 6.0 scrub starts 2024-09-21T08:30:49.850 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:49 smithi137 ceph-mon[29634]: 6.0 scrub ok 2024-09-21T08:30:49.850 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:49 smithi137 ceph-mon[29634]: pgmap v587: 168 pgs: 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 35 active+clean; 552 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 6.0 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 372/1440 objects degraded (25.833%) 2024-09-21T08:30:49.850 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:49 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T08:30:49.850 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:49 smithi137 ceph-mon[29634]: osdmap e455: 8 total, 6 up, 4 in 2024-09-21T08:30:49.850 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:49 smithi137 ceph-mon[41044]: 6.0 scrub starts 2024-09-21T08:30:49.850 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:49 smithi137 ceph-mon[41044]: 6.0 scrub ok 2024-09-21T08:30:49.850 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:49 smithi137 ceph-mon[41044]: pgmap v587: 168 pgs: 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 35 active+clean; 552 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 6.0 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 372/1440 objects degraded (25.833%) 2024-09-21T08:30:49.851 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:49 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T08:30:49.851 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:49 smithi137 ceph-mon[41044]: osdmap e455: 8 total, 6 up, 4 in 2024-09-21T08:30:49.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:49.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:49.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:49.886 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:49.987 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:49 smithi165 ceph-mon[32893]: 6.0 scrub starts 2024-09-21T08:30:50.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:49 smithi165 ceph-mon[32893]: 6.0 scrub ok 2024-09-21T08:30:50.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:49 smithi165 ceph-mon[32893]: pgmap v587: 168 pgs: 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 35 active+clean; 552 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 6.0 MiB/s rd, 1.1 MiB/s wr, 11 op/s; 372/1440 objects degraded (25.833%) 2024-09-21T08:30:50.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:49 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pg_num_actual", "val": "11"}]': finished 2024-09-21T08:30:50.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:49 smithi165 ceph-mon[32893]: osdmap e455: 8 total, 6 up, 4 in 2024-09-21T08:30:50.015 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:50.015 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:50.016 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:50.116 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:50.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:50.140 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:50.140 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:50.241 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:50.244 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:50.245 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:50.245 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:50.253 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:50.254 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:50.263 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:50.263 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:50.264 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:50.271 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:50.271 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:50.281 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:50.289 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:50.289 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:50.312 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:50.312 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:50.329 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:50.329 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:50.330 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:50.365 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:50.370 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:30:50.391 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:50.392 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:50.392 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:50.493 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:50.517 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:50.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:50.518 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:50.617 INFO:tasks.rados.rados.0.smithi137.stdout:1831: done (3 left) 2024-09-21T08:30:50.617 INFO:tasks.rados.rados.0.smithi137.stdout:1832: done (2 left) 2024-09-21T08:30:50.617 INFO:tasks.rados.rados.0.smithi137.stdout:1833: done (1 left) 2024-09-21T08:30:50.617 INFO:tasks.rados.rados.0.smithi137.stdout:1834: done (0 left) 2024-09-21T08:30:50.617 INFO:tasks.rados.rados.0.smithi137.stdout:1835: snap_create 2024-09-21T08:30:50.619 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:50.642 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:50.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:50.643 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:50.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:50 smithi137 ceph-mon[29634]: osdmap e456: 8 total, 6 up, 4 in 2024-09-21T08:30:50.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:50 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:50.678 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:50 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:50.679 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:50 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2079021018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:50 smithi137 ceph-mon[41044]: osdmap e456: 8 total, 6 up, 4 in 2024-09-21T08:30:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:50 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:50 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:50.679 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:50 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2079021018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:50.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:50.769 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:50.769 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:50.770 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:50.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:50.892 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:50.893 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:50.893 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:50.994 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:50 smithi165 ceph-mon[32893]: osdmap e456: 8 total, 6 up, 4 in 2024-09-21T08:30:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:50 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:50 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]: dispatch 2024-09-21T08:30:51.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:50 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2079021018' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:51.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:30:50 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:30:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:30:51.020 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:51.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:51.020 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:51.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:51.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:51.144 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:51.144 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:51.186 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:51.187 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:51.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:51.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:51.267 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:51.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:51.394 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:51.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:51.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:51.413 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:30:51.413 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:30:51.413 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:30:51.413 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:30:51.413 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.413 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:260 7.8 7:1fd29771:::smithi13779221-22:head [copy-from ver 15] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:30:51.413 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:30:51.413 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 364.85689835800002, 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025065418999999999, 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 260 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.414 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.415 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554325+0000", 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.667e-06 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554332+0000", 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1600000000000001e-06 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554346+0000", 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4011e-05 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.416 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554355+0000", 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9199999999999993e-06 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554524+0000", 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000169469 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554534+0000", 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5489999999999995e-06 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.417 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557746+0000", 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003212164 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557822+0000", 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6108999999999999e-05 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568552+0000", 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01072972 2024-09-21T08:30:51.418 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574773+0000", 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062208760000000002 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576999+0000", 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022266059999999999 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579275+0000", 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002276054 2024-09-21T08:30:51.419 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579335+0000", 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9775000000000003e-05 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579388+0000", 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2339000000000002e-05 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.420 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:261 7.8 7:1fd29771:::smithi13779221-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 364.85675379700001, 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020259856, 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 261 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.421 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554468+0000", 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0219999999999999e-06 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.422 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554475+0000", 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.81e-06 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554484+0000", 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0550000000000005e-06 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554491+0000", 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2080000000000003e-06 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.423 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554607+0000", 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011585400000000001 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554612+0000", 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0130000000000001e-06 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557969+0000", 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033576320000000002 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.424 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558007+0000", 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.735e-05 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574538+0000", 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016531694 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.425 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574630+0000", 2024-09-21T08:30:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1902000000000002e-05 2024-09-21T08:30:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574727+0000", 2024-09-21T08:30:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6316000000000004e-05 2024-09-21T08:30:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.426 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:332 7.3 7:c851e2c3:::smithi13779221-25:head [read 0~2403368,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:30:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:30:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 363.72507697600003, 2024-09-21T08:30:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90930888700000001, 2024-09-21T08:30:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.427 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 332 2024-09-21T08:30:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.428 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686147+0000", 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5370000000000002e-06 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.429 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686160+0000", 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2542e-05 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7927999999999998e-05 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686187+0000", 2024-09-21T08:30:51.430 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3610000000000007e-06 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686261+0000", 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4197999999999997e-05 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686271+0000", 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5259999999999999e-06 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.590700+0000", 2024-09-21T08:30:51.431 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90442903500000005 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.590838+0000", 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137863 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595452+0000", 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046148969999999997 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.432 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:333 7.3 7:c851e2c3:::smithi13779221-25:head [read 0~2403368,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 363.72501344699998, 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91060290700000002, 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 333 2024-09-21T08:30:51.433 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.434 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686208+0000", 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4900000000000005e-07 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686212+0000", 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9940000000000002e-06 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686221+0000", 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8789999999999992e-06 2024-09-21T08:30:51.435 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686225+0000", 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5059999999999997e-06 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686303+0000", 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7855000000000006e-05 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.436 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686310+0000", 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0829999999999998e-06 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595485+0000", 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90917505300000001 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595590+0000", 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105059 2024-09-21T08:30:51.437 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.596810+0000", 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012197289999999999 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:334 7.3 7:c851e2c3:::smithi13779221-25:head [sparse-read 0~1282710] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 363.72497644100002, 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91097041300000003, 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.438 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 334 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.439 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686245+0000", 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2600000000000002e-07 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686247+0000", 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7429999999999998e-06 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.440 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686255+0000", 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8369999999999997e-06 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686259+0000", 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0509999999999998e-06 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686326+0000", 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6845000000000001e-05 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:51.441 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686331+0000", 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.13e-06 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.596911+0000", 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.910579845 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.597015+0000", 2024-09-21T08:30:51.442 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103538 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.597215+0000", 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000199798 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:569 7.e 7:78d05d33:::smithi13779221-41:head [copy-from ver 7] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 340.864527716, 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033085196999999997, 2024-09-21T08:30:51.443 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 569 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.444 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546695+0000", 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3920000000000001e-06 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546704+0000", 2024-09-21T08:30:51.445 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9330000000000007e-06 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546720+0000", 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5651999999999999e-05 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546731+0000", 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1147e-05 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.446 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546794+0000", 2024-09-21T08:30:51.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3214999999999997e-05 2024-09-21T08:30:51.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:30:51.447 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546801+0000", 2024-09-21T08:30:51.447 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3350000000000001e-06 2024-09-21T08:30:51.447 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.447 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.447 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556379+0000", 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095778679999999998 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556505+0000", 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012529699999999999 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564765+0000", 2024-09-21T08:30:51.448 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082601020000000001 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576097+0000", 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011331938999999999 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576715+0000", 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00061843800000000002 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579665+0000", 2024-09-21T08:30:51.449 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029497 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579729+0000", 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4337000000000004e-05 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579778+0000", 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8841999999999998e-05 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.450 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:570 7.e 7:78d05d33:::smithi13779221-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 340.86438662500001, 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024747969000000002, 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 570 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.451 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546835+0000", 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4e-07 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.452 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546838+0000", 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4829999999999999e-06 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546845+0000", 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5810000000000002e-06 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546857+0000", 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2034e-05 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.453 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546900+0000", 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2945000000000003e-05 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546905+0000", 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2700000000000004e-06 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556676+0000", 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097709930000000004 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.454 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556717+0000", 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.104e-05 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571353+0000", 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014636287 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.455 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571465+0000", 2024-09-21T08:30:51.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111943 2024-09-21T08:30:51.456 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.456 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.456 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.456 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571582+0000", 2024-09-21T08:30:51.456 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011655299999999999 2024-09-21T08:30:51.456 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.457 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.457 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.457 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.457 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.457 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1571 7.18 7:1fd29771:::smithi13779221-22:head [copy-from ver 143] snapc 32=[32,31] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-21T08:30:51.457 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:16.499943+0000", 2024-09-21T08:30:51.457 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 274.91127738199998, 2024-09-21T08:30:51.458 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022751851999999999, 2024-09-21T08:30:51.458 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.458 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.458 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.458 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.458 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.458 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.459 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1571 2024-09-21T08:30:51.459 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.459 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.459 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.459 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.459 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499943+0000", 2024-09-21T08:30:51.459 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.460 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.460 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499943+0000", 2024-09-21T08:30:51.460 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.460 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.460 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.461 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.461 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499945+0000", 2024-09-21T08:30:51.461 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2230000000000001e-06 2024-09-21T08:30:51.461 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.461 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499952+0000", 2024-09-21T08:30:51.462 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.764e-06 2024-09-21T08:30:51.462 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.462 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.462 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.462 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499963+0000", 2024-09-21T08:30:51.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1124000000000001e-05 2024-09-21T08:30:51.463 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.463 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.463 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.463 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499969+0000", 2024-09-21T08:30:51.463 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.276e-06 2024-09-21T08:30:51.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.464 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.464 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.500029+0000", 2024-09-21T08:30:51.464 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0927000000000001e-05 2024-09-21T08:30:51.464 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.464 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.500130+0000", 2024-09-21T08:30:51.465 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010014900000000001 2024-09-21T08:30:51.465 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.465 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.465 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:30:51.465 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.509954+0000", 2024-09-21T08:30:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098240830000000008 2024-09-21T08:30:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.518293+0000", 2024-09-21T08:30:51.466 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0083390770000000003 2024-09-21T08:30:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522396+0000", 2024-09-21T08:30:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041036809999999996 2024-09-21T08:30:51.467 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522612+0000", 2024-09-21T08:30:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021587900000000001 2024-09-21T08:30:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.468 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522652+0000", 2024-09-21T08:30:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9635000000000003e-05 2024-09-21T08:30:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.469 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522695+0000", 2024-09-21T08:30:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3034000000000002e-05 2024-09-21T08:30:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.470 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1919 7.8 7:144c2ec5:::smithi13779221-45:head [delete] snapc 43=[40,3d,39] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:30:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.572348+0000", 2024-09-21T08:30:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 242.83887242099999, 2024-09-21T08:30:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0652029459999999, 2024-09-21T08:30:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.471 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.472 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.472 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.472 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.472 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.472 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1919 2024-09-21T08:30:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572348+0000", 2024-09-21T08:30:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.473 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572348+0000", 2024-09-21T08:30:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.474 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572350+0000", 2024-09-21T08:30:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.928e-06 2024-09-21T08:30:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572356+0000", 2024-09-21T08:30:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9449999999999998e-06 2024-09-21T08:30:51.475 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572366+0000", 2024-09-21T08:30:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0237e-05 2024-09-21T08:30:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.476 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572375+0000", 2024-09-21T08:30:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7940000000000008e-06 2024-09-21T08:30:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.477 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572776+0000", 2024-09-21T08:30:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040072699999999997 2024-09-21T08:30:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572789+0000", 2024-09-21T08:30:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2974e-05 2024-09-21T08:30:51.478 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.526784+0000", 2024-09-21T08:30:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95399481799999997 2024-09-21T08:30:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.479 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:30:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.526798+0000", 2024-09-21T08:30:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4428e-05 2024-09-21T08:30:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.480 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.636113+0000", 2024-09-21T08:30:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.109315396 2024-09-21T08:30:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.636265+0000", 2024-09-21T08:30:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015202999999999999 2024-09-21T08:30:51.481 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:30:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.636448+0000", 2024-09-21T08:30:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000182957 2024-09-21T08:30:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.482 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637097+0000", 2024-09-21T08:30:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064855999999999996 2024-09-21T08:30:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.483 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637266+0000", 2024-09-21T08:30:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016917399999999999 2024-09-21T08:30:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637374+0000", 2024-09-21T08:30:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107899 2024-09-21T08:30:51.484 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637451+0000", 2024-09-21T08:30:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7486999999999999e-05 2024-09-21T08:30:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.485 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637551+0000", 2024-09-21T08:30:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9592000000000005e-05 2024-09-21T08:30:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.486 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2116 7.b e228/157 7:df2b42d9:::smithi13779221-36:head v 228'352, mlcod=225'348)", 2024-09-21T08:30:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.532769+0000", 2024-09-21T08:30:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 229.87845175300001, 2024-09-21T08:30:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020877380000000001, 2024-09-21T08:30:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.487 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.532769+0000", 2024-09-21T08:30:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.488 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.532769+0000", 2024-09-21T08:30:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.532772+0000", 2024-09-21T08:30:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2729999999999998e-06 2024-09-21T08:30:51.489 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542209+0000", 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094370849999999996 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542235+0000", 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6072999999999999e-05 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.490 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542247+0000", 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2233e-05 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542328+0000", 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0271000000000004e-05 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542360+0000", 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2166999999999999e-05 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.491 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553599+0000", 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011239416 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553646+0000", 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6862e-05 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2233 7.8 7:17170b1f:::smithi13779221-4:head [write 2798226~596838] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.553031+0000", 2024-09-21T08:30:51.492 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 219.85818907800001, 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021618093000000001, 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2233 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553031+0000", 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.493 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553031+0000", 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553036+0000", 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7879999999999997e-06 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553403+0000", 2024-09-21T08:30:51.494 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036713799999999998 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553435+0000", 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1219e-05 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553451+0000", 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6841e-05 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.495 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553587+0000", 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013595600000000001 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553713+0000", 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012593700000000001 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553874+0000", 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160798 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.496 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.558304+0000", 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044295280000000003 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.569637+0000", 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011333609 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574517+0000", 2024-09-21T08:30:51.497 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048793129999999997 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574592+0000", 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5236999999999998e-05 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574650+0000", 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7729000000000002e-05 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.498 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2234 7.8 7:17170b1f:::smithi13779221-4:head [setxattr _header (56) in=7b,truncate 3445468] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.553487+0000", 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 219.85773357900001, 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021391090000000001, 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.499 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2234 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553487+0000", 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553487+0000", 2024-09-21T08:30:51.500 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553489+0000", 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.215e-06 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553495+0000", 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7389999999999996e-06 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553500+0000", 2024-09-21T08:30:51.501 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2360000000000004e-06 2024-09-21T08:30:51.502 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.502 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.502 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:51.503 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.503 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553505+0000", 2024-09-21T08:30:51.503 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3009999999999997e-06 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.554931+0000", 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00142592 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555047+0000", 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115139 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:30:51.504 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555183+0000", 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013642099999999999 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.558431+0000", 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032475770000000002 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.569645+0000", 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011214783000000001 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.505 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574711+0000", 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050659449999999996 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574768+0000", 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6415000000000001e-05 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574878+0000", 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110399 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.506 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2235 7.8 7:17170b1f:::smithi13779221-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 219.857707823, 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.02162265, 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2235 2024-09-21T08:30:51.507 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:30:51.508 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3000000000000001e-07 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553516+0000", 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5900000000000002e-06 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553522+0000", 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8980000000000001e-06 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.509 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553526+0000", 2024-09-21T08:30:51.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7360000000000001e-06 2024-09-21T08:30:51.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555581+0000", 2024-09-21T08:30:51.510 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020548799999999998 2024-09-21T08:30:51.510 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.510 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.510 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:30:51.510 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555650+0000", 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9046000000000003e-05 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574919+0000", 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019268464999999999 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.575027+0000", 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108526 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.511 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.575135+0000", 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107979 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2831 7.8 7:144c2ec5:::smithi13779221-45:head [copy-from ver 295] snapc 64=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:01.538037+0000", 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 169.87318345700001, 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020800902, 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.512 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2831 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538037+0000", 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538037+0000", 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.513 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538039+0000", 2024-09-21T08:30:51.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9099999999999999e-06 2024-09-21T08:30:51.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.514 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.514 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.514 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538047+0000", 2024-09-21T08:30:51.514 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.79e-06 2024-09-21T08:30:51.514 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.515 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538058+0000", 2024-09-21T08:30:51.515 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1141999999999999e-05 2024-09-21T08:30:51.515 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.515 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.515 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538068+0000", 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6330000000000008e-06 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538117+0000", 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9360000000000002e-05 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538218+0000", 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101282 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.516 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.548519+0000", 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010300971000000001 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.557646+0000", 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0091271570000000003 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558736+0000", 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001089996 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.517 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558773+0000", 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6785000000000001e-05 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558810+0000", 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7058999999999999e-05 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558838+0000", 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7817000000000002e-05 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.518 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3157 7.12 7:4ccee2c4:::smithi13779221-50:head [copy-from ver 217] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.573639+0000", 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 138.83758104699999, 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.02873676, 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3157 2024-09-21T08:30:51.519 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.520 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.520 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.520 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573639+0000", 2024-09-21T08:30:51.520 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.520 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.520 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.520 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.520 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573639+0000", 2024-09-21T08:30:51.520 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573642+0000", 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.041e-06 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573649+0000", 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6340000000000001e-06 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.521 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573662+0000", 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3263e-05 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573670+0000", 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.418e-06 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573726+0000", 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6103000000000001e-05 2024-09-21T08:30:51.522 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573839+0000", 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113186 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.586583+0000", 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012743775000000001 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594268+0000", 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076849559999999997 2024-09-21T08:30:51.523 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602175+0000", 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079072770000000007 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602261+0000", 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6271000000000001e-05 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602327+0000", 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5827000000000005e-05 2024-09-21T08:30:51.524 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602376+0000", 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9008999999999998e-05 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3159 7.12 7:4d793d31:::smithi13779221-9:head [copy-from ver 303] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.573715+0000", 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 138.8375054, 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019914487000000002, 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.525 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3159 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573715+0000", 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.526 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573715+0000", 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573716+0000", 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.06e-07 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573719+0000", 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.371e-06 2024-09-21T08:30:51.527 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573724+0000", 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4299999999999997e-06 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573729+0000", 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9400000000000001e-06 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.574567+0000", 2024-09-21T08:30:51.528 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00083730199999999995 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.574658+0000", 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1775999999999997e-05 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.580114+0000", 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054551759999999999 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.591315+0000", 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011201825 2024-09-21T08:30:51.529 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593389+0000", 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020731130000000001 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593531+0000", 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000142144 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.530 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593575+0000", 2024-09-21T08:30:51.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4229e-05 2024-09-21T08:30:51.531 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.531 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.531 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.531 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593630+0000", 2024-09-21T08:30:51.531 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4675000000000001e-05 2024-09-21T08:30:51.531 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.531 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.531 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.532 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.532 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:3441 7.15 e343/339 7:a9dd6c7d:::smithi13779221-47:head v 343'399, mlcod=0'0)", 2024-09-21T08:30:51.532 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:56.572197+0000", 2024-09-21T08:30:51.532 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 114.839023408, 2024-09-21T08:30:51.532 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019269069999999999, 2024-09-21T08:30:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.533 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572197+0000", 2024-09-21T08:30:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572197+0000", 2024-09-21T08:30:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.534 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572198+0000", 2024-09-21T08:30:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7100000000000002e-07 2024-09-21T08:30:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572500+0000", 2024-09-21T08:30:51.535 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030198600000000002 2024-09-21T08:30:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572508+0000", 2024-09-21T08:30:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8799999999999997e-06 2024-09-21T08:30:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.536 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572518+0000", 2024-09-21T08:30:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6269999999999995e-06 2024-09-21T08:30:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572589+0000", 2024-09-21T08:30:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0644999999999999e-05 2024-09-21T08:30:51.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572612+0000", 2024-09-21T08:30:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3623000000000001e-05 2024-09-21T08:30:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.538 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591357+0000", 2024-09-21T08:30:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018744255000000001 2024-09-21T08:30:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591466+0000", 2024-09-21T08:30:51.539 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109583 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:4006 7.9 e398/339 7:9247b5b1:::smithi13779221-17:head v 398'338, mlcod=376'335)", 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:29:51.586649+0000", 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 59.824571935000002, 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020193633999999999, 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.540 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586649+0000", 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586649+0000", 2024-09-21T08:30:51.541 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586650+0000", 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.545e-06 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587156+0000", 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050577 2024-09-21T08:30:51.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587177+0000", 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1353999999999999e-05 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587192+0000", 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5214e-05 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587422+0000", 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022968399999999999 2024-09-21T08:30:51.543 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587586+0000", 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016406799999999999 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.606793+0000", 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019206392999999999 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.606842+0000", 2024-09-21T08:30:51.544 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9605999999999999e-05 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:4168 7.b e409/293 7:d7506421:::smithi13779221-13:head v 409'639, mlcod=407'636)", 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 48.828841904999997, 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.083502604999999994, 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.545 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2e-07 2024-09-21T08:30:51.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582940+0000", 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00056103800000000003 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582952+0000", 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1732999999999999e-05 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582961+0000", 2024-09-21T08:30:51.547 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8000000000000004e-06 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.583031+0000", 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0870000000000004e-05 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.583059+0000", 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7206e-05 2024-09-21T08:30:51.548 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665820+0000", 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.082760995000000004 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665881+0000", 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1543000000000004e-05 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.549 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:4169 7.b e409/293 7:d7506421:::smithi13779221-13:head v 409'640, mlcod=407'636)", 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.589410+0000", 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 48.821811029000003, 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.080226373000000004, 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589410+0000", 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.550 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589410+0000", 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589412+0000", 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9840000000000001e-06 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589813+0000", 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040080299999999998 2024-09-21T08:30:51.551 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665555+0000", 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.075741557000000001 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665569+0000", 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3961000000000001e-05 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.552 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.668283+0000", 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027142099999999999 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.668307+0000", 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3487999999999999e-05 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669596+0000", 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012892540000000001 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:30:51.553 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:30:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:30:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669636+0000", 2024-09-21T08:30:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0116e-05 2024-09-21T08:30:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:30:51.554 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:30:51.554 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:30:51.575 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:51.576 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:51.576 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:1835: done (0 left) 2024-09-21T08:30:51.587 INFO:tasks.rados.rados.0.smithi137.stdout:1836: snap_remove snap 193 2024-09-21T08:30:51.677 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:51.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:51.699 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:51.699 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:51.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:51 smithi137 ceph-mon[29634]: pgmap v590: 178 pgs: 10 unknown, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 35 active+clean; 552 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 372/1440 objects degraded (25.833%) 2024-09-21T08:30:51.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:51 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T08:30:51.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:51 smithi137 ceph-mon[29634]: osdmap e457: 8 total, 6 up, 4 in 2024-09-21T08:30:51.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:51 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 372/1440 objects degraded (25.833%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:51.758 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:51 smithi137 ceph-mon[29634]: osdmap e458: 8 total, 6 up, 4 in 2024-09-21T08:30:51.758 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:51 smithi137 ceph-mon[41044]: pgmap v590: 178 pgs: 10 unknown, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 35 active+clean; 552 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 372/1440 objects degraded (25.833%) 2024-09-21T08:30:51.758 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:51 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T08:30:51.758 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:51 smithi137 ceph-mon[41044]: osdmap e457: 8 total, 6 up, 4 in 2024-09-21T08:30:51.759 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:51 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 372/1440 objects degraded (25.833%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:51.759 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:51 smithi137 ceph-mon[41044]: osdmap e458: 8 total, 6 up, 4 in 2024-09-21T08:30:51.800 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:51.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:51.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:51.822 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:51.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:51.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:51.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:51.945 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:51.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:51 smithi165 ceph-mon[32893]: pgmap v590: 178 pgs: 10 unknown, 3 active+clean+snaptrim, 72 active+undersized, 58 active+undersized+degraded, 35 active+clean; 552 MiB data, 1.2 GiB used, 356 GiB / 358 GiB avail; 372/1440 objects degraded (25.833%) 2024-09-21T08:30:51.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:51 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": ".mgr", "var": "pgp_num_actual", "val": "11"}]': finished 2024-09-21T08:30:51.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:51 smithi165 ceph-mon[32893]: osdmap e457: 8 total, 6 up, 4 in 2024-09-21T08:30:51.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:51 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 372/1440 objects degraded (25.833%), 58 pgs degraded, 130 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:51.946 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:51 smithi165 ceph-mon[32893]: osdmap e458: 8 total, 6 up, 4 in 2024-09-21T08:30:52.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:52.066 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:52.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:52.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:52.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:52.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T08:30:52.194 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:52.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:52.195 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:52.239 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:52.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:30:51 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:30:52.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:52.321 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:52.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:52.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:52.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:52.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:52.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:52.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:52.549 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:52.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:52.572 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:52.572 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:52.592 INFO:tasks.rados.rados.0.smithi137.stdout:1836: done (0 left) 2024-09-21T08:30:52.592 INFO:tasks.rados.rados.0.smithi137.stdout:1837: snap_remove snap 180 2024-09-21T08:30:52.673 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:52.694 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:52.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:52.695 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:52.748 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:52 smithi137 ceph-mon[29634]: 6.a scrub starts 2024-09-21T08:30:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:52 smithi137 ceph-mon[29634]: 6.a scrub ok 2024-09-21T08:30:52.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:52 smithi137 ceph-mon[29634]: osdmap e459: 8 total, 6 up, 4 in 2024-09-21T08:30:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:52 smithi137 ceph-mon[41044]: 6.a scrub starts 2024-09-21T08:30:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:52 smithi137 ceph-mon[41044]: 6.a scrub ok 2024-09-21T08:30:52.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:52 smithi137 ceph-mon[41044]: osdmap e459: 8 total, 6 up, 4 in 2024-09-21T08:30:52.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:52.817 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:52.817 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:52.818 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:52.918 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:52.944 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:52.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:52.945 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:53.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:52 smithi165 ceph-mon[32893]: 6.a scrub starts 2024-09-21T08:30:53.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:52 smithi165 ceph-mon[32893]: 6.a scrub ok 2024-09-21T08:30:53.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:52 smithi165 ceph-mon[32893]: osdmap e459: 8 total, 6 up, 4 in 2024-09-21T08:30:53.046 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:53.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:53.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:53.072 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:53.168 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1] out_osds: [4, 6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:30:53.168 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:30:53.169 INFO:tasks.thrashosds.thrasher:Adding osd 4 2024-09-21T08:30:53.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd in 4 2024-09-21T08:30:53.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:53.188 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:30:53.234 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:53.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:53.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:30:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:30:53.257 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:30:53.257 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:30:53.336 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:53.360 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:53.361 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:53.361 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:53.462 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:53.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:53.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:53.586 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:30:53.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:53.591 INFO:tasks.rados.rados.0.smithi137.stdout:1837: done (0 left) 2024-09-21T08:30:53.591 INFO:tasks.rados.rados.0.smithi137.stdout:1838: read oid 18 snap -1 2024-09-21T08:30:53.591 INFO:tasks.rados.rados.0.smithi137.stdout:1838: expect (ObjNum 590 snap 172 seq_num 590) 2024-09-21T08:30:53.592 INFO:tasks.rados.rados.0.smithi137.stdout:1839: write oid 40 current snap is 195 2024-09-21T08:30:53.592 INFO:tasks.rados.rados.0.smithi137.stdout:1839: seq_num 654 ranges {591778=419204,1637106=741879,2910049=81729} 2024-09-21T08:30:53.598 INFO:tasks.rados.rados.0.smithi137.stdout:1839: writing smithi13779221-40 from 591778 to 1010982 tid 1 2024-09-21T08:30:53.607 INFO:tasks.rados.rados.0.smithi137.stdout:1839: writing smithi13779221-40 from 1637106 to 2378985 tid 2 2024-09-21T08:30:53.611 INFO:tasks.rados.rados.0.smithi137.stdout:1839: writing smithi13779221-40 from 2910049 to 2991778 tid 3 2024-09-21T08:30:53.611 INFO:tasks.rados.rados.0.smithi137.stdout:1840: write oid 38 current snap is 195 2024-09-21T08:30:53.611 INFO:tasks.rados.rados.0.smithi137.stdout:1840: seq_num 655 ranges {608585=494453,1579235=601890,2821119=187466} 2024-09-21T08:30:53.617 INFO:tasks.rados.rados.0.smithi137.stdout:1840: writing smithi13779221-38 from 608585 to 1103038 tid 1 2024-09-21T08:30:53.619 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:53.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:53.620 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:53.623 INFO:tasks.rados.rados.0.smithi137.stdout:1840: writing smithi13779221-38 from 1579235 to 2181125 tid 2 2024-09-21T08:30:53.628 INFO:tasks.rados.rados.0.smithi137.stdout:1840: writing smithi13779221-38 from 2821119 to 3008585 tid 3 2024-09-21T08:30:53.628 INFO:tasks.rados.rados.0.smithi137.stdout:1841: snap_create 2024-09-21T08:30:53.628 INFO:tasks.rados.rados.0.smithi137.stdout:1838: expect (ObjNum 590 snap 172 seq_num 590) 2024-09-21T08:30:53.721 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:53.742 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:53.743 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:53.743 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:53.758 INFO:tasks.rados.rados.0.smithi137.stdout:1839: finishing write tid 1 to smithi13779221-40 2024-09-21T08:30:53.758 INFO:tasks.rados.rados.0.smithi137.stdout:1839: finishing write tid 2 to smithi13779221-40 2024-09-21T08:30:53.758 INFO:tasks.rados.rados.0.smithi137.stdout:1839: finishing write tid 3 to smithi13779221-40 2024-09-21T08:30:53.758 INFO:tasks.rados.rados.0.smithi137.stdout:1839: finishing write tid 4 to smithi13779221-40 2024-09-21T08:30:53.758 INFO:tasks.rados.rados.0.smithi137.stdout:1839: finishing write tid 5 to smithi13779221-40 2024-09-21T08:30:53.758 INFO:tasks.rados.rados.0.smithi137.stdout:1839: oid 40 updating version 0 to 332 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1839: oid 40 updating version 332 to 335 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1839: oid 40 version 335 is already newer than 333 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1839: oid 40 version 335 is already newer than 334 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 335 (ObjNum 654 snap 195 seq_num 654) dirty exists 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1839: left oid 40 (ObjNum 654 snap 195 seq_num 654) 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1840: finishing write tid 1 to smithi13779221-38 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1840: finishing write tid 2 to smithi13779221-38 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1840: finishing write tid 3 to smithi13779221-38 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1840: finishing write tid 4 to smithi13779221-38 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1840: finishing write tid 5 to smithi13779221-38 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1840: oid 38 updating version 0 to 762 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1840: oid 38 updating version 762 to 763 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1840: oid 38 version 763 is already newer than 758 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:1840: oid 38 version 763 is already newer than 759 2024-09-21T08:30:53.759 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 763 (ObjNum 655 snap 195 seq_num 655) dirty exists 2024-09-21T08:30:53.760 INFO:tasks.rados.rados.0.smithi137.stdout:1840: left oid 38 (ObjNum 655 snap 195 seq_num 655) 2024-09-21T08:30:53.838 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:53.838 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[29634]: pgmap v593: 178 pgs: 10 unknown, 3 active+clean+snaptrim, 73 active+undersized, 57 active+undersized+degraded, 35 active+clean; 575 MiB data, 1.3 GiB used, 356 GiB / 358 GiB avail; 385/1485 objects degraded (25.926%) 2024-09-21T08:30:53.838 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[29634]: Health check failed: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2024-09-21T08:30:53.838 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2652990832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:30:53.838 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:53.838 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2652990832' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:30:53.839 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[29634]: osdmap e460: 8 total, 6 up, 4 in 2024-09-21T08:30:53.839 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:53.839 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[41044]: pgmap v593: 178 pgs: 10 unknown, 3 active+clean+snaptrim, 73 active+undersized, 57 active+undersized+degraded, 35 active+clean; 575 MiB data, 1.3 GiB used, 356 GiB / 358 GiB avail; 385/1485 objects degraded (25.926%) 2024-09-21T08:30:53.839 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[41044]: Health check failed: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2024-09-21T08:30:53.839 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2652990832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:30:53.839 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:53.839 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2652990832' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:30:53.839 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:53 smithi137 ceph-mon[41044]: osdmap e460: 8 total, 6 up, 4 in 2024-09-21T08:30:53.844 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:53.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:53.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:53.868 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:53.969 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:53.983 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T08:30:53.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:53.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:53.997 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:53 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:30:54.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:53 smithi165 ceph-mon[32893]: pgmap v593: 178 pgs: 10 unknown, 3 active+clean+snaptrim, 73 active+undersized, 57 active+undersized+degraded, 35 active+clean; 575 MiB data, 1.3 GiB used, 356 GiB / 358 GiB avail; 385/1485 objects degraded (25.926%) 2024-09-21T08:30:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:53 smithi165 ceph-mon[32893]: Health check failed: Reduced data availability: 3 pgs inactive (PG_AVAILABILITY) 2024-09-21T08:30:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:53 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2652990832' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:30:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:53 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:30:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:53 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2652990832' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:30:54.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:53 smithi165 ceph-mon[32893]: osdmap e460: 8 total, 6 up, 4 in 2024-09-21T08:30:54.098 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:54.119 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:54.120 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:54.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:54.221 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:54.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:54.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:54.248 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:54.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:54.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:54.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:54.372 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:54.472 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:54.494 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:54.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:54.495 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:54.592 INFO:tasks.rados.rados.0.smithi137.stdout:1838: done (3 left) 2024-09-21T08:30:54.592 INFO:tasks.rados.rados.0.smithi137.stdout:1839: done (2 left) 2024-09-21T08:30:54.592 INFO:tasks.rados.rados.0.smithi137.stdout:1840: done (1 left) 2024-09-21T08:30:54.593 INFO:tasks.rados.rados.0.smithi137.stdout:1841: done (0 left) 2024-09-21T08:30:54.593 INFO:tasks.rados.rados.0.smithi137.stdout:1842: copy_from oid 46 from oid 43 current snap is 196 2024-09-21T08:30:54.593 INFO:tasks.rados.rados.0.smithi137.stdout:1843: write (excl) oid 28 current snap is 196 2024-09-21T08:30:54.593 INFO:tasks.rados.rados.0.smithi137.stdout:1843: seq_num 656 ranges {625392=569702,1637717=745548,2848542=176850} 2024-09-21T08:30:54.595 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:54.601 INFO:tasks.rados.rados.0.smithi137.stdout:1843: writing smithi13779221-28 from 625392 to 1195094 tid 1 2024-09-21T08:30:54.608 INFO:tasks.rados.rados.0.smithi137.stdout:1843: writing smithi13779221-28 from 1637717 to 2383265 tid 2 2024-09-21T08:30:54.612 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:54.612 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:54.612 INFO:tasks.rados.rados.0.smithi137.stdout:1843: writing smithi13779221-28 from 2848542 to 3025392 tid 3 2024-09-21T08:30:54.613 INFO:tasks.rados.rados.0.smithi137.stdout:1842: finishing copy_from to smithi13779221-46 2024-09-21T08:30:54.613 INFO:tasks.rados.rados.0.smithi137.stdout:1842: got expected ENOENT (src dne) 2024-09-21T08:30:54.613 INFO:tasks.rados.rados.0.smithi137.stdout:1842: finishing copy_from racing read to smithi13779221-46 2024-09-21T08:30:54.613 INFO:tasks.rados.rados.0.smithi137.stdout:1843: finishing write tid 1 to smithi13779221-28 2024-09-21T08:30:54.613 INFO:tasks.rados.rados.0.smithi137.stdout:1842: done (1 left) 2024-09-21T08:30:54.613 INFO:tasks.rados.rados.0.smithi137.stdout:1844: snap_create 2024-09-21T08:30:54.614 INFO:tasks.rados.rados.0.smithi137.stdout:1843: finishing write tid 2 to smithi13779221-28 2024-09-21T08:30:54.615 INFO:tasks.rados.rados.0.smithi137.stdout:1843: finishing write tid 3 to smithi13779221-28 2024-09-21T08:30:54.615 INFO:tasks.rados.rados.0.smithi137.stdout:1843: finishing write tid 4 to smithi13779221-28 2024-09-21T08:30:54.615 INFO:tasks.rados.rados.0.smithi137.stdout:1843: finishing write tid 5 to smithi13779221-28 2024-09-21T08:30:54.616 INFO:tasks.rados.rados.0.smithi137.stdout:1843: oid 28 updating version 0 to 380 2024-09-21T08:30:54.616 INFO:tasks.rados.rados.0.smithi137.stdout:1843: oid 28 version 380 is already newer than 378 2024-09-21T08:30:54.616 INFO:tasks.rados.rados.0.smithi137.stdout:1843: oid 28 version 380 is already newer than 379 2024-09-21T08:30:54.616 INFO:tasks.rados.rados.0.smithi137.stdout:1843: oid 28 updating version 380 to 381 2024-09-21T08:30:54.616 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 381 (ObjNum 656 snap 196 seq_num 656) dirty exists 2024-09-21T08:30:54.616 INFO:tasks.rados.rados.0.smithi137.stdout:1843: left oid 28 (ObjNum 656 snap 196 seq_num 656) 2024-09-21T08:30:54.629 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:54.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:54.631 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:54.732 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:54.754 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:54.755 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:54.755 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:54.856 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:54.884 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:54.884 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:54.884 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:54.985 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:55.011 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:55.012 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:55.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:55.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:55.138 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:55.139 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:55.139 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:55.240 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:55.265 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:55.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:55.267 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:55.367 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:55.393 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:55.394 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:55.395 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:55.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:55.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:55.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:55.519 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:55.596 INFO:tasks.rados.rados.0.smithi137.stdout:1843: done (1 left) 2024-09-21T08:30:55.596 INFO:tasks.rados.rados.0.smithi137.stdout:1844: done (0 left) 2024-09-21T08:30:55.596 INFO:tasks.rados.rados.0.smithi137.stdout:1845: write oid 18 current snap is 197 2024-09-21T08:30:55.597 INFO:tasks.rados.rados.0.smithi137.stdout:1845: seq_num 657 ranges {642199=644951,1696199=605559,2992318=49881} 2024-09-21T08:30:55.597 INFO:teuthology.orchestra.run.smithi137.stderr:marked in osd.4. 2024-09-21T08:30:55.603 INFO:tasks.rados.rados.0.smithi137.stdout:1845: writing smithi13779221-18 from 642199 to 1287150 tid 1 2024-09-21T08:30:55.609 INFO:tasks.rados.rados.0.smithi137.stdout:1845: writing smithi13779221-18 from 1696199 to 2301758 tid 2 2024-09-21T08:30:55.614 INFO:tasks.rados.rados.0.smithi137.stdout:1845: writing smithi13779221-18 from 2992318 to 3042199 tid 3 2024-09-21T08:30:55.614 INFO:tasks.rados.rados.0.smithi137.stdout:1846: delete oid 10 current snap is 197 2024-09-21T08:30:55.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:55.640 INFO:tasks.rados.rados.0.smithi137.stdout:1845: finishing write tid 1 to smithi13779221-18 2024-09-21T08:30:55.641 INFO:tasks.rados.rados.0.smithi137.stdout:1846: done (1 left) 2024-09-21T08:30:55.641 INFO:tasks.rados.rados.0.smithi137.stdout:1847: delete oid 7 current snap is 197 2024-09-21T08:30:55.641 INFO:tasks.rados.rados.0.smithi137.stdout:1845: finishing write tid 2 to smithi13779221-18 2024-09-21T08:30:55.641 INFO:tasks.rados.rados.0.smithi137.stdout:1845: finishing write tid 3 to smithi13779221-18 2024-09-21T08:30:55.642 INFO:tasks.rados.rados.0.smithi137.stdout:1845: finishing write tid 4 to smithi13779221-18 2024-09-21T08:30:55.642 INFO:tasks.rados.rados.0.smithi137.stdout:1845: finishing write tid 5 to smithi13779221-18 2024-09-21T08:30:55.642 INFO:tasks.rados.rados.0.smithi137.stdout:1845: oid 18 updating version 0 to 232 2024-09-21T08:30:55.642 INFO:tasks.rados.rados.0.smithi137.stdout:1845: oid 18 updating version 232 to 233 2024-09-21T08:30:55.642 INFO:tasks.rados.rados.0.smithi137.stdout:1845: oid 18 version 233 is already newer than 231 2024-09-21T08:30:55.642 INFO:tasks.rados.rados.0.smithi137.stdout:1845: oid 18 version 233 is already newer than 230 2024-09-21T08:30:55.643 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 233 (ObjNum 657 snap 197 seq_num 657) dirty exists 2024-09-21T08:30:55.643 INFO:tasks.rados.rados.0.smithi137.stdout:1845: left oid 18 (ObjNum 657 snap 197 seq_num 657) 2024-09-21T08:30:55.647 INFO:tasks.rados.rados.0.smithi137.stdout:1845: done (1 left) 2024-09-21T08:30:55.647 INFO:tasks.rados.rados.0.smithi137.stdout:1847: done (0 left) 2024-09-21T08:30:55.647 INFO:tasks.rados.rados.0.smithi137.stdout:1848: read oid 33 snap 186 2024-09-21T08:30:55.647 INFO:tasks.rados.rados.0.smithi137.stdout:1848: expect (ObjNum 624 snap 184 seq_num 624) 2024-09-21T08:30:55.647 INFO:tasks.rados.rados.0.smithi137.stdout:1849: write (excl) oid 50 current snap is 197 2024-09-21T08:30:55.647 INFO:tasks.rados.rados.0.smithi137.stdout:1849: seq_num 660 ranges {692620=470698,1755292=752886} 2024-09-21T08:30:55.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:55 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mon-a[29611]: 2024-09-21T08:30:55.375+0000 7f1058bb7640 -1 mon.a@0(leader).osd e461 definitely_dead 0 2024-09-21T08:30:55.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:55 smithi137 ceph-mon[29634]: pgmap v596: 178 pgs: 10 unknown, 3 active+clean+snaptrim, 73 active+undersized, 57 active+undersized+degraded, 35 active+clean; 575 MiB data, 1.3 GiB used, 356 GiB / 358 GiB avail; 385/1485 objects degraded (25.926%) 2024-09-21T08:30:55.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:55 smithi137 ceph-mon[29634]: osdmap e461: 8 total, 6 up, 4 in 2024-09-21T08:30:55.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:55 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3135445935' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-21T08:30:55.648 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:55 smithi137 ceph-mon[29634]: from='client.45803 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-21T08:30:55.648 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:55 smithi137 ceph-mon[41044]: pgmap v596: 178 pgs: 10 unknown, 3 active+clean+snaptrim, 73 active+undersized, 57 active+undersized+degraded, 35 active+clean; 575 MiB data, 1.3 GiB used, 356 GiB / 358 GiB avail; 385/1485 objects degraded (25.926%) 2024-09-21T08:30:55.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:55 smithi137 ceph-mon[41044]: osdmap e461: 8 total, 6 up, 4 in 2024-09-21T08:30:55.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:55 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3135445935' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-21T08:30:55.649 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:55 smithi137 ceph-mon[41044]: from='client.45803 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-21T08:30:55.650 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:30:55.651 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:55.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:55.652 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:55.653 INFO:tasks.rados.rados.0.smithi137.stdout:1849: writing smithi13779221-50 from 692620 to 1163318 tid 1 2024-09-21T08:30:55.661 INFO:tasks.rados.rados.0.smithi137.stdout:1849: writing smithi13779221-50 from 1755292 to 2508178 tid 2 2024-09-21T08:30:55.661 INFO:tasks.rados.rados.0.smithi137.stdout:1850: delete oid 20 current snap is 197 2024-09-21T08:30:55.662 INFO:tasks.rados.rados.0.smithi137.stdout:1849: finishing write tid 1 to smithi13779221-50 2024-09-21T08:30:55.663 INFO:tasks.rados.rados.0.smithi137.stdout:1850: done (2 left) 2024-09-21T08:30:55.663 INFO:tasks.rados.rados.0.smithi137.stdout:1851: copy_from oid 42 from oid 13 current snap is 197 2024-09-21T08:30:55.663 INFO:tasks.rados.rados.0.smithi137.stdout:1852: read oid 12 snap -1 2024-09-21T08:30:55.663 INFO:tasks.rados.rados.0.smithi137.stdout:1852: expect deleted 2024-09-21T08:30:55.664 INFO:tasks.rados.rados.0.smithi137.stdout:1853: copy_from oid 11 from oid 46 current snap is 197 2024-09-21T08:30:55.664 INFO:tasks.rados.rados.0.smithi137.stdout:1854: copy_from oid 3 from oid 4 current snap is 197 2024-09-21T08:30:55.664 INFO:tasks.rados.rados.0.smithi137.stdout:1855: snap_remove snap 184 2024-09-21T08:30:55.670 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:30:55.670 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:30:55.688 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:30:55.688 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:30:55.704 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:30:55.704 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:30:55.721 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:30:55.721 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:30:55.737 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:30:55.738 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:30:55.753 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:55.754 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:30:55.755 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:30:55.755 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:30:55.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:55.777 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:55.777 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:55.878 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:55.899 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:55.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:55.900 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:56.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:55 smithi165 ceph-mon[32893]: pgmap v596: 178 pgs: 10 unknown, 3 active+clean+snaptrim, 73 active+undersized, 57 active+undersized+degraded, 35 active+clean; 575 MiB data, 1.3 GiB used, 356 GiB / 358 GiB avail; 385/1485 objects degraded (25.926%) 2024-09-21T08:30:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:55 smithi165 ceph-mon[32893]: osdmap e461: 8 total, 6 up, 4 in 2024-09-21T08:30:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:55 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3135445935' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-21T08:30:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:55 smithi165 ceph-mon[32893]: from='client.45803 ' entity='client.admin' cmd=[{"prefix": "osd in", "ids": ["4"]}]: dispatch 2024-09-21T08:30:56.023 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:56.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:56.024 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:56.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:56.146 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:56.147 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:56.147 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:56.248 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:56.271 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:56.271 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:56.271 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:56.338 INFO:tasks.thrashosds.thrasher:Added osd 4 2024-09-21T08:30:56.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:56.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:56.396 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:56.396 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:56.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:56.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:56.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:56.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:56.542 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:30:56.587 INFO:tasks.rados.rados.0.smithi137.stdout:1855: done (6 left) 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:1856: snap_create 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:1849: finishing write tid 2 to smithi13779221-50 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:1849: finishing write tid 3 to smithi13779221-50 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:1849: finishing write tid 4 to smithi13779221-50 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:1849: oid 50 updating version 0 to 522 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:1849: oid 50 updating version 522 to 523 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:1849: oid 50 updating version 523 to 524 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 524 (ObjNum 660 snap 197 seq_num 660) dirty exists 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:1849: left oid 50 (ObjNum 660 snap 197 seq_num 660) 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:1851: finishing copy_from racing read to smithi13779221-42 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:1851: finishing copy_from to smithi13779221-42 2024-09-21T08:30:56.588 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 693 (ObjNum 599 snap 174 seq_num 599) dirty exists 2024-09-21T08:30:56.610 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:56.620 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:30:56.653 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:56.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:56.654 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:30:56.755 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:56.778 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:56.778 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:56.778 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:56.879 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:56.900 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:56.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:56.901 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:56.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:56 smithi137 ceph-mon[29634]: from='client.45803 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T08:30:56.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:56 smithi137 ceph-mon[29634]: osdmap e462: 8 total, 6 up, 5 in 2024-09-21T08:30:56.989 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:56 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3450595992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:56.990 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:56 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 385/1485 objects degraded (25.926%), 57 pgs degraded, 129 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:56.990 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:56 smithi137 ceph-mon[29634]: osdmap e463: 8 total, 6 up, 5 in 2024-09-21T08:30:56.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:56 smithi137 ceph-mon[41044]: from='client.45803 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T08:30:56.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:56 smithi137 ceph-mon[41044]: osdmap e462: 8 total, 6 up, 5 in 2024-09-21T08:30:56.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:56 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3450595992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:56.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:56 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 385/1485 objects degraded (25.926%), 57 pgs degraded, 129 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:56.990 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:56 smithi137 ceph-mon[41044]: osdmap e463: 8 total, 6 up, 5 in 2024-09-21T08:30:57.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:56 smithi165 ceph-mon[32893]: from='client.45803 ' entity='client.admin' cmd='[{"prefix": "osd in", "ids": ["4"]}]': finished 2024-09-21T08:30:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:56 smithi165 ceph-mon[32893]: osdmap e462: 8 total, 6 up, 5 in 2024-09-21T08:30:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:56 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3450595992' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:30:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:56 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 385/1485 objects degraded (25.926%), 57 pgs degraded, 129 pgs undersized (PG_DEGRADED) 2024-09-21T08:30:57.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:56 smithi165 ceph-mon[32893]: osdmap e463: 8 total, 6 up, 5 in 2024-09-21T08:30:57.018 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:30:57.018 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:30:57.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:57.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:57.026 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:57.063 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:57.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:57.149 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:57.150 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:57.150 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:57.251 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:57.273 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:57.273 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:57.274 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:57.374 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:57.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:57.396 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:57.397 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:57.497 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:57.518 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:57.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:57.522 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:57.591 INFO:tasks.rados.rados.0.smithi137.stdout:1849: done (6 left) 2024-09-21T08:30:57.591 INFO:tasks.rados.rados.0.smithi137.stdout:1851: done (5 left) 2024-09-21T08:30:57.591 INFO:tasks.rados.rados.0.smithi137.stdout:1852: done (4 left) 2024-09-21T08:30:57.591 INFO:tasks.rados.rados.0.smithi137.stdout:1856: done (3 left) 2024-09-21T08:30:57.591 INFO:tasks.rados.rados.0.smithi137.stdout:1857: read oid 47 snap 186 2024-09-21T08:30:57.591 INFO:tasks.rados.rados.0.smithi137.stdout:1857: expect deleted 2024-09-21T08:30:57.591 INFO:tasks.rados.rados.0.smithi137.stdout:1858: snap_remove snap 197 2024-09-21T08:30:57.623 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:57.653 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:57.654 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:57.654 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:57.694 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:57.694 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:57.755 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:57.779 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:57.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:57.780 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:57.880 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:57.906 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:57.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:57.907 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:58.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:57 smithi165 ceph-mon[32893]: pgmap v599: 178 pgs: 2 active+clean+remapped, 61 peering, 4 remapped+peering, 35 unknown, 1 active+clean+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 17 active+clean; 511 MiB data, 1.3 GiB used, 446 GiB / 447 GiB avail; 237/1287 objects degraded (18.415%) 2024-09-21T08:30:58.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:57 smithi165 ceph-mon[32893]: osdmap e464: 8 total, 6 up, 5 in 2024-09-21T08:30:58.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:57 smithi137 ceph-mon[29634]: pgmap v599: 178 pgs: 2 active+clean+remapped, 61 peering, 4 remapped+peering, 35 unknown, 1 active+clean+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 17 active+clean; 511 MiB data, 1.3 GiB used, 446 GiB / 447 GiB avail; 237/1287 objects degraded (18.415%) 2024-09-21T08:30:58.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:57 smithi137 ceph-mon[29634]: osdmap e464: 8 total, 6 up, 5 in 2024-09-21T08:30:58.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:57 smithi137 ceph-mon[41044]: pgmap v599: 178 pgs: 2 active+clean+remapped, 61 peering, 4 remapped+peering, 35 unknown, 1 active+clean+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 17 active+clean; 511 MiB data, 1.3 GiB used, 446 GiB / 447 GiB avail; 237/1287 objects degraded (18.415%) 2024-09-21T08:30:58.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:57 smithi137 ceph-mon[41044]: osdmap e464: 8 total, 6 up, 5 in 2024-09-21T08:30:58.032 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:58.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:58.033 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:58.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:58.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:58.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:58.156 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:58.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:58.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:58.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:58.279 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:30:58.379 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:58.401 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:58.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:58.402 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:58.426 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T08:30:58.504 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:30:58.530 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:58.531 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:58.531 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:30:58.591 INFO:tasks.rados.rados.0.smithi137.stdout:1858: done (4 left) 2024-09-21T08:30:58.591 INFO:tasks.rados.rados.0.smithi137.stdout:1859: delete oid 45 current snap is 198 2024-09-21T08:30:58.591 INFO:tasks.rados.rados.0.smithi137.stdout:1854: finishing copy_from to smithi13779221-3 2024-09-21T08:30:58.592 INFO:tasks.rados.rados.0.smithi137.stdout:1854: got expected ENOENT (src dne) 2024-09-21T08:30:58.592 INFO:tasks.rados.rados.0.smithi137.stdout:1854: finishing copy_from racing read to smithi13779221-3 2024-09-21T08:30:58.592 INFO:tasks.rados.rados.0.smithi137.stdout:1848: expect (ObjNum 624 snap 184 seq_num 624) 2024-09-21T08:30:58.632 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:58.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:58.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:58.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:58.747 INFO:tasks.rados.rados.0.smithi137.stdout:1853: finishing copy_from to smithi13779221-11 2024-09-21T08:30:58.747 INFO:tasks.rados.rados.0.smithi137.stdout:1853: got expected ENOENT (src dne) 2024-09-21T08:30:58.747 INFO:tasks.rados.rados.0.smithi137.stdout:1853: finishing copy_from racing read to smithi13779221-11 2024-09-21T08:30:58.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:58.780 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:58.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:58.781 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:58.882 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:58.903 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:58.904 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:58.904 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:58.969 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:30:59.005 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:59.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:30:59.028 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:59.028 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:59.028 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:59.130 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:59.151 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:59.151 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:59.152 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:59.252 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:59.273 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:59.274 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:59.274 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:59.375 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:30:59.397 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:59.397 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:59.398 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:30:59.496 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:30:59.496 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:30:59.499 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:59.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:59.520 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:59.521 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:59.592 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:30:59.608 INFO:tasks.rados.rados.0.smithi137.stdout:1848: done (4 left) 2024-09-21T08:30:59.608 INFO:tasks.rados.rados.0.smithi137.stdout:1853: done (3 left) 2024-09-21T08:30:59.608 INFO:tasks.rados.rados.0.smithi137.stdout:1854: done (2 left) 2024-09-21T08:30:59.608 INFO:tasks.rados.rados.0.smithi137.stdout:1857: done (1 left) 2024-09-21T08:30:59.608 INFO:tasks.rados.rados.0.smithi137.stdout:1859: done (0 left) 2024-09-21T08:30:59.608 INFO:tasks.rados.rados.0.smithi137.stdout:1860: rollback oid 46 current snap is 198 2024-09-21T08:30:59.608 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 46 to 176 2024-09-21T08:30:59.609 INFO:tasks.rados.rados.0.smithi137.stdout:1861: snap_remove snap 161 2024-09-21T08:30:59.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:59 smithi137 ceph-mon[29634]: pgmap v602: 178 pgs: 2 active+clean+remapped, 61 peering, 4 remapped+peering, 35 unknown, 1 active+clean+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 17 active+clean; 511 MiB data, 1.3 GiB used, 446 GiB / 447 GiB avail; 237/1287 objects degraded (18.415%) 2024-09-21T08:30:59.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:59 smithi137 ceph-mon[29634]: osdmap e465: 8 total, 6 up, 5 in 2024-09-21T08:30:59.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:59 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3577541318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:59.614 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:30:59 smithi137 ceph-mon[29634]: from='client.45815 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:59.614 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:59 smithi137 ceph-mon[41044]: pgmap v602: 178 pgs: 2 active+clean+remapped, 61 peering, 4 remapped+peering, 35 unknown, 1 active+clean+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 17 active+clean; 511 MiB data, 1.3 GiB used, 446 GiB / 447 GiB avail; 237/1287 objects degraded (18.415%) 2024-09-21T08:30:59.615 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:59 smithi137 ceph-mon[41044]: osdmap e465: 8 total, 6 up, 5 in 2024-09-21T08:30:59.615 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:59 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3577541318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:59.615 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:30:59 smithi137 ceph-mon[41044]: from='client.45815 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:30:59.621 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:30:59.650 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:59.651 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:59.652 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:30:59.753 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:30:59.774 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:30:59.775 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:59.775 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:30:59.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:30:59.901 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:30:59.902 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:30:59.902 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:00.003 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:59 smithi165 ceph-mon[32893]: pgmap v602: 178 pgs: 2 active+clean+remapped, 61 peering, 4 remapped+peering, 35 unknown, 1 active+clean+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 17 active+clean; 511 MiB data, 1.3 GiB used, 446 GiB / 447 GiB avail; 237/1287 objects degraded (18.415%) 2024-09-21T08:31:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:59 smithi165 ceph-mon[32893]: osdmap e465: 8 total, 6 up, 5 in 2024-09-21T08:31:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:59 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3577541318' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:00.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:30:59 smithi165 ceph-mon[32893]: from='client.45815 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:00.025 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:00.025 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:00.025 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:00.126 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:00.147 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:00.148 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:00.148 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:00.249 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:00.267 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:31:00.121+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:31:00.272 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:00.272 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:00.273 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:00.373 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:00.395 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:00.395 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:00.395 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:00.496 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:00.518 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:00.519 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:00.519 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:00.602 INFO:tasks.rados.rados.0.smithi137.stdout:1861: done (1 left) 2024-09-21T08:31:00.602 INFO:tasks.rados.rados.0.smithi137.stdout:1862: read oid 6 snap -1 2024-09-21T08:31:00.602 INFO:tasks.rados.rados.0.smithi137.stdout:1862: expect deleted 2024-09-21T08:31:00.602 INFO:tasks.rados.rados.0.smithi137.stdout:1863: read oid 5 snap -1 2024-09-21T08:31:00.602 INFO:tasks.rados.rados.0.smithi137.stdout:1863: expect (ObjNum 653 snap 193 seq_num 653) 2024-09-21T08:31:00.602 INFO:tasks.rados.rados.0.smithi137.stdout:1864: snap_create 2024-09-21T08:31:00.603 INFO:tasks.rados.rados.0.smithi137.stdout:1860: finishing rollback tid 1 to smithi13779221-46 2024-09-21T08:31:00.603 INFO:tasks.rados.rados.0.smithi137.stdout:1860: finishing rollback tid 2 to smithi13779221-46 2024-09-21T08:31:00.603 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 777 (ObjNum 499 snap 140 seq_num 499) dirty exists 2024-09-21T08:31:00.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:00.621 INFO:tasks.rados.rados.0.smithi137.stdout:1863: expect (ObjNum 653 snap 193 seq_num 653) 2024-09-21T08:31:00.642 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:00.643 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:00.643 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:00.744 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:00.766 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:00.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:00.767 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:00.867 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:00.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:00.891 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:00.891 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:00.981 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:31:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:31:00.981 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:00.981 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-mon[41044]: from='client.45815 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:00.981 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-mon[41044]: osdmap e466: 8 total, 6 up, 5 in 2024-09-21T08:31:00.981 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:00.981 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:00.982 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:00.982 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-mon[29634]: from='client.45815 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:00.982 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-mon[29634]: osdmap e466: 8 total, 6 up, 5 in 2024-09-21T08:31:00.982 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:00.982 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:00.992 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:01.007 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T08:31:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:00 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:00 smithi165 ceph-mon[32893]: from='client.45815 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:00 smithi165 ceph-mon[32893]: osdmap e466: 8 total, 6 up, 5 in 2024-09-21T08:31:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:01.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:01.014 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:01.014 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:01.015 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:01.053 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:01.116 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:01.137 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:01.137 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:01.138 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:01.238 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:01.260 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:01.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:01.261 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:01.261 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:01.339 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1, 4] out_osds: [6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:31:01.339 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:31:01.340 INFO:tasks.thrashosds.thrasher:Growing pool unique_pool_0 2024-09-21T08:31:01.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:31:01.362 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:01.404 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:01.405 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:01.405 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:01.506 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:01.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:01.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:01.530 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:01.558 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:01.570 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:01.570 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:01.588 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:01.589 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:01.598 INFO:tasks.rados.rados.0.smithi137.stdout:1860: done (3 left) 2024-09-21T08:31:01.599 INFO:tasks.rados.rados.0.smithi137.stdout:1862: done (2 left) 2024-09-21T08:31:01.599 INFO:tasks.rados.rados.0.smithi137.stdout:1863: done (1 left) 2024-09-21T08:31:01.599 INFO:tasks.rados.rados.0.smithi137.stdout:1864: done (0 left) 2024-09-21T08:31:01.599 INFO:tasks.rados.rados.0.smithi137.stdout:1865: delete oid 7 current snap is 199 2024-09-21T08:31:01.616 INFO:tasks.rados.rados.0.smithi137.stdout:1865: done (0 left) 2024-09-21T08:31:01.616 INFO:tasks.rados.rados.0.smithi137.stdout:1866: write oid 31 current snap is 199 2024-09-21T08:31:01.616 INFO:tasks.rados.rados.0.smithi137.stdout:1866: seq_num 664 ranges {759848=488047,1989220=760224} 2024-09-21T08:31:01.619 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:01.619 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:01.623 INFO:tasks.rados.rados.0.smithi137.stdout:1866: writing smithi13779221-31 from 759848 to 1247895 tid 1 2024-09-21T08:31:01.631 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:01.633 INFO:tasks.rados.rados.0.smithi137.stdout:1866: writing smithi13779221-31 from 1989220 to 2749444 tid 2 2024-09-21T08:31:01.633 INFO:tasks.rados.rados.0.smithi137.stdout:1867: write oid 7 current snap is 199 2024-09-21T08:31:01.633 INFO:tasks.rados.rados.0.smithi137.stdout:1867: seq_num 665 ranges {776655=563296,2047702=620235,3095349=81306} 2024-09-21T08:31:01.640 INFO:tasks.rados.rados.0.smithi137.stdout:1867: writing smithi13779221-7 from 776655 to 1339951 tid 1 2024-09-21T08:31:01.644 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:01.645 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:01.645 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:01.649 INFO:tasks.rados.rados.0.smithi137.stdout:1867: writing smithi13779221-7 from 2047702 to 2667937 tid 2 2024-09-21T08:31:01.652 INFO:tasks.rados.rados.0.smithi137.stdout:1867: writing smithi13779221-7 from 3095349 to 3176655 tid 3 2024-09-21T08:31:01.652 INFO:tasks.rados.rados.0.smithi137.stdout:1868: snap_create 2024-09-21T08:31:01.652 INFO:tasks.rados.rados.0.smithi137.stdout:1866: finishing write tid 1 to smithi13779221-31 2024-09-21T08:31:01.653 INFO:tasks.rados.rados.0.smithi137.stdout:1866: finishing write tid 2 to smithi13779221-31 2024-09-21T08:31:01.653 INFO:tasks.rados.rados.0.smithi137.stdout:1866: finishing write tid 3 to smithi13779221-31 2024-09-21T08:31:01.653 INFO:tasks.rados.rados.0.smithi137.stdout:1866: finishing write tid 4 to smithi13779221-31 2024-09-21T08:31:01.653 INFO:tasks.rados.rados.0.smithi137.stdout:1866: oid 31 updating version 0 to 360 2024-09-21T08:31:01.653 INFO:tasks.rados.rados.0.smithi137.stdout:1866: oid 31 updating version 360 to 361 2024-09-21T08:31:01.653 INFO:tasks.rados.rados.0.smithi137.stdout:1866: oid 31 updating version 361 to 362 2024-09-21T08:31:01.653 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 362 (ObjNum 664 snap 199 seq_num 664) dirty exists 2024-09-21T08:31:01.653 INFO:tasks.rados.rados.0.smithi137.stdout:1866: left oid 31 (ObjNum 664 snap 199 seq_num 664) 2024-09-21T08:31:01.653 INFO:tasks.rados.rados.0.smithi137.stdout:1867: finishing write tid 1 to smithi13779221-7 2024-09-21T08:31:01.653 INFO:tasks.rados.rados.0.smithi137.stdout:1867: finishing write tid 2 to smithi13779221-7 2024-09-21T08:31:01.655 INFO:tasks.rados.rados.0.smithi137.stdout:1867: finishing write tid 3 to smithi13779221-7 2024-09-21T08:31:01.655 INFO:tasks.rados.rados.0.smithi137.stdout:1867: finishing write tid 4 to smithi13779221-7 2024-09-21T08:31:01.656 INFO:tasks.rados.rados.0.smithi137.stdout:1867: finishing write tid 5 to smithi13779221-7 2024-09-21T08:31:01.656 INFO:tasks.rados.rados.0.smithi137.stdout:1867: oid 7 updating version 0 to 782 2024-09-21T08:31:01.656 INFO:tasks.rados.rados.0.smithi137.stdout:1867: oid 7 version 782 is already newer than 779 2024-09-21T08:31:01.656 INFO:tasks.rados.rados.0.smithi137.stdout:1867: oid 7 version 782 is already newer than 780 2024-09-21T08:31:01.656 INFO:tasks.rados.rados.0.smithi137.stdout:1867: oid 7 version 782 is already newer than 781 2024-09-21T08:31:01.656 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 782 (ObjNum 665 snap 199 seq_num 665) dirty exists 2024-09-21T08:31:01.656 INFO:tasks.rados.rados.0.smithi137.stdout:1867: left oid 7 (ObjNum 665 snap 199 seq_num 665) 2024-09-21T08:31:01.657 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:01.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:01.658 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:01.666 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:01.666 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:01.683 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:01.684 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:01.684 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:01.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:01 smithi165 ceph-mon[32893]: pgmap v605: 178 pgs: 2 active+clean+remapped, 61 peering, 4 remapped+peering, 35 unknown, 1 active+clean+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 17 active+clean; 511 MiB data, 1.3 GiB used, 446 GiB / 447 GiB avail; 237/1287 objects degraded (18.415%) 2024-09-21T08:31:01.728 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:01 smithi165 ceph-mon[32893]: osdmap e467: 8 total, 6 up, 5 in 2024-09-21T08:31:01.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:01 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2042715283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:01.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:01 smithi165 ceph-mon[32893]: Health check update: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T08:31:01.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:01 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 237/1287 objects degraded (18.415%), 33 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:01.729 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:01 smithi165 ceph-mon[32893]: osdmap e468: 8 total, 6 up, 5 in 2024-09-21T08:31:01.730 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:01.730 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:31:01.730 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:31:01.730 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:01.759 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:01.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:01.781 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:01.782 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:01.883 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:01.906 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:01.906 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:01.906 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:01.916 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[29634]: pgmap v605: 178 pgs: 2 active+clean+remapped, 61 peering, 4 remapped+peering, 35 unknown, 1 active+clean+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 17 active+clean; 511 MiB data, 1.3 GiB used, 446 GiB / 447 GiB avail; 237/1287 objects degraded (18.415%) 2024-09-21T08:31:01.916 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[29634]: osdmap e467: 8 total, 6 up, 5 in 2024-09-21T08:31:01.916 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2042715283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:01.916 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[29634]: Health check update: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T08:31:01.916 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 237/1287 objects degraded (18.415%), 33 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:01.916 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[29634]: osdmap e468: 8 total, 6 up, 5 in 2024-09-21T08:31:01.917 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[41044]: pgmap v605: 178 pgs: 2 active+clean+remapped, 61 peering, 4 remapped+peering, 35 unknown, 1 active+clean+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 17 active+clean; 511 MiB data, 1.3 GiB used, 446 GiB / 447 GiB avail; 237/1287 objects degraded (18.415%) 2024-09-21T08:31:01.917 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[41044]: osdmap e467: 8 total, 6 up, 5 in 2024-09-21T08:31:01.917 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2042715283' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:01.917 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[41044]: Health check update: Reduced data availability: 5 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T08:31:01.917 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 237/1287 objects degraded (18.415%), 33 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:01.917 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:01 smithi137 ceph-mon[41044]: osdmap e468: 8 total, 6 up, 5 in 2024-09-21T08:31:02.007 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:02.013 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:31:01 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:31:02.030 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:02.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:02.031 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:02.131 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:02.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:02.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:02.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:02.256 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:02.277 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:02.278 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:02.278 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:02.363 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:31:02.379 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:02.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:02.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:02.402 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:02.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:02.526 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:02.526 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:02.527 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:02.540 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T08:31:02.558 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:31:02.560 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:31:02.619 INFO:tasks.rados.rados.0.smithi137.stdout:1866: done (2 left) 2024-09-21T08:31:02.619 INFO:tasks.rados.rados.0.smithi137.stdout:1867: done (1 left) 2024-09-21T08:31:02.619 INFO:tasks.rados.rados.0.smithi137.stdout:1868: done (0 left) 2024-09-21T08:31:02.619 INFO:tasks.rados.rados.0.smithi137.stdout:1869: read oid 10 snap 189 2024-09-21T08:31:02.619 INFO:tasks.rados.rados.0.smithi137.stdout:1869: expect (ObjNum 576 snap 170 seq_num 576) 2024-09-21T08:31:02.620 INFO:tasks.rados.rados.0.smithi137.stdout:1870: copy_from oid 11 from oid 1 current snap is 200 2024-09-21T08:31:02.620 INFO:tasks.rados.rados.0.smithi137.stdout:1871: read oid 48 snap -1 2024-09-21T08:31:02.620 INFO:tasks.rados.rados.0.smithi137.stdout:1871: expect (ObjNum 497 snap 137 seq_num 497) 2024-09-21T08:31:02.620 INFO:tasks.rados.rados.0.smithi137.stdout:1872: snap_remove snap 181 2024-09-21T08:31:02.627 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:02.648 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:02.649 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:02.650 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:02.750 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:02 smithi165 ceph-mon[32893]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-21T08:31:02.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:02 smithi165 ceph-mon[32893]: osdmap e469: 8 total, 6 up, 5 in 2024-09-21T08:31:02.772 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:02.773 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:02.773 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:02.828 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:02.874 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:02.897 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:02.897 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:02.898 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:02.999 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:03.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:02 smithi137 ceph-mon[29634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-21T08:31:03.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:02 smithi137 ceph-mon[29634]: osdmap e469: 8 total, 6 up, 5 in 2024-09-21T08:31:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:02 smithi137 ceph-mon[41044]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 5 pgs inactive, 5 pgs peering) 2024-09-21T08:31:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:02 smithi137 ceph-mon[41044]: osdmap e469: 8 total, 6 up, 5 in 2024-09-21T08:31:03.019 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:03.020 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:03.020 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:03.051 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:03.052 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:03.121 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:03.143 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:03.143 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:03.143 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:03.197 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:03.198 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:31:03.198 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:31:03.198 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:31:03.198 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:03.244 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:03.266 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:03.266 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:03.267 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:03.324 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:03.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:03.389 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:03.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:03.390 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:03.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:03.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:03.514 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:03.514 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:03.615 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:03.623 INFO:tasks.rados.rados.0.smithi137.stdout:1872: done (3 left) 2024-09-21T08:31:03.623 INFO:tasks.rados.rados.0.smithi137.stdout:1873: read oid 26 snap -1 2024-09-21T08:31:03.624 INFO:tasks.rados.rados.0.smithi137.stdout:1873: expect (ObjNum 620 snap 182 seq_num 620) 2024-09-21T08:31:03.624 INFO:tasks.rados.rados.0.smithi137.stdout:1874: read oid 29 snap -1 2024-09-21T08:31:03.624 INFO:tasks.rados.rados.0.smithi137.stdout:1874: expect (ObjNum 640 snap 191 seq_num 640) 2024-09-21T08:31:03.624 INFO:tasks.rados.rados.0.smithi137.stdout:1875: write oid 39 current snap is 200 2024-09-21T08:31:03.624 INFO:tasks.rados.rados.0.smithi137.stdout:1875: seq_num 666 ranges {793462=638545,2106184=763893} 2024-09-21T08:31:03.633 INFO:tasks.rados.rados.0.smithi137.stdout:1875: writing smithi13779221-39 from 793462 to 1432007 tid 1 2024-09-21T08:31:03.642 INFO:tasks.rados.rados.0.smithi137.stdout:1875: writing smithi13779221-39 from 2106184 to 2870077 tid 2 2024-09-21T08:31:03.642 INFO:tasks.rados.rados.0.smithi137.stdout:1870: finishing copy_from to smithi13779221-11 2024-09-21T08:31:03.642 INFO:tasks.rados.rados.0.smithi137.stdout:1870: got expected ENOENT (src dne) 2024-09-21T08:31:03.642 INFO:tasks.rados.rados.0.smithi137.stdout:1870: finishing copy_from racing read to smithi13779221-11 2024-09-21T08:31:03.643 INFO:tasks.rados.rados.0.smithi137.stdout:1869: expect (ObjNum 576 snap 170 seq_num 576) 2024-09-21T08:31:03.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:03.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:03.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:03.684 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":608,"stamp":"2024-09-21T08:31:02.144815+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":582239484,"num_objects":485,"num_object_clones":236,"num_object_copies":1455,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":237,"num_objects_misplaced":100,"num_objects_unfound":0,"num_objects_dirty":485,"num_whiteouts":18,"num_read":3248,"num_read_kb":2093308,"num_write":1979,"num_write_kb":551358,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1485,"num_bytes_recovered":407498946,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":9853,"ondisk_log_size":9853,"up":423,"acting":476,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":610,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":1485752,"kb_used_data":1236812,"kb_used_omap":7,"kb_used_meta":248824,"kb_avail":467240008,"statfs":{"total":479975178240,"available":478453768192,"internally_reserved":0,"allocated":1266495488,"data_stored":1834265383,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7970,"internal_metadata":254795998},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":46512987,"num_objects":56,"num_object_clones":19,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":-29,"num_objects_degraded":0,"num_objects_misplaced":100,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":4,"num_read":427,"num_read_kb":383853,"num_write":271,"num_write_kb":73887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":530,"num_bytes_recovered":237817843,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001305"},"pg_stats":[{"pgid":"7.23","version":"179'19","reported_seq":857,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:31:00.603403+0000","last_change":"2024-09-21T08:31:00.603374+0000","last_active":"2024-09-21T08:31:00.603403+0000","last_peered":"2024-09-21T08:31:00.603403+0000","last_clean":"2024-09-21T08:31:00.603403+0000","last_became_active":"2024-09-21T08:30:56.626989+0000","last_became_peered":"2024-09-21T08:30:56.626989+0000","last_unstale":"2024-09-21T08:31:00.603403+0000","last_undegraded":"2024-09-21T08:31:00.603403+0000","last_fullsized":"2024-09-21T08:31:00.603403+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:28:10.421606+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.421606+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0026732460000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"7.22","version":"257'264","reported_seq":2236,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.603678+0000","last_change":"2024-09-21T08:31:00.604026+0000","last_active":"2024-09-21T08:31:00.603678+0000","last_peered":"2024-09-21T08:31:00.603678+0000","last_clean":"2024-09-21T08:31:00.603678+0000","last_became_active":"2024-09-21T08:30:59.603168+0000","last_became_peered":"2024-09-21T08:30:59.603168+0000","last_unstale":"2024-09-21T08:31:00.603678+0000","last_undegraded":"2024-09-21T08:31:00.603678+0000","last_fullsized":"2024-09-21T08:31:00.603678+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":6,"last_scrub":"257'264","last_scrub_stamp":"2024-09-21T08:27:48.727600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:48.727600+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035897519999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"7.21","version":"179'43","reported_seq":1692,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.605061+0000","last_change":"2024-09-21T08:31:00.606686+0000","last_active":"2024-09-21T08:31:00.605061+0000","last_peered":"2024-09-21T08:31:00.605061+0000","last_clean":"2024-09-21T08:31:00.605061+0000","last_became_active":"2024-09-21T08:30:57.612821+0000","last_became_peered":"2024-09-21T08:30:57.612821+0000","last_unstale":"2024-09-21T08:31:00.605061+0000","last_undegraded":"2024-09-21T08:31:00.605061+0000","last_fullsized":"2024-09-21T08:31:00.605061+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:36.934372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.934372+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0021831680000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"7.20","version":"432'172","reported_seq":1846,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.602202+0000","last_change":"2024-09-21T08:28:53.605282+0000","last_active":"2024-09-21T08:31:00.602202+0000","last_peered":"2024-09-21T08:31:00.602202+0000","last_clean":"2024-09-21T08:28:51.585842+0000","last_became_active":"2024-09-21T08:28:53.605282+0000","last_became_peered":"2024-09-21T08:28:53.605282+0000","last_unstale":"2024-09-21T08:31:00.602202+0000","last_undegraded":"2024-09-21T08:28:53.580783+0000","last_fullsized":"2024-09-21T08:28:53.580670+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"251'134","last_scrub_stamp":"2024-09-21T08:27:49.318143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.318143+0000","objects_scrubbed":1,"log_size":172,"log_dups_size":0,"ondisk_log_size":172,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":62,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.076740219999999998,"stat_sum":{"num_bytes":13440760,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":27,"num_read_kb":17523,"num_write":29,"num_write_kb":6079,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1848493,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"2"}]},{"pgid":"7.1b","version":"466'767","reported_seq":2977,"reported_epoch":467,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:31:00.607719+0000","last_change":"2024-09-21T08:30:59.605624+0000","last_active":"2024-09-21T08:31:00.607719+0000","last_peered":"2024-09-21T08:31:00.607719+0000","last_clean":"2024-09-21T08:31:00.607719+0000","last_became_active":"2024-09-21T08:30:59.601175+0000","last_became_peered":"2024-09-21T08:30:59.601175+0000","last_unstale":"2024-09-21T08:31:00.607719+0000","last_undegraded":"2024-09-21T08:31:00.607719+0000","last_fullsized":"2024-09-21T08:31:00.607719+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'499","last_scrub_stamp":"2024-09-21T08:28:09.328624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.328624+0000","objects_scrubbed":5,"log_size":767,"log_dups_size":0,"ondisk_log_size":767,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":6,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0029166660000000001,"stat_sum":{"num_bytes":47561707,"num_objects":23,"num_object_clones":19,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":143,"num_read_kb":153802,"num_write":92,"num_write_kb":37580,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":61029229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"}]},{"pgid":"3.1f","version":"0'0","reported_seq":323,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.607469+0000","last_change":"2024-09-21T08:30:56.618803+0000","last_active":"2024-09-21T08:31:00.607469+0000","last_peered":"2024-09-21T08:31:00.607469+0000","last_clean":"2024-09-21T08:31:00.607469+0000","last_became_active":"2024-09-21T08:30:56.617989+0000","last_became_peered":"2024-09-21T08:30:56.617989+0000","last_unstale":"2024-09-21T08:31:00.607469+0000","last_undegraded":"2024-09-21T08:31:00.607469+0000","last_fullsized":"2024-09-21T08:31:00.607469+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":29,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.600228+0000","last_change":"2024-09-21T08:30:56.662007+0000","last_active":"2024-09-21T08:31:00.600228+0000","last_peered":"2024-09-21T08:31:00.600228+0000","last_clean":"2024-09-21T08:31:00.600228+0000","last_became_active":"2024-09-21T08:30:56.628842+0000","last_became_peered":"2024-09-21T08:30:56.628842+0000","last_unstale":"2024-09-21T08:31:00.600228+0000","last_undegraded":"2024-09-21T08:31:00.600228+0000","last_fullsized":"2024-09-21T08:31:00.600228+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:44.058465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:44.058465+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:05.271406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":375,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.600546+0000","last_change":"2024-09-21T08:28:53.594621+0000","last_active":"2024-09-21T08:31:00.600546+0000","last_peered":"2024-09-21T08:31:00.600546+0000","last_clean":"2024-09-21T08:28:51.584740+0000","last_became_active":"2024-09-21T08:28:53.594621+0000","last_became_peered":"2024-09-21T08:28:53.594621+0000","last_unstale":"2024-09-21T08:31:00.600546+0000","last_undegraded":"2024-09-21T08:31:00.600546+0000","last_fullsized":"2024-09-21T08:28:53.576425+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.424482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.424482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995660+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":19,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602672+0000","last_change":"2024-09-21T08:30:56.631377+0000","last_active":"2024-09-21T08:31:00.602672+0000","last_peered":"2024-09-21T08:31:00.602672+0000","last_clean":"2024-09-21T08:31:00.602672+0000","last_became_active":"2024-09-21T08:30:56.630218+0000","last_became_peered":"2024-09-21T08:30:56.630218+0000","last_unstale":"2024-09-21T08:31:00.602672+0000","last_undegraded":"2024-09-21T08:31:00.602672+0000","last_fullsized":"2024-09-21T08:31:00.602672+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.464969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.464969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.964068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"248'68","reported_seq":264,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.604189+0000","last_change":"2024-09-21T08:31:00.604941+0000","last_active":"2024-09-21T08:31:00.604189+0000","last_peered":"2024-09-21T08:31:00.604189+0000","last_clean":"2024-09-21T08:31:00.604189+0000","last_became_active":"2024-09-21T08:30:57.616295+0000","last_became_peered":"2024-09-21T08:30:57.616295+0000","last_unstale":"2024-09-21T08:31:00.604189+0000","last_undegraded":"2024-09-21T08:31:00.604189+0000","last_fullsized":"2024-09-21T08:31:00.604189+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'68","last_scrub_stamp":"2024-09-21T08:27:35.919031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:35.919031+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0035676340000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":732,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.603374+0000","last_change":"2024-09-21T08:28:53.606497+0000","last_active":"2024-09-21T08:31:00.603374+0000","last_peered":"2024-09-21T08:31:00.603374+0000","last_clean":"2024-09-21T08:28:51.585547+0000","last_became_active":"2024-09-21T08:28:53.606497+0000","last_became_peered":"2024-09-21T08:28:53.606497+0000","last_unstale":"2024-09-21T08:31:00.603374+0000","last_undegraded":"2024-09-21T08:31:00.603374+0000","last_fullsized":"2024-09-21T08:28:53.581183+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.804294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.804294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995567+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":659,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.604236+0000","last_change":"2024-09-21T08:28:53.602981+0000","last_active":"2024-09-21T08:31:00.604236+0000","last_peered":"2024-09-21T08:31:00.604236+0000","last_clean":"2024-09-21T08:28:51.587203+0000","last_became_active":"2024-09-21T08:28:53.602981+0000","last_became_peered":"2024-09-21T08:28:53.602981+0000","last_unstale":"2024-09-21T08:31:00.604236+0000","last_undegraded":"2024-09-21T08:31:00.604236+0000","last_fullsized":"2024-09-21T08:28:53.581112+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.146798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.146798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995617+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":656,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.605087+0000","last_change":"2024-09-21T08:29:00.355702+0000","last_active":"2024-09-21T08:31:00.605087+0000","last_peered":"2024-09-21T08:31:00.605087+0000","last_clean":"2024-09-21T08:31:00.605087+0000","last_became_active":"2024-09-21T08:28:29.620169+0000","last_became_peered":"2024-09-21T08:28:29.620169+0000","last_unstale":"2024-09-21T08:31:00.605087+0000","last_undegraded":"2024-09-21T08:31:00.605087+0000","last_fullsized":"2024-09-21T08:31:00.605087+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:00.355591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:00.355591+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":19,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.599991+0000","last_change":"2024-09-21T08:30:56.628272+0000","last_active":"2024-09-21T08:31:00.599991+0000","last_peered":"2024-09-21T08:31:00.599991+0000","last_clean":"2024-09-21T08:31:00.599991+0000","last_became_active":"2024-09-21T08:30:56.627964+0000","last_became_peered":"2024-09-21T08:30:56.627964+0000","last_unstale":"2024-09-21T08:31:00.599991+0000","last_undegraded":"2024-09-21T08:31:00.599991+0000","last_fullsized":"2024-09-21T08:31:00.599991+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.340817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.340817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963713+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"455'207","reported_seq":1562,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.605357+0000","last_change":"2024-09-21T08:28:53.587606+0000","last_active":"2024-09-21T08:31:00.605357+0000","last_peered":"2024-09-21T08:31:00.605357+0000","last_clean":"2024-09-21T08:28:52.589310+0000","last_became_active":"2024-09-21T08:28:53.587606+0000","last_became_peered":"2024-09-21T08:28:53.587606+0000","last_unstale":"2024-09-21T08:31:00.605357+0000","last_undegraded":"2024-09-21T08:28:53.574680+0000","last_fullsized":"2024-09-21T08:28:53.574555+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"266'188","last_scrub_stamp":"2024-09-21T08:27:49.719124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.719124+0000","objects_scrubbed":1,"log_size":207,"log_dups_size":0,"ondisk_log_size":207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":61,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.034770314,"stat_sum":{"num_bytes":11158040,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":25,"num_read_kb":25354,"num_write":17,"num_write_kb":11591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"}]},{"pgid":"3.1d","version":"0'0","reported_seq":869,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.608113+0000","last_change":"2024-09-21T08:30:56.626716+0000","last_active":"2024-09-21T08:31:00.608113+0000","last_peered":"2024-09-21T08:31:00.608113+0000","last_clean":"2024-09-21T08:31:00.608113+0000","last_became_active":"2024-09-21T08:30:56.625894+0000","last_became_peered":"2024-09-21T08:30:56.625894+0000","last_unstale":"2024-09-21T08:31:00.608113+0000","last_undegraded":"2024-09-21T08:31:00.608113+0000","last_fullsized":"2024-09-21T08:31:00.608113+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.780545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.780545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816910+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":29,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.603682+0000","last_change":"2024-09-21T08:30:56.648653+0000","last_active":"2024-09-21T08:31:00.603682+0000","last_peered":"2024-09-21T08:31:00.603682+0000","last_clean":"2024-09-21T08:31:00.603682+0000","last_became_active":"2024-09-21T08:30:56.621427+0000","last_became_peered":"2024-09-21T08:30:56.621427+0000","last_unstale":"2024-09-21T08:31:00.603682+0000","last_undegraded":"2024-09-21T08:31:00.603682+0000","last_fullsized":"2024-09-21T08:31:00.603682+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:26.781541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:26.781541+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:43.277471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":827,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.605454+0000","last_change":"2024-09-21T08:30:57.625041+0000","last_active":"2024-09-21T08:31:00.605454+0000","last_peered":"2024-09-21T08:31:00.605454+0000","last_clean":"2024-09-21T08:31:00.605454+0000","last_became_active":"2024-09-21T08:30:57.624588+0000","last_became_peered":"2024-09-21T08:30:57.624588+0000","last_unstale":"2024-09-21T08:31:00.605454+0000","last_undegraded":"2024-09-21T08:31:00.605454+0000","last_fullsized":"2024-09-21T08:31:00.605454+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816824+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":778,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.600630+0000","last_change":"2024-09-21T08:28:53.586342+0000","last_active":"2024-09-21T08:31:00.600630+0000","last_peered":"2024-09-21T08:31:00.600630+0000","last_clean":"2024-09-21T08:28:51.584564+0000","last_became_active":"2024-09-21T08:28:53.586342+0000","last_became_peered":"2024-09-21T08:28:53.586342+0000","last_unstale":"2024-09-21T08:31:00.600630+0000","last_undegraded":"2024-09-21T08:31:00.600630+0000","last_fullsized":"2024-09-21T08:28:53.575506+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.291226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.291226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995611+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"453'265","reported_seq":1994,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.604464+0000","last_change":"2024-09-21T08:28:53.587953+0000","last_active":"2024-09-21T08:31:00.604464+0000","last_peered":"2024-09-21T08:31:00.604464+0000","last_clean":"2024-09-21T08:28:51.550342+0000","last_became_active":"2024-09-21T08:28:53.587953+0000","last_became_peered":"2024-09-21T08:28:53.587953+0000","last_unstale":"2024-09-21T08:31:00.604464+0000","last_undegraded":"2024-09-21T08:28:53.574363+0000","last_fullsized":"2024-09-21T08:28:53.574234+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"319'218","last_scrub_stamp":"2024-09-21T08:28:32.867460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:32.867460+0000","objects_scrubbed":4,"log_size":265,"log_dups_size":0,"ondisk_log_size":265,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":58,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027063920000000002,"stat_sum":{"num_bytes":27140451,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":58,"num_read_kb":68205,"num_write":63,"num_write_kb":31018,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7346792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":11}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"3.1c","version":"0'0","reported_seq":855,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.606606+0000","last_change":"2024-09-21T08:30:56.627917+0000","last_active":"2024-09-21T08:31:00.606606+0000","last_peered":"2024-09-21T08:31:00.606606+0000","last_clean":"2024-09-21T08:31:00.606606+0000","last_became_active":"2024-09-21T08:30:56.627614+0000","last_became_peered":"2024-09-21T08:30:56.627614+0000","last_unstale":"2024-09-21T08:31:00.606606+0000","last_undegraded":"2024-09-21T08:31:00.606606+0000","last_fullsized":"2024-09-21T08:31:00.606606+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816761+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":25,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.601449+0000","last_change":"2024-09-21T08:30:56.659860+0000","last_active":"2024-09-21T08:31:00.601449+0000","last_peered":"2024-09-21T08:31:00.601449+0000","last_clean":"2024-09-21T08:31:00.601449+0000","last_became_active":"2024-09-21T08:30:56.627915+0000","last_became_peered":"2024-09-21T08:30:56.627915+0000","last_unstale":"2024-09-21T08:31:00.601449+0000","last_undegraded":"2024-09-21T08:31:00.601449+0000","last_fullsized":"2024-09-21T08:31:00.601449+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:27:26.356987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:26.356987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963800+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602307+0000","last_change":"2024-09-21T08:30:57.619332+0000","last_active":"2024-09-21T08:31:00.602307+0000","last_peered":"2024-09-21T08:31:00.602307+0000","last_clean":"2024-09-21T08:31:00.602307+0000","last_became_active":"2024-09-21T08:30:57.617614+0000","last_became_peered":"2024-09-21T08:30:57.617614+0000","last_unstale":"2024-09-21T08:31:00.602307+0000","last_undegraded":"2024-09-21T08:31:00.602307+0000","last_fullsized":"2024-09-21T08:31:00.602307+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.850373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.964116+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":869,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.606246+0000","last_change":"2024-09-21T08:30:04.034656+0000","last_active":"2024-09-21T08:31:00.606246+0000","last_peered":"2024-09-21T08:31:00.606246+0000","last_clean":"2024-09-21T08:31:00.606246+0000","last_became_active":"2024-09-21T08:28:04.705450+0000","last_became_peered":"2024-09-21T08:28:04.705450+0000","last_unstale":"2024-09-21T08:31:00.606246+0000","last_undegraded":"2024-09-21T08:31:00.606246+0000","last_fullsized":"2024-09-21T08:31:00.606246+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:30:04.034575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:04.034575+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:12.159925+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1f","version":"247'32","reported_seq":270,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.608342+0000","last_change":"2024-09-21T08:31:00.608570+0000","last_active":"2024-09-21T08:31:00.608342+0000","last_peered":"2024-09-21T08:31:00.608342+0000","last_clean":"2024-09-21T08:31:00.608342+0000","last_became_active":"2024-09-21T08:30:56.617686+0000","last_became_peered":"2024-09-21T08:30:56.617686+0000","last_unstale":"2024-09-21T08:31:00.608342+0000","last_undegraded":"2024-09-21T08:31:00.608342+0000","last_fullsized":"2024-09-21T08:31:00.608342+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:27:51.078531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.078531+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0062082229999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"6.1e","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602164+0000","last_change":"2024-09-21T08:30:57.620780+0000","last_active":"2024-09-21T08:31:00.602164+0000","last_peered":"2024-09-21T08:31:00.602164+0000","last_clean":"2024-09-21T08:31:00.602164+0000","last_became_active":"2024-09-21T08:30:57.617563+0000","last_became_peered":"2024-09-21T08:30:57.617563+0000","last_unstale":"2024-09-21T08:31:00.602164+0000","last_undegraded":"2024-09-21T08:31:00.602164+0000","last_fullsized":"2024-09-21T08:31:00.602164+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.855448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.855448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963412+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.601174+0000","last_change":"2024-09-21T08:30:57.614029+0000","last_active":"2024-09-21T08:31:00.601174+0000","last_peered":"2024-09-21T08:31:00.601174+0000","last_clean":"2024-09-21T08:31:00.601174+0000","last_became_active":"2024-09-21T08:30:57.611889+0000","last_became_peered":"2024-09-21T08:30:57.611889+0000","last_unstale":"2024-09-21T08:31:00.601174+0000","last_undegraded":"2024-09-21T08:31:00.601174+0000","last_fullsized":"2024-09-21T08:31:00.601174+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963380+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":911,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602109+0000","last_change":"2024-09-21T08:30:56.660342+0000","last_active":"2024-09-21T08:31:00.602109+0000","last_peered":"2024-09-21T08:31:00.602109+0000","last_clean":"2024-09-21T08:31:00.602109+0000","last_became_active":"2024-09-21T08:30:56.630563+0000","last_became_peered":"2024-09-21T08:30:56.630563+0000","last_unstale":"2024-09-21T08:31:00.602109+0000","last_undegraded":"2024-09-21T08:31:00.602109+0000","last_fullsized":"2024-09-21T08:31:00.602109+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:37.352879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:37.352879+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087544+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":379,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.603525+0000","last_change":"2024-09-21T08:30:56.628090+0000","last_active":"2024-09-21T08:31:00.603525+0000","last_peered":"2024-09-21T08:31:00.603525+0000","last_clean":"2024-09-21T08:31:00.603525+0000","last_became_active":"2024-09-21T08:30:56.627667+0000","last_became_peered":"2024-09-21T08:30:56.627667+0000","last_unstale":"2024-09-21T08:31:00.603525+0000","last_undegraded":"2024-09-21T08:31:00.603525+0000","last_fullsized":"2024-09-21T08:31:00.603525+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:30.310628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.310628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"450'310","reported_seq":2136,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:31:00.604077+0000","last_change":"2024-09-21T08:30:59.609687+0000","last_active":"2024-09-21T08:31:00.604077+0000","last_peered":"2024-09-21T08:31:00.604077+0000","last_clean":"2024-09-21T08:31:00.604077+0000","last_became_active":"2024-09-21T08:30:59.606410+0000","last_became_peered":"2024-09-21T08:30:59.606410+0000","last_unstale":"2024-09-21T08:31:00.604077+0000","last_undegraded":"2024-09-21T08:31:00.604077+0000","last_fullsized":"2024-09-21T08:31:00.604077+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'236","last_scrub_stamp":"2024-09-21T08:28:09.464279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.464279+0000","objects_scrubbed":2,"log_size":310,"log_dups_size":0,"ondisk_log_size":310,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":64,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.0792350470000001,"stat_sum":{"num_bytes":41783176,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":80,"num_read_kb":84122,"num_write":63,"num_write_kb":22455,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":36880118,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"6.1f","version":"0'0","reported_seq":724,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.607911+0000","last_change":"2024-09-21T08:30:37.732703+0000","last_active":"2024-09-21T08:31:00.607911+0000","last_peered":"2024-09-21T08:31:00.607911+0000","last_clean":"2024-09-21T08:31:00.607911+0000","last_became_active":"2024-09-21T08:28:04.725392+0000","last_became_peered":"2024-09-21T08:28:04.725392+0000","last_unstale":"2024-09-21T08:31:00.607911+0000","last_undegraded":"2024-09-21T08:31:00.607911+0000","last_fullsized":"2024-09-21T08:31:00.607911+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:37.732599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:37.732599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:56.617806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":19,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602500+0000","last_change":"2024-09-21T08:30:56.630745+0000","last_active":"2024-09-21T08:31:00.602500+0000","last_peered":"2024-09-21T08:31:00.602500+0000","last_clean":"2024-09-21T08:31:00.602500+0000","last_became_active":"2024-09-21T08:30:56.630058+0000","last_became_peered":"2024-09-21T08:30:56.630058+0000","last_unstale":"2024-09-21T08:31:00.602500+0000","last_undegraded":"2024-09-21T08:31:00.602500+0000","last_fullsized":"2024-09-21T08:31:00.602500+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:33.237691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.237691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963502+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":402,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602829+0000","last_change":"2024-09-21T08:30:56.656296+0000","last_active":"2024-09-21T08:31:00.602829+0000","last_peered":"2024-09-21T08:31:00.602829+0000","last_clean":"2024-09-21T08:31:00.602829+0000","last_became_active":"2024-09-21T08:30:56.626214+0000","last_became_peered":"2024-09-21T08:30:56.626214+0000","last_unstale":"2024-09-21T08:31:00.602829+0000","last_undegraded":"2024-09-21T08:31:00.602829+0000","last_fullsized":"2024-09-21T08:31:00.602829+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:29:15.349124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:15.349124+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087503+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.600872+0000","last_change":"2024-09-21T08:30:57.611316+0000","last_active":"2024-09-21T08:31:00.600872+0000","last_peered":"2024-09-21T08:31:00.600872+0000","last_clean":"2024-09-21T08:31:00.600872+0000","last_became_active":"2024-09-21T08:30:57.610910+0000","last_became_peered":"2024-09-21T08:30:57.610910+0000","last_unstale":"2024-09-21T08:31:00.600872+0000","last_undegraded":"2024-09-21T08:31:00.600872+0000","last_fullsized":"2024-09-21T08:31:00.600872+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:09.116412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.116412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963928+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1d","version":"248'84","reported_seq":620,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.603058+0000","last_change":"2024-09-21T08:28:53.595944+0000","last_active":"2024-09-21T08:31:00.603058+0000","last_peered":"2024-09-21T08:31:00.603058+0000","last_clean":"2024-09-21T08:28:51.587189+0000","last_became_active":"2024-09-21T08:28:53.595944+0000","last_became_peered":"2024-09-21T08:28:53.595944+0000","last_unstale":"2024-09-21T08:31:00.603058+0000","last_undegraded":"2024-09-21T08:31:00.603058+0000","last_fullsized":"2024-09-21T08:28:53.578716+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'84","last_scrub_stamp":"2024-09-21T08:28:12.352559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.352559+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.072813132000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"6.1c","version":"73'1","reported_seq":361,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.606899+0000","last_change":"2024-09-21T08:28:53.589802+0000","last_active":"2024-09-21T08:31:00.606899+0000","last_peered":"2024-09-21T08:31:00.606899+0000","last_clean":"2024-09-21T08:28:51.557351+0000","last_became_active":"2024-09-21T08:28:53.589802+0000","last_became_peered":"2024-09-21T08:28:53.589802+0000","last_unstale":"2024-09-21T08:31:00.606899+0000","last_undegraded":"2024-09-21T08:28:53.577740+0000","last_fullsized":"2024-09-21T08:28:53.577630+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:27:14.826047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.826047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"198'2","reported_seq":380,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.600724+0000","last_change":"2024-09-21T08:30:57.634005+0000","last_active":"2024-09-21T08:31:00.600724+0000","last_peered":"2024-09-21T08:31:00.600724+0000","last_clean":"2024-09-21T08:31:00.600724+0000","last_became_active":"2024-09-21T08:30:57.611269+0000","last_became_peered":"2024-09-21T08:30:57.611269+0000","last_unstale":"2024-09-21T08:31:00.600724+0000","last_undegraded":"2024-09-21T08:31:00.600724+0000","last_fullsized":"2024-09-21T08:31:00.600724+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'2","last_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_deep_scrub":"198'2","last_deep_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_clean_scrub_stamp":"2024-09-21T08:26:32.819313+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087281+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":872,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.604877+0000","last_change":"2024-09-21T08:28:53.597832+0000","last_active":"2024-09-21T08:31:00.604877+0000","last_peered":"2024-09-21T08:31:00.604877+0000","last_clean":"2024-09-21T08:28:51.587327+0000","last_became_active":"2024-09-21T08:28:53.597832+0000","last_became_peered":"2024-09-21T08:28:53.597832+0000","last_unstale":"2024-09-21T08:31:00.604877+0000","last_undegraded":"2024-09-21T08:28:53.582409+0000","last_fullsized":"2024-09-21T08:28:53.582308+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"464'19","reported_seq":25,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602162+0000","last_change":"2024-09-21T08:30:57.629872+0000","last_active":"2024-09-21T08:31:00.602162+0000","last_peered":"2024-09-21T08:31:00.602162+0000","last_clean":"2024-09-21T08:31:00.602162+0000","last_became_active":"2024-09-21T08:30:57.613278+0000","last_became_peered":"2024-09-21T08:30:57.613278+0000","last_unstale":"2024-09-21T08:31:00.602162+0000","last_undegraded":"2024-09-21T08:31:00.602162+0000","last_fullsized":"2024-09-21T08:31:00.602162+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'15","last_scrub_stamp":"2024-09-21T08:28:14.153130+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.153130+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.964086+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'129","reported_seq":1095,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.605313+0000","last_change":"2024-09-21T08:30:59.600965+0000","last_active":"2024-09-21T08:31:00.605313+0000","last_peered":"2024-09-21T08:31:00.605313+0000","last_clean":"2024-09-21T08:31:00.605313+0000","last_became_active":"2024-09-21T08:30:59.600508+0000","last_became_peered":"2024-09-21T08:30:59.600508+0000","last_unstale":"2024-09-21T08:31:00.605313+0000","last_undegraded":"2024-09-21T08:31:00.605313+0000","last_fullsized":"2024-09-21T08:31:00.605313+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.842769+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":322019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"463'268","reported_seq":2234,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:31:00.608076+0000","last_change":"2024-09-21T08:31:00.608047+0000","last_active":"2024-09-21T08:31:00.608076+0000","last_peered":"2024-09-21T08:31:00.608076+0000","last_clean":"2024-09-21T08:31:00.608076+0000","last_became_active":"2024-09-21T08:28:32.557643+0000","last_became_peered":"2024-09-21T08:28:32.557643+0000","last_unstale":"2024-09-21T08:31:00.608076+0000","last_undegraded":"2024-09-21T08:31:00.608076+0000","last_fullsized":"2024-09-21T08:31:00.608076+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'254","last_scrub_stamp":"2024-09-21T08:30:46.778996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:46.778996+0000","objects_scrubbed":4,"log_size":268,"log_dups_size":0,"ondisk_log_size":268,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:57.029862+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.030263589,"stat_sum":{"num_bytes":13010315,"num_objects":6,"num_object_clones":4,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":52,"num_read_kb":58170,"num_write":32,"num_write_kb":12039,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":6235496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.f","version":"73'15","reported_seq":404,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.604974+0000","last_change":"2024-09-21T08:30:57.648167+0000","last_active":"2024-09-21T08:31:00.604974+0000","last_peered":"2024-09-21T08:31:00.604974+0000","last_clean":"2024-09-21T08:31:00.604974+0000","last_became_active":"2024-09-21T08:30:57.617493+0000","last_became_peered":"2024-09-21T08:30:57.617493+0000","last_unstale":"2024-09-21T08:31:00.604974+0000","last_undegraded":"2024-09-21T08:31:00.604974+0000","last_fullsized":"2024-09-21T08:31:00.604974+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:26:46.777115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.777115+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087339+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":372,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.601816+0000","last_change":"2024-09-21T08:30:57.611521+0000","last_active":"2024-09-21T08:31:00.601816+0000","last_peered":"2024-09-21T08:31:00.601816+0000","last_clean":"2024-09-21T08:31:00.601816+0000","last_became_active":"2024-09-21T08:30:57.611080+0000","last_became_peered":"2024-09-21T08:30:57.611080+0000","last_unstale":"2024-09-21T08:31:00.601816+0000","last_undegraded":"2024-09-21T08:31:00.601816+0000","last_fullsized":"2024-09-21T08:31:00.601816+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:45.434055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:45.434055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087210+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"463'12","reported_seq":25,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602276+0000","last_change":"2024-09-21T08:30:56.622801+0000","last_active":"2024-09-21T08:31:00.602276+0000","last_peered":"2024-09-21T08:31:00.602276+0000","last_clean":"2024-09-21T08:31:00.602276+0000","last_became_active":"2024-09-21T08:30:56.607770+0000","last_became_peered":"2024-09-21T08:30:56.607770+0000","last_unstale":"2024-09-21T08:31:00.602276+0000","last_undegraded":"2024-09-21T08:31:00.602276+0000","last_fullsized":"2024-09-21T08:31:00.602276+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:26:44.741275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.741275+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963562+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":844,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.609203+0000","last_change":"2024-09-21T08:30:21.800509+0000","last_active":"2024-09-21T08:31:00.609203+0000","last_peered":"2024-09-21T08:31:00.609203+0000","last_clean":"2024-09-21T08:31:00.609203+0000","last_became_active":"2024-09-21T08:28:29.619967+0000","last_became_peered":"2024-09-21T08:28:29.619967+0000","last_unstale":"2024-09-21T08:31:00.609203+0000","last_undegraded":"2024-09-21T08:31:00.609203+0000","last_fullsized":"2024-09-21T08:31:00.609203+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:21.800427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:21.800427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:50.788583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'129","reported_seq":793,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.601089+0000","last_change":"2024-09-21T08:30:59.600955+0000","last_active":"2024-09-21T08:31:00.601089+0000","last_peered":"2024-09-21T08:31:00.601089+0000","last_clean":"2024-09-21T08:31:00.601089+0000","last_became_active":"2024-09-21T08:30:59.600613+0000","last_became_peered":"2024-09-21T08:30:59.600613+0000","last_unstale":"2024-09-21T08:31:00.601089+0000","last_undegraded":"2024-09-21T08:31:00.601089+0000","last_fullsized":"2024-09-21T08:31:00.601089+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963279+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"179'19","reported_seq":1795,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.608457+0000","last_change":"2024-09-21T08:31:00.608810+0000","last_active":"2024-09-21T08:31:00.608457+0000","last_peered":"2024-09-21T08:31:00.608457+0000","last_clean":"2024-09-21T08:31:00.608457+0000","last_became_active":"2024-09-21T08:30:57.613435+0000","last_became_peered":"2024-09-21T08:30:57.613435+0000","last_unstale":"2024-09-21T08:31:00.608457+0000","last_undegraded":"2024-09-21T08:31:00.608457+0000","last_fullsized":"2024-09-21T08:31:00.608457+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:27:36.268364+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.268364+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00080262599999999997,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4,1],"avail_no_missing":["2","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.0","version":"73'18","reported_seq":801,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.601551+0000","last_change":"2024-09-21T08:28:53.593785+0000","last_active":"2024-09-21T08:31:00.601551+0000","last_peered":"2024-09-21T08:31:00.601551+0000","last_clean":"2024-09-21T08:28:51.584787+0000","last_became_active":"2024-09-21T08:28:53.593785+0000","last_became_peered":"2024-09-21T08:28:53.593785+0000","last_unstale":"2024-09-21T08:31:00.601551+0000","last_undegraded":"2024-09-21T08:28:53.574919+0000","last_fullsized":"2024-09-21T08:28:53.574229+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-09-21T08:26:54.003173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:54.003173+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995553+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":581,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.606614+0000","last_change":"2024-09-21T08:30:57.623808+0000","last_active":"2024-09-21T08:31:00.606614+0000","last_peered":"2024-09-21T08:31:00.606614+0000","last_clean":"2024-09-21T08:31:00.606614+0000","last_became_active":"2024-09-21T08:30:57.622744+0000","last_became_peered":"2024-09-21T08:30:57.622744+0000","last_unstale":"2024-09-21T08:31:00.606614+0000","last_undegraded":"2024-09-21T08:31:00.606614+0000","last_fullsized":"2024-09-21T08:31:00.606614+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.601091+0000","last_change":"2024-09-21T08:30:57.625683+0000","last_active":"2024-09-21T08:31:00.601091+0000","last_peered":"2024-09-21T08:31:00.601091+0000","last_clean":"2024-09-21T08:31:00.601091+0000","last_became_active":"2024-09-21T08:30:57.619764+0000","last_became_peered":"2024-09-21T08:30:57.619764+0000","last_unstale":"2024-09-21T08:31:00.601091+0000","last_undegraded":"2024-09-21T08:31:00.601091+0000","last_fullsized":"2024-09-21T08:31:00.601091+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:32.813973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:32.813973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602767+0000","last_change":"2024-09-21T08:30:57.619543+0000","last_active":"2024-09-21T08:31:00.602767+0000","last_peered":"2024-09-21T08:31:00.602767+0000","last_clean":"2024-09-21T08:31:00.602767+0000","last_became_active":"2024-09-21T08:30:57.617414+0000","last_became_peered":"2024-09-21T08:30:57.617414+0000","last_unstale":"2024-09-21T08:31:00.602767+0000","last_undegraded":"2024-09-21T08:31:00.602767+0000","last_fullsized":"2024-09-21T08:31:00.602767+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:16.341564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.341564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963656+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"76'129","reported_seq":1099,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.605254+0000","last_change":"2024-09-21T08:30:59.604070+0000","last_active":"2024-09-21T08:31:00.605254+0000","last_peered":"2024-09-21T08:31:00.605254+0000","last_clean":"2024-09-21T08:31:00.605254+0000","last_became_active":"2024-09-21T08:30:59.603597+0000","last_became_peered":"2024-09-21T08:30:59.603597+0000","last_unstale":"2024-09-21T08:31:00.605254+0000","last_undegraded":"2024-09-21T08:31:00.605254+0000","last_fullsized":"2024-09-21T08:31:00.605254+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.842785+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":977379,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"462'353","reported_seq":2072,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.604079+0000","last_change":"2024-09-21T08:28:53.605477+0000","last_active":"2024-09-21T08:31:00.604079+0000","last_peered":"2024-09-21T08:31:00.604079+0000","last_clean":"2024-09-21T08:28:52.666434+0000","last_became_active":"2024-09-21T08:28:53.605477+0000","last_became_peered":"2024-09-21T08:28:53.605477+0000","last_unstale":"2024-09-21T08:31:00.604079+0000","last_undegraded":"2024-09-21T08:28:53.586515+0000","last_fullsized":"2024-09-21T08:28:53.586400+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'273","last_scrub_stamp":"2024-09-21T08:27:50.318620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.318620+0000","objects_scrubbed":3,"log_size":353,"log_dups_size":0,"ondisk_log_size":353,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":64,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.073725586999999995,"stat_sum":{"num_bytes":20243096,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":66,"num_read_kb":73718,"num_write":42,"num_write_kb":14479,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":3845252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":14}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.1","version":"239'21","reported_seq":34,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.600869+0000","last_change":"2024-09-21T08:30:57.699489+0000","last_active":"2024-09-21T08:31:00.600869+0000","last_peered":"2024-09-21T08:31:00.600869+0000","last_clean":"2024-09-21T08:31:00.600869+0000","last_became_active":"2024-09-21T08:30:57.619835+0000","last_became_peered":"2024-09-21T08:30:57.619835+0000","last_unstale":"2024-09-21T08:31:00.600869+0000","last_undegraded":"2024-09-21T08:31:00.600869+0000","last_fullsized":"2024-09-21T08:31:00.600869+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21","last_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_deep_scrub":"239'21","last_deep_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.969642+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963642+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[{"shards":"2,4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"66'1","reported_seq":819,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.605007+0000","last_change":"2024-09-21T08:30:57.774125+0000","last_active":"2024-09-21T08:31:00.605007+0000","last_peered":"2024-09-21T08:31:00.605007+0000","last_clean":"2024-09-21T08:31:00.605007+0000","last_became_active":"2024-09-21T08:30:57.621298+0000","last_became_peered":"2024-09-21T08:30:57.621298+0000","last_unstale":"2024-09-21T08:31:00.605007+0000","last_undegraded":"2024-09-21T08:31:00.605007+0000","last_fullsized":"2024-09-21T08:31:00.605007+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:24:55.889065+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:55.889065+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087163+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"463'17","reported_seq":432,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.601565+0000","last_change":"2024-09-21T08:30:56.651846+0000","last_active":"2024-09-21T08:31:00.601565+0000","last_peered":"2024-09-21T08:31:00.601565+0000","last_clean":"2024-09-21T08:31:00.601565+0000","last_became_active":"2024-09-21T08:30:56.630142+0000","last_became_peered":"2024-09-21T08:30:56.630142+0000","last_unstale":"2024-09-21T08:31:00.601565+0000","last_undegraded":"2024-09-21T08:31:00.601565+0000","last_fullsized":"2024-09-21T08:31:00.601565+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'16","last_scrub_stamp":"2024-09-21T08:30:43.041103+0000","last_deep_scrub":"263'14","last_deep_scrub_stamp":"2024-09-21T08:27:36.998538+0000","last_clean_scrub_stamp":"2024-09-21T08:30:43.041103+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:07.558107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":823,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.607504+0000","last_change":"2024-09-21T08:28:53.591306+0000","last_active":"2024-09-21T08:31:00.607504+0000","last_peered":"2024-09-21T08:31:00.607504+0000","last_clean":"2024-09-21T08:28:51.554109+0000","last_became_active":"2024-09-21T08:28:53.591306+0000","last_became_peered":"2024-09-21T08:28:53.591306+0000","last_unstale":"2024-09-21T08:31:00.607504+0000","last_undegraded":"2024-09-21T08:31:00.607504+0000","last_fullsized":"2024-09-21T08:28:53.580235+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.344185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.344185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.703314+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"76'129","reported_seq":1086,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.606077+0000","last_change":"2024-09-21T08:30:54.613129+0000","last_active":"2024-09-21T08:31:00.606077+0000","last_peered":"2024-09-21T08:31:00.606077+0000","last_clean":"2024-09-21T08:31:00.606077+0000","last_became_active":"2024-09-21T08:30:54.612724+0000","last_became_peered":"2024-09-21T08:30:54.612724+0000","last_unstale":"2024-09-21T08:31:00.606077+0000","last_undegraded":"2024-09-21T08:31:00.606077+0000","last_fullsized":"2024-09-21T08:31:00.606077+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.842735+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":322019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,1],"avail_no_missing":["5","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"179'43","reported_seq":272,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602407+0000","last_change":"2024-09-21T08:31:00.602782+0000","last_active":"2024-09-21T08:31:00.602407+0000","last_peered":"2024-09-21T08:31:00.602407+0000","last_clean":"2024-09-21T08:31:00.602407+0000","last_became_active":"2024-09-21T08:30:57.622886+0000","last_became_peered":"2024-09-21T08:30:57.622886+0000","last_unstale":"2024-09-21T08:31:00.602407+0000","last_undegraded":"2024-09-21T08:31:00.602407+0000","last_fullsized":"2024-09-21T08:31:00.602407+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:50.158232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.158232+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0023786570000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.2","version":"73'10","reported_seq":420,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602993+0000","last_change":"2024-09-21T08:30:56.663150+0000","last_active":"2024-09-21T08:31:00.602993+0000","last_peered":"2024-09-21T08:31:00.602993+0000","last_clean":"2024-09-21T08:31:00.602993+0000","last_became_active":"2024-09-21T08:30:56.639664+0000","last_became_peered":"2024-09-21T08:30:56.639664+0000","last_unstale":"2024-09-21T08:31:00.602993+0000","last_undegraded":"2024-09-21T08:31:00.602993+0000","last_fullsized":"2024-09-21T08:31:00.602993+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:29:58.300806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.300806+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:12.150130+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":871,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.605934+0000","last_change":"2024-09-21T08:28:53.588911+0000","last_active":"2024-09-21T08:31:00.605934+0000","last_peered":"2024-09-21T08:31:00.605934+0000","last_clean":"2024-09-21T08:28:51.554689+0000","last_became_active":"2024-09-21T08:28:53.588911+0000","last_became_peered":"2024-09-21T08:28:53.588911+0000","last_unstale":"2024-09-21T08:31:00.605934+0000","last_undegraded":"2024-09-21T08:28:53.576074+0000","last_fullsized":"2024-09-21T08:28:53.575948+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:16.003225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.003225+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"340'16","reported_seq":638,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.606677+0000","last_change":"2024-09-21T08:28:53.594258+0000","last_active":"2024-09-21T08:31:00.606677+0000","last_peered":"2024-09-21T08:31:00.606677+0000","last_clean":"2024-09-21T08:28:51.556539+0000","last_became_active":"2024-09-21T08:28:53.594258+0000","last_became_peered":"2024-09-21T08:28:53.594258+0000","last_unstale":"2024-09-21T08:31:00.606677+0000","last_undegraded":"2024-09-21T08:28:53.574644+0000","last_fullsized":"2024-09-21T08:28:53.574523+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'13","last_scrub_stamp":"2024-09-21T08:27:27.874952+0000","last_deep_scrub":"176'11","last_deep_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_clean_scrub_stamp":"2024-09-21T08:27:27.874952+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":667,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.604167+0000","last_change":"2024-09-21T08:30:47.733247+0000","last_active":"2024-09-21T08:31:00.604167+0000","last_peered":"2024-09-21T08:31:00.604167+0000","last_clean":"2024-09-21T08:31:00.604167+0000","last_became_active":"2024-09-21T08:28:37.565624+0000","last_became_peered":"2024-09-21T08:28:37.565624+0000","last_unstale":"2024-09-21T08:31:00.604167+0000","last_undegraded":"2024-09-21T08:31:00.604167+0000","last_fullsized":"2024-09-21T08:31:00.604167+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:47.733148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:47.733148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:00.075734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"76'129","reported_seq":793,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.601612+0000","last_change":"2024-09-21T08:30:59.604276+0000","last_active":"2024-09-21T08:31:00.601612+0000","last_peered":"2024-09-21T08:31:00.601612+0000","last_clean":"2024-09-21T08:31:00.601612+0000","last_became_active":"2024-09-21T08:30:59.602487+0000","last_became_peered":"2024-09-21T08:30:59.602487+0000","last_unstale":"2024-09-21T08:31:00.601612+0000","last_undegraded":"2024-09-21T08:31:00.601612+0000","last_fullsized":"2024-09-21T08:31:00.601612+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963331+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"467'220","reported_seq":2271,"reported_epoch":467,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:31:00.606764+0000","last_change":"2024-09-21T08:31:00.606764+0000","last_active":"2024-09-21T08:31:00.606764+0000","last_peered":"2024-09-21T08:31:00.606764+0000","last_clean":"2024-09-21T08:31:00.606764+0000","last_became_active":"2024-09-21T08:28:07.549600+0000","last_became_peered":"2024-09-21T08:28:07.549600+0000","last_unstale":"2024-09-21T08:31:00.606764+0000","last_undegraded":"2024-09-21T08:31:00.606764+0000","last_fullsized":"2024-09-21T08:31:00.606764+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'166","last_scrub_stamp":"2024-09-21T08:29:32.253921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.253921+0000","objects_scrubbed":3,"log_size":220,"log_dups_size":0,"ondisk_log_size":220,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.034853467999999999,"stat_sum":{"num_bytes":12502908,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":57,"num_read_kb":69788,"num_write":29,"num_write_kb":10125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1484609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.3","version":"73'19","reported_seq":389,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.605970+0000","last_change":"2024-09-21T08:28:53.593536+0000","last_active":"2024-09-21T08:31:00.605970+0000","last_peered":"2024-09-21T08:31:00.605970+0000","last_clean":"2024-09-21T08:28:51.557090+0000","last_became_active":"2024-09-21T08:28:53.593536+0000","last_became_peered":"2024-09-21T08:28:53.593536+0000","last_unstale":"2024-09-21T08:31:00.605970+0000","last_undegraded":"2024-09-21T08:28:53.574981+0000","last_fullsized":"2024-09-21T08:28:53.574872+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:27:55.260127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:55.260127+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":372,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.603449+0000","last_change":"2024-09-21T08:30:46.074689+0000","last_active":"2024-09-21T08:31:00.603449+0000","last_peered":"2024-09-21T08:31:00.603449+0000","last_clean":"2024-09-21T08:31:00.603449+0000","last_became_active":"2024-09-21T08:28:04.723998+0000","last_became_peered":"2024-09-21T08:28:04.723998+0000","last_unstale":"2024-09-21T08:31:00.603449+0000","last_undegraded":"2024-09-21T08:31:00.603449+0000","last_fullsized":"2024-09-21T08:31:00.603449+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:46.074613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:30:46.074613+0000","last_clean_scrub_stamp":"2024-09-21T08:30:46.074613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:51.291896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":319,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.607635+0000","last_change":"2024-09-21T08:28:53.588949+0000","last_active":"2024-09-21T08:31:00.607635+0000","last_peered":"2024-09-21T08:31:00.607635+0000","last_clean":"2024-09-21T08:28:51.557587+0000","last_became_active":"2024-09-21T08:28:53.588949+0000","last_became_peered":"2024-09-21T08:28:53.588949+0000","last_unstale":"2024-09-21T08:31:00.607635+0000","last_undegraded":"2024-09-21T08:31:00.607635+0000","last_fullsized":"2024-09-21T08:28:53.575963+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.339126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.339126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":367,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.600554+0000","last_change":"2024-09-21T08:28:53.600582+0000","last_active":"2024-09-21T08:31:00.600554+0000","last_peered":"2024-09-21T08:31:00.600554+0000","last_clean":"2024-09-21T08:28:51.584621+0000","last_became_active":"2024-09-21T08:28:53.600582+0000","last_became_peered":"2024-09-21T08:28:53.600582+0000","last_unstale":"2024-09-21T08:31:00.600554+0000","last_undegraded":"2024-09-21T08:31:00.600554+0000","last_fullsized":"2024-09-21T08:28:53.576129+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:46.263984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.263984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995536+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'129","reported_seq":1094,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.604971+0000","last_change":"2024-09-21T08:30:59.600857+0000","last_active":"2024-09-21T08:31:00.604971+0000","last_peered":"2024-09-21T08:31:00.604971+0000","last_clean":"2024-09-21T08:31:00.604971+0000","last_became_active":"2024-09-21T08:30:59.600493+0000","last_became_peered":"2024-09-21T08:30:59.600493+0000","last_unstale":"2024-09-21T08:31:00.604971+0000","last_undegraded":"2024-09-21T08:31:00.604971+0000","last_fullsized":"2024-09-21T08:31:00.604971+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.842754+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":322019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"466'194","reported_seq":2082,"reported_epoch":467,"state":"active+clean+remapped+snaptrim","last_fresh":"2024-09-21T08:31:00.604735+0000","last_change":"2024-09-21T08:30:59.607958+0000","last_active":"2024-09-21T08:31:00.604735+0000","last_peered":"2024-09-21T08:31:00.604735+0000","last_clean":"2024-09-21T08:31:00.604735+0000","last_became_active":"2024-09-21T08:30:59.605652+0000","last_became_peered":"2024-09-21T08:30:59.605652+0000","last_unstale":"2024-09-21T08:31:00.604735+0000","last_undegraded":"2024-09-21T08:31:00.604735+0000","last_fullsized":"2024-09-21T08:31:00.604735+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":194,"log_dups_size":0,"ondisk_log_size":194,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":67,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.074195671000000005,"stat_sum":{"num_bytes":28995274,"num_objects":12,"num_object_clones":10,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":0,"num_read":71,"num_read_kb":59407,"num_write":70,"num_write_kb":36181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":87,"num_bytes_recovered":54540496,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":12}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"1"}]},{"pgid":"4.4","version":"340'35","reported_seq":426,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.602744+0000","last_change":"2024-09-21T08:28:53.603157+0000","last_active":"2024-09-21T08:31:00.602744+0000","last_peered":"2024-09-21T08:31:00.602744+0000","last_clean":"2024-09-21T08:28:51.586037+0000","last_became_active":"2024-09-21T08:28:53.603157+0000","last_became_peered":"2024-09-21T08:28:53.603157+0000","last_unstale":"2024-09-21T08:31:00.602744+0000","last_undegraded":"2024-09-21T08:28:53.579248+0000","last_fullsized":"2024-09-21T08:28:53.579125+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'32","last_scrub_stamp":"2024-09-21T08:27:11.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:11.407384+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995544+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":19,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.601091+0000","last_change":"2024-09-21T08:30:56.627449+0000","last_active":"2024-09-21T08:31:00.601091+0000","last_peered":"2024-09-21T08:31:00.601091+0000","last_clean":"2024-09-21T08:31:00.601091+0000","last_became_active":"2024-09-21T08:30:56.627283+0000","last_became_peered":"2024-09-21T08:30:56.627283+0000","last_unstale":"2024-09-21T08:31:00.601091+0000","last_undegraded":"2024-09-21T08:31:00.601091+0000","last_fullsized":"2024-09-21T08:31:00.601091+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:14.370256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.370256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":895,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.606398+0000","last_change":"2024-09-21T08:28:53.594367+0000","last_active":"2024-09-21T08:31:00.606398+0000","last_peered":"2024-09-21T08:31:00.606398+0000","last_clean":"2024-09-21T08:28:51.557303+0000","last_became_active":"2024-09-21T08:28:53.594367+0000","last_became_peered":"2024-09-21T08:28:53.594367+0000","last_unstale":"2024-09-21T08:31:00.606398+0000","last_undegraded":"2024-09-21T08:28:53.574945+0000","last_fullsized":"2024-09-21T08:28:53.574836+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":812,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602947+0000","last_change":"2024-09-21T08:30:57.609064+0000","last_active":"2024-09-21T08:31:00.602947+0000","last_peered":"2024-09-21T08:31:00.602947+0000","last_clean":"2024-09-21T08:31:00.602947+0000","last_became_active":"2024-09-21T08:30:57.608038+0000","last_became_peered":"2024-09-21T08:30:57.608038+0000","last_unstale":"2024-09-21T08:31:00.602947+0000","last_undegraded":"2024-09-21T08:31:00.602947+0000","last_fullsized":"2024-09-21T08:31:00.602947+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:45.237804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.237804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087354+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602525+0000","last_change":"2024-09-21T08:30:57.620758+0000","last_active":"2024-09-21T08:31:00.602525+0000","last_peered":"2024-09-21T08:31:00.602525+0000","last_clean":"2024-09-21T08:31:00.602525+0000","last_became_active":"2024-09-21T08:30:57.618058+0000","last_became_peered":"2024-09-21T08:30:57.618058+0000","last_unstale":"2024-09-21T08:31:00.602525+0000","last_undegraded":"2024-09-21T08:31:00.602525+0000","last_fullsized":"2024-09-21T08:31:00.602525+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.844729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.844729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.964036+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"76'129","reported_seq":1084,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.600794+0000","last_change":"2024-09-21T08:30:59.601231+0000","last_active":"2024-09-21T08:31:00.600794+0000","last_peered":"2024-09-21T08:31:00.600794+0000","last_clean":"2024-09-21T08:31:00.600794+0000","last_became_active":"2024-09-21T08:30:59.600899+0000","last_became_peered":"2024-09-21T08:30:59.600899+0000","last_unstale":"2024-09-21T08:31:00.600794+0000","last_undegraded":"2024-09-21T08:31:00.600794+0000","last_fullsized":"2024-09-21T08:31:00.600794+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963298+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":322019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"467'155","reported_seq":2063,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.610289+0000","last_change":"2024-09-21T08:31:00.610572+0000","last_active":"2024-09-21T08:31:00.610289+0000","last_peered":"2024-09-21T08:31:00.610289+0000","last_clean":"2024-09-21T08:31:00.610289+0000","last_became_active":"2024-09-21T08:30:59.604883+0000","last_became_peered":"2024-09-21T08:30:59.604883+0000","last_unstale":"2024-09-21T08:31:00.610289+0000","last_undegraded":"2024-09-21T08:31:00.610289+0000","last_fullsized":"2024-09-21T08:31:00.610289+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"445'141","last_scrub_stamp":"2024-09-21T08:30:40.016003+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:30:40.016003+0000","objects_scrubbed":4,"log_size":155,"log_dups_size":0,"ondisk_log_size":155,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:07.980253+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.0088667610000000008,"stat_sum":{"num_bytes":7318695,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":1,"num_read":39,"num_read_kb":47754,"num_write":20,"num_write_kb":10608,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":18164798,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.7","version":"73'13","reported_seq":405,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.605060+0000","last_change":"2024-09-21T08:28:53.595533+0000","last_active":"2024-09-21T08:31:00.605060+0000","last_peered":"2024-09-21T08:31:00.605060+0000","last_clean":"2024-09-21T08:28:51.587779+0000","last_became_active":"2024-09-21T08:28:53.595533+0000","last_became_peered":"2024-09-21T08:28:53.595533+0000","last_unstale":"2024-09-21T08:31:00.605060+0000","last_undegraded":"2024-09-21T08:28:53.580029+0000","last_fullsized":"2024-09-21T08:28:53.579372+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-09-21T08:28:16.386950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.386950+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995638+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":378,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.601778+0000","last_change":"2024-09-21T08:29:56.368212+0000","last_active":"2024-09-21T08:31:00.601778+0000","last_peered":"2024-09-21T08:31:00.601778+0000","last_clean":"2024-09-21T08:31:00.601778+0000","last_became_active":"2024-09-21T08:28:29.629320+0000","last_became_peered":"2024-09-21T08:28:29.629320+0000","last_unstale":"2024-09-21T08:31:00.601778+0000","last_undegraded":"2024-09-21T08:31:00.601778+0000","last_fullsized":"2024-09-21T08:31:00.601778+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:56.368144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:29:56.368144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:02.232771+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":846,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.601762+0000","last_change":"2024-09-21T08:30:56.650473+0000","last_active":"2024-09-21T08:31:00.601762+0000","last_peered":"2024-09-21T08:31:00.601762+0000","last_clean":"2024-09-21T08:31:00.601762+0000","last_became_active":"2024-09-21T08:30:56.629748+0000","last_became_peered":"2024-09-21T08:30:56.629748+0000","last_unstale":"2024-09-21T08:31:00.601762+0000","last_undegraded":"2024-09-21T08:31:00.601762+0000","last_fullsized":"2024-09-21T08:31:00.601762+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:59.262463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:27:59.262463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087436+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":824,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.607791+0000","last_change":"2024-09-21T08:28:53.598122+0000","last_active":"2024-09-21T08:31:00.607791+0000","last_peered":"2024-09-21T08:31:00.607791+0000","last_clean":"2024-09-21T08:28:51.556093+0000","last_became_active":"2024-09-21T08:28:53.598122+0000","last_became_peered":"2024-09-21T08:28:53.598122+0000","last_unstale":"2024-09-21T08:31:00.607791+0000","last_undegraded":"2024-09-21T08:31:00.607791+0000","last_fullsized":"2024-09-21T08:28:53.584774+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.374076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.374076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.703337+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":722,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.605942+0000","last_change":"2024-09-21T08:28:53.587481+0000","last_active":"2024-09-21T08:31:00.605942+0000","last_peered":"2024-09-21T08:31:00.605942+0000","last_clean":"2024-09-21T08:28:51.554685+0000","last_became_active":"2024-09-21T08:28:53.587481+0000","last_became_peered":"2024-09-21T08:28:53.587481+0000","last_unstale":"2024-09-21T08:31:00.605942+0000","last_undegraded":"2024-09-21T08:31:00.605942+0000","last_fullsized":"2024-09-21T08:28:53.575943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:08.831386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:08.831386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"76'129","reported_seq":1102,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.603556+0000","last_change":"2024-09-21T08:30:59.606331+0000","last_active":"2024-09-21T08:31:00.603556+0000","last_peered":"2024-09-21T08:31:00.603556+0000","last_clean":"2024-09-21T08:31:00.603556+0000","last_became_active":"2024-09-21T08:30:59.605942+0000","last_became_peered":"2024-09-21T08:30:59.605942+0000","last_unstale":"2024-09-21T08:31:00.603556+0000","last_undegraded":"2024-09-21T08:31:00.603556+0000","last_fullsized":"2024-09-21T08:31:00.603556+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087127+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":28,"num_write":17,"num_write_kb":220,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":322019,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"462'307","reported_seq":1970,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.601939+0000","last_change":"2024-09-21T08:28:53.600711+0000","last_active":"2024-09-21T08:31:00.601939+0000","last_peered":"2024-09-21T08:31:00.601939+0000","last_clean":"2024-09-21T08:28:52.661593+0000","last_became_active":"2024-09-21T08:28:53.600711+0000","last_became_peered":"2024-09-21T08:28:53.600711+0000","last_unstale":"2024-09-21T08:31:00.601939+0000","last_undegraded":"2024-09-21T08:28:53.576301+0000","last_fullsized":"2024-09-21T08:28:53.576176+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'245","last_scrub_stamp":"2024-09-21T08:26:30.864040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.864040+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":62,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.075799959,"stat_sum":{"num_bytes":8372441,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":49,"num_read_kb":51018,"num_write":35,"num_write_kb":10072,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":1853978,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.6","version":"73'12","reported_seq":833,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.605069+0000","last_change":"2024-09-21T08:28:53.603118+0000","last_active":"2024-09-21T08:31:00.605069+0000","last_peered":"2024-09-21T08:31:00.605069+0000","last_clean":"2024-09-21T08:28:51.587443+0000","last_became_active":"2024-09-21T08:28:53.603118+0000","last_became_peered":"2024-09-21T08:28:53.603118+0000","last_unstale":"2024-09-21T08:31:00.605069+0000","last_undegraded":"2024-09-21T08:28:53.582699+0000","last_fullsized":"2024-09-21T08:28:53.582597+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:26:07.384640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:07.384640+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995457+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.601353+0000","last_change":"2024-09-21T08:30:57.626533+0000","last_active":"2024-09-21T08:31:00.601353+0000","last_peered":"2024-09-21T08:31:00.601353+0000","last_clean":"2024-09-21T08:31:00.601353+0000","last_became_active":"2024-09-21T08:30:57.621632+0000","last_became_peered":"2024-09-21T08:30:57.621632+0000","last_unstale":"2024-09-21T08:31:00.601353+0000","last_undegraded":"2024-09-21T08:31:00.601353+0000","last_fullsized":"2024-09-21T08:31:00.601353+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.917891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.917891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1300,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.600888+0000","last_change":"2024-09-21T08:28:53.594845+0000","last_active":"2024-09-21T08:31:00.600888+0000","last_peered":"2024-09-21T08:31:00.600888+0000","last_clean":"2024-09-21T08:28:52.670294+0000","last_became_active":"2024-09-21T08:28:53.594845+0000","last_became_peered":"2024-09-21T08:28:53.594845+0000","last_unstale":"2024-09-21T08:31:00.600888+0000","last_undegraded":"2024-09-21T08:28:53.577170+0000","last_fullsized":"2024-09-21T08:28:53.575736+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995418+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":427,"num_read_kb":422,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":841,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.607565+0000","last_change":"2024-09-21T08:30:56.627295+0000","last_active":"2024-09-21T08:31:00.607565+0000","last_peered":"2024-09-21T08:31:00.607565+0000","last_clean":"2024-09-21T08:31:00.607565+0000","last_became_active":"2024-09-21T08:30:56.626598+0000","last_became_peered":"2024-09-21T08:30:56.626598+0000","last_unstale":"2024-09-21T08:31:00.607565+0000","last_undegraded":"2024-09-21T08:31:00.607565+0000","last_fullsized":"2024-09-21T08:31:00.607565+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.812789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.812789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816838+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":378,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.603352+0000","last_change":"2024-09-21T08:30:56.638995+0000","last_active":"2024-09-21T08:31:00.603352+0000","last_peered":"2024-09-21T08:31:00.603352+0000","last_clean":"2024-09-21T08:31:00.603352+0000","last_became_active":"2024-09-21T08:30:56.638712+0000","last_became_peered":"2024-09-21T08:30:56.638712+0000","last_unstale":"2024-09-21T08:31:00.603352+0000","last_undegraded":"2024-09-21T08:31:00.603352+0000","last_fullsized":"2024-09-21T08:31:00.603352+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:17.424515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:17.424515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087487+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"453'161","reported_seq":1877,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.606718+0000","last_change":"2024-09-21T08:28:53.589874+0000","last_active":"2024-09-21T08:31:00.606718+0000","last_peered":"2024-09-21T08:31:00.606718+0000","last_clean":"2024-09-21T08:28:51.554645+0000","last_became_active":"2024-09-21T08:28:53.589874+0000","last_became_peered":"2024-09-21T08:28:53.589874+0000","last_unstale":"2024-09-21T08:31:00.606718+0000","last_undegraded":"2024-09-21T08:28:53.577395+0000","last_fullsized":"2024-09-21T08:28:53.577254+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'138","last_scrub_stamp":"2024-09-21T08:28:41.855754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:41.855754+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":58,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027851600000000001,"stat_sum":{"num_bytes":13036677,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":52,"num_read_kb":63284,"num_write":28,"num_write_kb":9661,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26,"num_bytes_recovered":9282889,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"4.5","version":"464'24","reported_seq":39,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.601565+0000","last_change":"2024-09-21T08:30:57.660855+0000","last_active":"2024-09-21T08:31:00.601565+0000","last_peered":"2024-09-21T08:31:00.601565+0000","last_clean":"2024-09-21T08:31:00.601565+0000","last_became_active":"2024-09-21T08:30:57.616133+0000","last_became_peered":"2024-09-21T08:30:57.616133+0000","last_unstale":"2024-09-21T08:31:00.601565+0000","last_undegraded":"2024-09-21T08:31:00.601565+0000","last_fullsized":"2024-09-21T08:31:00.601565+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-21T08:28:13.956509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.956509+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963913+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4,5","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":738,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.609750+0000","last_change":"2024-09-21T08:30:56.629168+0000","last_active":"2024-09-21T08:31:00.609750+0000","last_peered":"2024-09-21T08:31:00.609750+0000","last_clean":"2024-09-21T08:31:00.609750+0000","last_became_active":"2024-09-21T08:30:56.628829+0000","last_became_peered":"2024-09-21T08:30:56.628829+0000","last_unstale":"2024-09-21T08:31:00.609750+0000","last_undegraded":"2024-09-21T08:31:00.609750+0000","last_fullsized":"2024-09-21T08:31:00.609750+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.045353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.045353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":1085,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.600745+0000","last_change":"2024-09-21T08:30:49.608935+0000","last_active":"2024-09-21T08:31:00.600745+0000","last_peered":"2024-09-21T08:31:00.600745+0000","last_clean":"2024-09-21T08:28:51.584594+0000","last_became_active":"2024-09-21T08:30:49.608935+0000","last_became_peered":"2024-09-21T08:30:49.608935+0000","last_unstale":"2024-09-21T08:31:00.600745+0000","last_undegraded":"2024-09-21T08:31:00.600745+0000","last_fullsized":"2024-09-21T08:30:49.602844+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:02.006058+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":322018,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":302,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.606724+0000","last_change":"2024-09-21T08:28:53.589976+0000","last_active":"2024-09-21T08:31:00.606724+0000","last_peered":"2024-09-21T08:31:00.606724+0000","last_clean":"2024-09-21T08:28:51.557022+0000","last_became_active":"2024-09-21T08:28:53.589976+0000","last_became_peered":"2024-09-21T08:28:53.589976+0000","last_unstale":"2024-09-21T08:31:00.606724+0000","last_undegraded":"2024-09-21T08:31:00.606724+0000","last_fullsized":"2024-09-21T08:28:53.576557+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":855,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.607708+0000","last_change":"2024-09-21T08:30:56.627506+0000","last_active":"2024-09-21T08:31:00.607708+0000","last_peered":"2024-09-21T08:31:00.607708+0000","last_clean":"2024-09-21T08:31:00.607708+0000","last_became_active":"2024-09-21T08:30:56.627007+0000","last_became_peered":"2024-09-21T08:30:56.627007+0000","last_unstale":"2024-09-21T08:31:00.607708+0000","last_undegraded":"2024-09-21T08:31:00.607708+0000","last_fullsized":"2024-09-21T08:31:00.607708+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816746+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"455'154","reported_seq":725,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.603624+0000","last_change":"2024-09-21T08:28:53.596348+0000","last_active":"2024-09-21T08:31:00.603624+0000","last_peered":"2024-09-21T08:31:00.603624+0000","last_clean":"2024-09-21T08:28:52.661631+0000","last_became_active":"2024-09-21T08:28:53.596348+0000","last_became_peered":"2024-09-21T08:28:53.596348+0000","last_unstale":"2024-09-21T08:31:00.603624+0000","last_undegraded":"2024-09-21T08:28:53.582490+0000","last_fullsized":"2024-09-21T08:28:53.579369+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'93","last_scrub_stamp":"2024-09-21T08:28:11.373358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:11.373358+0000","objects_scrubbed":3,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":67,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.070749670000000001,"stat_sum":{"num_bytes":27948773,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":2,"num_read":31,"num_read_kb":17875,"num_write":41,"num_write_kb":11195,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":6571636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":14}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"1"}]},{"pgid":"4.e","version":"73'11","reported_seq":31,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602061+0000","last_change":"2024-09-21T08:30:56.656178+0000","last_active":"2024-09-21T08:31:00.602061+0000","last_peered":"2024-09-21T08:31:00.602061+0000","last_clean":"2024-09-21T08:31:00.602061+0000","last_became_active":"2024-09-21T08:30:56.626093+0000","last_became_peered":"2024-09-21T08:30:56.626093+0000","last_unstale":"2024-09-21T08:31:00.602061+0000","last_undegraded":"2024-09-21T08:31:00.602061+0000","last_fullsized":"2024-09-21T08:31:00.602061+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:28:14.416729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.416729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963985+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.601271+0000","last_change":"2024-09-21T08:30:57.630426+0000","last_active":"2024-09-21T08:31:00.601271+0000","last_peered":"2024-09-21T08:31:00.601271+0000","last_clean":"2024-09-21T08:31:00.601271+0000","last_became_active":"2024-09-21T08:30:57.626173+0000","last_became_peered":"2024-09-21T08:30:57.626173+0000","last_unstale":"2024-09-21T08:31:00.601271+0000","last_undegraded":"2024-09-21T08:31:00.601271+0000","last_fullsized":"2024-09-21T08:31:00.601271+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:28.347474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:28.347474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":864,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.607804+0000","last_change":"2024-09-21T08:30:56.619286+0000","last_active":"2024-09-21T08:31:00.607804+0000","last_peered":"2024-09-21T08:31:00.607804+0000","last_clean":"2024-09-21T08:31:00.607804+0000","last_became_active":"2024-09-21T08:30:56.618484+0000","last_became_peered":"2024-09-21T08:30:56.618484+0000","last_unstale":"2024-09-21T08:31:00.607804+0000","last_undegraded":"2024-09-21T08:31:00.607804+0000","last_fullsized":"2024-09-21T08:31:00.607804+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:55.150058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:32.961381+0000","last_clean_scrub_stamp":"2024-09-21T08:29:55.150058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.386629+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":711,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.607045+0000","last_change":"2024-09-21T08:28:53.601979+0000","last_active":"2024-09-21T08:31:00.607045+0000","last_peered":"2024-09-21T08:31:00.607045+0000","last_clean":"2024-09-21T08:28:51.557371+0000","last_became_active":"2024-09-21T08:28:53.601979+0000","last_became_peered":"2024-09-21T08:28:53.601979+0000","last_unstale":"2024-09-21T08:31:00.607045+0000","last_undegraded":"2024-09-21T08:31:00.607045+0000","last_fullsized":"2024-09-21T08:28:53.577942+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.977956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"76'129","reported_seq":1099,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.603935+0000","last_change":"2024-09-21T08:30:59.604641+0000","last_active":"2024-09-21T08:31:00.603935+0000","last_peered":"2024-09-21T08:31:00.603935+0000","last_clean":"2024-09-21T08:31:00.603935+0000","last_became_active":"2024-09-21T08:30:59.604289+0000","last_became_peered":"2024-09-21T08:30:59.604289+0000","last_unstale":"2024-09-21T08:31:00.603935+0000","last_undegraded":"2024-09-21T08:31:00.603935+0000","last_fullsized":"2024-09-21T08:31:00.603935+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.842801+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":847395,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"256'199","reported_seq":2112,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.607812+0000","last_change":"2024-09-21T08:31:00.608117+0000","last_active":"2024-09-21T08:31:00.607812+0000","last_peered":"2024-09-21T08:31:00.607812+0000","last_clean":"2024-09-21T08:31:00.607812+0000","last_became_active":"2024-09-21T08:28:32.558440+0000","last_became_peered":"2024-09-21T08:28:32.558440+0000","last_unstale":"2024-09-21T08:31:00.607812+0000","last_undegraded":"2024-09-21T08:31:00.607812+0000","last_fullsized":"2024-09-21T08:31:00.607812+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'199","last_scrub_stamp":"2024-09-21T08:30:17.742629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:17.742629+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:40.989634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000541191,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.d","version":"345'25","reported_seq":37,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.601686+0000","last_change":"2024-09-21T08:30:56.655936+0000","last_active":"2024-09-21T08:31:00.601686+0000","last_peered":"2024-09-21T08:31:00.601686+0000","last_clean":"2024-09-21T08:31:00.601686+0000","last_became_active":"2024-09-21T08:30:56.617615+0000","last_became_peered":"2024-09-21T08:30:56.617615+0000","last_unstale":"2024-09-21T08:31:00.601686+0000","last_undegraded":"2024-09-21T08:31:00.601686+0000","last_fullsized":"2024-09-21T08:31:00.601686+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'25","last_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_deep_scrub":"345'25","last_deep_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.241931+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.964132+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":19,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.603542+0000","last_change":"2024-09-21T08:30:56.631627+0000","last_active":"2024-09-21T08:31:00.603542+0000","last_peered":"2024-09-21T08:31:00.603542+0000","last_clean":"2024-09-21T08:31:00.603542+0000","last_became_active":"2024-09-21T08:30:56.630646+0000","last_became_peered":"2024-09-21T08:30:56.630646+0000","last_unstale":"2024-09-21T08:31:00.603542+0000","last_undegraded":"2024-09-21T08:31:00.603542+0000","last_fullsized":"2024-09-21T08:31:00.603542+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:30.909902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.909902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":378,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602330+0000","last_change":"2024-09-21T08:30:57.609293+0000","last_active":"2024-09-21T08:31:00.602330+0000","last_peered":"2024-09-21T08:31:00.602330+0000","last_clean":"2024-09-21T08:31:00.602330+0000","last_became_active":"2024-09-21T08:30:57.608826+0000","last_became_peered":"2024-09-21T08:30:57.608826+0000","last_unstale":"2024-09-21T08:31:00.602330+0000","last_undegraded":"2024-09-21T08:31:00.602330+0000","last_fullsized":"2024-09-21T08:31:00.602330+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":19,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602069+0000","last_change":"2024-09-21T08:30:56.617373+0000","last_active":"2024-09-21T08:31:00.602069+0000","last_peered":"2024-09-21T08:31:00.602069+0000","last_clean":"2024-09-21T08:31:00.602069+0000","last_became_active":"2024-09-21T08:30:56.617090+0000","last_became_peered":"2024-09-21T08:30:56.617090+0000","last_unstale":"2024-09-21T08:31:00.602069+0000","last_undegraded":"2024-09-21T08:31:00.602069+0000","last_fullsized":"2024-09-21T08:31:00.602069+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:00.276774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:00.276774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.854053+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"76'129","reported_seq":788,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.605816+0000","last_change":"2024-09-21T08:30:54.608124+0000","last_active":"2024-09-21T08:31:00.605816+0000","last_peered":"2024-09-21T08:31:00.605816+0000","last_clean":"2024-09-21T08:31:00.605816+0000","last_became_active":"2024-09-21T08:30:54.607659+0000","last_became_peered":"2024-09-21T08:30:54.607659+0000","last_unstale":"2024-09-21T08:31:00.605816+0000","last_undegraded":"2024-09-21T08:31:00.605816+0000","last_fullsized":"2024-09-21T08:31:00.605816+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53670,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":28,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":268349,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5,1],"avail_no_missing":["2","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"467'104","reported_seq":326,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.611987+0000","last_change":"2024-09-21T08:31:00.612304+0000","last_active":"2024-09-21T08:31:00.611987+0000","last_peered":"2024-09-21T08:31:00.611987+0000","last_clean":"2024-09-21T08:31:00.611987+0000","last_became_active":"2024-09-21T08:30:57.622924+0000","last_became_peered":"2024-09-21T08:30:57.622924+0000","last_unstale":"2024-09-21T08:31:00.611987+0000","last_undegraded":"2024-09-21T08:31:00.611987+0000","last_fullsized":"2024-09-21T08:31:00.611987+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:28:10.372149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.372149+0000","objects_scrubbed":1,"log_size":104,"log_dups_size":0,"ondisk_log_size":104,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00949061,"stat_sum":{"num_bytes":13171734,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":29,"num_read_kb":28274,"num_write":39,"num_write_kb":16470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":23448304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[{"shards":"2,4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.c","version":"73'10","reported_seq":30,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.600865+0000","last_change":"2024-09-21T08:30:57.643080+0000","last_active":"2024-09-21T08:31:00.600865+0000","last_peered":"2024-09-21T08:31:00.600865+0000","last_clean":"2024-09-21T08:31:00.600865+0000","last_became_active":"2024-09-21T08:30:57.611720+0000","last_became_peered":"2024-09-21T08:30:57.611720+0000","last_unstale":"2024-09-21T08:31:00.600865+0000","last_undegraded":"2024-09-21T08:31:00.600865+0000","last_fullsized":"2024-09-21T08:31:00.600865+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:26:33.823950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.823950+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963546+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.600616+0000","last_change":"2024-09-21T08:30:57.609872+0000","last_active":"2024-09-21T08:31:00.600616+0000","last_peered":"2024-09-21T08:31:00.600616+0000","last_clean":"2024-09-21T08:31:00.600616+0000","last_became_active":"2024-09-21T08:30:57.609402+0000","last_became_peered":"2024-09-21T08:30:57.609402+0000","last_unstale":"2024-09-21T08:31:00.600616+0000","last_undegraded":"2024-09-21T08:31:00.600616+0000","last_fullsized":"2024-09-21T08:31:00.600616+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:54.216528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:54.216528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963816+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"340'10","reported_seq":355,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.609830+0000","last_change":"2024-09-21T08:28:53.589819+0000","last_active":"2024-09-21T08:31:00.609830+0000","last_peered":"2024-09-21T08:31:00.609830+0000","last_clean":"2024-09-21T08:28:51.555429+0000","last_became_active":"2024-09-21T08:28:53.589819+0000","last_became_peered":"2024-09-21T08:28:53.589819+0000","last_unstale":"2024-09-21T08:31:00.609830+0000","last_undegraded":"2024-09-21T08:28:53.578694+0000","last_fullsized":"2024-09-21T08:28:53.577943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:27:51.312273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.312273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":19,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602155+0000","last_change":"2024-09-21T08:30:56.627132+0000","last_active":"2024-09-21T08:31:00.602155+0000","last_peered":"2024-09-21T08:31:00.602155+0000","last_clean":"2024-09-21T08:31:00.602155+0000","last_became_active":"2024-09-21T08:30:56.626799+0000","last_became_peered":"2024-09-21T08:30:56.626799+0000","last_unstale":"2024-09-21T08:31:00.602155+0000","last_undegraded":"2024-09-21T08:31:00.602155+0000","last_fullsized":"2024-09-21T08:31:00.602155+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:40.458054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:40.458054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:07.464713+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"467'376","reported_seq":2256,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.607352+0000","last_change":"2024-09-21T08:31:00.608255+0000","last_active":"2024-09-21T08:31:00.607352+0000","last_peered":"2024-09-21T08:31:00.607352+0000","last_clean":"2024-09-21T08:31:00.607352+0000","last_became_active":"2024-09-21T08:30:59.599438+0000","last_became_peered":"2024-09-21T08:30:59.599438+0000","last_unstale":"2024-09-21T08:31:00.607352+0000","last_undegraded":"2024-09-21T08:31:00.607352+0000","last_fullsized":"2024-09-21T08:31:00.607352+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'178","last_scrub_stamp":"2024-09-21T08:26:28.217842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.217842+0000","objects_scrubbed":4,"log_size":376,"log_dups_size":0,"ondisk_log_size":376,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0078853410000000006,"stat_sum":{"num_bytes":19126741,"num_objects":12,"num_object_clones":9,"num_object_copies":36,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":12,"num_objects_unfound":0,"num_objects_dirty":12,"num_whiteouts":3,"num_read":127,"num_read_kb":155669,"num_write":97,"num_write_kb":37199,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":44827516,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[{"shards":"2,4,5","objects":12}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.b","version":"73'9","reported_seq":902,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602013+0000","last_change":"2024-09-21T08:30:56.654171+0000","last_active":"2024-09-21T08:31:00.602013+0000","last_peered":"2024-09-21T08:31:00.602013+0000","last_clean":"2024-09-21T08:31:00.602013+0000","last_became_active":"2024-09-21T08:30:56.626474+0000","last_became_peered":"2024-09-21T08:30:56.626474+0000","last_unstale":"2024-09-21T08:31:00.602013+0000","last_undegraded":"2024-09-21T08:31:00.602013+0000","last_fullsized":"2024-09-21T08:31:00.602013+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:25:22.369811+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.369811+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087180+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":872,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.605708+0000","last_change":"2024-09-21T08:28:59.403264+0000","last_active":"2024-09-21T08:31:00.605708+0000","last_peered":"2024-09-21T08:31:00.605708+0000","last_clean":"2024-09-21T08:31:00.605708+0000","last_became_active":"2024-09-21T08:28:29.609526+0000","last_became_peered":"2024-09-21T08:28:29.609526+0000","last_unstale":"2024-09-21T08:31:00.605708+0000","last_undegraded":"2024-09-21T08:31:00.605708+0000","last_fullsized":"2024-09-21T08:31:00.605708+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:59.403194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.403194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":19,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602585+0000","last_change":"2024-09-21T08:30:56.631993+0000","last_active":"2024-09-21T08:31:00.602585+0000","last_peered":"2024-09-21T08:31:00.602585+0000","last_clean":"2024-09-21T08:31:00.602585+0000","last_became_active":"2024-09-21T08:30:56.630411+0000","last_became_peered":"2024-09-21T08:30:56.630411+0000","last_unstale":"2024-09-21T08:31:00.602585+0000","last_undegraded":"2024-09-21T08:31:00.602585+0000","last_fullsized":"2024-09-21T08:31:00.602585+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963427+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":325,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.606796+0000","last_change":"2024-09-21T08:28:53.591119+0000","last_active":"2024-09-21T08:31:00.606796+0000","last_peered":"2024-09-21T08:31:00.606796+0000","last_clean":"2024-09-21T08:28:51.552940+0000","last_became_active":"2024-09-21T08:28:53.591119+0000","last_became_peered":"2024-09-21T08:28:53.591119+0000","last_unstale":"2024-09-21T08:31:00.606796+0000","last_undegraded":"2024-09-21T08:31:00.606796+0000","last_fullsized":"2024-09-21T08:28:53.575921+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:36.308161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:36.308161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.703325+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"466'375","reported_seq":2349,"reported_epoch":467,"state":"active+clean+remapped+snaptrim","last_fresh":"2024-09-21T08:31:00.602215+0000","last_change":"2024-09-21T08:30:59.608701+0000","last_active":"2024-09-21T08:31:00.602215+0000","last_peered":"2024-09-21T08:31:00.602215+0000","last_clean":"2024-09-21T08:31:00.602215+0000","last_became_active":"2024-09-21T08:30:59.605649+0000","last_became_peered":"2024-09-21T08:30:59.605649+0000","last_unstale":"2024-09-21T08:31:00.602215+0000","last_undegraded":"2024-09-21T08:31:00.602215+0000","last_fullsized":"2024-09-21T08:31:00.602215+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":375,"log_dups_size":0,"ondisk_log_size":375,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":68,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.0805860919999999,"stat_sum":{"num_bytes":40081525,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":19,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":183,"num_read_kb":192871,"num_write":119,"num_write_kb":39699,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":116,"num_bytes_recovered":57090966,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":19}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.a","version":"73'19","reported_seq":387,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.602549+0000","last_change":"2024-09-21T08:28:53.600149+0000","last_active":"2024-09-21T08:31:00.602549+0000","last_peered":"2024-09-21T08:31:00.602549+0000","last_clean":"2024-09-21T08:28:51.585660+0000","last_became_active":"2024-09-21T08:28:53.600149+0000","last_became_peered":"2024-09-21T08:28:53.600149+0000","last_unstale":"2024-09-21T08:31:00.602549+0000","last_undegraded":"2024-09-21T08:28:53.586423+0000","last_fullsized":"2024-09-21T08:28:53.586235+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995442+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":739,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.607748+0000","last_change":"2024-09-21T08:30:57.613094+0000","last_active":"2024-09-21T08:31:00.607748+0000","last_peered":"2024-09-21T08:31:00.607748+0000","last_clean":"2024-09-21T08:31:00.607748+0000","last_became_active":"2024-09-21T08:30:57.612659+0000","last_became_peered":"2024-09-21T08:30:57.612659+0000","last_unstale":"2024-09-21T08:31:00.607748+0000","last_undegraded":"2024-09-21T08:31:00.607748+0000","last_fullsized":"2024-09-21T08:31:00.607748+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.901202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.901202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816964+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":822,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.607081+0000","last_change":"2024-09-21T08:28:53.592593+0000","last_active":"2024-09-21T08:31:00.607081+0000","last_peered":"2024-09-21T08:31:00.607081+0000","last_clean":"2024-09-21T08:28:51.554029+0000","last_became_active":"2024-09-21T08:28:53.592593+0000","last_became_peered":"2024-09-21T08:28:53.592593+0000","last_unstale":"2024-09-21T08:31:00.607081+0000","last_undegraded":"2024-09-21T08:31:00.607081+0000","last_fullsized":"2024-09-21T08:28:53.577826+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:15.362045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:15.362045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.703217+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":317,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.600280+0000","last_change":"2024-09-21T08:28:53.586960+0000","last_active":"2024-09-21T08:31:00.600280+0000","last_peered":"2024-09-21T08:31:00.600280+0000","last_clean":"2024-09-21T08:28:51.584413+0000","last_became_active":"2024-09-21T08:28:53.586960+0000","last_became_peered":"2024-09-21T08:28:53.586960+0000","last_unstale":"2024-09-21T08:31:00.600280+0000","last_undegraded":"2024-09-21T08:31:00.600280+0000","last_fullsized":"2024-09-21T08:28:53.575166+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995433+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"451'116","reported_seq":1224,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:31:00.607051+0000","last_change":"2024-09-21T08:31:00.607007+0000","last_active":"2024-09-21T08:31:00.607051+0000","last_peered":"2024-09-21T08:31:00.607051+0000","last_clean":"2024-09-21T08:31:00.607051+0000","last_became_active":"2024-09-21T08:28:37.567400+0000","last_became_peered":"2024-09-21T08:28:37.567400+0000","last_unstale":"2024-09-21T08:31:00.607051+0000","last_undegraded":"2024-09-21T08:31:00.607051+0000","last_fullsized":"2024-09-21T08:31:00.607051+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'114","last_scrub_stamp":"2024-09-21T08:30:41.677928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:41.677928+0000","objects_scrubbed":2,"log_size":116,"log_dups_size":0,"ondisk_log_size":116,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:55.503706+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.033119902999999999,"stat_sum":{"num_bytes":5680964,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":14,"num_read_kb":4732,"num_write":20,"num_write_kb":3134,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1183872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.9","version":"73'12","reported_seq":30,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.601106+0000","last_change":"2024-09-21T08:30:57.948055+0000","last_active":"2024-09-21T08:31:00.601106+0000","last_peered":"2024-09-21T08:31:00.601106+0000","last_clean":"2024-09-21T08:31:00.601106+0000","last_became_active":"2024-09-21T08:30:57.624413+0000","last_became_peered":"2024-09-21T08:30:57.624413+0000","last_unstale":"2024-09-21T08:31:00.601106+0000","last_undegraded":"2024-09-21T08:31:00.601106+0000","last_fullsized":"2024-09-21T08:31:00.601106+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:41.463820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.463820+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602842+0000","last_change":"2024-09-21T08:30:57.621571+0000","last_active":"2024-09-21T08:31:00.602842+0000","last_peered":"2024-09-21T08:31:00.602842+0000","last_clean":"2024-09-21T08:31:00.602842+0000","last_became_active":"2024-09-21T08:30:57.619377+0000","last_became_peered":"2024-09-21T08:30:57.619377+0000","last_unstale":"2024-09-21T08:31:00.602842+0000","last_undegraded":"2024-09-21T08:31:00.602842+0000","last_fullsized":"2024-09-21T08:31:00.602842+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:51.769044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.769044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963624+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":835,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.601437+0000","last_change":"2024-09-21T08:29:59.267735+0000","last_active":"2024-09-21T08:31:00.601437+0000","last_peered":"2024-09-21T08:31:00.601437+0000","last_clean":"2024-09-21T08:31:00.601437+0000","last_became_active":"2024-09-21T08:28:37.566192+0000","last_became_peered":"2024-09-21T08:28:37.566192+0000","last_unstale":"2024-09-21T08:31:00.601437+0000","last_undegraded":"2024-09-21T08:31:00.601437+0000","last_fullsized":"2024-09-21T08:31:00.601437+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:59.267668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:29:59.267668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.614991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":731,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602161+0000","last_change":"2024-09-21T08:30:57.612738+0000","last_active":"2024-09-21T08:31:00.602161+0000","last_peered":"2024-09-21T08:31:00.602161+0000","last_clean":"2024-09-21T08:31:00.602161+0000","last_became_active":"2024-09-21T08:30:57.611217+0000","last_became_peered":"2024-09-21T08:30:57.611217+0000","last_unstale":"2024-09-21T08:31:00.602161+0000","last_undegraded":"2024-09-21T08:31:00.602161+0000","last_fullsized":"2024-09-21T08:31:00.602161+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087195+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"466'777","reported_seq":2830,"reported_epoch":467,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:31:00.603957+0000","last_change":"2024-09-21T08:31:00.603927+0000","last_active":"2024-09-21T08:31:00.603957+0000","last_peered":"2024-09-21T08:31:00.603957+0000","last_clean":"2024-09-21T08:31:00.603957+0000","last_became_active":"2024-09-21T08:28:07.549709+0000","last_became_peered":"2024-09-21T08:28:07.549709+0000","last_unstale":"2024-09-21T08:31:00.603957+0000","last_undegraded":"2024-09-21T08:31:00.603957+0000","last_fullsized":"2024-09-21T08:31:00.603957+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'577","last_scrub_stamp":"2024-09-21T08:29:32.265609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.265609+0000","objects_scrubbed":10,"log_size":777,"log_dups_size":0,"ondisk_log_size":777,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":1,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.19148454200000001,"stat_sum":{"num_bytes":41992706,"num_objects":19,"num_object_clones":15,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":1,"num_read":150,"num_read_kb":149973,"num_write":93,"num_write_kb":37638,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":39,"num_bytes_recovered":4385271,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"7"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.8","version":"73'15","reported_seq":977,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.604428+0000","last_change":"2024-09-21T08:30:57.696703+0000","last_active":"2024-09-21T08:31:00.604428+0000","last_peered":"2024-09-21T08:31:00.604428+0000","last_clean":"2024-09-21T08:31:00.604428+0000","last_became_active":"2024-09-21T08:30:57.622063+0000","last_became_peered":"2024-09-21T08:30:57.622063+0000","last_unstale":"2024-09-21T08:31:00.604428+0000","last_undegraded":"2024-09-21T08:31:00.604428+0000","last_fullsized":"2024-09-21T08:31:00.604428+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816793+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"66'2","reported_seq":24,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602906+0000","last_change":"2024-09-21T08:30:57.689264+0000","last_active":"2024-09-21T08:31:00.602906+0000","last_peered":"2024-09-21T08:31:00.602906+0000","last_clean":"2024-09-21T08:31:00.602906+0000","last_became_active":"2024-09-21T08:30:57.620311+0000","last_became_peered":"2024-09-21T08:30:57.620311+0000","last_unstale":"2024-09-21T08:31:00.602906+0000","last_undegraded":"2024-09-21T08:31:00.602906+0000","last_fullsized":"2024-09-21T08:31:00.602906+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'2","last_scrub_stamp":"2024-09-21T08:24:31.747410+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:31.747410+0000","objects_scrubbed":2,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963350+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4,5","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"464'18","reported_seq":25,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.600430+0000","last_change":"2024-09-21T08:30:57.632693+0000","last_active":"2024-09-21T08:31:00.600430+0000","last_peered":"2024-09-21T08:31:00.600430+0000","last_clean":"2024-09-21T08:31:00.600430+0000","last_became_active":"2024-09-21T08:30:57.613452+0000","last_became_peered":"2024-09-21T08:30:57.613452+0000","last_unstale":"2024-09-21T08:31:00.600430+0000","last_undegraded":"2024-09-21T08:31:00.600430+0000","last_fullsized":"2024-09-21T08:31:00.600430+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'10","last_scrub_stamp":"2024-09-21T08:25:42.708458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.708458+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963445+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":830,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.609510+0000","last_change":"2024-09-21T08:30:50.793864+0000","last_active":"2024-09-21T08:31:00.609510+0000","last_peered":"2024-09-21T08:31:00.609510+0000","last_clean":"2024-09-21T08:31:00.609510+0000","last_became_active":"2024-09-21T08:28:29.619471+0000","last_became_peered":"2024-09-21T08:28:29.619471+0000","last_unstale":"2024-09-21T08:31:00.609510+0000","last_undegraded":"2024-09-21T08:31:00.609510+0000","last_fullsized":"2024-09-21T08:31:00.609510+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:50.793789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:50.793789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:08.828948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"180'50","reported_seq":1607,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.607173+0000","last_change":"2024-09-21T08:28:53.592066+0000","last_active":"2024-09-21T08:31:00.607173+0000","last_peered":"2024-09-21T08:31:00.607173+0000","last_clean":"2024-09-21T08:28:51.555673+0000","last_became_active":"2024-09-21T08:28:53.592066+0000","last_became_peered":"2024-09-21T08:28:53.592066+0000","last_unstale":"2024-09-21T08:31:00.607173+0000","last_undegraded":"2024-09-21T08:31:00.607173+0000","last_fullsized":"2024-09-21T08:28:53.577992+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:26:50.014780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:50.014780+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":58,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020276460000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"3.10","version":"0'0","reported_seq":621,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.604304+0000","last_change":"2024-09-21T08:29:39.177460+0000","last_active":"2024-09-21T08:31:00.604304+0000","last_peered":"2024-09-21T08:31:00.604304+0000","last_clean":"2024-09-21T08:31:00.604304+0000","last_became_active":"2024-09-21T08:28:29.610987+0000","last_became_peered":"2024-09-21T08:28:29.610987+0000","last_unstale":"2024-09-21T08:31:00.604304+0000","last_undegraded":"2024-09-21T08:31:00.604304+0000","last_fullsized":"2024-09-21T08:31:00.604304+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.177382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":772,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.604224+0000","last_change":"2024-09-21T08:28:53.602197+0000","last_active":"2024-09-21T08:31:00.604224+0000","last_peered":"2024-09-21T08:31:00.604224+0000","last_clean":"2024-09-21T08:28:51.587601+0000","last_became_active":"2024-09-21T08:28:53.602197+0000","last_became_peered":"2024-09-21T08:28:53.602197+0000","last_unstale":"2024-09-21T08:31:00.604224+0000","last_undegraded":"2024-09-21T08:28:53.578880+0000","last_fullsized":"2024-09-21T08:28:53.578772+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-09-21T08:27:30.806522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.806522+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995560+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":853,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.607698+0000","last_change":"2024-09-21T08:28:53.600847+0000","last_active":"2024-09-21T08:31:00.607698+0000","last_peered":"2024-09-21T08:31:00.607698+0000","last_clean":"2024-09-21T08:28:51.557075+0000","last_became_active":"2024-09-21T08:28:53.600847+0000","last_became_peered":"2024-09-21T08:28:53.600847+0000","last_unstale":"2024-09-21T08:31:00.607698+0000","last_undegraded":"2024-09-21T08:31:00.607698+0000","last_fullsized":"2024-09-21T08:28:53.578503+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.795325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.601485+0000","last_change":"2024-09-21T08:30:57.616327+0000","last_active":"2024-09-21T08:31:00.601485+0000","last_peered":"2024-09-21T08:31:00.601485+0000","last_clean":"2024-09-21T08:31:00.601485+0000","last_became_active":"2024-09-21T08:30:57.615385+0000","last_became_peered":"2024-09-21T08:30:57.615385+0000","last_unstale":"2024-09-21T08:31:00.601485+0000","last_undegraded":"2024-09-21T08:31:00.601485+0000","last_fullsized":"2024-09-21T08:31:00.601485+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:10.851510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.851510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963700+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.15","version":"462'448","reported_seq":2265,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.604182+0000","last_change":"2024-09-21T08:28:53.603910+0000","last_active":"2024-09-21T08:31:00.604182+0000","last_peered":"2024-09-21T08:31:00.604182+0000","last_clean":"2024-09-21T08:28:51.587746+0000","last_became_active":"2024-09-21T08:28:53.603910+0000","last_became_peered":"2024-09-21T08:28:53.603910+0000","last_unstale":"2024-09-21T08:31:00.604182+0000","last_undegraded":"2024-09-21T08:28:53.582931+0000","last_fullsized":"2024-09-21T08:28:53.582809+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'262","last_scrub_stamp":"2024-09-21T08:26:29.875830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.875830+0000","objects_scrubbed":3,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":64,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.074129519000000005,"stat_sum":{"num_bytes":42555630,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":170,"num_read_kb":185534,"num_write":90,"num_write_kb":42335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":8623856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.16","version":"73'9","reported_seq":325,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.606029+0000","last_change":"2024-09-21T08:28:53.589261+0000","last_active":"2024-09-21T08:31:00.606029+0000","last_peered":"2024-09-21T08:31:00.606029+0000","last_clean":"2024-09-21T08:28:51.555305+0000","last_became_active":"2024-09-21T08:28:53.589261+0000","last_became_peered":"2024-09-21T08:28:53.589261+0000","last_unstale":"2024-09-21T08:31:00.606029+0000","last_undegraded":"2024-09-21T08:28:53.576380+0000","last_fullsized":"2024-09-21T08:28:53.576277+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:10.183035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.183035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.600547+0000","last_change":"2024-09-21T08:30:57.621029+0000","last_active":"2024-09-21T08:31:00.600547+0000","last_peered":"2024-09-21T08:31:00.600547+0000","last_clean":"2024-09-21T08:31:00.600547+0000","last_became_active":"2024-09-21T08:30:57.619257+0000","last_became_peered":"2024-09-21T08:30:57.619257+0000","last_unstale":"2024-09-21T08:31:00.600547+0000","last_undegraded":"2024-09-21T08:31:00.600547+0000","last_fullsized":"2024-09-21T08:31:00.600547+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:46.719545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:46.719545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963460+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":373,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.605150+0000","last_change":"2024-09-21T08:28:53.602669+0000","last_active":"2024-09-21T08:31:00.605150+0000","last_peered":"2024-09-21T08:31:00.605150+0000","last_clean":"2024-09-21T08:28:51.587829+0000","last_became_active":"2024-09-21T08:28:53.602669+0000","last_became_peered":"2024-09-21T08:28:53.602669+0000","last_unstale":"2024-09-21T08:31:00.605150+0000","last_undegraded":"2024-09-21T08:31:00.605150+0000","last_fullsized":"2024-09-21T08:28:53.580175+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:05.814753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.814753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995449+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":837,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.607314+0000","last_change":"2024-09-21T08:30:57.613466+0000","last_active":"2024-09-21T08:31:00.607314+0000","last_peered":"2024-09-21T08:31:00.607314+0000","last_clean":"2024-09-21T08:31:00.607314+0000","last_became_active":"2024-09-21T08:30:57.613069+0000","last_became_peered":"2024-09-21T08:30:57.613069+0000","last_unstale":"2024-09-21T08:31:00.607314+0000","last_undegraded":"2024-09-21T08:31:00.607314+0000","last_fullsized":"2024-09-21T08:31:00.607314+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:10.076564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.076564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.703301+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4,5],"avail_no_missing":["2","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"467'188","reported_seq":2105,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.610602+0000","last_change":"2024-09-21T08:31:00.610866+0000","last_active":"2024-09-21T08:31:00.610602+0000","last_peered":"2024-09-21T08:31:00.610602+0000","last_clean":"2024-09-21T08:31:00.610602+0000","last_became_active":"2024-09-21T08:30:59.600861+0000","last_became_peered":"2024-09-21T08:30:59.600861+0000","last_unstale":"2024-09-21T08:31:00.610602+0000","last_undegraded":"2024-09-21T08:31:00.610602+0000","last_fullsized":"2024-09-21T08:31:00.610602+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'102","last_scrub_stamp":"2024-09-21T08:26:33.255256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.255256+0000","objects_scrubbed":1,"log_size":188,"log_dups_size":0,"ondisk_log_size":188,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0082809670000000002,"stat_sum":{"num_bytes":9899595,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":98,"num_read_kb":141433,"num_write":40,"num_write_kb":9384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":16651842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[{"shards":"2,4,5","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6d","length":"17"},{"start":"85","length":"d"},{"start":"93","length":"6"},{"start":"9a","length":"1"},{"start":"9c","length":"8"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"1"},{"start":"b6","length":"1"},{"start":"ba","length":"2"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.15","version":"73'9","reported_seq":917,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.606970+0000","last_change":"2024-09-21T08:28:53.601176+0000","last_active":"2024-09-21T08:31:00.606970+0000","last_peered":"2024-09-21T08:31:00.606970+0000","last_clean":"2024-09-21T08:28:51.554870+0000","last_became_active":"2024-09-21T08:28:53.601176+0000","last_became_peered":"2024-09-21T08:28:53.601176+0000","last_unstale":"2024-09-21T08:31:00.606970+0000","last_undegraded":"2024-09-21T08:28:53.577554+0000","last_fullsized":"2024-09-21T08:28:53.576455+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:28:14.535334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.535334+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":800,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.601060+0000","last_change":"2024-09-21T08:28:53.605117+0000","last_active":"2024-09-21T08:31:00.601060+0000","last_peered":"2024-09-21T08:31:00.601060+0000","last_clean":"2024-09-21T08:28:51.584984+0000","last_became_active":"2024-09-21T08:28:53.605117+0000","last_became_peered":"2024-09-21T08:28:53.605117+0000","last_unstale":"2024-09-21T08:31:00.601060+0000","last_undegraded":"2024-09-21T08:31:00.601060+0000","last_fullsized":"2024-09-21T08:28:53.584510+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.361096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.361096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"340'16","reported_seq":827,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.603575+0000","last_change":"2024-09-21T08:28:53.600719+0000","last_active":"2024-09-21T08:31:00.603575+0000","last_peered":"2024-09-21T08:31:00.603575+0000","last_clean":"2024-09-21T08:28:51.585913+0000","last_became_active":"2024-09-21T08:28:53.600719+0000","last_became_peered":"2024-09-21T08:28:53.600719+0000","last_unstale":"2024-09-21T08:31:00.603575+0000","last_undegraded":"2024-09-21T08:28:53.576377+0000","last_fullsized":"2024-09-21T08:28:53.576220+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995402+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":19,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602708+0000","last_change":"2024-09-21T08:30:56.611027+0000","last_active":"2024-09-21T08:31:00.602708+0000","last_peered":"2024-09-21T08:31:00.602708+0000","last_clean":"2024-09-21T08:31:00.602708+0000","last_became_active":"2024-09-21T08:30:56.610628+0000","last_became_peered":"2024-09-21T08:30:56.610628+0000","last_unstale":"2024-09-21T08:31:00.602708+0000","last_undegraded":"2024-09-21T08:31:00.602708+0000","last_fullsized":"2024-09-21T08:31:00.602708+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.364388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.364388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.964052+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"179'63","reported_seq":1161,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.606998+0000","last_change":"2024-09-21T08:28:53.590100+0000","last_active":"2024-09-21T08:31:00.606998+0000","last_peered":"2024-09-21T08:31:00.606998+0000","last_clean":"2024-09-21T08:28:51.556223+0000","last_became_active":"2024-09-21T08:28:53.590100+0000","last_became_peered":"2024-09-21T08:28:53.590100+0000","last_unstale":"2024-09-21T08:31:00.606998+0000","last_undegraded":"2024-09-21T08:31:00.606998+0000","last_fullsized":"2024-09-21T08:28:53.579666+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'63","last_scrub_stamp":"2024-09-21T08:28:13.511512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.511512+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":62,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010228017000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.14","version":"104'15","reported_seq":379,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.602860+0000","last_change":"2024-09-21T08:28:53.602117+0000","last_active":"2024-09-21T08:31:00.602860+0000","last_peered":"2024-09-21T08:31:00.602860+0000","last_clean":"2024-09-21T08:28:51.587113+0000","last_became_active":"2024-09-21T08:28:53.602117+0000","last_became_peered":"2024-09-21T08:28:53.602117+0000","last_unstale":"2024-09-21T08:31:00.602860+0000","last_undegraded":"2024-09-21T08:28:53.578695+0000","last_fullsized":"2024-09-21T08:28:53.578596+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995426+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":22,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.601276+0000","last_change":"2024-09-21T08:30:57.691227+0000","last_active":"2024-09-21T08:31:00.601276+0000","last_peered":"2024-09-21T08:31:00.601276+0000","last_clean":"2024-09-21T08:31:00.601276+0000","last_became_active":"2024-09-21T08:30:57.616000+0000","last_became_peered":"2024-09-21T08:30:57.616000+0000","last_unstale":"2024-09-21T08:31:00.601276+0000","last_undegraded":"2024-09-21T08:31:00.601276+0000","last_fullsized":"2024-09-21T08:31:00.601276+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:28:16.875544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.875544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963970+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"463'15","reported_seq":977,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.608113+0000","last_change":"2024-09-21T08:30:56.635835+0000","last_active":"2024-09-21T08:31:00.608113+0000","last_peered":"2024-09-21T08:31:00.608113+0000","last_clean":"2024-09-21T08:31:00.608113+0000","last_became_active":"2024-09-21T08:30:56.626363+0000","last_became_peered":"2024-09-21T08:30:56.626363+0000","last_unstale":"2024-09-21T08:31:00.608113+0000","last_undegraded":"2024-09-21T08:31:00.608113+0000","last_fullsized":"2024-09-21T08:31:00.608113+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:03.816810+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602343+0000","last_change":"2024-09-21T08:30:57.620328+0000","last_active":"2024-09-21T08:31:00.602343+0000","last_peered":"2024-09-21T08:31:00.602343+0000","last_clean":"2024-09-21T08:31:00.602343+0000","last_became_active":"2024-09-21T08:30:57.617666+0000","last_became_peered":"2024-09-21T08:30:57.617666+0000","last_unstale":"2024-09-21T08:31:00.602343+0000","last_undegraded":"2024-09-21T08:31:00.602343+0000","last_fullsized":"2024-09-21T08:31:00.602343+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:48.251366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:48.251366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963577+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"430'161","reported_seq":983,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.602074+0000","last_change":"2024-09-21T08:28:53.604012+0000","last_active":"2024-09-21T08:31:00.602074+0000","last_peered":"2024-09-21T08:31:00.602074+0000","last_clean":"2024-09-21T08:28:53.029949+0000","last_became_active":"2024-09-21T08:28:53.604012+0000","last_became_peered":"2024-09-21T08:28:53.604012+0000","last_unstale":"2024-09-21T08:31:00.602074+0000","last_undegraded":"2024-09-21T08:28:53.580392+0000","last_fullsized":"2024-09-21T08:28:53.580253+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'94","last_scrub_stamp":"2024-09-21T08:26:27.208234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.208234+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":64,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.075698227000000007,"stat_sum":{"num_bytes":12337203,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":22,"num_read_kb":13076,"num_write":38,"num_write_kb":17037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10770304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"}]},{"pgid":"4.13","version":"73'11","reported_seq":31,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.600980+0000","last_change":"2024-09-21T08:30:56.661267+0000","last_active":"2024-09-21T08:31:00.600980+0000","last_peered":"2024-09-21T08:31:00.600980+0000","last_clean":"2024-09-21T08:31:00.600980+0000","last_became_active":"2024-09-21T08:30:56.626643+0000","last_became_peered":"2024-09-21T08:30:56.626643+0000","last_unstale":"2024-09-21T08:31:00.600980+0000","last_undegraded":"2024-09-21T08:31:00.600980+0000","last_fullsized":"2024-09-21T08:31:00.600980+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:30:25.153729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:25.153729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:35.550050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602040+0000","last_change":"2024-09-21T08:30:57.617620+0000","last_active":"2024-09-21T08:31:00.602040+0000","last_peered":"2024-09-21T08:31:00.602040+0000","last_clean":"2024-09-21T08:31:00.602040+0000","last_became_active":"2024-09-21T08:30:57.616182+0000","last_became_peered":"2024-09-21T08:30:57.616182+0000","last_unstale":"2024-09-21T08:31:00.602040+0000","last_undegraded":"2024-09-21T08:31:00.602040+0000","last_fullsized":"2024-09-21T08:31:00.602040+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:06.889127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:06.889127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963671+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":367,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.603132+0000","last_change":"2024-09-21T08:28:53.600540+0000","last_active":"2024-09-21T08:31:00.603132+0000","last_peered":"2024-09-21T08:31:00.603132+0000","last_clean":"2024-09-21T08:28:51.585918+0000","last_became_active":"2024-09-21T08:28:53.600540+0000","last_became_peered":"2024-09-21T08:28:53.600540+0000","last_unstale":"2024-09-21T08:31:00.603132+0000","last_undegraded":"2024-09-21T08:31:00.603132+0000","last_fullsized":"2024-09-21T08:28:53.586917+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":577,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.606339+0000","last_change":"2024-09-21T08:30:56.618534+0000","last_active":"2024-09-21T08:31:00.606339+0000","last_peered":"2024-09-21T08:31:00.606339+0000","last_clean":"2024-09-21T08:31:00.606339+0000","last_became_active":"2024-09-21T08:30:56.618134+0000","last_became_peered":"2024-09-21T08:30:56.618134+0000","last_unstale":"2024-09-21T08:31:00.606339+0000","last_undegraded":"2024-09-21T08:31:00.606339+0000","last_fullsized":"2024-09-21T08:31:00.606339+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:23.816669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:23.816669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:31.339821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"455'203","reported_seq":1736,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.601463+0000","last_change":"2024-09-21T08:28:53.605541+0000","last_active":"2024-09-21T08:31:00.601463+0000","last_peered":"2024-09-21T08:31:00.601463+0000","last_clean":"2024-09-21T08:28:52.665793+0000","last_became_active":"2024-09-21T08:28:53.605541+0000","last_became_peered":"2024-09-21T08:28:53.605541+0000","last_unstale":"2024-09-21T08:31:00.601463+0000","last_undegraded":"2024-09-21T08:28:53.581187+0000","last_fullsized":"2024-09-21T08:28:53.581055+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'44","last_scrub_stamp":"2024-09-21T08:26:27.882963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.882963+0000","objects_scrubbed":1,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0806039080000001,"stat_sum":{"num_bytes":44372956,"num_objects":23,"num_object_clones":21,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":77,"num_read_kb":68079,"num_write":99,"num_write_kb":40253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":11110430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6d","length":"4"},{"start":"73","length":"1"}]},{"pgid":"4.12","version":"73'9","reported_seq":398,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602779+0000","last_change":"2024-09-21T08:30:42.016780+0000","last_active":"2024-09-21T08:31:00.602779+0000","last_peered":"2024-09-21T08:31:00.602779+0000","last_clean":"2024-09-21T08:31:00.602779+0000","last_became_active":"2024-09-21T08:28:29.636404+0000","last_became_peered":"2024-09-21T08:28:29.636404+0000","last_unstale":"2024-09-21T08:31:00.602779+0000","last_undegraded":"2024-09-21T08:31:00.602779+0000","last_fullsized":"2024-09-21T08:31:00.602779+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:42.016697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:42.016697+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:01.081137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602600+0000","last_change":"2024-09-21T08:30:57.617367+0000","last_active":"2024-09-21T08:31:00.602600+0000","last_peered":"2024-09-21T08:31:00.602600+0000","last_clean":"2024-09-21T08:31:00.602600+0000","last_became_active":"2024-09-21T08:30:57.615854+0000","last_became_peered":"2024-09-21T08:30:57.615854+0000","last_unstale":"2024-09-21T08:31:00.602600+0000","last_undegraded":"2024-09-21T08:31:00.602600+0000","last_fullsized":"2024-09-21T08:31:00.602600+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:49.790900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:49.790900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":685,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.601836+0000","last_change":"2024-09-21T08:30:56.626544+0000","last_active":"2024-09-21T08:31:00.601836+0000","last_peered":"2024-09-21T08:31:00.601836+0000","last_clean":"2024-09-21T08:31:00.601836+0000","last_became_active":"2024-09-21T08:30:56.626197+0000","last_became_peered":"2024-09-21T08:30:56.626197+0000","last_unstale":"2024-09-21T08:31:00.601836+0000","last_undegraded":"2024-09-21T08:31:00.601836+0000","last_fullsized":"2024-09-21T08:31:00.601836+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:13.940546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:13.940546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.087369+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":791,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.604635+0000","last_change":"2024-09-21T08:30:56.627439+0000","last_active":"2024-09-21T08:31:00.604635+0000","last_peered":"2024-09-21T08:31:00.604635+0000","last_clean":"2024-09-21T08:31:00.604635+0000","last_became_active":"2024-09-21T08:30:56.627130+0000","last_became_peered":"2024-09-21T08:30:56.627130+0000","last_unstale":"2024-09-21T08:31:00.604635+0000","last_undegraded":"2024-09-21T08:31:00.604635+0000","last_fullsized":"2024-09-21T08:31:00.604635+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:02.065082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:02.065082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:25.458006+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.12","version":"462'411","reported_seq":2217,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.615238+0000","last_change":"2024-09-21T08:28:53.587108+0000","last_active":"2024-09-21T08:31:00.615238+0000","last_peered":"2024-09-21T08:31:00.615238+0000","last_clean":"2024-09-21T08:28:52.589762+0000","last_became_active":"2024-09-21T08:28:53.587108+0000","last_became_peered":"2024-09-21T08:28:53.587108+0000","last_unstale":"2024-09-21T08:31:00.615238+0000","last_undegraded":"2024-09-21T08:28:53.574106+0000","last_fullsized":"2024-09-21T08:28:53.573924+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'200","last_scrub_stamp":"2024-09-21T08:26:27.417028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.417028+0000","objects_scrubbed":3,"log_size":411,"log_dups_size":0,"ondisk_log_size":411,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":62,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.027231871000000001,"stat_sum":{"num_bytes":45513352,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":22,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":2,"num_read":157,"num_read_kb":186641,"num_write":102,"num_write_kb":49221,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":8981699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":22}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.11","version":"73'11","reported_seq":408,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.606355+0000","last_change":"2024-09-21T08:28:53.589901+0000","last_active":"2024-09-21T08:31:00.606355+0000","last_peered":"2024-09-21T08:31:00.606355+0000","last_clean":"2024-09-21T08:28:51.555481+0000","last_became_active":"2024-09-21T08:28:53.589901+0000","last_became_peered":"2024-09-21T08:28:53.589901+0000","last_unstale":"2024-09-21T08:31:00.606355+0000","last_undegraded":"2024-09-21T08:28:53.577098+0000","last_fullsized":"2024-09-21T08:28:53.576477+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":849,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.607513+0000","last_change":"2024-09-21T08:28:53.600628+0000","last_active":"2024-09-21T08:31:00.607513+0000","last_peered":"2024-09-21T08:31:00.607513+0000","last_clean":"2024-09-21T08:28:51.555529+0000","last_became_active":"2024-09-21T08:28:53.600628+0000","last_became_peered":"2024-09-21T08:28:53.600628+0000","last_unstale":"2024-09-21T08:31:00.607513+0000","last_undegraded":"2024-09-21T08:31:00.607513+0000","last_fullsized":"2024-09-21T08:28:53.577666+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.600030+0000","last_change":"2024-09-21T08:30:57.618001+0000","last_active":"2024-09-21T08:31:00.600030+0000","last_peered":"2024-09-21T08:31:00.600030+0000","last_clean":"2024-09-21T08:31:00.600030+0000","last_became_active":"2024-09-21T08:30:57.617550+0000","last_became_peered":"2024-09-21T08:30:57.617550+0000","last_unstale":"2024-09-21T08:31:00.600030+0000","last_undegraded":"2024-09-21T08:31:00.600030+0000","last_fullsized":"2024-09-21T08:31:00.600030+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.964100+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":19,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.600628+0000","last_change":"2024-09-21T08:30:56.608178+0000","last_active":"2024-09-21T08:31:00.600628+0000","last_peered":"2024-09-21T08:31:00.600628+0000","last_clean":"2024-09-21T08:31:00.600628+0000","last_became_active":"2024-09-21T08:30:56.607833+0000","last_became_peered":"2024-09-21T08:30:56.607833+0000","last_unstale":"2024-09-21T08:31:00.600628+0000","last_undegraded":"2024-09-21T08:31:00.600628+0000","last_fullsized":"2024-09-21T08:31:00.600628+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:58.167184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.167184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.964147+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.13","version":"455'68","reported_seq":1713,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.605653+0000","last_change":"2024-09-21T08:28:53.588606+0000","last_active":"2024-09-21T08:31:00.605653+0000","last_peered":"2024-09-21T08:31:00.605653+0000","last_clean":"2024-09-21T08:28:52.586222+0000","last_became_active":"2024-09-21T08:28:53.588606+0000","last_became_peered":"2024-09-21T08:28:53.588606+0000","last_unstale":"2024-09-21T08:31:00.605653+0000","last_undegraded":"2024-09-21T08:28:53.574192+0000","last_fullsized":"2024-09-21T08:28:53.574061+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'22","last_scrub_stamp":"2024-09-21T08:26:29.353795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.353795+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":62,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019195647,"stat_sum":{"num_bytes":14428994,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":56,"num_read_kb":78729,"num_write":31,"num_write_kb":16265,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5630444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":6}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"4.10","version":"73'4","reported_seq":390,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.601621+0000","last_change":"2024-09-21T08:30:56.659256+0000","last_active":"2024-09-21T08:31:00.601621+0000","last_peered":"2024-09-21T08:31:00.601621+0000","last_clean":"2024-09-21T08:31:00.601621+0000","last_became_active":"2024-09-21T08:30:56.638177+0000","last_became_peered":"2024-09-21T08:30:56.638177+0000","last_unstale":"2024-09-21T08:31:00.601621+0000","last_undegraded":"2024-09-21T08:31:00.601621+0000","last_fullsized":"2024-09-21T08:31:00.601621+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-09-21T08:29:39.416397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.416397+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:05.614359+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":817,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.607067+0000","last_change":"2024-09-21T08:30:24.826266+0000","last_active":"2024-09-21T08:31:00.607067+0000","last_peered":"2024-09-21T08:31:00.607067+0000","last_clean":"2024-09-21T08:31:00.607067+0000","last_became_active":"2024-09-21T08:28:29.611592+0000","last_became_peered":"2024-09-21T08:28:29.611592+0000","last_unstale":"2024-09-21T08:31:00.607067+0000","last_undegraded":"2024-09-21T08:31:00.607067+0000","last_fullsized":"2024-09-21T08:31:00.607067+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:24.826198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:30:24.826198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:28.562894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.600137+0000","last_change":"2024-09-21T08:30:57.621325+0000","last_active":"2024-09-21T08:31:00.600137+0000","last_peered":"2024-09-21T08:31:00.600137+0000","last_clean":"2024-09-21T08:31:00.600137+0000","last_became_active":"2024-09-21T08:30:57.618219+0000","last_became_peered":"2024-09-21T08:30:57.618219+0000","last_unstale":"2024-09-21T08:31:00.600137+0000","last_undegraded":"2024-09-21T08:31:00.600137+0000","last_fullsized":"2024-09-21T08:31:00.600137+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":18,"reported_epoch":467,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:00.602858+0000","last_change":"2024-09-21T08:30:57.611382+0000","last_active":"2024-09-21T08:31:00.602858+0000","last_peered":"2024-09-21T08:31:00.602858+0000","last_clean":"2024-09-21T08:31:00.602858+0000","last_became_active":"2024-09-21T08:30:57.610902+0000","last_became_peered":"2024-09-21T08:30:57.610902+0000","last_unstale":"2024-09-21T08:31:00.602858+0000","last_undegraded":"2024-09-21T08:31:00.602858+0000","last_fullsized":"2024-09-21T08:31:00.602858+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.736816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.736816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:04.963609+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"461'222","reported_seq":1957,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.606418+0000","last_change":"2024-09-21T08:28:53.600314+0000","last_active":"2024-09-21T08:31:00.606418+0000","last_peered":"2024-09-21T08:31:00.606418+0000","last_clean":"2024-09-21T08:28:51.556402+0000","last_became_active":"2024-09-21T08:28:53.600314+0000","last_became_peered":"2024-09-21T08:28:53.600314+0000","last_unstale":"2024-09-21T08:31:00.606418+0000","last_undegraded":"2024-09-21T08:28:53.575998+0000","last_fullsized":"2024-09-21T08:28:53.575875+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'167","last_scrub_stamp":"2024-09-21T08:27:49.143452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.143452+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":62,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.015256377999999999,"stat_sum":{"num_bytes":19969264,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":97092,"num_write":35,"num_write_kb":17179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1129140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":10}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6d","length":"4"},{"start":"74","length":"2"}]},{"pgid":"6.1d","version":"0'0","reported_seq":378,"reported_epoch":467,"state":"active+clean","last_fresh":"2024-09-21T08:31:00.602752+0000","last_change":"2024-09-21T08:30:56.627685+0000","last_active":"2024-09-21T08:31:00.602752+0000","last_peered":"2024-09-21T08:31:00.602752+0000","last_clean":"2024-09-21T08:31:00.602752+0000","last_became_active":"2024-09-21T08:30:56.627251+0000","last_became_peered":"2024-09-21T08:30:56.627251+0000","last_unstale":"2024-09-21T08:31:00.602752+0000","last_undegraded":"2024-09-21T08:31:00.602752+0000","last_fullsized":"2024-09-21T08:31:00.602752+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:28.069450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:28.069450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:51.381406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":688,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.600547+0000","last_change":"2024-09-21T08:28:53.585838+0000","last_active":"2024-09-21T08:31:00.600547+0000","last_peered":"2024-09-21T08:31:00.600547+0000","last_clean":"2024-09-21T08:28:51.584499+0000","last_became_active":"2024-09-21T08:28:53.585838+0000","last_became_peered":"2024-09-21T08:28:53.585838+0000","last_unstale":"2024-09-21T08:31:00.600547+0000","last_undegraded":"2024-09-21T08:31:00.600547+0000","last_fullsized":"2024-09-21T08:28:53.575428+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995410+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":883,"reported_epoch":467,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:00.607665+0000","last_change":"2024-09-21T08:28:53.603028+0000","last_active":"2024-09-21T08:31:00.607665+0000","last_peered":"2024-09-21T08:31:00.607665+0000","last_clean":"2024-09-21T08:28:51.556942+0000","last_became_active":"2024-09-21T08:28:53.603028+0000","last_became_peered":"2024-09-21T08:28:53.603028+0000","last_unstale":"2024-09-21T08:31:00.607665+0000","last_undegraded":"2024-09-21T08:28:53.581121+0000","last_fullsized":"2024-09-21T08:28:53.581005+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":266,"reported_epoch":467,"state":"active+undersized","last_fresh":"2024-09-21T08:31:00.603667+0000","last_change":"2024-09-21T08:28:53.601908+0000","last_active":"2024-09-21T08:31:00.603667+0000","last_peered":"2024-09-21T08:31:00.603667+0000","last_clean":"2024-09-21T08:23:33.544062+0000","last_became_active":"2024-09-21T08:28:53.601908+0000","last_became_peered":"2024-09-21T08:28:53.601908+0000","last_unstale":"2024-09-21T08:31:00.603667+0000","last_undegraded":"2024-09-21T08:31:00.603667+0000","last_fullsized":"2024-09-21T08:28:53.577043+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.872291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.872291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:00.995652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":36,"stat_sum":{"num_bytes":581642977,"num_objects":286,"num_object_clones":236,"num_object_copies":858,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":153,"num_objects_misplaced":54,"num_objects_unfound":0,"num_objects_dirty":286,"num_whiteouts":18,"num_read":1909,"num_read_kb":2092123,"num_write":1364,"num_write_kb":548897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":803,"num_bytes_recovered":402917430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1422598144,"data_stored":2022973324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7866,"ondisk_log_size":7866,"up":82,"acting":92,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":79,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":74,"acting":84,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":676,"num_read_kb":433,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":12388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626688,"data_stored":16766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":77,"acting":83,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":5459,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":8622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":7,"ondisk_log_size":7,"up":76,"acting":91,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":427,"num_read_kb":422,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":212,"num_read_kb":306,"num_write":179,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":4561934,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1419,"ondisk_log_size":1419,"up":28,"acting":32,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070533,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":228312,"kb_used_data":174916,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93516840,"statfs":{"total":95995035648,"available":95761244160,"internally_reserved":0,"allocated":179113984,"data_stored":227784827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397605,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659240,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":322476,"kb_used_data":260760,"kb_used_omap":1,"kb_used_meta":61694,"kb_avail":93422676,"statfs":{"total":95995035648,"available":95664820224,"internally_reserved":0,"allocated":267018240,"data_stored":395111134,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":63175109},"hb_peers":[1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986286,"num_pgs":100,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":205708,"kb_used_data":170360,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93539444,"statfs":{"total":95995035648,"available":95784390656,"internally_reserved":0,"allocated":174448640,"data_stored":227142026,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182514,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411448,"num_pgs":116,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":397884,"kb_used_data":352800,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93347268,"statfs":{"total":95995035648,"available":95587602432,"internally_reserved":0,"allocated":361267200,"data_stored":555790803,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574972,"num_pgs":136,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":331372,"kb_used_data":277976,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93413780,"statfs":{"total":95995035648,"available":95655710720,"internally_reserved":0,"allocated":284647424,"data_stored":428436593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74776576,"data_stored":86348592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":281063424,"data_stored":426243647,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":357806080,"data_stored":553655131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70848512,"data_stored":84468350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":170676224,"data_stored":224688491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":263196672,"data_stored":392656388,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":176136192,"data_stored":227042310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:31:03.691 INFO:tasks.cephadm.ceph_manager.ceph:increase pool size by 10 2024-09-21T08:31:03.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pool set unique_pool_0 pg_num 46 2024-09-21T08:31:03.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:03.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:03.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:03.771 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:03.857 INFO:tasks.rados.rados.0.smithi137.stdout:1871: expect (ObjNum 497 snap 137 seq_num 497) 2024-09-21T08:31:03.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:03.895 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:03.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:03.896 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:03.908 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:03 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:03.908 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:03 smithi137 ceph-mon[41044]: pgmap v608: 178 pgs: 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 94 MiB/s rd, 18 MiB/s wr, 174 op/s; 237/1455 objects degraded (16.289%); 100/1455 objects misplaced (6.873%); 57 MiB/s, 132 objects/s recovering 2024-09-21T08:31:03.908 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:03 smithi137 ceph-mon[41044]: from='client.45827 v1:172.21.15.137:0/3095770183' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:31:03.909 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:03 smithi137 ceph-mon[41044]: osdmap e470: 8 total, 6 up, 5 in 2024-09-21T08:31:03.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:03 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:03.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:03 smithi137 ceph-mon[29634]: pgmap v608: 178 pgs: 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 94 MiB/s rd, 18 MiB/s wr, 174 op/s; 237/1455 objects degraded (16.289%); 100/1455 objects misplaced (6.873%); 57 MiB/s, 132 objects/s recovering 2024-09-21T08:31:03.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:03 smithi137 ceph-mon[29634]: from='client.45827 v1:172.21.15.137:0/3095770183' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:31:03.909 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:03 smithi137 ceph-mon[29634]: osdmap e470: 8 total, 6 up, 5 in 2024-09-21T08:31:03.974 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:03 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:03.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:03 smithi165 ceph-mon[32893]: pgmap v608: 178 pgs: 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 94 MiB/s rd, 18 MiB/s wr, 174 op/s; 237/1455 objects degraded (16.289%); 100/1455 objects misplaced (6.873%); 57 MiB/s, 132 objects/s recovering 2024-09-21T08:31:03.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:03 smithi165 ceph-mon[32893]: from='client.45827 v1:172.21.15.137:0/3095770183' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:31:03.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:03 smithi165 ceph-mon[32893]: osdmap e470: 8 total, 6 up, 5 in 2024-09-21T08:31:03.996 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:04.001 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T08:31:04.015 INFO:tasks.rados.rados.0.smithi137.stdout:1874: expect (ObjNum 640 snap 191 seq_num 640) 2024-09-21T08:31:04.018 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:04.019 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:04.019 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:04.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:04.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:04.142 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:04.142 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:04.172 INFO:tasks.rados.rados.0.smithi137.stdout:1873: expect (ObjNum 620 snap 182 seq_num 620) 2024-09-21T08:31:04.243 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:04.245 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:1875: finishing write tid 1 to smithi13779221-39 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:1875: finishing write tid 2 to smithi13779221-39 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:1875: finishing write tid 3 to smithi13779221-39 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:1875: finishing write tid 4 to smithi13779221-39 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:1875: oid 39 updating version 0 to 240 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:1875: oid 39 version 240 is already newer than 239 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:1875: oid 39 updating version 240 to 241 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 241 (ObjNum 666 snap 200 seq_num 666) dirty exists 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:1875: left oid 39 (ObjNum 666 snap 200 seq_num 666) 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:1869: done (5 left) 2024-09-21T08:31:04.253 INFO:tasks.rados.rados.0.smithi137.stdout:1870: done (4 left) 2024-09-21T08:31:04.254 INFO:tasks.rados.rados.0.smithi137.stdout:1871: done (3 left) 2024-09-21T08:31:04.254 INFO:tasks.rados.rados.0.smithi137.stdout:1873: done (2 left) 2024-09-21T08:31:04.254 INFO:tasks.rados.rados.0.smithi137.stdout:1874: done (1 left) 2024-09-21T08:31:04.254 INFO:tasks.rados.rados.0.smithi137.stdout:1875: done (0 left) 2024-09-21T08:31:04.254 INFO:tasks.rados.rados.0.smithi137.stdout:1876: snap_remove snap 151 2024-09-21T08:31:04.264 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:04.264 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:04.265 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:04.314 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:04.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:04.386 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:04.387 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:04.387 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:04.488 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:04.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:04.510 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:04.510 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:04.611 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:04.633 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:04.633 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:04.634 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:04.668 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:04.668 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:31:04.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:31:04.668 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:31:04.668 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.668 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44417.0:46 5.1f e70/69 5:f95f44c2:::notify.0:head v 70'1, mlcod=0'0)", 2024-09-21T08:31:04.668 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.569323+0000", 2024-09-21T08:31:04.668 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 450.09580328999999, 2024-09-21T08:31:04.668 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021424550000000001, 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569323+0000", 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569323+0000", 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.669 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569324+0000", 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.009e-06 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569331+0000", 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.528e-06 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.670 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569343+0000", 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2483999999999999e-05 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569351+0000", 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5079999999999999e-06 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569398+0000", 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6708000000000001e-05 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.671 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569414+0000", 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6829000000000001e-05 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571435+0000", 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020206299999999998 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571466+0000", 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0759000000000001e-05 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.672 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.44387.0:46 5.1f e70/69 5:f95f44c2:::notify.0:head v 70'2, mlcod=0'0)", 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.569550+0000", 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 450.09557624299998, 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019394410000000001, 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569550+0000", 2024-09-21T08:31:04.673 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569550+0000", 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569551+0000", 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2500000000000005e-07 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.674 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569557+0000", 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.412e-06 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569564+0000", 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1420000000000004e-06 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569570+0000", 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.6250000000000001e-06 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.675 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.570055+0000", 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048423599999999998 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.570072+0000", 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7007000000000002e-05 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571468+0000", 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013962219999999999 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.676 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.571490+0000", 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2072000000000001e-05 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:52 5.e 5:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.576342+0000", 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 450.08878474800002, 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011620549999999999, 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.677 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 52 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576342+0000", 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.678 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576342+0000", 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576344+0000", 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3659999999999999e-06 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576351+0000", 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7129999999999999e-06 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.679 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576362+0000", 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0740999999999999e-05 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576367+0000", 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5670000000000002e-06 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576412+0000", 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5079e-05 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.680 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576503+0000", 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0703000000000002e-05 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576585+0000", 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1724999999999994e-05 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577152+0000", 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000567054 2024-09-21T08:31:04.681 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.682 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.682 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.682 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577187+0000", 2024-09-21T08:31:04.682 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:31:04.683 INFO:tasks.rados.rados.0.smithi137.stdout:1876: done (0 left) 2024-09-21T08:31:04.683 INFO:tasks.rados.rados.0.smithi137.stdout:1877: snap_remove snap 186 2024-09-21T08:31:04.683 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5212000000000001e-05 2024-09-21T08:31:04.683 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.683 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.683 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.683 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577426+0000", 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023901099999999999 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577463+0000", 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6820000000000003e-05 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577504+0000", 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1063999999999999e-05 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.684 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:52 5.e 5:7759931f:::notify.3:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.576451+0000", 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 450.08867551200001, 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001253133, 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:31:04.685 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 52 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576451+0000", 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576451+0000", 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.686 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576453+0000", 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8050000000000001e-06 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576458+0000", 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3299999999999998e-06 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576468+0000", 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8700000000000004e-06 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.687 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576476+0000", 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6729999999999992e-06 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576767+0000", 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029066200000000002 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576845+0000", 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8517999999999996e-05 2024-09-21T08:31:04.688 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.576961+0000", 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011566900000000001 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577297+0000", 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033612900000000001 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577569+0000", 2024-09-21T08:31:04.689 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027198899999999999 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577623+0000", 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3841000000000002e-05 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577667+0000", 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4152e-05 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.690 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577704+0000", 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7494999999999999e-05 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:53 5.e 5:7759931f:::notify.3:head [watch watch cookie 94074441556624] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.577867+0000", 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 450.08726006699999, 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001803189, 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.691 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 53 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577867+0000", 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577867+0000", 2024-09-21T08:31:04.692 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577868+0000", 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1289999999999999e-06 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577872+0000", 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4050000000000002e-06 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.693 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577880+0000", 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1130000000000001e-06 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.577885+0000", 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8080000000000004e-06 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578144+0000", 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025872800000000001 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.694 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578215+0000", 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1330999999999999e-05 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578696+0000", 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00048043800000000002 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578769+0000", 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3302999999999999e-05 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.695 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.578911+0000", 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014194000000000001 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579286+0000", 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037537600000000002 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579511+0000", 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022471700000000001 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.696 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579547+0000", 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5947e-05 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579599+0000", 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1929999999999999e-05 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.579670+0000", 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1024000000000003e-05 2024-09-21T08:31:04.697 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:206 4.18 4:1e85421f:gc::gc.1:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e72)", 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.590771+0000", 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.07435591799998, 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023079490000000001, 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.698 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 206 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590771+0000", 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590771+0000", 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.699 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590773+0000", 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4739999999999999e-06 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590782+0000", 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1290000000000002e-06 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590795+0000", 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2456000000000001e-05 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.700 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590805+0000", 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0111e-05 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590876+0000", 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1111000000000004e-05 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.590965+0000", 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9061999999999995e-05 2024-09-21T08:31:04.701 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.591134+0000", 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000169044 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.591516+0000", 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000381362 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.591767+0000", 2024-09-21T08:31:04.702 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025140299999999998 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.593025+0000", 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012583200000000001 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.593054+0000", 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8863000000000001e-05 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.593079+0000", 2024-09-21T08:31:04.703 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4613999999999998e-05 2024-09-21T08:31:04.704 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.704 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.704 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.704 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.704 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.704 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:210 4.9 4:954be8d2:gc::gc.5:head [create,call version.check_conds in=18b,call rgw_gc.rgw_gc_queue_init in=23b,call version.set in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.704 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.600055+0000", 2024-09-21T08:31:04.704 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.06507152099999, 2024-09-21T08:31:04.704 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012045479999999999, 2024-09-21T08:31:04.705 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.705 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.705 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.705 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.705 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.705 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.705 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 210 2024-09-21T08:31:04.705 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.705 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.705 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600055+0000", 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600055+0000", 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600057+0000", 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9149999999999999e-06 2024-09-21T08:31:04.706 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600063+0000", 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6230000000000002e-06 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600076+0000", 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3128e-05 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600083+0000", 2024-09-21T08:31:04.707 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2439999999999996e-06 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600148+0000", 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5118999999999998e-05 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600224+0000", 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6283000000000001e-05 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:31:04.708 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600400+0000", 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000175794 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.600700+0000", 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029931899999999998 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601121+0000", 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042133399999999999 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.709 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601205+0000", 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4358999999999996e-05 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601232+0000", 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6902000000000001e-05 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.601260+0000", 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7528000000000001e-05 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.710 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:238 4.18 4:1a734c59:::obj_delete_at_hint.0000000000:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.633924+0000", 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.03120265000001, 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001234052, 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.711 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 238 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633924+0000", 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633924+0000", 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.712 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633925+0000", 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1600000000000001e-07 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633928+0000", 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1810000000000001e-06 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633933+0000", 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4399999999999996e-06 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.713 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633937+0000", 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4249999999999998e-06 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.633973+0000", 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6247e-05 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634046+0000", 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2575000000000002e-05 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.714 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:31:04.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634151+0000", 2024-09-21T08:31:04.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010584199999999999 2024-09-21T08:31:04.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.715 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.715 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634527+0000", 2024-09-21T08:31:04.715 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00037577699999999999 2024-09-21T08:31:04.715 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.715 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.634630+0000", 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102846 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.635070+0000", 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000439796 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.635121+0000", 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0847999999999998e-05 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.716 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.635158+0000", 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7358999999999999e-05 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:255 4.9 4:921ecdb1:gc::gc.25:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.636496+0000", 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.02863099699999, 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015126289999999999, 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.717 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 255 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636496+0000", 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.718 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636496+0000", 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636497+0000", 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2099999999999995e-07 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636507+0000", 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9620000000000005e-06 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.719 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636514+0000", 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5270000000000001e-06 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636519+0000", 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7589999999999997e-06 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636564+0000", 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4919999999999997e-05 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.720 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636637+0000", 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3219999999999994e-05 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.636792+0000", 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015530599999999999 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637091+0000", 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029869500000000003 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.721 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637625+0000", 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053441199999999995 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637927+0000", 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00030120199999999998 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.637974+0000", 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6999999999999997e-05 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.722 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.638008+0000", 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4705000000000001e-05 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:287 4.e 4:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.640319+0000", 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.02480810999998, 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0013081729999999999, 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.723 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.724 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.724 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.724 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 287 2024-09-21T08:31:04.724 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.724 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.724 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.724 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.725 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:04 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3805776988' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:04.725 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:04 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3805776988' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:04.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640319+0000", 2024-09-21T08:31:04.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.726 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.726 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.726 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640319+0000", 2024-09-21T08:31:04.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640320+0000", 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4300000000000001e-07 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640323+0000", 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.715e-06 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640331+0000", 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5619999999999998e-06 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640337+0000", 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8579999999999996e-06 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640379+0000", 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2290000000000003e-05 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640452+0000", 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2948999999999996e-05 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640582+0000", 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013043000000000001 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.640890+0000", 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000307614 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641132+0000", 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000241728 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641472+0000", 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034040600000000002 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641530+0000", 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8112999999999997e-05 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.641627+0000", 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6564999999999994e-05 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:300 4.1 4:856b9b26:reshard::reshard.0000000002:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.642024+0000", 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.02310274299998, 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011698909999999999, 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.731 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 300 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642024+0000", 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642024+0000", 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642025+0000", 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.17e-06 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642030+0000", 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0370000000000002e-06 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642040+0000", 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3929999999999997e-06 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642045+0000", 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0470000000000001e-06 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642083+0000", 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8195000000000003e-05 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642167+0000", 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3720000000000005e-05 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642315+0000", 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014880500000000001 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642777+0000", 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00046160999999999999 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643089+0000", 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031199199999999998 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643132+0000", 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3343999999999997e-05 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643171+0000", 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9088999999999998e-05 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643194+0000", 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2489000000000001e-05 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:302 4.e 4:703a9e16:::obj_delete_at_hint.0000000002:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.642059+0000", 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.023067696, 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012335429999999999, 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.737 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 302 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642059+0000", 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642059+0000", 2024-09-21T08:31:04.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642060+0000", 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2900000000000004e-07 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642062+0000", 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5160000000000001e-06 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642066+0000", 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7979999999999999e-06 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642069+0000", 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0010000000000002e-06 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642105+0000", 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6066999999999999e-05 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.740 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642153+0000", 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7642000000000002e-05 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,6", 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642252+0000", 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9345000000000005e-05 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642627+0000", 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000374745 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.741 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.642847+0000", 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022007999999999999 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643187+0000", 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034015599999999999 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643250+0000", 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2829000000000004e-05 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643293+0000", 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2834999999999999e-05 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:316 4.1 4:856b9b26:reshard::reshard.0000000002:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.643653+0000", 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.02147326599999, 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011504950000000001, 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.743 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 316 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643653+0000", 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643653+0000", 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643655+0000", 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2309999999999999e-06 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643659+0000", 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3669999999999999e-06 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643669+0000", 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0081999999999999e-05 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643675+0000", 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8039999999999998e-06 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643727+0000", 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2144000000000001e-05 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643822+0000", 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4899000000000001e-05 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.643982+0000", 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000159887 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644499+0000", 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051742300000000004 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644668+0000", 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016911000000000001 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644716+0000", 2024-09-21T08:31:04.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7833000000000001e-05 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644768+0000", 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1634000000000001e-05 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644804+0000", 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6081000000000002e-05 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:329 4.1a 4:5909e671:gc::gc.27:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.644844+0000", 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 447.02028282399999, 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012130229999999999, 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 329 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644844+0000", 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644844+0000", 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644845+0000", 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3030000000000001e-06 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644850+0000", 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9200000000000003e-06 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644859+0000", 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0529999999999996e-06 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644864+0000", 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6949999999999999e-06 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644905+0000", 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1433999999999997e-05 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.644949+0000", 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4024999999999999e-05 2024-09-21T08:31:04.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645045+0000", 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5297999999999995e-05 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645301+0000", 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025601699999999999 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645800+0000", 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000499834 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.645965+0000", 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016433600000000001 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.646021+0000", 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6687999999999997e-05 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.646057+0000", 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5420000000000003e-05 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:453 4.1 4:80b6fe23:reshard::reshard.0000000011:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.670680+0000", 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 446.99444690899998, 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0011835179999999999, 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.756 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 453 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670680+0000", 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670680+0000", 2024-09-21T08:31:04.757 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670681+0000", 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2049999999999999e-06 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670690+0000", 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.3230000000000004e-06 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670698+0000", 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7810000000000005e-06 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670703+0000", 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6879999999999998e-06 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670757+0000", 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4660999999999997e-05 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.670860+0000", 2024-09-21T08:31:04.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102471 2024-09-21T08:31:04.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:31:04.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671030+0000", 2024-09-21T08:31:04.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016985599999999999 2024-09-21T08:31:04.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671457+0000", 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042770500000000001 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671715+0000", 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025777299999999999 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671761+0000", 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5836e-05 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671812+0000", 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0719e-05 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.671863+0000", 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1499999999999998e-05 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:462 4.1 4:80b6fe23:reshard::reshard.0000000011:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.672625+0000", 2024-09-21T08:31:04.763 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 446.992502081, 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014703749999999999, 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 462 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672625+0000", 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672625+0000", 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672626+0000", 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.139e-06 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672630+0000", 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7190000000000001e-06 2024-09-21T08:31:04.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672640+0000", 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0459999999999994e-06 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672644+0000", 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7609999999999998e-06 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672684+0000", 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9431000000000002e-05 2024-09-21T08:31:04.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672739+0000", 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4994999999999998e-05 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,6", 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.672855+0000", 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115975 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.673270+0000", 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041560499999999999 2024-09-21T08:31:04.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.673495+0000", 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022474600000000001 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.674004+0000", 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050865299999999997 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.674061+0000", 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6925999999999997e-05 2024-09-21T08:31:04.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.674095+0000", 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4378999999999997e-05 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:501 4.1a 4:5c7e8a6d:reshard::reshard.0000000015:head [call lock.unlock in=10b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.769 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.684553+0000", 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 446.98057392800001, 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015740999999999999, 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 501 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684553+0000", 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684553+0000", 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684554+0000", 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6000000000000002e-07 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684563+0000", 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0790000000000007e-06 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684570+0000", 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8379999999999996e-06 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684575+0000", 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5910000000000003e-06 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684605+0000", 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0256999999999998e-05 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684651+0000", 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5160999999999997e-05 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,3", 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.684747+0000", 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6220999999999999e-05 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.685024+0000", 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027716299999999998 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.685367+0000", 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034277100000000002 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.686053+0000", 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00068657400000000002 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.686090+0000", 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6801000000000002e-05 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.686127+0000", 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6783999999999999e-05 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.775 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:697 4.1b 4:db9d130d:gc::gc.24:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.750453+0000", 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 446.91467392700002, 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012349640000000001, 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 697 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750453+0000", 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750453+0000", 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750454+0000", 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3999999999999999e-06 2024-09-21T08:31:04.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750462+0000", 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4279999999999999e-06 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750473+0000", 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1362999999999999e-05 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750480+0000", 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8360000000000003e-06 2024-09-21T08:31:04.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750531+0000", 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1008999999999999e-05 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750610+0000", 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9122000000000006e-05 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.750784+0000", 2024-09-21T08:31:04.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174329 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751214+0000", 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00042964799999999998 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751607+0000", 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039261899999999998 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751636+0000", 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9655999999999999e-05 2024-09-21T08:31:04.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751663+0000", 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7254e-05 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.751688+0000", 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4300000000000001e-05 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:780 4.1b 4:dc490aaa:::obj_delete_at_hint.0000000067:head [call lock.lock in=8b] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e73)", 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:37.810765+0000", 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 446.85436207800001, 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023807759999999998, 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 780 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810765+0000", 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810765+0000", 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810766+0000", 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.497e-06 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810773+0000", 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8689999999999998e-06 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810784+0000", 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0675e-05 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810789+0000", 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7490000000000004e-06 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810832+0000", 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2174e-05 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.810916+0000", 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4040000000000002e-05 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,3", 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.811045+0000", 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012987299999999999 2024-09-21T08:31:04.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.811286+0000", 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000240743 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.811623+0000", 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033676300000000001 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.813096+0000", 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014725059999999999 2024-09-21T08:31:04.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.813122+0000", 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6744e-05 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:37.813145+0000", 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3142999999999998e-05 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.787 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDScrubReserve(3.f,QREQUEST,e:79,from:7,reservation_nonce:3)", 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 392.92279495100001, 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003249558, 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:04.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742334+0000", 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6809999999999998e-06 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742342+0000", 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6930000000000007e-06 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:04.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742355+0000", 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2556e-05 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742366+0000", 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0994e-05 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742426+0000", 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0509e-05 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.790 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742436+0000", 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8060000000000006e-06 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745540+0000", 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031038630000000001 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745553+0000", 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2669e-05 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:04.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745581+0000", 2024-09-21T08:31:04.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8787000000000001e-05 2024-09-21T08:31:04.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.792 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.792 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:04.792 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:04.792 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:04.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:04.815 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:04.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:04.816 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:04.917 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:04.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:04.939 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:04.940 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:05.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:04 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3805776988' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:05.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:05.053 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:31:05.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:05.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:05.063 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:05.164 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:05.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:05.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:05.186 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:05.287 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:05.308 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:05.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:05.309 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:05.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:05.431 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:05.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:05.432 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:05.533 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:05.554 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:05.554 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:05.555 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:05.641 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:05.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:05.678 INFO:teuthology.orchestra.run.smithi137.stderr:set pool 7 pg_num to 46 2024-09-21T08:31:05.678 INFO:tasks.rados.rados.0.smithi137.stdout:1877: done (0 left) 2024-09-21T08:31:05.678 INFO:tasks.rados.rados.0.smithi137.stdout:1878: snap_create 2024-09-21T08:31:05.680 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:05.681 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:05.681 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:05.762 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[41044]: pgmap v611: 178 pgs: 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 94 MiB/s rd, 18 MiB/s wr, 174 op/s; 237/1455 objects degraded (16.289%); 100/1455 objects misplaced (6.873%); 57 MiB/s, 132 objects/s recovering 2024-09-21T08:31:05.763 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:05.763 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3805776988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:31:05.763 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[41044]: osdmap e471: 8 total, 6 up, 5 in 2024-09-21T08:31:05.763 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1168206557' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]: dispatch 2024-09-21T08:31:05.763 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[41044]: from='client.45839 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]: dispatch 2024-09-21T08:31:05.763 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[29634]: pgmap v611: 178 pgs: 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 94 MiB/s rd, 18 MiB/s wr, 174 op/s; 237/1455 objects degraded (16.289%); 100/1455 objects misplaced (6.873%); 57 MiB/s, 132 objects/s recovering 2024-09-21T08:31:05.763 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:05.763 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3805776988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:31:05.763 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[29634]: osdmap e471: 8 total, 6 up, 5 in 2024-09-21T08:31:05.763 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1168206557' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]: dispatch 2024-09-21T08:31:05.764 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:05 smithi137 ceph-mon[29634]: from='client.45839 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]: dispatch 2024-09-21T08:31:05.782 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:05.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:05.804 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:05.804 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:05.905 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:05.926 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:05 smithi165 ceph-mon[32893]: pgmap v611: 178 pgs: 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 2 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 94 MiB/s rd, 18 MiB/s wr, 174 op/s; 237/1455 objects degraded (16.289%); 100/1455 objects misplaced (6.873%); 57 MiB/s, 132 objects/s recovering 2024-09-21T08:31:05.926 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:05 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:05.926 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:05 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3805776988' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:31:05.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:05 smithi165 ceph-mon[32893]: osdmap e471: 8 total, 6 up, 5 in 2024-09-21T08:31:05.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:05 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1168206557' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]: dispatch 2024-09-21T08:31:05.927 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:05 smithi165 ceph-mon[32893]: from='client.45839 ' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]: dispatch 2024-09-21T08:31:05.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:05.928 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:05.928 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:05.957 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T08:31:06.029 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:06.050 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:06.051 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:06.051 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:06.152 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:06.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:06.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:06.174 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:06.203 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:06.275 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:06.298 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:06.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:06.299 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:06.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:06.425 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:06.425 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:06.426 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:06.526 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:06.548 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:06.549 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:06.549 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:06.650 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:06.663 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:06.663 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:31:06.663 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:31:06.663 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:06.672 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:06.673 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:06.673 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:06.692 INFO:tasks.rados.rados.0.smithi137.stdout:1878: done (0 left) 2024-09-21T08:31:06.692 INFO:tasks.rados.rados.0.smithi137.stdout:1879: snap_create 2024-09-21T08:31:06.774 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:06.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:06.796 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:06.796 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:06.897 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:06.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:06.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:06.919 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:06.949 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: 1.5 scrub starts 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: 1.5 scrub ok 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: 1.3 scrub starts 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: 1.3 scrub ok 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: from='client.45839 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]': finished 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: osdmap e472: 8 total, 6 up, 5 in 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: 7.3 scrub starts 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: 7.3 scrub ok 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: 1.7 scrub starts 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: 1.7 scrub ok 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:06.950 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 237/1455 objects degraded (16.289%), 33 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: 1.5 scrub starts 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: 1.5 scrub ok 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: 1.3 scrub starts 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: 1.3 scrub ok 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: from='client.45839 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]': finished 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: osdmap e472: 8 total, 6 up, 5 in 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: 7.3 scrub starts 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: 7.3 scrub ok 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: 1.7 scrub starts 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: 1.7 scrub ok 2024-09-21T08:31:06.951 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:06.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:06.952 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:06 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 237/1455 objects degraded (16.289%), 33 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:06.983 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:06.983 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:06.984 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:06.984 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:06.984 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:06.991 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: 1.5 scrub starts 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: 1.5 scrub ok 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: 1.3 scrub starts 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: 1.3 scrub ok 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: from='client.45839 ' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num", "val": "46"}]': finished 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: osdmap e472: 8 total, 6 up, 5 in 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: 7.3 scrub starts 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: 7.3 scrub ok 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: 1.7 scrub starts 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: 1.7 scrub ok 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:06.992 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:06 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 237/1455 objects degraded (16.289%), 33 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:06.998 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:06.998 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:07.015 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:07.015 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:07.021 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:07.031 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:07.031 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:07.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:07.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:07.043 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:07.048 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:07.048 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:07.049 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:07.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:07.165 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:07.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:07.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:07.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:07.288 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:07.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:07.289 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:07.390 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:07.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:07.412 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:07.413 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:07.513 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:07.525 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:31:07.535 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:07.535 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:07.535 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:07.637 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:07.658 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:07.659 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:07.659 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:07.689 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[29634]: 1.2 scrub starts 2024-09-21T08:31:07.689 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[29634]: 1.2 scrub ok 2024-09-21T08:31:07.690 INFO:tasks.rados.rados.0.smithi137.stdout:1879: done (0 left) 2024-09-21T08:31:07.690 INFO:tasks.rados.rados.0.smithi137.stdout:1880: write (excl) oid 42 current snap is 202 2024-09-21T08:31:07.690 INFO:tasks.rados.rados.0.smithi137.stdout:1880: seq_num 667 ranges {410269=713794,1648313=507551,2917489=292780} 2024-09-21T08:31:07.698 INFO:tasks.rados.rados.0.smithi137.stdout:1880: writing smithi13779221-42 from 410269 to 1124063 tid 1 2024-09-21T08:31:07.705 INFO:tasks.rados.rados.0.smithi137.stdout:1880: writing smithi13779221-42 from 1648313 to 2155864 tid 2 2024-09-21T08:31:07.712 INFO:tasks.rados.rados.0.smithi137.stdout:1880: writing smithi13779221-42 from 2917489 to 3210269 tid 3 2024-09-21T08:31:07.712 INFO:tasks.rados.rados.0.smithi137.stdout:1881: read oid 50 snap 130 2024-09-21T08:31:07.712 INFO:tasks.rados.rados.0.smithi137.stdout:1881: expect (ObjNum 463 snap 124 seq_num 463) 2024-09-21T08:31:07.712 INFO:tasks.rados.rados.0.smithi137.stdout:1882: snap_remove snap 179 2024-09-21T08:31:07.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T08:31:07.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:07.801 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:07.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:07.802 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:07.835 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:07.838 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:07.878 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:07.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:07.924 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:07.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:07.925 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:07.986 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:07.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[29634]: pgmap v614: 178 pgs: 2 active+clean+remapped+snaptrim_wait, 50 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 557 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 1 op/s; 238/1449 objects degraded (16.425%); 97/1449 objects misplaced (6.694%) 2024-09-21T08:31:07.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]': finished 2024-09-21T08:31:07.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[29634]: osdmap e473: 8 total, 6 up, 5 in 2024-09-21T08:31:07.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[29634]: 1.9 scrub starts 2024-09-21T08:31:07.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[29634]: 1.9 scrub ok 2024-09-21T08:31:07.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[29634]: 1.1 scrub starts 2024-09-21T08:31:07.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[29634]: 1.1 scrub ok 2024-09-21T08:31:07.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2019540830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:07.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[41044]: 1.2 scrub starts 2024-09-21T08:31:07.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[41044]: 1.2 scrub ok 2024-09-21T08:31:07.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[41044]: pgmap v614: 178 pgs: 2 active+clean+remapped+snaptrim_wait, 50 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 557 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 1 op/s; 238/1449 objects degraded (16.425%); 97/1449 objects misplaced (6.694%) 2024-09-21T08:31:07.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]': finished 2024-09-21T08:31:07.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[41044]: osdmap e473: 8 total, 6 up, 5 in 2024-09-21T08:31:07.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[41044]: 1.9 scrub starts 2024-09-21T08:31:07.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[41044]: 1.9 scrub ok 2024-09-21T08:31:07.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[41044]: 1.1 scrub starts 2024-09-21T08:31:07.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[41044]: 1.1 scrub ok 2024-09-21T08:31:07.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:07 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2019540830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:07 smithi165 ceph-mon[32893]: 1.2 scrub starts 2024-09-21T08:31:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:07 smithi165 ceph-mon[32893]: 1.2 scrub ok 2024-09-21T08:31:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:07 smithi165 ceph-mon[32893]: pgmap v614: 178 pgs: 2 active+clean+remapped+snaptrim_wait, 50 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 557 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 1 op/s; 238/1449 objects degraded (16.425%); 97/1449 objects misplaced (6.694%) 2024-09-21T08:31:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:07 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pg_num_actual", "val": "46"}]': finished 2024-09-21T08:31:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:07 smithi165 ceph-mon[32893]: osdmap e473: 8 total, 6 up, 5 in 2024-09-21T08:31:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:07 smithi165 ceph-mon[32893]: 1.9 scrub starts 2024-09-21T08:31:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:07 smithi165 ceph-mon[32893]: 1.9 scrub ok 2024-09-21T08:31:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:07 smithi165 ceph-mon[32893]: 1.1 scrub starts 2024-09-21T08:31:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:07 smithi165 ceph-mon[32893]: 1.1 scrub ok 2024-09-21T08:31:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:07 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2019540830' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:08.026 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:08.046 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:08.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:08.047 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:08.148 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:08.169 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:08.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:08.170 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:08.271 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:08.292 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:08.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:08.293 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:08.394 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:08.415 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:08.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:08.416 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:08.517 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:08.538 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:08.539 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:08.539 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:08.605 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:08.605 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:08.640 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:08.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:08.662 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:08.662 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:08.687 INFO:tasks.rados.rados.0.smithi137.stdout:1882: done (2 left) 2024-09-21T08:31:08.687 INFO:tasks.rados.rados.0.smithi137.stdout:1883: snap_remove snap 187 2024-09-21T08:31:08.687 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:31:08.763 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: 1.6 scrub starts 2024-09-21T08:31:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: 1.6 scrub ok 2024-09-21T08:31:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: osdmap e474: 8 total, 6 up, 5 in 2024-09-21T08:31:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: 6.14 scrub starts 2024-09-21T08:31:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: 6.14 scrub ok 2024-09-21T08:31:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: 1.a scrub starts 2024-09-21T08:31:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: 1.a scrub ok 2024-09-21T08:31:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: pgmap v617: 188 pgs: 10 unknown, 2 active+clean+remapped+snaptrim_wait, 50 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 557 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 1 op/s; 238/1449 objects degraded (16.425%); 97/1449 objects misplaced (6.694%) 2024-09-21T08:31:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:08.764 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:08.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1748429100' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:08.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: from='client.45851 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:08.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:31:08.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]': finished 2024-09-21T08:31:08.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: from='client.45851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:08.765 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:08 smithi165 ceph-mon[32893]: osdmap e475: 8 total, 6 up, 5 in 2024-09-21T08:31:08.785 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:08.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:08.786 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:08.887 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:08.908 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:08.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:08.909 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:09.009 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:09.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: 1.6 scrub starts 2024-09-21T08:31:09.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: 1.6 scrub ok 2024-09-21T08:31:09.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: osdmap e474: 8 total, 6 up, 5 in 2024-09-21T08:31:09.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: 6.14 scrub starts 2024-09-21T08:31:09.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: 6.14 scrub ok 2024-09-21T08:31:09.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: 1.a scrub starts 2024-09-21T08:31:09.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: 1.a scrub ok 2024-09-21T08:31:09.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: pgmap v617: 188 pgs: 10 unknown, 2 active+clean+remapped+snaptrim_wait, 50 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 557 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 1 op/s; 238/1449 objects degraded (16.425%); 97/1449 objects misplaced (6.694%) 2024-09-21T08:31:09.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1748429100' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: from='client.45851 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]': finished 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: from='client.45851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[29634]: osdmap e475: 8 total, 6 up, 5 in 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: 1.6 scrub starts 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: 1.6 scrub ok 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: osdmap e474: 8 total, 6 up, 5 in 2024-09-21T08:31:09.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: 6.14 scrub starts 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: 6.14 scrub ok 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: 1.a scrub starts 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: 1.a scrub ok 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: pgmap v617: 188 pgs: 10 unknown, 2 active+clean+remapped+snaptrim_wait, 50 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 557 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 12 MiB/s rd, 1 op/s; 238/1449 objects degraded (16.425%); 97/1449 objects misplaced (6.694%) 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd=[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]: dispatch 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1748429100' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: from='client.45851 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' cmd='[{"prefix": "osd pool set", "pool": "unique_pool_0", "var": "pgp_num_actual", "val": "46"}]': finished 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: from='client.45851 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:09.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:08 smithi137 ceph-mon[41044]: osdmap e475: 8 total, 6 up, 5 in 2024-09-21T08:31:09.031 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:09.032 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:09.032 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:09.133 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:09.155 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:09.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:09.156 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:09.257 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:09.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:09.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:09.279 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:09.353 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T08:31:09.440 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:09.463 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:09.463 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:09.464 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:09.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:09.586 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:09.586 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:09.587 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:09.687 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:09.693 INFO:tasks.rados.rados.0.smithi137.stdout:1883: done (2 left) 2024-09-21T08:31:09.693 INFO:tasks.rados.rados.0.smithi137.stdout:1884: write oid 40 current snap is 202 2024-09-21T08:31:09.694 INFO:tasks.rados.rados.0.smithi137.stdout:1884: seq_num 668 ranges {427076=789043,1706795=767562,2944912=282164} 2024-09-21T08:31:09.695 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:09.700 INFO:tasks.rados.rados.0.smithi137.stdout:1884: writing smithi13779221-40 from 427076 to 1216119 tid 1 2024-09-21T08:31:09.707 INFO:tasks.rados.rados.0.smithi137.stdout:1884: writing smithi13779221-40 from 1706795 to 2474357 tid 2 2024-09-21T08:31:09.712 INFO:tasks.rados.rados.0.smithi137.stdout:1884: writing smithi13779221-40 from 2944912 to 3227076 tid 3 2024-09-21T08:31:09.712 INFO:tasks.rados.rados.0.smithi137.stdout:1885: write oid 26 current snap is 202 2024-09-21T08:31:09.712 INFO:tasks.rados.rados.0.smithi137.stdout:1880: finishing write tid 1 to smithi13779221-42 2024-09-21T08:31:09.712 INFO:tasks.rados.rados.0.smithi137.stdout:1885: seq_num 669 ranges {443883=464292,1365277=511220,2572335=671548} 2024-09-21T08:31:09.716 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:09.716 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:09.717 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:09.717 INFO:tasks.rados.rados.0.smithi137.stdout:1885: writing smithi13779221-26 from 443883 to 908175 tid 1 2024-09-21T08:31:09.722 INFO:tasks.rados.rados.0.smithi137.stdout:1885: writing smithi13779221-26 from 1365277 to 1876497 tid 2 2024-09-21T08:31:09.730 INFO:tasks.rados.rados.0.smithi137.stdout:1885: writing smithi13779221-26 from 2572335 to 3243883 tid 3 2024-09-21T08:31:09.730 INFO:tasks.rados.rados.0.smithi137.stdout:1886: copy_from oid 8 from oid 15 current snap is 202 2024-09-21T08:31:09.730 INFO:tasks.rados.rados.0.smithi137.stdout:1880: finishing write tid 2 to smithi13779221-42 2024-09-21T08:31:09.730 INFO:tasks.rados.rados.0.smithi137.stdout:1880: finishing write tid 3 to smithi13779221-42 2024-09-21T08:31:09.730 INFO:tasks.rados.rados.0.smithi137.stdout:1880: finishing write tid 4 to smithi13779221-42 2024-09-21T08:31:09.730 INFO:tasks.rados.rados.0.smithi137.stdout:1880: finishing write tid 5 to smithi13779221-42 2024-09-21T08:31:09.730 INFO:tasks.rados.rados.0.smithi137.stdout:1880: oid 42 updating version 0 to 697 2024-09-21T08:31:09.730 INFO:tasks.rados.rados.0.smithi137.stdout:1880: oid 42 version 697 is already newer than 695 2024-09-21T08:31:09.731 INFO:tasks.rados.rados.0.smithi137.stdout:1880: oid 42 version 697 is already newer than 694 2024-09-21T08:31:09.731 INFO:tasks.rados.rados.0.smithi137.stdout:1880: oid 42 version 697 is already newer than 696 2024-09-21T08:31:09.731 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 697 (ObjNum 667 snap 202 seq_num 667) dirty exists 2024-09-21T08:31:09.731 INFO:tasks.rados.rados.0.smithi137.stdout:1880: left oid 42 (ObjNum 667 snap 202 seq_num 667) 2024-09-21T08:31:09.731 INFO:tasks.rados.rados.0.smithi137.stdout:1881: expect (ObjNum 463 snap 124 seq_num 463) 2024-09-21T08:31:09.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:09.838 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:31:09.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:09.840 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:09.840 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:09.940 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:09.957 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:09 smithi165 ceph-mon[32893]: 7.8 scrub starts 2024-09-21T08:31:09.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:09 smithi165 ceph-mon[32893]: 7.8 scrub ok 2024-09-21T08:31:09.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:09 smithi165 ceph-mon[32893]: 5.c scrub starts 2024-09-21T08:31:09.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:09 smithi165 ceph-mon[32893]: 5.c scrub ok 2024-09-21T08:31:09.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:09 smithi165 ceph-mon[32893]: 1.4 deep-scrub starts 2024-09-21T08:31:09.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:09 smithi165 ceph-mon[32893]: 1.4 deep-scrub ok 2024-09-21T08:31:09.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:09 smithi165 ceph-mon[32893]: 3.6 deep-scrub starts 2024-09-21T08:31:09.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:09 smithi165 ceph-mon[32893]: 3.6 deep-scrub ok 2024-09-21T08:31:09.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:09 smithi165 ceph-mon[32893]: osdmap e476: 8 total, 6 up, 5 in 2024-09-21T08:31:09.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:09.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:09.962 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:10.013 INFO:tasks.rados.rados.0.smithi137.stdout:1885: finishing write tid 1 to smithi13779221-26 2024-09-21T08:31:10.013 INFO:tasks.rados.rados.0.smithi137.stdout:1880: done (4 left) 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1881: done (3 left) 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1887: snap_create 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1885: finishing write tid 2 to smithi13779221-26 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1885: finishing write tid 3 to smithi13779221-26 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1885: finishing write tid 4 to smithi13779221-26 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1885: finishing write tid 5 to smithi13779221-26 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1885: oid 26 updating version 0 to 401 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1885: oid 26 updating version 401 to 404 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1885: oid 26 version 404 is already newer than 402 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1885: oid 26 version 404 is already newer than 403 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 404 (ObjNum 669 snap 202 seq_num 669) dirty exists 2024-09-21T08:31:10.014 INFO:tasks.rados.rados.0.smithi137.stdout:1885: left oid 26 (ObjNum 669 snap 202 seq_num 669) 2024-09-21T08:31:10.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[29634]: 7.8 scrub starts 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[29634]: 7.8 scrub ok 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[29634]: 5.c scrub starts 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[29634]: 5.c scrub ok 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[29634]: 1.4 deep-scrub starts 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[29634]: 1.4 deep-scrub ok 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[29634]: 3.6 deep-scrub starts 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[29634]: 3.6 deep-scrub ok 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[29634]: osdmap e476: 8 total, 6 up, 5 in 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[41044]: 7.8 scrub starts 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[41044]: 7.8 scrub ok 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[41044]: 5.c scrub starts 2024-09-21T08:31:10.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[41044]: 5.c scrub ok 2024-09-21T08:31:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[41044]: 1.4 deep-scrub starts 2024-09-21T08:31:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[41044]: 1.4 deep-scrub ok 2024-09-21T08:31:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[41044]: 3.6 deep-scrub starts 2024-09-21T08:31:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[41044]: 3.6 deep-scrub ok 2024-09-21T08:31:10.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:09 smithi137 ceph-mon[41044]: osdmap e476: 8 total, 6 up, 5 in 2024-09-21T08:31:10.062 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:10.085 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:10.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:10.086 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:10.088 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:10.140 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:10.140 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:10.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:10.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:10.209 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:10.210 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:10.310 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:10.332 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:10.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:10.333 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:10.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:10.456 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:10.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:10.457 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:10.544 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:10.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:10.561 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:10.561 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:10.579 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:10.579 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:10.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:10.581 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:10.581 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:10.595 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:10.595 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:10.611 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:10.612 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:10.628 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:10.628 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:10.644 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:10.644 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:10.645 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:10.682 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:10.690 INFO:tasks.rados.rados.0.smithi137.stdout:1885: done (3 left) 2024-09-21T08:31:10.690 INFO:tasks.rados.rados.0.smithi137.stdout:1887: done (2 left) 2024-09-21T08:31:10.690 INFO:tasks.rados.rados.0.smithi137.stdout:1888: write (excl) oid 18 current snap is 203 2024-09-21T08:31:10.690 INFO:tasks.rados.rados.0.smithi137.stdout:1888: seq_num 670 ranges {460690=655894,1540112=771231,2716111=539778} 2024-09-21T08:31:10.697 INFO:tasks.rados.rados.0.smithi137.stdout:1888: writing smithi13779221-18 from 460690 to 1116584 tid 1 2024-09-21T08:31:10.705 INFO:tasks.rados.rados.0.smithi137.stdout:1888: writing smithi13779221-18 from 1540112 to 2311343 tid 2 2024-09-21T08:31:10.714 INFO:tasks.rados.rados.0.smithi137.stdout:1888: writing smithi13779221-18 from 2716111 to 3255889 tid 3 2024-09-21T08:31:10.714 INFO:tasks.rados.rados.0.smithi137.stdout:1889: rollback oid 35 current snap is 203 2024-09-21T08:31:10.714 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 35 to 163 2024-09-21T08:31:10.714 INFO:tasks.rados.rados.0.smithi137.stdout:1888: finishing write tid 1 to smithi13779221-18 2024-09-21T08:31:10.714 INFO:tasks.rados.rados.0.smithi137.stdout:1890: write (excl) oid 29 current snap is 203 2024-09-21T08:31:10.714 INFO:tasks.rados.rados.0.smithi137.stdout:1890: seq_num 671 ranges {477497=731143,1998594=514889,3143534=133963} 2024-09-21T08:31:10.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:10.718 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:10.719 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:10.719 INFO:tasks.rados.rados.0.smithi137.stdout:1890: writing smithi13779221-29 from 477497 to 1208640 tid 1 2024-09-21T08:31:10.727 INFO:tasks.rados.rados.0.smithi137.stdout:1890: writing smithi13779221-29 from 1998594 to 2513483 tid 2 2024-09-21T08:31:10.731 INFO:tasks.rados.rados.0.smithi137.stdout:1890: writing smithi13779221-29 from 3143534 to 3277497 tid 3 2024-09-21T08:31:10.732 INFO:tasks.rados.rados.0.smithi137.stdout:1891: write oid 27 current snap is 203 2024-09-21T08:31:10.732 INFO:tasks.rados.rados.0.smithi137.stdout:1886: finishing copy_from racing read to smithi13779221-8 2024-09-21T08:31:10.732 INFO:tasks.rados.rados.0.smithi137.stdout:1891: seq_num 672 ranges {494304=406392,1540723=774900,2770957=523347} 2024-09-21T08:31:10.737 INFO:tasks.rados.rados.0.smithi137.stdout:1891: writing smithi13779221-27 from 494304 to 900696 tid 1 2024-09-21T08:31:10.745 INFO:tasks.rados.rados.0.smithi137.stdout:1891: writing smithi13779221-27 from 1540723 to 2315623 tid 2 2024-09-21T08:31:10.751 INFO:tasks.rados.rados.0.smithi137.stdout:1891: writing smithi13779221-27 from 2770957 to 3294304 tid 3 2024-09-21T08:31:10.751 INFO:tasks.rados.rados.0.smithi137.stdout:1892: read oid 9 snap -1 2024-09-21T08:31:10.751 INFO:tasks.rados.rados.0.smithi137.stdout:1892: expect deleted 2024-09-21T08:31:10.751 INFO:tasks.rados.rados.0.smithi137.stdout:1893: read oid 30 snap -1 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:1893: expect (ObjNum 649 snap 192 seq_num 649) 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:1894: snap_create 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:1884: finishing write tid 1 to smithi13779221-40 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:1889: finishing rollback tid 0 to smithi13779221-35 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:1889: finishing rollback tid 1 to smithi13779221-35 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 735 (ObjNum 289 snap 0 seq_num 0) dirty dne 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:1888: finishing write tid 2 to smithi13779221-18 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:1886: finishing copy_from to smithi13779221-8 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 331 (ObjNum 651 snap 192 seq_num 651) dirty exists 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:1884: finishing write tid 2 to smithi13779221-40 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:1888: finishing write tid 3 to smithi13779221-18 2024-09-21T08:31:10.752 INFO:tasks.rados.rados.0.smithi137.stdout:1888: finishing write tid 4 to smithi13779221-18 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1884: finishing write tid 3 to smithi13779221-40 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1888: finishing write tid 5 to smithi13779221-18 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1888: oid 18 updating version 0 to 235 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1888: oid 18 updating version 235 to 236 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1888: oid 18 updating version 236 to 237 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1888: oid 18 version 237 is already newer than 234 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 18 v 237 (ObjNum 670 snap 203 seq_num 670) dirty exists 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1888: left oid 18 (ObjNum 670 snap 203 seq_num 670) 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1884: finishing write tid 4 to smithi13779221-40 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1884: finishing write tid 5 to smithi13779221-40 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1884: oid 40 updating version 0 to 339 2024-09-21T08:31:10.753 INFO:tasks.rados.rados.0.smithi137.stdout:1884: oid 40 version 339 is already newer than 338 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1884: oid 40 version 339 is already newer than 337 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1884: oid 40 version 339 is already newer than 336 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 339 (ObjNum 668 snap 202 seq_num 668) dirty exists 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1884: left oid 40 (ObjNum 668 snap 202 seq_num 668) 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1890: finishing write tid 1 to smithi13779221-29 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1890: finishing write tid 2 to smithi13779221-29 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1890: finishing write tid 3 to smithi13779221-29 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1890: finishing write tid 4 to smithi13779221-29 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1890: finishing write tid 5 to smithi13779221-29 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1890: oid 29 updating version 0 to 793 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1890: oid 29 updating version 793 to 795 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1890: oid 29 version 795 is already newer than 792 2024-09-21T08:31:10.754 INFO:tasks.rados.rados.0.smithi137.stdout:1890: oid 29 version 795 is already newer than 794 2024-09-21T08:31:10.755 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 795 (ObjNum 671 snap 203 seq_num 671) dirty exists 2024-09-21T08:31:10.755 INFO:tasks.rados.rados.0.smithi137.stdout:1890: left oid 29 (ObjNum 671 snap 203 seq_num 671) 2024-09-21T08:31:10.755 INFO:tasks.rados.rados.0.smithi137.stdout:1891: finishing write tid 1 to smithi13779221-27 2024-09-21T08:31:10.755 INFO:tasks.rados.rados.0.smithi137.stdout:1891: finishing write tid 2 to smithi13779221-27 2024-09-21T08:31:10.757 INFO:tasks.rados.rados.0.smithi137.stdout:1891: finishing write tid 3 to smithi13779221-27 2024-09-21T08:31:10.757 INFO:tasks.rados.rados.0.smithi137.stdout:1891: finishing write tid 4 to smithi13779221-27 2024-09-21T08:31:10.762 INFO:tasks.rados.rados.0.smithi137.stdout:1893: expect (ObjNum 649 snap 192 seq_num 649) 2024-09-21T08:31:10.768 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:31:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:31:10.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[29634]: 7.4 scrub starts 2024-09-21T08:31:10.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[29634]: 7.4 scrub ok 2024-09-21T08:31:10.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[29634]: 1.8 scrub starts 2024-09-21T08:31:10.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[29634]: 1.8 scrub ok 2024-09-21T08:31:10.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[29634]: pgmap v620: 188 pgs: 10 unknown, 2 active+clean+remapped+snaptrim_wait, 50 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 557 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 238/1449 objects degraded (16.425%); 97/1449 objects misplaced (6.694%) 2024-09-21T08:31:10.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4212561615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:10.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[29634]: osdmap e477: 8 total, 6 up, 5 in 2024-09-21T08:31:10.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[41044]: 7.4 scrub starts 2024-09-21T08:31:10.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[41044]: 7.4 scrub ok 2024-09-21T08:31:10.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[41044]: 1.8 scrub starts 2024-09-21T08:31:10.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[41044]: 1.8 scrub ok 2024-09-21T08:31:10.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[41044]: pgmap v620: 188 pgs: 10 unknown, 2 active+clean+remapped+snaptrim_wait, 50 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 557 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 238/1449 objects degraded (16.425%); 97/1449 objects misplaced (6.694%) 2024-09-21T08:31:10.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4212561615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:10.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:10 smithi137 ceph-mon[41044]: osdmap e477: 8 total, 6 up, 5 in 2024-09-21T08:31:10.819 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:10.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:10.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:10.841 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:10.932 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:10 smithi165 ceph-mon[32893]: 7.4 scrub starts 2024-09-21T08:31:10.932 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:10 smithi165 ceph-mon[32893]: 7.4 scrub ok 2024-09-21T08:31:10.932 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:10 smithi165 ceph-mon[32893]: 1.8 scrub starts 2024-09-21T08:31:10.932 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:10 smithi165 ceph-mon[32893]: 1.8 scrub ok 2024-09-21T08:31:10.932 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:10 smithi165 ceph-mon[32893]: pgmap v620: 188 pgs: 10 unknown, 2 active+clean+remapped+snaptrim_wait, 50 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+snaptrim, 1 active+clean+remapped+snaptrim, 25 active+undersized, 33 active+undersized+degraded, 60 active+clean; 557 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 238/1449 objects degraded (16.425%); 97/1449 objects misplaced (6.694%) 2024-09-21T08:31:10.932 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:10 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4212561615' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:10.932 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:10 smithi165 ceph-mon[32893]: osdmap e477: 8 total, 6 up, 5 in 2024-09-21T08:31:10.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:10.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T08:31:10.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:10.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:10.965 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:10.981 INFO:tasks.rados.rados.0.smithi137.stdout:1891: finishing write tid 5 to smithi13779221-27 2024-09-21T08:31:10.981 INFO:tasks.rados.rados.0.smithi137.stdout:1891: oid 27 updating version 0 to 389 2024-09-21T08:31:10.981 INFO:tasks.rados.rados.0.smithi137.stdout:1891: oid 27 updating version 389 to 392 2024-09-21T08:31:10.981 INFO:tasks.rados.rados.0.smithi137.stdout:1891: oid 27 version 392 is already newer than 391 2024-09-21T08:31:10.981 INFO:tasks.rados.rados.0.smithi137.stdout:1891: oid 27 version 392 is already newer than 390 2024-09-21T08:31:10.981 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 392 (ObjNum 672 snap 203 seq_num 672) dirty exists 2024-09-21T08:31:10.981 INFO:tasks.rados.rados.0.smithi137.stdout:1891: left oid 27 (ObjNum 672 snap 203 seq_num 672) 2024-09-21T08:31:11.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:11.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:11.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:11.090 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:11.190 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:11.205 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:11.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:11.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:11.213 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:11.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:11.335 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:11.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:11.336 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:11.345 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:11.346 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:11.436 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:11.442 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:31:11.459 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:11.460 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:11.460 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:11.561 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:11.583 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:11.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:11.584 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:11.595 INFO:tasks.rados.rados.0.smithi137.stdout:1884: done (8 left) 2024-09-21T08:31:11.595 INFO:tasks.rados.rados.0.smithi137.stdout:1886: done (7 left) 2024-09-21T08:31:11.595 INFO:tasks.rados.rados.0.smithi137.stdout:1888: done (6 left) 2024-09-21T08:31:11.595 INFO:tasks.rados.rados.0.smithi137.stdout:1889: done (5 left) 2024-09-21T08:31:11.595 INFO:tasks.rados.rados.0.smithi137.stdout:1890: done (4 left) 2024-09-21T08:31:11.595 INFO:tasks.rados.rados.0.smithi137.stdout:1891: done (3 left) 2024-09-21T08:31:11.595 INFO:tasks.rados.rados.0.smithi137.stdout:1892: done (2 left) 2024-09-21T08:31:11.595 INFO:tasks.rados.rados.0.smithi137.stdout:1893: done (1 left) 2024-09-21T08:31:11.596 INFO:tasks.rados.rados.0.smithi137.stdout:1894: done (0 left) 2024-09-21T08:31:11.596 INFO:tasks.rados.rados.0.smithi137.stdout:1895: write (excl) oid 12 current snap is 204 2024-09-21T08:31:11.596 INFO:tasks.rados.rados.0.smithi137.stdout:1895: seq_num 673 ranges {511111=481641,1599205=518558,2682027=489947} 2024-09-21T08:31:11.601 INFO:tasks.rados.rados.0.smithi137.stdout:1895: writing smithi13779221-12 from 511111 to 992752 tid 1 2024-09-21T08:31:11.608 INFO:tasks.rados.rados.0.smithi137.stdout:1895: writing smithi13779221-12 from 1599205 to 2117763 tid 2 2024-09-21T08:31:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:1895: writing smithi13779221-12 from 2682027 to 3171974 tid 3 2024-09-21T08:31:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:1896: delete oid 30 current snap is 204 2024-09-21T08:31:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:1895: finishing write tid 1 to smithi13779221-12 2024-09-21T08:31:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:1895: finishing write tid 2 to smithi13779221-12 2024-09-21T08:31:11.635 INFO:tasks.rados.rados.0.smithi137.stdout:1895: finishing write tid 3 to smithi13779221-12 2024-09-21T08:31:11.635 INFO:tasks.rados.rados.0.smithi137.stdout:1895: finishing write tid 4 to smithi13779221-12 2024-09-21T08:31:11.635 INFO:tasks.rados.rados.0.smithi137.stdout:1896: done (1 left) 2024-09-21T08:31:11.635 INFO:tasks.rados.rados.0.smithi137.stdout:1897: delete oid 3 current snap is 204 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1895: finishing write tid 5 to smithi13779221-12 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1895: oid 12 updating version 0 to 528 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1895: oid 12 version 528 is already newer than 527 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1895: oid 12 version 528 is already newer than 526 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1895: oid 12 version 528 is already newer than 525 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 528 (ObjNum 673 snap 204 seq_num 673) dirty exists 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1895: left oid 12 (ObjNum 673 snap 204 seq_num 673) 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1895: done (1 left) 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1897: done (0 left) 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1898: read oid 20 snap -1 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1898: expect deleted 2024-09-21T08:31:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:1899: snap_create 2024-09-21T08:31:11.657 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:11.657 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:31:11.657 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:31:11.657 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:11.685 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:11.697 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:11.707 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:11.707 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:11.707 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:11.736 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1, 4] out_osds: [6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:31:11.736 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:31:11.736 INFO:tasks.thrashosds.thrasher:inject_pause on osd.2 2024-09-21T08:31:11.736 INFO:tasks.thrashosds.thrasher:Testing filestore_inject_stall pause injection for duration 3 2024-09-21T08:31:11.736 INFO:tasks.thrashosds.thrasher:Checking after 0, should_be_down=False 2024-09-21T08:31:11.736 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 config set filestore_inject_stall 3 2024-09-21T08:31:11.809 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:11.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:11.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:11.831 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:11.932 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:11.953 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:11.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:11.954 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:11.966 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:11 smithi165 ceph-mon[32893]: 3.f deep-scrub starts 2024-09-21T08:31:11.967 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:11 smithi165 ceph-mon[32893]: 3.f deep-scrub ok 2024-09-21T08:31:11.967 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:11 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 238/1449 objects degraded (16.425%), 33 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:11.967 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:11 smithi165 ceph-mon[32893]: osdmap e478: 8 total, 6 up, 5 in 2024-09-21T08:31:12.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:11 smithi137 ceph-mon[29634]: 3.f deep-scrub starts 2024-09-21T08:31:12.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:11 smithi137 ceph-mon[29634]: 3.f deep-scrub ok 2024-09-21T08:31:12.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:11 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 238/1449 objects degraded (16.425%), 33 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:12.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:11 smithi137 ceph-mon[29634]: osdmap e478: 8 total, 6 up, 5 in 2024-09-21T08:31:12.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:11 smithi137 ceph-mon[41044]: 3.f deep-scrub starts 2024-09-21T08:31:12.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:11 smithi137 ceph-mon[41044]: 3.f deep-scrub ok 2024-09-21T08:31:12.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:11 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 238/1449 objects degraded (16.425%), 33 pgs degraded, 57 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:12.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:11 smithi137 ceph-mon[41044]: osdmap e478: 8 total, 6 up, 5 in 2024-09-21T08:31:12.055 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:12.077 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:12.078 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:12.078 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:12.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:12.201 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:12.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:12.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:12.219 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:12.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:31:11 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:31:12.303 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:12.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T08:31:12.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:12.325 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:12.326 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:12.427 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:12.449 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:12.450 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:12.450 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:12.551 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:12.570 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:12.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:12.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:12.574 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:12.607 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:31:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:1898: done (1 left) 2024-09-21T08:31:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:1899: done (0 left) 2024-09-21T08:31:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:1900: rollback oid 7 current snap is 205 2024-09-21T08:31:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 7 to 144 2024-09-21T08:31:12.607 INFO:tasks.rados.rados.0.smithi137.stdout:1901: read oid 15 snap -1 2024-09-21T08:31:12.608 INFO:tasks.rados.rados.0.smithi137.stdout:1901: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:12.608 INFO:tasks.rados.rados.0.smithi137.stdout:1902: snap_remove snap 203 2024-09-21T08:31:12.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:12.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:12.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:12.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:12.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:12.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:12.821 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:12.821 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:12.885 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:31:12.885 INFO:teuthology.orchestra.run.smithi137.stdout: "success": "filestore_inject_stall = '' (not observed, change may require restart) " 2024-09-21T08:31:12.885 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:31:12.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:12.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[41044]: pgmap v623: 188 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 2 active+clean+snaptrim, 4 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 4 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 1 active+recovering+undersized+degraded+remapped, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 239/1437 objects degraded (16.632%); 118/1437 objects misplaced (8.212%) 2024-09-21T08:31:12.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1705180514' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:12.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[41044]: from='client.45863 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:12.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:12.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[41044]: from='client.45863 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:31:12.886 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[41044]: osdmap e479: 8 total, 6 up, 5 in 2024-09-21T08:31:12.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:12.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[29634]: pgmap v623: 188 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 2 active+clean+snaptrim, 4 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 4 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 1 active+recovering+undersized+degraded+remapped, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 239/1437 objects degraded (16.632%); 118/1437 objects misplaced (8.212%) 2024-09-21T08:31:12.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1705180514' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:12.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[29634]: from='client.45863 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:12.886 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:12.887 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[29634]: from='client.45863 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:31:12.887 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:12 smithi137 ceph-mon[29634]: osdmap e479: 8 total, 6 up, 5 in 2024-09-21T08:31:12.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:12.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:12.945 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:12.945 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:12.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:12 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:12.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:12 smithi165 ceph-mon[32893]: pgmap v623: 188 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 2 active+clean+snaptrim, 4 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 4 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 1 active+recovering+undersized+degraded+remapped, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 239/1437 objects degraded (16.632%); 118/1437 objects misplaced (8.212%) 2024-09-21T08:31:12.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:12 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1705180514' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:12.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:12 smithi165 ceph-mon[32893]: from='client.45863 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:12.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:12 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:12.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:12 smithi165 ceph-mon[32893]: from='client.45863 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:31:12.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:12 smithi165 ceph-mon[32893]: osdmap e479: 8 total, 6 up, 5 in 2024-09-21T08:31:13.046 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:13.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:13.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:13.068 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:13.088 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:13.088 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:31:13.089 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:31:13.089 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:31:13.089 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:13.168 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:13.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:13.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:13.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:13.291 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:13.312 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:13.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:13.313 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:13.346 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:31:13.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:13.435 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:13.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:13.436 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:13.537 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:13.559 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:13.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:13.560 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:13.613 INFO:tasks.rados.rados.0.smithi137.stdout:1902: done (2 left) 2024-09-21T08:31:13.613 INFO:tasks.rados.rados.0.smithi137.stdout:1903: rollback oid 42 current snap is 205 2024-09-21T08:31:13.614 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 42 to 190 2024-09-21T08:31:13.614 INFO:tasks.rados.rados.0.smithi137.stdout:1900: finishing rollback tid 0 to smithi13779221-7 2024-09-21T08:31:13.614 INFO:tasks.rados.rados.0.smithi137.stdout:1900: finishing rollback tid 1 to smithi13779221-7 2024-09-21T08:31:13.614 INFO:tasks.rados.rados.0.smithi137.stdout:1900: finishing rollback tid 2 to smithi13779221-7 2024-09-21T08:31:13.614 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 794 (ObjNum 443 snap 114 seq_num 443) dirty exists 2024-09-21T08:31:13.614 INFO:tasks.rados.rados.0.smithi137.stdout:1901: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:13.661 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:13.684 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:13.684 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:13.685 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:13.722 INFO:tasks.rados.rados.0.smithi137.stdout:1903: finishing rollback tid 0 to smithi13779221-42 2024-09-21T08:31:13.722 INFO:tasks.rados.rados.0.smithi137.stdout:1903: finishing rollback tid 1 to smithi13779221-42 2024-09-21T08:31:13.722 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 699 (ObjNum 49 snap 0 seq_num 1953066355) dirty dne 2024-09-21T08:31:13.722 INFO:tasks.rados.rados.0.smithi137.stdout:1900: done (2 left) 2024-09-21T08:31:13.722 INFO:tasks.rados.rados.0.smithi137.stdout:1901: done (1 left) 2024-09-21T08:31:13.722 INFO:tasks.rados.rados.0.smithi137.stdout:1903: done (0 left) 2024-09-21T08:31:13.722 INFO:tasks.rados.rados.0.smithi137.stdout:1904: snap_remove snap 162 2024-09-21T08:31:13.786 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:13.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:13.808 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:13.808 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:13.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T08:31:13.909 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:13.930 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:13.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:13.931 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:14.031 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:14.044 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:14.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:14.055 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:14.055 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:14.087 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:14.157 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:14.179 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:14.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:14.179 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:14.280 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:14.303 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:14.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:14.305 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:14.405 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:14.427 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:14.428 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:14.428 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:14.462 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:14.477 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:14.478 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:14.494 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:14.494 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:14.510 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:14.510 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:14.527 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:14.527 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:31:14.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:31:14.527 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:31:14.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.527 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:31:14.527 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 459.95724414400001, 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049090319999999998, 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567958+0000", 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567959+0000", 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.762e-06 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567965+0000", 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8769999999999998e-06 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567980+0000", 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4742e-05 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.567990+0000", 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8800000000000003e-06 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.530 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568114+0000", 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012444499999999999 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568250+0000", 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000135205 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568390+0000", 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000140335 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569250+0000", 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00085971599999999995 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572132+0000", 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028820819999999998 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572759+0000", 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062750100000000004 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572810+0000", 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0488000000000002e-05 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572867+0000", 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6999000000000001e-05 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:46 5.1f 5:f95f44c2:::notify.0:head [create] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:31:14.533 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 459.95677733500003, 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045637589999999997, 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 46 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568424+0000", 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568426+0000", 2024-09-21T08:31:14.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6470000000000001e-06 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568432+0000", 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.395e-06 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568443+0000", 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1070000000000001e-05 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568456+0000", 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3862e-05 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568688+0000", 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023171699999999999 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568773+0000", 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5307000000000002e-05 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:31:14.537 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.568890+0000", 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116082 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.569279+0000", 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00038986899999999999 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572238+0000", 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029581009999999999 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572906+0000", 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000668945 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572955+0000", 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8455000000000003e-05 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.572988+0000", 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3309000000000002e-05 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44417.0:47 5.1f 5:f95f44c2:::notify.0:head [watch watch cookie 94074441540560] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 459.95199625499998, 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026954729999999999, 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44417", 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/1716164338", 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 47 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.541 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:14.542 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:14.542 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:14.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:31:14.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573206+0000", 2024-09-21T08:31:14.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573208+0000", 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9970000000000001e-06 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573213+0000", 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5679999999999999e-06 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573224+0000", 2024-09-21T08:31:14.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1270000000000001e-05 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573233+0000", 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8939999999999999e-06 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573770+0000", 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053668100000000001 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:31:14.545 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.573841+0000", 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.135e-05 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.574869+0000", 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010281719999999999 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.574911+0000", 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1631000000000002e-05 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 4,7", 2024-09-21T08:31:14.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575071+0000", 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015977 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575463+0000", 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039199299999999999 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575737+0000", 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00027415300000000002 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575785+0000", 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7970999999999999e-05 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575832+0000", 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7120999999999998e-05 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.575901+0000", 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8901999999999999e-05 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.548 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:192 4.5 4:a2ace8b3:::data_log.0:head [call fifo.get_meta in=12b] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e70)", 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 459.90937556900002, 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031080779999999998, 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 192 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.549 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615826+0000", 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615827+0000", 2024-09-21T08:31:14.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1629999999999999e-06 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615831+0000", 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0790000000000002e-06 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615841+0000", 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8279999999999998e-06 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615847+0000", 2024-09-21T08:31:14.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2660000000000001e-06 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615879+0000", 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2512999999999999e-05 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.615924+0000", 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4709999999999997e-05 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:23:34.618934+0000", 2024-09-21T08:31:14.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030105190000000001 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1084 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94837804042368 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 396.365584222, 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40449157200000002, 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:14.553 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1084 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159618+0000", 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159622+0000", 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1400000000000002e-06 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159631+0000", 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9670000000000007e-06 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159655+0000", 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4037999999999999e-05 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159667+0000", 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2427e-05 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159756+0000", 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9103000000000004e-05 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159772+0000", 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5877000000000001e-05 2024-09-21T08:31:14.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.554979+0000", 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.39520667999999998 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.555262+0000", 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00028320500000000002 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.560260+0000", 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049979630000000002 2024-09-21T08:31:14.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.560331+0000", 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1048000000000001e-05 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.564109+0000", 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037781239999999999 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1085 4.5 4:a60f9269:lc::lua_package_allowlist:head [watch reconnect cookie 94837808833664 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 396.36544499000001, 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.40437364799999997, 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1085 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159757+0000", 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159758+0000", 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.539e-06 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159764+0000", 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1330000000000004e-06 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159773+0000", 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3250000000000008e-06 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159779+0000", 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5400000000000001e-06 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159844+0000", 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4580999999999997e-05 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159854+0000", 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.893e-06 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552880+0000", 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.393026184 2024-09-21T08:31:14.562 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.553195+0000", 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031526299999999998 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559454+0000", 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062587809999999997 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.559541+0000", 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7379999999999993e-05 2024-09-21T08:31:14.563 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.564130+0000", 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0045890289999999997 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:265 7.b e93/80 7:dd97af7e:::smithi13779221-30:head v 93'26, mlcod=86'23)", 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 387.96144993600001, 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036488330000000002, 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.564 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563752+0000", 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.563753+0000", 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4000000000000001e-07 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565430+0000", 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0016774439999999999 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565446+0000", 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5540999999999999e-05 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565455+0000", 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9580000000000002e-06 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565513+0000", 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7312000000000003e-05 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.565542+0000", 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8856999999999998e-05 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.567363+0000", 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0018212300000000001 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.567401+0000", 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7851000000000001e-05 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:321 7.c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 6] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:31:14.568 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 387.71557457099999, 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013109218000000001, 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 321 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:31:14.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809627+0000", 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809632+0000", 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9559999999999996e-06 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809648+0000", 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5814000000000001e-05 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809680+0000", 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1732e-05 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809696+0000", 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6293000000000001e-05 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809815+0000", 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119442 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809903+0000", 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7033999999999994e-05 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.815434+0000", 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055318420000000004 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.820591+0000", 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051568229999999996 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.821125+0000", 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053413400000000002 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822656+0000", 2024-09-21T08:31:14.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001531035 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822697+0000", 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0488999999999998e-05 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.822736+0000", 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9623999999999999e-05 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:322 7.c 7:3cae00e7:::smithi13779221-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 387.715476319, 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0093384190000000006, 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 322 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809725+0000", 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809727+0000", 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.13e-06 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809734+0000", 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9569999999999996e-06 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809744+0000", 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0633e-05 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.809753+0000", 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7090000000000008e-06 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.810046+0000", 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029301499999999999 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.810079+0000", 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3293000000000001e-05 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.818920+0000", 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0088405299999999992 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.818982+0000", 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1999000000000001e-05 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:46.819064+0000", 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2152999999999998e-05 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:329 7.c 7:361ee62b:::smithi13779221-40:head [sparse-read 0~1272916,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 386.839023467, 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90863698400000004, 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 329 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686182+0000", 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0459999999999999e-06 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686196+0000", 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4097000000000001e-05 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686225+0000", 2024-09-21T08:31:14.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8846000000000001e-05 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686240+0000", 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4256e-05 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686362+0000", 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012248200000000001 2024-09-21T08:31:14.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686390+0000", 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7546e-05 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.591277+0000", 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90488744700000001 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.591416+0000", 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013935900000000001 2024-09-21T08:31:14.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.594815+0000", 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033989049999999998 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:330 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2655473,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 386.83893452799998, 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91026091200000003, 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.585 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.586 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:14.586 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:14.586 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 330 2024-09-21T08:31:14.586 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.586 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.586 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.586 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.586 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686267+0000", 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686269+0000", 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7060000000000001e-06 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.587 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686277+0000", 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4470000000000006e-06 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686294+0000", 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6099999999999998e-05 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686308+0000", 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4516e-05 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.588 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686428+0000", 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119834 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686436+0000", 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1450000000000008e-06 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.595025+0000", 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.90858932599999997 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.589 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.595096+0000", 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.0821999999999994e-05 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596528+0000", 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0014320159999999999 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:331 7.c 7:361ee62b:::smithi13779221-40:head [read 0~2655473,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:31:14.590 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 386.83887503599999, 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91109795900000001, 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 331 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:31:14.591 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686327+0000", 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686328+0000", 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.387e-06 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686336+0000", 2024-09-21T08:31:14.592 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4030000000000001e-06 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686350+0000", 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4713e-05 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686364+0000", 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3716999999999999e-05 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686455+0000", 2024-09-21T08:31:14.593 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0737999999999998e-05 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:47.686460+0000", 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1290000000000004e-06 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596614+0000", 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.91015405299999996 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.596687+0000", 2024-09-21T08:31:14.594 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2700000000000005e-05 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.597425+0000", 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00073811899999999997 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:338 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:31:14.595 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 385.95737009599998, 2024-09-21T08:31:14.597 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038704049999999999, 2024-09-21T08:31:14.597 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.597 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.597 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 338 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.598 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567832+0000", 2024-09-21T08:31:14.599 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.600 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:14.600 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:14.600 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.600 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.600 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567835+0000", 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0019999999999998e-06 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567843+0000", 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0090000000000007e-06 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567861+0000", 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8748000000000001e-05 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.601 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567872+0000", 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0173000000000001e-05 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567975+0000", 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103398 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568085+0000", 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109691 2024-09-21T08:31:14.602 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571702+0000", 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003617384 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:339 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:31:14.603 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 385.95729499700002, 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054334199999999996, 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 339 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.604 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567907+0000", 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567908+0000", 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.082e-06 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.605 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567913+0000", 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.536e-06 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567922+0000", 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0730000000000005e-06 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.567928+0000", 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4489999999999996e-06 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.606 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571741+0000", 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038125590000000001 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.571859+0000", 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000118249 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.573340+0000", 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001481472 2024-09-21T08:31:14.607 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:340 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~2336140,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e95)", 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 385.95709647199999, 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0082399729999999994, 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:14.608 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 340 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568105+0000", 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.609 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568107+0000", 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1850000000000001e-06 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568112+0000", 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8019999999999997e-06 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568122+0000", 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6970000000000005e-06 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.610 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.568129+0000", 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2119999999999997e-06 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.575080+0000", 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069511850000000003 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.575168+0000", 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7613000000000003e-05 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.611 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.576345+0000", 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001177279 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.7 95/94 [PushReplyOp(7:e1536981:::smithi13779221-32:head)])", 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 385.86851097700003, 2024-09-21T08:31:14.612 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030062309999999998, 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656691+0000", 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.613 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656692+0000", 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0690000000000001e-06 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656696+0000", 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3590000000000001e-06 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656705+0000", 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5639999999999997e-06 2024-09-21T08:31:14.614 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656710+0000", 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9289999999999997e-06 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.656749+0000", 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9026999999999998e-05 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.659697+0000", 2024-09-21T08:31:14.615 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029482829999999999 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.c 95/94 [PushReplyOp(7:3cae00e7:::smithi13779221-28:5)])", 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 385.84707694899998, 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028000960000000002, 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.616 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678125+0000", 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678126+0000", 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.327e-06 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.617 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678131+0000", 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4920000000000004e-06 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678141+0000", 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9839999999999996e-06 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678146+0000", 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.446e-06 2024-09-21T08:31:14.618 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.678184+0000", 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8259000000000001e-05 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.680925+0000", 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002740588 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.619 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPushReply(7.c 95/94 [PushReplyOp(7:3cae00e7:::smithi13779221-28:head)])", 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 385.83677390399998, 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027171439999999999, 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.620 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688428+0000", 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688429+0000", 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.195e-06 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688434+0000", 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999996e-06 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.621 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688443+0000", 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4769999999999992e-06 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688448+0000", 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3499999999999996e-06 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.688491+0000", 2024-09-21T08:31:14.622 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2216999999999999e-05 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:48.691145+0000", 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026543550000000002 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:353 7.c 7:361ee62b:::smithi13779221-40:head [copy-from ver 7] snapc 6=[] ondisk+write+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 384.622910354, 2024-09-21T08:31:14.623 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014212783, 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 353 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.624 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902291+0000", 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902296+0000", 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1509999999999997e-06 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902323+0000", 2024-09-21T08:31:14.625 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7491000000000002e-05 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902342+0000", 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8522999999999999e-05 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902350+0000", 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8119999999999997e-06 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902448+0000", 2024-09-21T08:31:14.626 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.7867000000000003e-05 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902544+0000", 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5612999999999994e-05 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.907790+0000", 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052465360000000004 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:14.627 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.913037+0000", 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052469129999999998 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916372+0000", 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003334914 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916404+0000", 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1327e-05 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.628 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916459+0000", 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4917000000000003e-05 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.916504+0000", 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5719000000000001e-05 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:354 7.c 7:361ee62b:::smithi13779221-40:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e96)", 2024-09-21T08:31:14.629 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 384.62282940900002, 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0089418770000000009, 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 354 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:31:14.630 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902372+0000", 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902373+0000", 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6499999999999998e-07 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902376+0000", 2024-09-21T08:31:14.631 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.616e-06 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902402+0000", 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5890000000000001e-05 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902407+0000", 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0019999999999997e-06 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902704+0000", 2024-09-21T08:31:14.632 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00029720499999999998 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.902740+0000", 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6383e-05 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911116+0000", 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.008375136 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911208+0000", 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2791e-05 2024-09-21T08:31:14.633 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:14.634 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:14.634 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:14.634 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:49.911314+0000", 2024-09-21T08:31:14.634 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000105989 2024-09-21T08:31:14.634 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.634 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.634 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.634 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:14.634 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:14.634 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:14.635 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:14.635 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:14.635 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:14.636 INFO:tasks.rados.rados.0.smithi137.stdout:1904: done (0 left) 2024-09-21T08:31:14.636 INFO:tasks.rados.rados.0.smithi137.stdout:1905: read oid 6 snap 173 2024-09-21T08:31:14.636 INFO:tasks.rados.rados.0.smithi137.stdout:1905: expect (ObjNum 593 snap 172 seq_num 593) 2024-09-21T08:31:14.636 INFO:tasks.rados.rados.0.smithi137.stdout:1906: write (excl) oid 22 current snap is 205 2024-09-21T08:31:14.636 INFO:tasks.rados.rados.0.smithi137.stdout:1906: seq_num 676 ranges {561532=707388,2058298=782238} 2024-09-21T08:31:14.636 INFO:tasks.rados.rados.0.smithi137.stdout:1906: writing smithi13779221-22 from 561532 to 1268920 tid 1 2024-09-21T08:31:14.637 INFO:tasks.rados.rados.0.smithi137.stdout:1906: writing smithi13779221-22 from 2058298 to 2840536 tid 2 2024-09-21T08:31:14.637 INFO:tasks.rados.rados.0.smithi137.stdout:1907: read oid 4 snap -1 2024-09-21T08:31:14.637 INFO:tasks.rados.rados.0.smithi137.stdout:1907: expect deleted 2024-09-21T08:31:14.637 INFO:tasks.rados.rados.0.smithi137.stdout:1908: write oid 21 current snap is 205 2024-09-21T08:31:14.637 INFO:tasks.rados.rados.0.smithi137.stdout:1908: seq_num 677 ranges {578339=498990,1833133=525896,2791719=586620} 2024-09-21T08:31:14.638 INFO:tasks.rados.rados.0.smithi137.stdout:1908: writing smithi13779221-21 from 578339 to 1077329 tid 1 2024-09-21T08:31:14.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:14.639 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:14.639 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:14.643 INFO:tasks.rados.rados.0.smithi137.stdout:1908: writing smithi13779221-21 from 1833133 to 2359029 tid 2 2024-09-21T08:31:14.649 INFO:tasks.rados.rados.0.smithi137.stdout:1908: writing smithi13779221-21 from 2791719 to 3378339 tid 3 2024-09-21T08:31:14.649 INFO:tasks.rados.rados.0.smithi137.stdout:1909: read oid 41 snap 200 2024-09-21T08:31:14.649 INFO:tasks.rados.rados.0.smithi137.stdout:1909: expect deleted 2024-09-21T08:31:14.649 INFO:tasks.rados.rados.0.smithi137.stdout:1910: snap_remove snap 196 2024-09-21T08:31:14.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:14.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:14.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:14.763 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:14.833 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:14 smithi137 ceph-mon[29634]: osdmap e480: 8 total, 6 up, 5 in 2024-09-21T08:31:14.833 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:14 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/41425971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:14.834 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:14 smithi137 ceph-mon[41044]: osdmap e480: 8 total, 6 up, 5 in 2024-09-21T08:31:14.834 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:14 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/41425971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:14.846 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:14 smithi165 ceph-mon[32893]: osdmap e480: 8 total, 6 up, 5 in 2024-09-21T08:31:14.847 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:14 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/41425971' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:14.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:14.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:14.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:14.888 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:14.989 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:15.011 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:15.012 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:15.012 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:15.113 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:15.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:15.135 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:15.135 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:15.235 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:15.235 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:15.235 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:15.259 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:15.260 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:15.260 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:15.279 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T08:31:15.361 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:15.382 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:15.383 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:15.383 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:15.484 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:15.506 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:15.507 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:15.507 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:15.517 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:31:15 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:31:15.138+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:31:15.523 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:15.541 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:31:15.608 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:15.627 INFO:tasks.rados.rados.0.smithi137.stdout:1910: done (5 left) 2024-09-21T08:31:15.627 INFO:tasks.rados.rados.0.smithi137.stdout:1911: write oid 45 current snap is 205 2024-09-21T08:31:15.627 INFO:tasks.rados.rados.0.smithi137.stdout:1911: seq_num 678 ranges {595146=574239,1891615=669554,3219142=176004} 2024-09-21T08:31:15.632 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:15.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:15.633 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:15.633 INFO:tasks.rados.rados.0.smithi137.stdout:1911: writing smithi13779221-45 from 595146 to 1169385 tid 1 2024-09-21T08:31:15.643 INFO:tasks.rados.rados.0.smithi137.stdout:1911: writing smithi13779221-45 from 1891615 to 2561169 tid 2 2024-09-21T08:31:15.647 INFO:tasks.rados.rados.0.smithi137.stdout:1911: writing smithi13779221-45 from 3219142 to 3395146 tid 3 2024-09-21T08:31:15.648 INFO:tasks.rados.rados.0.smithi137.stdout:1912: read oid 18 snap -1 2024-09-21T08:31:15.648 INFO:tasks.rados.rados.0.smithi137.stdout:1912: expect (ObjNum 670 snap 203 seq_num 670) 2024-09-21T08:31:15.648 INFO:tasks.rados.rados.0.smithi137.stdout:1913: read oid 15 snap -1 2024-09-21T08:31:15.648 INFO:tasks.rados.rados.0.smithi137.stdout:1913: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:15.648 INFO:tasks.rados.rados.0.smithi137.stdout:1914: read oid 48 snap 144 2024-09-21T08:31:15.648 INFO:tasks.rados.rados.0.smithi137.stdout:1914: expect (ObjNum 453 snap 119 seq_num 453) 2024-09-21T08:31:15.648 INFO:tasks.rados.rados.0.smithi137.stdout:1915: delete oid 26 current snap is 205 2024-09-21T08:31:15.648 INFO:tasks.rados.rados.0.smithi137.stdout:1905: expect (ObjNum 593 snap 172 seq_num 593) 2024-09-21T08:31:15.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:15.755 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:15.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:15.756 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:15.789 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:15 smithi137 ceph-mon[29634]: pgmap v626: 188 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 2 active+clean+snaptrim, 4 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 4 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 1 active+recovering+undersized+degraded+remapped, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 239/1437 objects degraded (16.632%); 118/1437 objects misplaced (8.212%) 2024-09-21T08:31:15.789 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:15 smithi137 ceph-mon[29634]: osdmap e481: 8 total, 6 up, 5 in 2024-09-21T08:31:15.789 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:15 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:15.790 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:15 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:15.790 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:15 smithi137 ceph-mon[41044]: pgmap v626: 188 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 2 active+clean+snaptrim, 4 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 4 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 1 active+recovering+undersized+degraded+remapped, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 239/1437 objects degraded (16.632%); 118/1437 objects misplaced (8.212%) 2024-09-21T08:31:15.790 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:15 smithi137 ceph-mon[41044]: osdmap e481: 8 total, 6 up, 5 in 2024-09-21T08:31:15.790 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:15 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:15.790 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:15 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:15.857 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:15.879 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:15.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:15.880 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:15.934 INFO:tasks.rados.rados.0.smithi137.stdout:1906: finishing write tid 1 to smithi13779221-22 2024-09-21T08:31:15.934 INFO:tasks.rados.rados.0.smithi137.stdout:1906: finishing write tid 2 to smithi13779221-22 2024-09-21T08:31:15.934 INFO:tasks.rados.rados.0.smithi137.stdout:1905: done (9 left) 2024-09-21T08:31:15.934 INFO:tasks.rados.rados.0.smithi137.stdout:1907: done (8 left) 2024-09-21T08:31:15.934 INFO:tasks.rados.rados.0.smithi137.stdout:1915: done (7 left) 2024-09-21T08:31:15.934 INFO:tasks.rados.rados.0.smithi137.stdout:1916: rollback oid 7 current snap is 205 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 7 to 194 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1906: finishing write tid 3 to smithi13779221-22 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1906: finishing write tid 4 to smithi13779221-22 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1906: oid 22 updating version 0 to 364 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1906: oid 22 version 364 is already newer than 363 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1906: oid 22 updating version 364 to 365 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 365 (ObjNum 676 snap 205 seq_num 676) dirty exists 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1906: left oid 22 (ObjNum 676 snap 205 seq_num 676) 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1908: finishing write tid 1 to smithi13779221-21 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1908: finishing write tid 2 to smithi13779221-21 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1908: finishing write tid 3 to smithi13779221-21 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1908: finishing write tid 4 to smithi13779221-21 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1908: finishing write tid 5 to smithi13779221-21 2024-09-21T08:31:15.935 INFO:tasks.rados.rados.0.smithi137.stdout:1908: oid 21 updating version 0 to 518 2024-09-21T08:31:15.936 INFO:tasks.rados.rados.0.smithi137.stdout:1908: oid 21 version 518 is already newer than 517 2024-09-21T08:31:15.936 INFO:tasks.rados.rados.0.smithi137.stdout:1908: oid 21 updating version 518 to 519 2024-09-21T08:31:15.936 INFO:tasks.rados.rados.0.smithi137.stdout:1908: oid 21 version 519 is already newer than 516 2024-09-21T08:31:15.936 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 519 (ObjNum 677 snap 205 seq_num 677) dirty exists 2024-09-21T08:31:15.936 INFO:tasks.rados.rados.0.smithi137.stdout:1908: left oid 21 (ObjNum 677 snap 205 seq_num 677) 2024-09-21T08:31:15.936 INFO:tasks.rados.rados.0.smithi137.stdout:1911: finishing write tid 1 to smithi13779221-45 2024-09-21T08:31:15.936 INFO:tasks.rados.rados.0.smithi137.stdout:1911: finishing write tid 2 to smithi13779221-45 2024-09-21T08:31:15.936 INFO:tasks.rados.rados.0.smithi137.stdout:1914: expect (ObjNum 453 snap 119 seq_num 453) 2024-09-21T08:31:15.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:16.001 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:16.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:16.002 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:15 smithi165 ceph-mon[32893]: pgmap v626: 188 pgs: 2 active+undersized+remapped, 1 active+recovering+undersized+remapped, 2 active+clean+snaptrim, 4 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 4 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 1 active+recovering+undersized+degraded+remapped, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 555 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 239/1437 objects degraded (16.632%); 118/1437 objects misplaced (8.212%) 2024-09-21T08:31:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:15 smithi165 ceph-mon[32893]: osdmap e481: 8 total, 6 up, 5 in 2024-09-21T08:31:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:15 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:16.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:15 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:16.026 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:16.026 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:16.103 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:16.124 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:16.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:16.125 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:16.142 INFO:tasks.rados.rados.0.smithi137.stdout:1913: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:16.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:16.244 INFO:tasks.rados.rados.0.smithi137.stdout:1911: finishing write tid 3 to smithi13779221-45 2024-09-21T08:31:16.244 INFO:tasks.rados.rados.0.smithi137.stdout:1911: finishing write tid 4 to smithi13779221-45 2024-09-21T08:31:16.244 INFO:tasks.rados.rados.0.smithi137.stdout:1911: finishing write tid 5 to smithi13779221-45 2024-09-21T08:31:16.244 INFO:tasks.rados.rados.0.smithi137.stdout:1911: oid 45 updating version 0 to 443 2024-09-21T08:31:16.244 INFO:tasks.rados.rados.0.smithi137.stdout:1911: oid 45 version 443 is already newer than 442 2024-09-21T08:31:16.244 INFO:tasks.rados.rados.0.smithi137.stdout:1911: oid 45 updating version 443 to 444 2024-09-21T08:31:16.244 INFO:tasks.rados.rados.0.smithi137.stdout:1911: oid 45 updating version 444 to 445 2024-09-21T08:31:16.244 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 445 (ObjNum 678 snap 205 seq_num 678) dirty exists 2024-09-21T08:31:16.244 INFO:tasks.rados.rados.0.smithi137.stdout:1911: left oid 45 (ObjNum 678 snap 205 seq_num 678) 2024-09-21T08:31:16.244 INFO:tasks.rados.rados.0.smithi137.stdout:1912: expect (ObjNum 670 snap 203 seq_num 670) 2024-09-21T08:31:16.247 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:16.248 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:16.248 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:16.349 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:16.370 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:16.371 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:16.371 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:16.434 INFO:tasks.rados.rados.0.smithi137.stdout:1916: finishing rollback tid 0 to smithi13779221-7 2024-09-21T08:31:16.434 INFO:tasks.rados.rados.0.smithi137.stdout:1916: finishing rollback tid 1 to smithi13779221-7 2024-09-21T08:31:16.434 INFO:tasks.rados.rados.0.smithi137.stdout:1916: finishing rollback tid 2 to smithi13779221-7 2024-09-21T08:31:16.434 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 797 (ObjNum 420 snap 103 seq_num 420) dirty exists 2024-09-21T08:31:16.434 INFO:tasks.rados.rados.0.smithi137.stdout:1906: done (7 left) 2024-09-21T08:31:16.434 INFO:tasks.rados.rados.0.smithi137.stdout:1908: done (6 left) 2024-09-21T08:31:16.434 INFO:tasks.rados.rados.0.smithi137.stdout:1909: done (5 left) 2024-09-21T08:31:16.434 INFO:tasks.rados.rados.0.smithi137.stdout:1911: done (4 left) 2024-09-21T08:31:16.435 INFO:tasks.rados.rados.0.smithi137.stdout:1912: done (3 left) 2024-09-21T08:31:16.435 INFO:tasks.rados.rados.0.smithi137.stdout:1913: done (2 left) 2024-09-21T08:31:16.435 INFO:tasks.rados.rados.0.smithi137.stdout:1914: done (1 left) 2024-09-21T08:31:16.435 INFO:tasks.rados.rados.0.smithi137.stdout:1916: done (0 left) 2024-09-21T08:31:16.435 INFO:tasks.rados.rados.0.smithi137.stdout:1917: copy_from oid 39 from oid 26 current snap is 205 2024-09-21T08:31:16.435 INFO:tasks.rados.rados.0.smithi137.stdout:1918: snap_create 2024-09-21T08:31:16.435 INFO:tasks.rados.rados.0.smithi137.stdout:1917: finishing copy_from to smithi13779221-39 2024-09-21T08:31:16.435 INFO:tasks.rados.rados.0.smithi137.stdout:1917: got expected ENOENT (src dne) 2024-09-21T08:31:16.435 INFO:tasks.rados.rados.0.smithi137.stdout:1917: finishing copy_from racing read to smithi13779221-39 2024-09-21T08:31:16.472 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:16.494 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:16.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:16.495 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:16.593 INFO:tasks.rados.rados.0.smithi137.stdout:1917: done (1 left) 2024-09-21T08:31:16.594 INFO:tasks.rados.rados.0.smithi137.stdout:1918: done (0 left) 2024-09-21T08:31:16.594 INFO:tasks.rados.rados.0.smithi137.stdout:1919: snap_create 2024-09-21T08:31:16.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:16.617 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:16.618 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:16.618 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:16.719 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:16.741 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:16.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:16.742 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:16.754 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T08:31:16.783 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:16.843 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:16.864 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:16.865 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:16.865 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:16.906 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:16 smithi137 ceph-mon[29634]: osdmap e482: 8 total, 6 up, 5 in 2024-09-21T08:31:16.906 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:16 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 239/1437 objects degraded (16.632%), 36 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:16.907 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:16 smithi137 ceph-mon[29634]: osdmap e483: 8 total, 6 up, 5 in 2024-09-21T08:31:16.907 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:16 smithi137 ceph-mon[41044]: osdmap e482: 8 total, 6 up, 5 in 2024-09-21T08:31:16.907 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:16 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 239/1437 objects degraded (16.632%), 36 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:16.907 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:16 smithi137 ceph-mon[41044]: osdmap e483: 8 total, 6 up, 5 in 2024-09-21T08:31:16.965 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:16.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:16.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:16.987 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:16 smithi165 ceph-mon[32893]: osdmap e482: 8 total, 6 up, 5 in 2024-09-21T08:31:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:16 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 239/1437 objects degraded (16.632%), 36 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:16 smithi165 ceph-mon[32893]: osdmap e483: 8 total, 6 up, 5 in 2024-09-21T08:31:17.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:17.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:17.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:17.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:17.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:17.233 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:17.234 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:17.234 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:17.236 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:31:17.335 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:17.349 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:17.356 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:17.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:17.357 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:17.458 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:17.479 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:17.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:17.480 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:17.581 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:17.601 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:31:17.602 INFO:tasks.rados.rados.0.smithi137.stdout:1919: done (0 left) 2024-09-21T08:31:17.602 INFO:tasks.rados.rados.0.smithi137.stdout:1920: delete oid 42 current snap is 207 2024-09-21T08:31:17.606 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:17.606 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:17.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:17.608 INFO:tasks.rados.rados.0.smithi137.stdout:1920: done (0 left) 2024-09-21T08:31:17.608 INFO:tasks.rados.rados.0.smithi137.stdout:1921: write (excl) oid 46 current snap is 207 2024-09-21T08:31:17.608 INFO:tasks.rados.rados.0.smithi137.stdout:1921: seq_num 681 ranges {645567=799986,1950708=533234,3185058=260509} 2024-09-21T08:31:17.617 INFO:tasks.rados.rados.0.smithi137.stdout:1921: writing smithi13779221-46 from 645567 to 1445553 tid 1 2024-09-21T08:31:17.627 INFO:tasks.rados.rados.0.smithi137.stdout:1921: writing smithi13779221-46 from 1950708 to 2483942 tid 2 2024-09-21T08:31:17.635 INFO:tasks.rados.rados.0.smithi137.stdout:1921: writing smithi13779221-46 from 3185058 to 3445567 tid 3 2024-09-21T08:31:17.635 INFO:tasks.rados.rados.0.smithi137.stdout:1922: read oid 21 snap 194 2024-09-21T08:31:17.635 INFO:tasks.rados.rados.0.smithi137.stdout:1922: expect deleted 2024-09-21T08:31:17.635 INFO:tasks.rados.rados.0.smithi137.stdout:1921: finishing write tid 1 to smithi13779221-46 2024-09-21T08:31:17.635 INFO:tasks.rados.rados.0.smithi137.stdout:1921: finishing write tid 2 to smithi13779221-46 2024-09-21T08:31:17.635 INFO:tasks.rados.rados.0.smithi137.stdout:1923: rollback oid 30 current snap is 207 2024-09-21T08:31:17.635 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 30 to 191 2024-09-21T08:31:17.635 INFO:tasks.rados.rados.0.smithi137.stdout:1924: read oid 15 snap -1 2024-09-21T08:31:17.635 INFO:tasks.rados.rados.0.smithi137.stdout:1924: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:17.636 INFO:tasks.rados.rados.0.smithi137.stdout:1925: snap_create 2024-09-21T08:31:17.638 INFO:tasks.rados.rados.0.smithi137.stdout:1921: finishing write tid 3 to smithi13779221-46 2024-09-21T08:31:17.638 INFO:tasks.rados.rados.0.smithi137.stdout:1921: finishing write tid 4 to smithi13779221-46 2024-09-21T08:31:17.641 INFO:tasks.rados.rados.0.smithi137.stdout:1921: finishing write tid 5 to smithi13779221-46 2024-09-21T08:31:17.641 INFO:tasks.rados.rados.0.smithi137.stdout:1921: oid 46 updating version 0 to 803 2024-09-21T08:31:17.641 INFO:tasks.rados.rados.0.smithi137.stdout:1921: oid 46 updating version 803 to 806 2024-09-21T08:31:17.641 INFO:tasks.rados.rados.0.smithi137.stdout:1921: oid 46 version 806 is already newer than 804 2024-09-21T08:31:17.641 INFO:tasks.rados.rados.0.smithi137.stdout:1921: oid 46 version 806 is already newer than 805 2024-09-21T08:31:17.641 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 806 (ObjNum 681 snap 207 seq_num 681) dirty exists 2024-09-21T08:31:17.641 INFO:tasks.rados.rados.0.smithi137.stdout:1921: left oid 46 (ObjNum 681 snap 207 seq_num 681) 2024-09-21T08:31:17.641 INFO:tasks.rados.rados.0.smithi137.stdout:1924: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:17.707 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:17.730 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:17.731 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:17.731 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:17.742 INFO:tasks.rados.rados.0.smithi137.stdout:1923: finishing rollback tid 1 to smithi13779221-30 2024-09-21T08:31:17.742 INFO:tasks.rados.rados.0.smithi137.stdout:1923: finishing rollback tid 2 to smithi13779221-30 2024-09-21T08:31:17.743 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 795 (ObjNum 626 snap 184 seq_num 626) dirty exists 2024-09-21T08:31:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:17 smithi165 ceph-mon[32893]: pgmap v629: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 566 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 20 op/s; 237/1443 objects degraded (16.424%); 111/1443 objects misplaced (7.692%); 285 KiB/s, 0 objects/s recovering 2024-09-21T08:31:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:17 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1810060097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:17 smithi165 ceph-mon[32893]: from='client.45875 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:17 smithi165 ceph-mon[32893]: from='client.45875 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:17.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:17 smithi165 ceph-mon[32893]: osdmap e484: 8 total, 6 up, 5 in 2024-09-21T08:31:17.767 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:17 smithi137 ceph-mon[41044]: pgmap v629: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 566 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 20 op/s; 237/1443 objects degraded (16.424%); 111/1443 objects misplaced (7.692%); 285 KiB/s, 0 objects/s recovering 2024-09-21T08:31:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:17 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1810060097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:17 smithi137 ceph-mon[41044]: from='client.45875 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:17 smithi137 ceph-mon[41044]: from='client.45875 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:17.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:17 smithi137 ceph-mon[41044]: osdmap e484: 8 total, 6 up, 5 in 2024-09-21T08:31:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:17 smithi137 ceph-mon[29634]: pgmap v629: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 566 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 20 op/s; 237/1443 objects degraded (16.424%); 111/1443 objects misplaced (7.692%); 285 KiB/s, 0 objects/s recovering 2024-09-21T08:31:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:17 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1810060097' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:17 smithi137 ceph-mon[29634]: from='client.45875 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:17 smithi137 ceph-mon[29634]: from='client.45875 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:17.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:17 smithi137 ceph-mon[29634]: osdmap e484: 8 total, 6 up, 5 in 2024-09-21T08:31:17.832 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:17.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:17.854 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:17.854 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:17.955 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:17.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:17.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:17.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:18.069 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:18.069 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:18.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:18.102 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:18.102 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:18.103 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:18.204 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:18.226 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:18.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:18.227 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:18.328 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:18.349 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:18.349 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:18.350 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:18.450 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:18.472 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:18.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:18.473 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:18.574 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:18.599 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:18.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:18.600 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:18.644 INFO:tasks.rados.rados.0.smithi137.stdout:1921: done (4 left) 2024-09-21T08:31:18.644 INFO:tasks.rados.rados.0.smithi137.stdout:1922: done (3 left) 2024-09-21T08:31:18.644 INFO:tasks.rados.rados.0.smithi137.stdout:1923: done (2 left) 2024-09-21T08:31:18.644 INFO:tasks.rados.rados.0.smithi137.stdout:1924: done (1 left) 2024-09-21T08:31:18.644 INFO:tasks.rados.rados.0.smithi137.stdout:1925: done (0 left) 2024-09-21T08:31:18.644 INFO:tasks.rados.rados.0.smithi137.stdout:1926: snap_create 2024-09-21T08:31:18.671 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T08:31:18.701 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:18.740 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:18.741 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:18.741 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:18.842 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:18.868 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:18.868 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:18.869 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:18.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:18.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:18.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:18.996 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:19.028 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1, 4] out_osds: [6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:31:19.028 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:31:19.028 INFO:tasks.thrashosds.thrasher:thrash_pg_upmap_items 2024-09-21T08:31:19.028 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd dump -f json-pretty 2024-09-21T08:31:19.078 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:19.097 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:19.119 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:19.122 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:19.122 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:19.123 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:19.223 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:19.249 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:19.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:19.250 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:19.351 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:19.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:19.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:19.372 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:19.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:19.498 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:19.499 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:19.499 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:19.600 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:19.626 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:19.626 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:19.627 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:19.644 INFO:tasks.rados.rados.0.smithi137.stdout:1926: done (0 left) 2024-09-21T08:31:19.644 INFO:tasks.rados.rados.0.smithi137.stdout:1927: snap_remove snap 189 2024-09-21T08:31:19.727 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:19.751 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:19.751 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:19.752 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:19.850 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:422 7.d 7:b31d99ce:::smithi13779221-21:head [write 443718~615603] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 383.34297497099999, 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034885076000000001, 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.851 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 422 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.852 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504978+0000", 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0090000000000002e-06 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506268+0000", 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001290444 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506279+0000", 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0526999999999999e-05 2024-09-21T08:31:19.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506288+0000", 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5470000000000002e-06 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506343+0000", 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5287000000000001e-05 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506354+0000", 2024-09-21T08:31:19.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0798999999999999e-05 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.525308+0000", 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018953616 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529390+0000", 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040819029999999996 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533318+0000", 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039282190000000002 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533380+0000", 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1890000000000005e-05 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533709+0000", 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032948400000000001 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536916+0000", 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032063399999999998 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539553+0000", 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026370909999999998 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539658+0000", 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105585 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539723+0000", 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4343000000000004e-05 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539861+0000", 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137992 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:423 7.d 7:b31d99ce:::smithi13779221-21:head [write 1564138~470341] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:31:19.858 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 383.34158530299999, 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033782404000000002, 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 423 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506366+0000", 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0449999999999999e-06 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507012+0000", 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064568899999999996 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507021+0000", 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2550000000000003e-06 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507029+0000", 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2559999999999996e-06 2024-09-21T08:31:19.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507079+0000", 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0957e-05 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507088+0000", 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3650000000000004e-06 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529457+0000", 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022369185999999999 2024-09-21T08:31:19.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529526+0000", 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8683999999999995e-05 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535157+0000", 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056315849999999997 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535244+0000", 2024-09-21T08:31:19.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6840000000000002e-05 2024-09-21T08:31:19.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:31:19.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535398+0000", 2024-09-21T08:31:19.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015404399999999999 2024-09-21T08:31:19.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.864 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:19.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539569+0000", 2024-09-21T08:31:19.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004170361 2024-09-21T08:31:19.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539613+0000", 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4101999999999999e-05 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539985+0000", 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037279300000000001 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540037+0000", 2024-09-21T08:31:19.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1345999999999999e-05 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540148+0000", 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110896 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:424 7.d 7:b31d99ce:::smithi13779221-21:head [write 2633066~610652] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 383.33459628899999, 2024-09-21T08:31:19.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027015847999999999, 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 424 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513356+0000", 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8789999999999999e-06 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513753+0000", 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039670999999999999 2024-09-21T08:31:19.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513764+0000", 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.154e-05 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513772+0000", 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9149999999999996e-06 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513833+0000", 2024-09-21T08:31:19.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1072000000000001e-05 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513843+0000", 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8349999999999999e-06 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529644+0000", 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015801084 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529688+0000", 2024-09-21T08:31:19.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3568e-05 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536031+0000", 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063434659999999999 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536135+0000", 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103599 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536291+0000", 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155668 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539575+0000", 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032840450000000002 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539925+0000", 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034988899999999998 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540200+0000", 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027585000000000001 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540250+0000", 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9413000000000003e-05 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540370+0000", 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120315 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.874 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:425 7.d 7:b31d99ce:::smithi13779221-21:head [setxattr _header (56) in=7b,truncate 3243718] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 383.33409364900001, 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030130239, 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 425 2024-09-21T08:31:19.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513858+0000", 2024-09-21T08:31:19.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9199999999999999e-07 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513863+0000", 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7650000000000001e-06 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513869+0000", 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9880000000000001e-06 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513874+0000", 2024-09-21T08:31:19.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.643e-06 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515915+0000", 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020407279999999999 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515925+0000", 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0051000000000001e-05 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529722+0000", 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013796780999999999 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529804+0000", 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1663999999999996e-05 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540423+0000", 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010619635000000001 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540671+0000", 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024766600000000001 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540795+0000", 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012383899999999999 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541324+0000", 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052910200000000004 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.542654+0000", 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013304370000000001 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543785+0000", 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001130603 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543841+0000", 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5939000000000001e-05 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543987+0000", 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014640599999999999 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:426 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 383.332134704, 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028482668999999999, 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.882 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 426 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515818+0000", 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7999999999999999e-06 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515825+0000", 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0790000000000003e-06 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.884 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515840+0000", 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4765e-05 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515857+0000", 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7414999999999998e-05 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515968+0000", 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111161 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.885 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:31:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515975+0000", 2024-09-21T08:31:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7499999999999997e-06 2024-09-21T08:31:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529841+0000", 2024-09-21T08:31:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013866387000000001 2024-09-21T08:31:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.886 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529951+0000", 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010953100000000001 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541110+0000", 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011159591 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541190+0000", 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9243e-05 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544032+0000", 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028424050000000001 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544118+0000", 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5798999999999996e-05 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544299+0000", 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180743 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.888 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:430 7.d 7:b0b32695:::smithi13779221-19:head [delete] snapc a=[9] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 383.33196402599998, 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028508957000000001, 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 430 2024-09-21T08:31:19.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:31:19.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1900000000000001e-07 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515991+0000", 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6899999999999998e-06 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515997+0000", 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2999999999999998e-06 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516003+0000", 2024-09-21T08:31:19.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0220000000000001e-06 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516045+0000", 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1069000000000003e-05 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516052+0000", 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9890000000000003e-06 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529980+0000", 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013927999999999999 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.532305+0000", 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023250580000000001 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541672+0000", 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093675080000000001 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541749+0000", 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6829000000000006e-05 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541934+0000", 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018531 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543560+0000", 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00162525 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543661+0000", 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101604 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544339+0000", 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067745299999999995 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544394+0000", 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5515e-05 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544496+0000", 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010144100000000001 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:571 7.1 7:88bf4ad2:::smithi13779221-6:head [copy-from ver 13] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:31:19.896 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 369.30089737200001, 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037746406000000003, 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 571 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547056+0000", 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.146e-06 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547067+0000", 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0609e-05 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547090+0000", 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2714999999999999e-05 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547100+0000", 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9920000000000002e-06 2024-09-21T08:31:19.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547203+0000", 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102828 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547210+0000", 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3629999999999996e-06 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553577+0000", 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063672829999999996 2024-09-21T08:31:19.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553669+0000", 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1856999999999998e-05 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573941+0000", 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020272168 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.583019+0000", 2024-09-21T08:31:19.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090777630000000008 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584643+0000", 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016245319999999999 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584686+0000", 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2259999999999999e-05 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584748+0000", 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2049000000000002e-05 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584800+0000", 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1841000000000001e-05 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:572 7.1 7:88bf4ad2:::smithi13779221-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:31:19.903 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 369.30082834299998, 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032210599999999999, 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 572 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:31:19.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547123+0000", 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5300000000000003e-07 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547138+0000", 2024-09-21T08:31:19.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4718999999999999e-05 2024-09-21T08:31:19.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547145+0000", 2024-09-21T08:31:19.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2520000000000002e-06 2024-09-21T08:31:19.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547151+0000", 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7649999999999999e-06 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547263+0000", 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112001 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547269+0000", 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8919999999999997e-06 2024-09-21T08:31:19.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553824+0000", 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065557499999999999 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553873+0000", 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8727999999999998e-05 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579133+0000", 2024-09-21T08:31:19.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025259964999999999 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579227+0000", 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3517999999999995e-05 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579333+0000", 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010615699999999999 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.910 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3155 7.9 7:9247b5b1:::smithi13779221-17:head [copy-from ver 183] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 167.275454529, 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029797267999999998, 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.911 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3155 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572497+0000", 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4899999999999995e-07 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572500+0000", 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.05e-06 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572505+0000", 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2820000000000004e-06 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572510+0000", 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8969999999999999e-06 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573522+0000", 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001012481 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573635+0000", 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112253 2024-09-21T08:31:19.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.583897+0000", 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010262288 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594011+0000", 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010113641 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594202+0000", 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000191418 2024-09-21T08:31:19.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602174+0000", 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079717079999999992 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602233+0000", 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9565999999999999e-05 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602293+0000", 2024-09-21T08:31:19.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0134999999999999e-05 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3441 7.15 7:a9dd6c7d:::smithi13779221-47:head [write 783325~557161,stat] snapc 84=[84,83,82,81,7f,7e,7d,7c,78,77,72,71,69] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 143.28460469500001, 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028447395, 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.917 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3441 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563352+0000", 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9039999999999997e-06 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563878+0000", 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052587200000000004 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563908+0000", 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0685999999999997e-05 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563922+0000", 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3738e-05 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.564007+0000", 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5283000000000003e-05 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.564017+0000", 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4909999999999994e-06 2024-09-21T08:31:19.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571275+0000", 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072579860000000001 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571419+0000", 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014457700000000001 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571666+0000", 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024687599999999999 2024-09-21T08:31:19.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591189+0000", 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019522923000000001 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591633+0000", 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044407499999999999 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591717+0000", 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4010000000000004e-05 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591793+0000", 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5974000000000001e-05 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4000 7.b 7:d6c5a045:::smithi13779221-46:head [copy-from ver 529] snapc a7=[a7,a6,a5,a4,a3,a2,a1,9e,9d,9c,9b,9a,99,98,94,93,92,91,90,8c,8b,88,86,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 88.279626348999997, 2024-09-21T08:31:19.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037495776000000001, 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4000 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568328+0000", 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.349e-06 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568346+0000", 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8839e-05 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568362+0000", 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5432999999999999e-05 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568371+0000", 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3009999999999995e-06 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568435+0000", 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4199e-05 2024-09-21T08:31:19.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568441+0000", 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0050000000000001e-06 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.575820+0000", 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073784970000000004 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.575934+0000", 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011455699999999999 2024-09-21T08:31:19.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.928 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:31:19.928 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587076+0000", 2024-09-21T08:31:19.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011141287 2024-09-21T08:31:19.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.595892+0000", 2024-09-21T08:31:19.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088164750000000007 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605259+0000", 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093663830000000007 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605662+0000", 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040333900000000001 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605744+0000", 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2560999999999998e-05 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605820+0000", 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5550999999999995e-05 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4168 7.b 7:d7506421:::smithi13779221-13:head [write 447218~719401,stat] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.581231+0000", 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 77.266719460000004, 2024-09-21T08:31:19.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.085401058000000002, 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4168 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581231+0000", 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581231+0000", 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581234+0000", 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9909999999999998e-06 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581576+0000", 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000342109 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581675+0000", 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8947e-05 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581686+0000", 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1039999999999999e-05 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582031+0000", 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000344333 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582110+0000", 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9197999999999996e-05 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582254+0000", 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000143881 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.584348+0000", 2024-09-21T08:31:19.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020943440000000002 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.585410+0000", 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001061782 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.666486+0000", 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081075990000000001 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.936 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.666565+0000", 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9076e-05 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.666632+0000", 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7366999999999994e-05 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4169 7.b 7:d7506421:::smithi13779221-13:head [write 1889370~660531] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.588798+0000", 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 77.259152993000001, 2024-09-21T08:31:19.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081088987000000001, 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4169 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588798+0000", 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588798+0000", 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588800+0000", 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.903e-06 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588978+0000", 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017865599999999999 2024-09-21T08:31:19.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588995+0000", 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6696000000000001e-05 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589003+0000", 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8520000000000004e-06 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589056+0000", 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3665000000000002e-05 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589145+0000", 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8455000000000006e-05 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589276+0000", 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000131032 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.590553+0000", 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001277572 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.591751+0000", 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001197137 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669765+0000", 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.078014230000000004 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669824+0000", 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9651999999999997e-05 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669887+0000", 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2137000000000005e-05 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.943 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4170 7.b 7:d7506421:::smithi13779221-13:head [write 2963386~614253] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.595002+0000", 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 77.252948880000005, 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.075006211000000003, 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4170 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595002+0000", 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595002+0000", 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595004+0000", 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4559999999999999e-06 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595379+0000", 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037446400000000003 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595396+0000", 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7303999999999999e-05 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595403+0000", 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7689999999999999e-06 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595459+0000", 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6373999999999999e-05 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595547+0000", 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7869999999999997e-05 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595681+0000", 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133857 2024-09-21T08:31:19.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599798+0000", 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041171790000000003 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599868+0000", 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9839000000000007e-05 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669931+0000", 2024-09-21T08:31:19.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070063149000000005 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669973+0000", 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1838999999999997e-05 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670008+0000", 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5111000000000003e-05 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.949 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4171 7.b 7:d7506421:::smithi13779221-13:head [setxattr _header (58) in=7b,truncate 3647218] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.595420+0000", 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 77.252530864999997, 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074711124000000004, 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4171 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.950 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595420+0000", 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595420+0000", 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595421+0000", 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3099999999999996e-07 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595425+0000", 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3109999999999999e-06 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595431+0000", 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3600000000000001e-06 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595435+0000", 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2780000000000002e-06 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596387+0000", 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00095243599999999997 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596459+0000", 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2077999999999997e-05 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596640+0000", 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018089500000000001 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599908+0000", 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032677769999999999 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599918+0000", 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0059e-05 2024-09-21T08:31:19.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670040+0000", 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070122219 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670079+0000", 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8436000000000003e-05 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670131+0000", 2024-09-21T08:31:19.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2343999999999999e-05 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4172 7.b 7:d7506421:::smithi13779221-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.595449+0000", 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 77.252501146, 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074831616000000004, 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.956 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4172 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595449+0000", 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595449+0000", 2024-09-21T08:31:19.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595450+0000", 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.85e-07 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595453+0000", 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7879999999999998e-06 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595460+0000", 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.131e-06 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595466+0000", 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7910000000000001e-06 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596937+0000", 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001470823 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.597004+0000", 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7550000000000002e-05 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670154+0000", 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.073150017999999997 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670214+0000", 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9565999999999999e-05 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670281+0000", 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7163999999999996e-05 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4344 7.b 7:d4aadaae:::smithi13779221-7:head [copy-from ver 229] snapc af=[af,ae,ad,ac,ab,a9,a7,a5,a4,a3,9c,9b,9a,99,94,93,92,8b,84,7e] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:15.597787+0000", 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 64.250163274000002, 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033249305, 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.961 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4344 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597787+0000", 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597787+0000", 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597788+0000", 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3000000000000001e-07 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597792+0000", 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1749999999999998e-06 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597796+0000", 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9219999999999999e-06 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597800+0000", 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2069999999999997e-06 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598500+0000", 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069988899999999998 2024-09-21T08:31:19.964 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598508+0000", 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8310000000000001e-06 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610088+0000", 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011580653 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610205+0000", 2024-09-21T08:31:19.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116882 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.618623+0000", 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0084175770000000007 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.629957+0000", 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011334222999999999 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.630882+0000", 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00092455300000000005 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.630902+0000", 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0522000000000001e-05 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.630982+0000", 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9491000000000002e-05 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.631037+0000", 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4950000000000001e-05 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4345 7.b 7:d4aadaae:::smithi13779221-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e422)", 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:15.597855+0000", 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 64.250095142999996, 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027086114000000001, 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:31:19.968 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4345 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597855+0000", 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597855+0000", 2024-09-21T08:31:19.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597856+0000", 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6199999999999996e-07 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597860+0000", 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6270000000000002e-06 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598442+0000", 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058186999999999996 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598450+0000", 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0420000000000003e-06 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.599133+0000", 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00068317699999999998 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.971 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.599141+0000", 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240000000000007e-06 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610453+0000", 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011312546999999999 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610511+0000", 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7732999999999998e-05 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.624495+0000", 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013983647 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.624674+0000", 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017928099999999999 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.624942+0000", 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000267404 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.973 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4390 7.10 7:0979c720:::smithi13779221-16:head [copy-from ver 498] snapc b2=[b2,b1,b0,af,ae,ad,ac,a9,a7,a5,a4,a3,9c,9b,99,94,93,92,84,7e] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:21.587905+0000", 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 58.260046037000002, 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028087958, 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4390 2024-09-21T08:31:19.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587905+0000", 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587905+0000", 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:31:19.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587906+0000", 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.404e-06 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587911+0000", 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.305e-06 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587921+0000", 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3780000000000007e-06 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:31:19.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587926+0000", 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0889999999999999e-06 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587979+0000", 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3261999999999999e-05 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.588091+0000", 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112184 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-09-21T08:31:19.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.608584+0000", 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020492429999999999 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615479+0000", 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00689502 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615773+0000", 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029482400000000001 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:31:19.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615854+0000", 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0430000000000001e-05 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615993+0000", 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000138632 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:19.979 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:31:19.980 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:19 smithi137 ceph-mon[29634]: pgmap v632: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 566 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 20 op/s; 237/1443 objects degraded (16.424%); 111/1443 objects misplaced (7.692%); 285 KiB/s, 0 objects/s recovering 2024-09-21T08:31:19.980 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:19 smithi137 ceph-mon[29634]: osdmap e485: 8 total, 6 up, 5 in 2024-09-21T08:31:19.980 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:19 smithi137 ceph-mon[41044]: pgmap v632: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 566 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 20 op/s; 237/1443 objects degraded (16.424%); 111/1443 objects misplaced (7.692%); 285 KiB/s, 0 objects/s recovering 2024-09-21T08:31:19.980 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:19 smithi137 ceph-mon[41044]: osdmap e485: 8 total, 6 up, 5 in 2024-09-21T08:31:19.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:19.981 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:19.981 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:19 smithi165 ceph-mon[32893]: pgmap v632: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 566 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 18 MiB/s rd, 1.4 MiB/s wr, 20 op/s; 237/1443 objects degraded (16.424%); 111/1443 objects misplaced (7.692%); 285 KiB/s, 0 objects/s recovering 2024-09-21T08:31:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:19 smithi165 ceph-mon[32893]: osdmap e485: 8 total, 6 up, 5 in 2024-09-21T08:31:20.025 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:20.025 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:20.025 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:20.025 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:20.025 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:20.025 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:20.026 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:20.026 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:20.026 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:20.026 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:20.026 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:20.026 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:20.026 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:20.026 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:20.083 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:20.109 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:20.110 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:20.110 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:20.211 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:20.232 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:20.232 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:20.233 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:20.333 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:20.359 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:20.359 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:20.360 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:20.460 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:20.481 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:20.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:20.482 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:20.583 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:20.608 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:20.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:20.609 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:20.643 INFO:tasks.rados.rados.0.smithi137.stdout:1927: done (0 left) 2024-09-21T08:31:20.644 INFO:tasks.rados.rados.0.smithi137.stdout:1928: read oid 19 snap -1 2024-09-21T08:31:20.644 INFO:tasks.rados.rados.0.smithi137.stdout:1928: expect deleted 2024-09-21T08:31:20.644 INFO:tasks.rados.rados.0.smithi137.stdout:1929: copy_from oid 42 from oid 28 current snap is 209 2024-09-21T08:31:20.644 INFO:tasks.rados.rados.0.smithi137.stdout:1930: write oid 30 current snap is 209 2024-09-21T08:31:20.644 INFO:tasks.rados.rados.0.smithi137.stdout:1930: seq_num 682 ranges {662374=475235,1609190=676892,2812481=624101} 2024-09-21T08:31:20.650 INFO:tasks.rados.rados.0.smithi137.stdout:1930: writing smithi13779221-30 from 662374 to 1137609 tid 1 2024-09-21T08:31:20.657 INFO:tasks.rados.rados.0.smithi137.stdout:1930: writing smithi13779221-30 from 1609190 to 2286082 tid 2 2024-09-21T08:31:20.664 INFO:tasks.rados.rados.0.smithi137.stdout:1930: writing smithi13779221-30 from 2812481 to 3436582 tid 3 2024-09-21T08:31:20.664 INFO:tasks.rados.rados.0.smithi137.stdout:1931: write (excl) oid 40 current snap is 209 2024-09-21T08:31:20.664 INFO:tasks.rados.rados.0.smithi137.stdout:1931: seq_num 683 ranges {679181=550484,1667672=536903,2839904=435373} 2024-09-21T08:31:20.670 INFO:tasks.rados.rados.0.smithi137.stdout:1931: writing smithi13779221-40 from 679181 to 1229665 tid 1 2024-09-21T08:31:20.676 INFO:tasks.rados.rados.0.smithi137.stdout:1931: writing smithi13779221-40 from 1667672 to 2204575 tid 2 2024-09-21T08:31:20.682 INFO:tasks.rados.rados.0.smithi137.stdout:1931: writing smithi13779221-40 from 2839904 to 3275277 tid 3 2024-09-21T08:31:20.682 INFO:tasks.rados.rados.0.smithi137.stdout:1932: copy_from oid 7 from oid 18 current snap is 209 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1930: finishing write tid 1 to smithi13779221-30 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1930: finishing write tid 2 to smithi13779221-30 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1929: finishing copy_from racing read to smithi13779221-42 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1930: finishing write tid 3 to smithi13779221-30 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1930: finishing write tid 4 to smithi13779221-30 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1930: finishing write tid 5 to smithi13779221-30 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1930: oid 30 updating version 0 to 804 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1930: oid 30 version 804 is already newer than 802 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1930: oid 30 version 804 is already newer than 801 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1930: oid 30 version 804 is already newer than 803 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 804 (ObjNum 682 snap 209 seq_num 682) dirty exists 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1930: left oid 30 (ObjNum 682 snap 209 seq_num 682) 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:1929: finishing copy_from to smithi13779221-42 2024-09-21T08:31:20.683 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 381 (ObjNum 656 snap 196 seq_num 656) dirty exists 2024-09-21T08:31:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:1931: finishing write tid 1 to smithi13779221-40 2024-09-21T08:31:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:1931: finishing write tid 2 to smithi13779221-40 2024-09-21T08:31:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:1928: done (4 left) 2024-09-21T08:31:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:1929: done (3 left) 2024-09-21T08:31:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:1930: done (2 left) 2024-09-21T08:31:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:1933: write (excl) oid 2 current snap is 209 2024-09-21T08:31:20.684 INFO:tasks.rados.rados.0.smithi137.stdout:1933: seq_num 684 ranges {695988=625733,2009801=680561,3150974=345014} 2024-09-21T08:31:20.689 INFO:tasks.rados.rados.0.smithi137.stdout:1933: writing smithi13779221-2 from 695988 to 1321721 tid 1 2024-09-21T08:31:20.697 INFO:tasks.rados.rados.0.smithi137.stdout:1933: writing smithi13779221-2 from 2009801 to 2690362 tid 2 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:1933: writing smithi13779221-2 from 3150974 to 3495988 tid 3 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:1934: delete oid 30 current snap is 209 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:1931: finishing write tid 3 to smithi13779221-40 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:1931: finishing write tid 4 to smithi13779221-40 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:1931: finishing write tid 5 to smithi13779221-40 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:1931: oid 40 updating version 0 to 342 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:1931: oid 40 version 342 is already newer than 341 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:1931: oid 40 version 342 is already newer than 340 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:1931: oid 40 updating version 342 to 343 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 343 (ObjNum 683 snap 209 seq_num 683) dirty exists 2024-09-21T08:31:20.702 INFO:tasks.rados.rados.0.smithi137.stdout:1931: left oid 40 (ObjNum 683 snap 209 seq_num 683) 2024-09-21T08:31:20.703 INFO:tasks.rados.rados.0.smithi137.stdout:1932: finishing copy_from racing read to smithi13779221-7 2024-09-21T08:31:20.703 INFO:tasks.rados.rados.0.smithi137.stdout:1932: finishing copy_from to smithi13779221-7 2024-09-21T08:31:20.703 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 237 (ObjNum 670 snap 203 seq_num 670) dirty exists 2024-09-21T08:31:20.703 INFO:tasks.rados.rados.0.smithi137.stdout:1933: finishing write tid 1 to smithi13779221-2 2024-09-21T08:31:20.703 INFO:tasks.rados.rados.0.smithi137.stdout:1933: finishing write tid 2 to smithi13779221-2 2024-09-21T08:31:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:1931: done (3 left) 2024-09-21T08:31:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:1932: done (2 left) 2024-09-21T08:31:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:1934: done (1 left) 2024-09-21T08:31:20.704 INFO:tasks.rados.rados.0.smithi137.stdout:1935: snap_remove snap 194 2024-09-21T08:31:20.710 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:20.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:20.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:20.732 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:20 smithi137 ceph-mon[29634]: osdmap e486: 8 total, 6 up, 5 in 2024-09-21T08:31:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:20 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2803052051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:20 smithi137 ceph-mon[29634]: osdmap e487: 8 total, 6 up, 5 in 2024-09-21T08:31:20.769 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:31:20 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:31:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:31:20.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:20 smithi137 ceph-mon[41044]: osdmap e486: 8 total, 6 up, 5 in 2024-09-21T08:31:20.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:20 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2803052051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:20.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:20 smithi137 ceph-mon[41044]: osdmap e487: 8 total, 6 up, 5 in 2024-09-21T08:31:20.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:20.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:20.855 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:20.855 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:20.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:20.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:20.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:20.978 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:20 smithi165 ceph-mon[32893]: osdmap e486: 8 total, 6 up, 5 in 2024-09-21T08:31:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:20 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2803052051' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:20 smithi165 ceph-mon[32893]: osdmap e487: 8 total, 6 up, 5 in 2024-09-21T08:31:21.042 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:31:21.078 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:21.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:21.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:21.105 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:21.154 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:21.154 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:21.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:21.227 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:21.227 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:21.227 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:21.328 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:21.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:21.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:21.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:21.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:21.480 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:21.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:21.481 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:21.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:21.594 INFO:tasks.rados.rados.0.smithi137.stdout:1935: done (1 left) 2024-09-21T08:31:21.594 INFO:tasks.rados.rados.0.smithi137.stdout:1936: snap_create 2024-09-21T08:31:21.595 INFO:tasks.rados.rados.0.smithi137.stdout:1933: finishing write tid 3 to smithi13779221-2 2024-09-21T08:31:21.595 INFO:tasks.rados.rados.0.smithi137.stdout:1933: finishing write tid 4 to smithi13779221-2 2024-09-21T08:31:21.595 INFO:tasks.rados.rados.0.smithi137.stdout:1933: finishing write tid 5 to smithi13779221-2 2024-09-21T08:31:21.595 INFO:tasks.rados.rados.0.smithi137.stdout:1933: oid 2 updating version 0 to 177 2024-09-21T08:31:21.595 INFO:tasks.rados.rados.0.smithi137.stdout:1933: oid 2 version 177 is already newer than 175 2024-09-21T08:31:21.595 INFO:tasks.rados.rados.0.smithi137.stdout:1933: oid 2 version 177 is already newer than 174 2024-09-21T08:31:21.595 INFO:tasks.rados.rados.0.smithi137.stdout:1933: oid 2 version 177 is already newer than 176 2024-09-21T08:31:21.595 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 177 (ObjNum 684 snap 209 seq_num 684) dirty exists 2024-09-21T08:31:21.595 INFO:tasks.rados.rados.0.smithi137.stdout:1933: left oid 2 (ObjNum 684 snap 209 seq_num 684) 2024-09-21T08:31:21.610 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:21.611 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:21.611 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:21.712 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:21.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:21.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:21.735 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:21 smithi137 ceph-mon[29634]: pgmap v635: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 566 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 237/1443 objects degraded (16.424%); 111/1443 objects misplaced (7.692%) 2024-09-21T08:31:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:21 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:21 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 237/1443 objects degraded (16.424%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:21.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:21 smithi137 ceph-mon[29634]: osdmap e488: 8 total, 6 up, 5 in 2024-09-21T08:31:21.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:21 smithi137 ceph-mon[41044]: pgmap v635: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 566 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 237/1443 objects degraded (16.424%); 111/1443 objects misplaced (7.692%) 2024-09-21T08:31:21.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:21 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:21.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:21 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 237/1443 objects degraded (16.424%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:21.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:21 smithi137 ceph-mon[41044]: osdmap e488: 8 total, 6 up, 5 in 2024-09-21T08:31:21.829 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T08:31:21.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:21.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:21.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:21.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:21.888 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:21.928 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:21.963 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:21.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:21 smithi165 ceph-mon[32893]: pgmap v635: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 566 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 237/1443 objects degraded (16.424%); 111/1443 objects misplaced (7.692%) 2024-09-21T08:31:21.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:21 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:21.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:21 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 237/1443 objects degraded (16.424%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:21.975 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:21 smithi165 ceph-mon[32893]: osdmap e488: 8 total, 6 up, 5 in 2024-09-21T08:31:21.984 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:21.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:21.985 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:22.086 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:22.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:22.113 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:22.113 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:22.214 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:22.239 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:22.239 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:22.240 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:22.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:31:21 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:31:22.340 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:22.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:22.365 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:22.365 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:22.466 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:22.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:22.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:22.493 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:22.530 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:31:22.530 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:31:22.530 INFO:teuthology.orchestra.run.smithi137.stdout: "epoch": 488, 2024-09-21T08:31:22.530 INFO:teuthology.orchestra.run.smithi137.stdout: "fsid": "ec8566d6-77f1-11ef-baf6-efdc52797490", 2024-09-21T08:31:22.530 INFO:teuthology.orchestra.run.smithi137.stdout: "created": "2024-09-21T08:18:21.794737+0000", 2024-09-21T08:31:22.530 INFO:teuthology.orchestra.run.smithi137.stdout: "modified": "2024-09-21T08:31:21.590160+0000", 2024-09-21T08:31:22.530 INFO:teuthology.orchestra.run.smithi137.stdout: "last_up_change": "2024-09-21T08:29:23.558248+0000", 2024-09-21T08:31:22.530 INFO:teuthology.orchestra.run.smithi137.stdout: "last_in_change": "2024-09-21T08:30:55.587299+0000", 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": "noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_num": 5806080, 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_set": [ 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "nodeep-scrub", 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "noscrub", 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "pglog_hardlimit", 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "purged_snapdirs", 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "recovery_deletes", 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "sortbitwise" 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_version": 18, 2024-09-21T08:31:22.531 INFO:teuthology.orchestra.run.smithi137.stdout: "full_ratio": 0.94999998807907104, 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "backfillfull_ratio": 0.89999997615814209, 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "nearfull_ratio": 0.85000002384185791, 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_snapshot": "", 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_max": 7, 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "max_osd": 8, 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "require_min_compat_client": "luminous", 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "min_compat_client": "luminous", 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "require_osd_release": "squid", 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "allow_crimson": false, 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "pools": [ 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 1, 2024-09-21T08:31:22.532 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".mgr", 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:21:15.037508+0000", 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:31:22.533 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 11, 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 11, 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 11, 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 11, 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 11, 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:31:22.534 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "457", 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "455", 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:31:22.535 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:31:22.536 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:31:22.537 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_max": 32, 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_min": 1 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: "mgr": {} 2024-09-21T08:31:22.538 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.6400001049041748, 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.6400001049041748, 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.8200000524520874, 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.8200000524520874, 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:31:22.539 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 2, 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "datapool", 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:28.246186+0000", 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:31:22.540 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 3, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 3, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 3, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 3, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 3, 2024-09-21T08:31:22.541 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "68", 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:31:22.542 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 2, 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 68, 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:31:22.543 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.544 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:31:22.545 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "rbd": {} 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 5.3400001525878906, 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 5.3400001525878906, 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 2.6700000762939453, 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 2.6700000762939453, 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:31:22.546 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 3, 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": ".rgw.root", 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:29.209879+0000", 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:31:22.547 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:31:22.548 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "67", 2024-09-21T08:31:22.549 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:31:22.550 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:31:22.551 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:31:22.552 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.2599999904632568, 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.2599999904632568, 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:31:22.553 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.6299999952316284, 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.6299999952316284, 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 4, 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.log", 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:30.851833+0000", 2024-09-21T08:31:22.554 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:31:22.555 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:31:22.556 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "69", 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:31:22.557 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:31:22.558 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:31:22.559 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 3.7599999904632568, 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 3.7599999904632568, 2024-09-21T08:31:22.560 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.8799999952316284, 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.8799999952316284, 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 5, 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.control", 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:32.798630+0000", 2024-09-21T08:31:22.561 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:31:22.562 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:31:22.563 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "71", 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:31:22.564 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:31:22.565 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:31:22.566 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.567 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:31:22.568 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:31:22.568 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.7599999904632568, 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.7599999904632568, 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.3799999952316284, 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.3799999952316284, 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 6, 2024-09-21T08:31:22.569 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "default.rgw.meta", 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:23:34.858938+0000", 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 1, 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool", 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:31:22.570 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 32, 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 32, 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 32, 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 32, 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 32, 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:31:22.571 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "73", 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "0", 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:31:22.572 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 0, 2024-09-21T08:31:22.573 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 0, 2024-09-21T08:31:22.573 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:31:22.573 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:31:22.573 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:31:22.573 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:31:22.573 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:31:22.573 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:31:22.573 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:31:22.573 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:31:22.574 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:31:22.575 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: "options": { 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_bias": 4 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: "rgw": {} 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.7599999904632568, 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.7599999904632568, 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:31:22.576 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.3799999952316284, 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.3799999952316284, 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_name": "unique_pool_0", 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: "create_time": "2024-09-21T08:24:33.441067+0000", 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: "flags": 8193, 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: "flags_names": "hashpspool,selfmanaged_snaps", 2024-09-21T08:31:22.577 INFO:teuthology.orchestra.run.smithi137.stdout: "type": 1, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 3, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "min_size": 2, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_rule": 0, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_count": 0, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_target": 0, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_barrier": 0, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "peering_crush_bucket_mandatory_member": 2147483647, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "is_stretch_pool": false, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "object_hash": 2, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_autoscale_mode": "off", 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num": 46, 2024-09-21T08:31:22.578 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num": 46, 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_placement_num_target": 46, 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_target": 46, 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_num_pending": 46, 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "last_pg_merge_meta": { 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "source_pgid": "0.0", 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "ready_epoch": 0, 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_started": 0, 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "last_epoch_clean": 0, 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "source_version": "0'0", 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "target_version": "0'0" 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.579 INFO:teuthology.orchestra.run.smithi137.stdout: "last_change": "488", 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend": "0", 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_prenautilus": "0", 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "last_force_op_resend_preluminous": "473", 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "auid": 0, 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_mode": "selfmanaged", 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_seq": 210, 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "snap_epoch": 488, 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "pool_snaps": [], 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps": "[]", 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_bytes": 0, 2024-09-21T08:31:22.580 INFO:teuthology.orchestra.run.smithi137.stdout: "quota_max_objects": 0, 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "tiers": [], 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "tier_of": -1, 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "read_tier": -1, 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "write_tier": -1, 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_mode": "none", 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_bytes": 0, 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "target_max_objects": 0, 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_ratio_micro": 400000, 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_target_full_ratio_micro": 800000, 2024-09-21T08:31:22.581 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_flush_age": 0, 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: "cache_min_evict_age": 0, 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profile": "", 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_params": { 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "none" 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_period": 0, 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_count": 0, 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: "use_gmt_hitset": true, 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: "min_read_recency_for_promote": 0, 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: "min_write_recency_for_promote": 0, 2024-09-21T08:31:22.582 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_grade_decay_rate": 0, 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: "hit_set_search_last_n": 0, 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: "grade_table": [], 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: "stripe_width": 0, 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: "expected_num_objects": 0, 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: "fast_read": false, 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: "options": {}, 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: "application_metadata": { 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: "rados": {} 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: "read_balance": { 2024-09-21T08:31:22.583 INFO:teuthology.orchestra.run.smithi137.stdout: "score_type": "Fair distribution", 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: "score_acting": 2.9600000381469727, 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: "score_stable": 2.9600000381469727, 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: "optimal_score": 0.5, 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_acting": 1.4800000190734863, 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: "raw_score_stable": 1.4800000190734863, 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity_weighted": 0.81856918334960938, 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity": 0.81999999284744263, 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: "average_primary_affinity_weighted": 0.81999999284744263 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:31:22.584 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.585 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.585 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-21T08:31:22.585 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "1b007d1d-6c6b-4a68-9e68-2c5fd7d26977", 2024-09-21T08:31:22.585 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 0, 2024-09-21T08:31:22.585 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:31:22.585 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:31:22.585 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 8, 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 317, 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 370, 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.586 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6801", 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6802", 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.587 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6804", 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:31:22.588 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6803", 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3938795530 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6801/3938795530", 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6802/3938795530", 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6804/3938795530", 2024-09-21T08:31:22.589 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6803/3938795530", 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: "exists" 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "8651068e-694f-4bd4-9917-5c55670aca5d", 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:31:22.590 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 13, 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 478, 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6805", 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.591 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6806", 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:31:22.592 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.593 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.593 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6808", 2024-09-21T08:31:22.593 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:31:22.593 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.593 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.593 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.593 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:31:22.593 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.593 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6807", 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 2743268361 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6805/2743268361", 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6806/2743268361", 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6808/2743268361", 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6807/2743268361", 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:31:22.594 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "2a22f0f7-c610-4995-98a7-d4785efb1e5d", 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0.3052520751953125, 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:31:22.595 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 18, 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 475, 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6809", 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.596 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6810", 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.597 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6812", 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.598 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6811", 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1500397557 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6809/1500397557", 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6810/1500397557", 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6812/1500397557", 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6811/1500397557", 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:31:22.599 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "73f21606-38be-4578-a86b-bb417217c0e0", 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 25, 2024-09-21T08:31:22.600 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 243, 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6813", 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.601 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6814", 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.602 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6816", 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.137:6815", 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 617155021 2024-09-21T08:31:22.603 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.137:6813/617155021", 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.137:6814/617155021", 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.137:6816/617155021", 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.137:6815/617155021", 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.604 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "e5f6762b-65f5-4dc7-8e54-9087149ae6ae", 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 31, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 478, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:31:22.605 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6800", 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.606 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6801", 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6803", 2024-09-21T08:31:22.607 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6802", 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 70360958 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.608 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6800/70360958", 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6801/70360958", 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6803/70360958", 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6802/70360958", 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.609 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "f7f8c456-b77f-4435-a860-202e64dda8dd", 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 1, 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 41, 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 478, 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:31:22.610 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6804", 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.611 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6805", 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6807", 2024-09-21T08:31:22.612 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6806", 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1835779380 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.613 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6804/1835779380", 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6805/1835779380", 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6807/1835779380", 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6806/1835779380", 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-21T08:31:22.614 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "b8b3df16-73fa-41a5-a93f-117c599d488b", 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 1, 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 0, 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0, 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 1, 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 49, 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 94, 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 0, 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:31:22.615 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6808", 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.616 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6809", 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6811", 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:31:22.617 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6810", 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 3820354208 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.618 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.619 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6808/3820354208", 2024-09-21T08:31:22.619 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6809/3820354208", 2024-09-21T08:31:22.619 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6811/3820354208", 2024-09-21T08:31:22.619 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6810/3820354208", 2024-09-21T08:31:22.619 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:31:22.619 INFO:teuthology.orchestra.run.smithi137.stdout: "exists", 2024-09-21T08:31:22.619 INFO:teuthology.orchestra.run.smithi137.stdout: "up" 2024-09-21T08:31:22.619 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.619 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.619 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "uuid": "ddf234e4-1494-40bd-a31b-333b01717c3f", 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "up": 0, 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "in": 1, 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "weight": 0.9804229736328125, 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_affinity": 0.2433013916015625, 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_begin": 0, 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "last_clean_end": 0, 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "up_from": 59, 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "up_thru": 323, 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "down_at": 339, 2024-09-21T08:31:22.620 INFO:teuthology.orchestra.run.smithi137.stdout: "lost_at": 0, 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addrs": { 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6812", 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addrs": { 2024-09-21T08:31:22.621 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.622 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.622 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6813", 2024-09-21T08:31:22.622 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:31:22.622 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.622 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.622 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.622 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addrs": { 2024-09-21T08:31:22.622 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.622 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.623 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.623 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6815", 2024-09-21T08:31:22.623 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:31:22.623 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.623 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.623 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.623 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addrs": { 2024-09-21T08:31:22.623 INFO:teuthology.orchestra.run.smithi137.stdout: "addrvec": [ 2024-09-21T08:31:22.623 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.624 INFO:teuthology.orchestra.run.smithi137.stdout: "type": "v1", 2024-09-21T08:31:22.624 INFO:teuthology.orchestra.run.smithi137.stdout: "addr": "172.21.15.165:6814", 2024-09-21T08:31:22.624 INFO:teuthology.orchestra.run.smithi137.stdout: "nonce": 1933524273 2024-09-21T08:31:22.624 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.624 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.624 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.624 INFO:teuthology.orchestra.run.smithi137.stdout: "public_addr": "172.21.15.165:6812/1933524273", 2024-09-21T08:31:22.624 INFO:teuthology.orchestra.run.smithi137.stdout: "cluster_addr": "172.21.15.165:6813/1933524273", 2024-09-21T08:31:22.624 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_back_addr": "172.21.15.165:6815/1933524273", 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: "heartbeat_front_addr": "172.21.15.165:6814/1933524273", 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: "state": [ 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: "exists" 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: "osd_xinfo": [ 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 0, 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:31:22.625 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:28:33.169954+0000", 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 369 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 1, 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:31:22.626 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:31:21.021760+0000", 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 2, 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:31:22.627 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:31:22.628 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:30:40.190958+0000", 2024-09-21T08:31:22.628 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:31:22.628 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.628 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.628 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 3, 2024-09-21T08:31:22.628 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:31:22.628 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:31:22.628 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:31:22.628 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:31:22.628 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:30:39.895217+0000", 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 4, 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:30:10.561114+0000", 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:31:22.629 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 5, 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:30:17.716787+0000", 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 6, 2024-09-21T08:31:22.630 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 65536, 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:30:22.018855+0000", 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 0 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "osd": 7, 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "down_stamp": "0.000000", 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_probability": 0, 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "laggy_interval": 0, 2024-09-21T08:31:22.631 INFO:teuthology.orchestra.run.smithi137.stdout: "features": 4540701547738038271, 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: "old_weight": 0, 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: "last_purged_snaps_scrub": "2024-09-21T08:28:07.349774+0000", 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: "dead_epoch": 338 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap": [ 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.10", 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.632 INFO:teuthology.orchestra.run.smithi137.stdout: 3, 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: 7 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.1d", 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: 3, 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.633 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_items": [ 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.c", 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: "mappings": [ 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: "from": 7, 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: "to": 5 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.634 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_upmap_primaries": [], 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: "pg_temp": [ 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "1.4", 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.635 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.636 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "1.6", 2024-09-21T08:31:22.636 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.636 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.636 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.636 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.636 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.636 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.636 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.636 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "1.7", 2024-09-21T08:31:22.636 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "1.9", 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.637 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.1", 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.638 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.6", 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.7", 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.639 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.640 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.640 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.640 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.640 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.640 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.8", 2024-09-21T08:31:22.640 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.640 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.640 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.640 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.640 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.9", 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.b", 2024-09-21T08:31:22.641 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.d", 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.642 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.e", 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.643 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.f", 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.11", 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.644 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.13", 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.645 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.14", 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.15", 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.646 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.19", 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.647 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "3.1b", 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.1", 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.648 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.5", 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.649 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.650 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.8", 2024-09-21T08:31:22.650 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.650 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.650 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.650 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.650 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.650 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.650 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.650 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.9", 2024-09-21T08:31:22.650 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.c", 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.651 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "4.f", 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.1", 2024-09-21T08:31:22.652 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.5", 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.653 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.9", 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.c", 2024-09-21T08:31:22.654 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.10", 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.655 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.11", 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.1a", 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.656 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.1b", 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.657 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.1c", 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "5.1f", 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.658 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.2", 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.659 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.6", 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.b", 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.660 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.12", 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.661 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.14", 2024-09-21T08:31:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.15", 2024-09-21T08:31:22.662 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.16", 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.663 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "6.1e", 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.664 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.1", 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.3", 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.665 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.7", 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.8", 2024-09-21T08:31:22.666 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.9", 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.667 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.f", 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.16", 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.668 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.1a", 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.21", 2024-09-21T08:31:22.669 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.22", 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.670 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.25", 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: 1 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.671 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.27", 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.28", 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.672 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.29", 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: 5, 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: 4 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.673 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.674 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.674 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:22.675 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.2a", 2024-09-21T08:31:22.675 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.675 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.675 INFO:teuthology.orchestra.run.smithi137.stdout: 4, 2024-09-21T08:31:22.676 INFO:teuthology.orchestra.run.smithi137.stdout: 2 2024-09-21T08:31:22.676 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.676 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.676 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.676 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.2b", 2024-09-21T08:31:22.676 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.676 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.676 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.676 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.676 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: "pgid": "7.2c", 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: "osds": [ 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: 1, 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: 2, 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: 5 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: "primary_temp": [], 2024-09-21T08:31:22.677 INFO:teuthology.orchestra.run.smithi137.stdout: "blocklist": { 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1478868457": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1738156348": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/2643910317": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/705906958": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3936900881": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/223603733": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/1718301516": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/2580000250": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3166961425": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/700535732": "2024-09-22T08:18:48.673163+0000", 2024-09-21T08:31:22.678 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1900990534": "2024-09-22T08:19:12.726560+0000", 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:6800/1825289826": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3010106803": "2024-09-22T08:23:59.682278+0000", 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: "172.21.15.137:0/3442439903": "2024-09-22T08:23:59.682278+0000" 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: "range_blocklist": {}, 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: "erasure_code_profiles": { 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: "default": { 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: "crush-failure-domain": "osd", 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: "k": "2", 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: "m": "1", 2024-09-21T08:31:22.679 INFO:teuthology.orchestra.run.smithi137.stdout: "plugin": "jerasure", 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: "technique": "reed_sol_van" 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: "removed_snaps_queue": [ 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 105, 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.680 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 110, 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 22 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 133, 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 13 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 147, 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 8 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.681 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 156, 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 9 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 166, 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 9 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 177, 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.682 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 180, 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 4 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 186, 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 4 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 191, 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.683 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 195, 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 2 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 198, 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 2 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 205, 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.684 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: "new_removed_snaps": [ 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: "pool": 7, 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: "snaps": [ 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: "begin": 196, 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: "length": 1 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.685 INFO:teuthology.orchestra.run.smithi137.stdout: ], 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout: "new_purged_snaps": [], 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout: "crush_node_flags": {}, 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout: "device_class_flags": {}, 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode": { 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_enabled": false, 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_bucket_count": 0, 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout: "degraded_stretch_mode": 0, 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout: "recovering_stretch_mode": 0, 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout: "stretch_mode_bucket": 0 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:31:22.686 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:31:22.688 INFO:tasks.rados.rados.0.smithi137.stdout:1933: done (1 left) 2024-09-21T08:31:22.688 INFO:tasks.rados.rados.0.smithi137.stdout:1936: done (0 left) 2024-09-21T08:31:22.688 INFO:tasks.rados.rados.0.smithi137.stdout:1937: delete oid 12 current snap is 210 2024-09-21T08:31:22.688 INFO:tasks.rados.rados.0.smithi137.stdout:1937: done (0 left) 2024-09-21T08:31:22.688 INFO:tasks.rados.rados.0.smithi137.stdout:1938: read oid 5 snap -1 2024-09-21T08:31:22.688 INFO:tasks.rados.rados.0.smithi137.stdout:1938: expect (ObjNum 653 snap 193 seq_num 653) 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1939: delete oid 27 current snap is 210 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1939: done (1 left) 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1940: copy_from oid 31 from oid 14 current snap is 210 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1941: write oid 25 current snap is 210 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1941: seq_num 688 ranges {763216=643082,1843729=687899,3260666=302550} 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1941: writing smithi13779221-25 from 763216 to 1406298 tid 1 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1941: writing smithi13779221-25 from 1843729 to 2531628 tid 2 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1941: writing smithi13779221-25 from 3260666 to 3563216 tid 3 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1942: write (excl) oid 13 current snap is 210 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1942: seq_num 689 ranges {780023=718331,1902211=547910,3004442=575581} 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1942: writing smithi13779221-13 from 780023 to 1498354 tid 1 2024-09-21T08:31:22.689 INFO:tasks.rados.rados.0.smithi137.stdout:1942: writing smithi13779221-13 from 1902211 to 2450121 tid 2 2024-09-21T08:31:22.690 INFO:tasks.rados.rados.0.smithi137.stdout:1942: writing smithi13779221-13 from 3004442 to 3580023 tid 3 2024-09-21T08:31:22.690 INFO:tasks.rados.rados.0.smithi137.stdout:1943: write (excl) oid 17 current snap is 210 2024-09-21T08:31:22.690 INFO:tasks.rados.rados.0.smithi137.stdout:1938: expect (ObjNum 653 snap 193 seq_num 653) 2024-09-21T08:31:22.697 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:22.698 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:22.698 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:22.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:22.820 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:22.821 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:22.821 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:22.867 INFO:tasks.rados.rados.0.smithi137.stdout:1941: finishing write tid 1 to smithi13779221-25 2024-09-21T08:31:22.867 INFO:tasks.rados.rados.0.smithi137.stdout:1940: finishing copy_from racing read to smithi13779221-31 2024-09-21T08:31:22.867 INFO:tasks.rados.rados.0.smithi137.stdout:1940: finishing copy_from to smithi13779221-31 2024-09-21T08:31:22.867 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 400 (ObjNum 619 snap 182 seq_num 619) dirty exists 2024-09-21T08:31:22.867 INFO:tasks.rados.rados.0.smithi137.stdout:1941: finishing write tid 2 to smithi13779221-25 2024-09-21T08:31:22.868 INFO:tasks.rados.rados.0.smithi137.stdout:1943: seq_num 690 ranges {796830=793580,2360693=691568} 2024-09-21T08:31:22.875 INFO:tasks.rados.rados.0.smithi137.stdout:1943: writing smithi13779221-17 from 796830 to 1590410 tid 1 2024-09-21T08:31:22.884 INFO:tasks.rados.rados.0.smithi137.stdout:1943: writing smithi13779221-17 from 2360693 to 3052261 tid 2 2024-09-21T08:31:22.884 INFO:tasks.rados.rados.0.smithi137.stdout:1938: done (4 left) 2024-09-21T08:31:22.884 INFO:tasks.rados.rados.0.smithi137.stdout:1940: done (3 left) 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1944: snap_create 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1941: finishing write tid 3 to smithi13779221-25 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1941: finishing write tid 4 to smithi13779221-25 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1941: finishing write tid 5 to smithi13779221-25 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1941: oid 25 updating version 0 to 352 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1941: oid 25 version 352 is already newer than 351 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1941: oid 25 updating version 352 to 353 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1941: oid 25 updating version 353 to 354 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 25 v 354 (ObjNum 688 snap 210 seq_num 688) dirty exists 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1941: left oid 25 (ObjNum 688 snap 210 seq_num 688) 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1942: finishing write tid 1 to smithi13779221-13 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1942: finishing write tid 2 to smithi13779221-13 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1942: finishing write tid 3 to smithi13779221-13 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1942: finishing write tid 4 to smithi13779221-13 2024-09-21T08:31:22.885 INFO:tasks.rados.rados.0.smithi137.stdout:1942: finishing write tid 5 to smithi13779221-13 2024-09-21T08:31:22.886 INFO:tasks.rados.rados.0.smithi137.stdout:1942: oid 13 updating version 0 to 818 2024-09-21T08:31:22.886 INFO:tasks.rados.rados.0.smithi137.stdout:1942: oid 13 version 818 is already newer than 816 2024-09-21T08:31:22.886 INFO:tasks.rados.rados.0.smithi137.stdout:1942: oid 13 version 818 is already newer than 817 2024-09-21T08:31:22.886 INFO:tasks.rados.rados.0.smithi137.stdout:1942: oid 13 updating version 818 to 819 2024-09-21T08:31:22.886 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 819 (ObjNum 689 snap 210 seq_num 689) dirty exists 2024-09-21T08:31:22.886 INFO:tasks.rados.rados.0.smithi137.stdout:1942: left oid 13 (ObjNum 689 snap 210 seq_num 689) 2024-09-21T08:31:22.886 INFO:tasks.rados.rados.0.smithi137.stdout:1943: finishing write tid 1 to smithi13779221-17 2024-09-21T08:31:22.886 INFO:tasks.rados.rados.0.smithi137.stdout:1943: finishing write tid 2 to smithi13779221-17 2024-09-21T08:31:22.887 INFO:tasks.rados.rados.0.smithi137.stdout:1943: finishing write tid 3 to smithi13779221-17 2024-09-21T08:31:22.887 INFO:tasks.rados.rados.0.smithi137.stdout:1943: finishing write tid 4 to smithi13779221-17 2024-09-21T08:31:22.887 INFO:tasks.rados.rados.0.smithi137.stdout:1943: oid 17 updating version 0 to 738 2024-09-21T08:31:22.887 INFO:tasks.rados.rados.0.smithi137.stdout:1943: oid 17 version 738 is already newer than 736 2024-09-21T08:31:22.888 INFO:tasks.rados.rados.0.smithi137.stdout:1943: oid 17 version 738 is already newer than 737 2024-09-21T08:31:22.888 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 738 (ObjNum 690 snap 210 seq_num 690) dirty exists 2024-09-21T08:31:22.888 INFO:tasks.rados.rados.0.smithi137.stdout:1943: left oid 17 (ObjNum 690 snap 210 seq_num 690) 2024-09-21T08:31:22.922 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:22.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:22.946 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:22.946 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:22 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:31:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:22 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:31:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:22 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1040308314' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:31:23.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:22 smithi165 ceph-mon[32893]: osdmap e489: 8 total, 6 up, 5 in 2024-09-21T08:31:23.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:22 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:31:23.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:22 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:31:23.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:22 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1040308314' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:31:23.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:22 smithi137 ceph-mon[29634]: osdmap e489: 8 total, 6 up, 5 in 2024-09-21T08:31:23.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:22 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:31:23.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:22 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:31:23.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:22 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1040308314' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json-pretty"}]: dispatch 2024-09-21T08:31:23.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:22 smithi137 ceph-mon[41044]: osdmap e489: 8 total, 6 up, 5 in 2024-09-21T08:31:23.048 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:23.070 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:23.071 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:23.071 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:23.155 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:31:23.172 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:23.219 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:23.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:23.220 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:23.321 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:23.347 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:23.347 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:23.348 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:23.449 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:23.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:23.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:23.471 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:23.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:23.592 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:23.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:23.593 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:23.674 INFO:tasks.rados.rados.0.smithi137.stdout:1941: done (3 left) 2024-09-21T08:31:23.674 INFO:tasks.rados.rados.0.smithi137.stdout:1942: done (2 left) 2024-09-21T08:31:23.674 INFO:tasks.rados.rados.0.smithi137.stdout:1943: done (1 left) 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:1944: done (0 left) 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:1945: read oid 21 snap 204 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:1945: expect deleted 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:1946: read oid 45 snap -1 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:1946: expect (ObjNum 678 snap 205 seq_num 678) 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:1947: rollback oid 16 current snap is 211 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 16 to 163 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:1948: read oid 5 snap -1 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:1948: expect (ObjNum 653 snap 193 seq_num 653) 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:1949: write oid 41 current snap is 211 2024-09-21T08:31:23.675 INFO:tasks.rados.rados.0.smithi137.stdout:1949: seq_num 691 ranges {413637=468829,1619175=435226,2542935=758255} 2024-09-21T08:31:23.679 INFO:tasks.rados.rados.0.smithi137.stdout:1949: writing smithi13779221-41 from 413637 to 882466 tid 1 2024-09-21T08:31:23.687 INFO:tasks.rados.rados.0.smithi137.stdout:1949: writing smithi13779221-41 from 1619175 to 2054401 tid 2 2024-09-21T08:31:23.694 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:23.695 INFO:tasks.rados.rados.0.smithi137.stdout:1949: writing smithi13779221-41 from 2542935 to 3301190 tid 3 2024-09-21T08:31:23.696 INFO:tasks.rados.rados.0.smithi137.stdout:1950: read oid 19 snap -1 2024-09-21T08:31:23.696 INFO:tasks.rados.rados.0.smithi137.stdout:1950: expect deleted 2024-09-21T08:31:23.696 INFO:tasks.rados.rados.0.smithi137.stdout:1951: snap_create 2024-09-21T08:31:23.696 INFO:tasks.rados.rados.0.smithi137.stdout:1947: finishing rollback tid 0 to smithi13779221-16 2024-09-21T08:31:23.696 INFO:tasks.rados.rados.0.smithi137.stdout:1947: finishing rollback tid 1 to smithi13779221-16 2024-09-21T08:31:23.696 INFO:tasks.rados.rados.0.smithi137.stdout:1948: expect (ObjNum 653 snap 193 seq_num 653) 2024-09-21T08:31:23.719 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:23.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:23.720 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:23.731 INFO:tasks.thrashosds.thrasher:j is {'epoch': 488, 'fsid': 'ec8566d6-77f1-11ef-baf6-efdc52797490', 'created': '2024-09-21T08:18:21.794737+0000', 'modified': '2024-09-21T08:31:21.590160+0000', 'last_up_change': '2024-09-21T08:29:23.558248+0000', 'last_in_change': '2024-09-21T08:30:55.587299+0000', 'flags': 'noscrub,nodeep-scrub,sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit', 'flags_num': 5806080, 'flags_set': ['nodeep-scrub', 'noscrub', 'pglog_hardlimit', 'purged_snapdirs', 'recovery_deletes', 'sortbitwise'], 'crush_version': 18, 'full_ratio': 0.949999988079071, 'backfillfull_ratio': 0.8999999761581421, 'nearfull_ratio': 0.8500000238418579, 'cluster_snapshot': '', 'pool_max': 7, 'max_osd': 8, 'require_min_compat_client': 'luminous', 'min_compat_client': 'luminous', 'require_osd_release': 'squid', 'allow_crimson': False, 'pools': [{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-09-21T08:21:15.037508+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 11, 'pg_placement_num': 11, 'pg_placement_num_target': 11, 'pg_num_target': 11, 'pg_num_pending': 11, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '457', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '455', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.640000104904175, 'score_stable': 3.640000104904175, 'optimal_score': 0.5, 'raw_score_acting': 1.8200000524520874, 'raw_score_stable': 1.8200000524520874, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 2, 'pool_name': 'datapool', 'create_time': '2024-09-21T08:23:28.246186+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 3, 'pg_placement_num': 3, 'pg_placement_num_target': 3, 'pg_num_target': 3, 'pg_num_pending': 3, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '68', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 2, 'snap_epoch': 68, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 5.340000152587891, 'score_stable': 5.340000152587891, 'optimal_score': 0.5, 'raw_score_acting': 2.6700000762939453, 'raw_score_stable': 2.6700000762939453, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 3, 'pool_name': '.rgw.root', 'create_time': '2024-09-21T08:23:29.209879+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '67', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.259999990463257, 'score_stable': 3.259999990463257, 'optimal_score': 0.5, 'raw_score_acting': 1.6299999952316284, 'raw_score_stable': 1.6299999952316284, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 4, 'pool_name': 'default.rgw.log', 'create_time': '2024-09-21T08:23:30.851833+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '69', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 3.759999990463257, 'score_stable': 3.759999990463257, 'optimal_score': 0.5, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 5, 'pool_name': 'default.rgw.control', 'create_time': '2024-09-21T08:23:32.798630+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.759999990463257, 'score_stable': 2.759999990463257, 'optimal_score': 0.5, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 6, 'pool_name': 'default.rgw.meta', 'create_time': '2024-09-21T08:23:34.858938+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '73', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4}, 'application_metadata': {'rgw': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.759999990463257, 'score_stable': 2.759999990463257, 'optimal_score': 0.5, 'raw_score_acting': 1.3799999952316284, 'raw_score_stable': 1.3799999952316284, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}, {'pool': 7, 'pool_name': 'unique_pool_0', 'create_time': '2024-09-21T08:24:33.441067+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'is_stretch_pool': False, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 46, 'pg_placement_num': 46, 'pg_placement_num_target': 46, 'pg_num_target': 46, 'pg_num_pending': 46, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '488', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '473', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 210, 'snap_epoch': 488, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rados': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.9600000381469727, 'score_stable': 2.9600000381469727, 'optimal_score': 0.5, 'raw_score_acting': 1.4800000190734863, 'raw_score_stable': 1.4800000190734863, 'primary_affinity_weighted': 0.8185691833496094, 'average_primary_affinity': 0.8199999928474426, 'average_primary_affinity_weighted': 0.8199999928474426}}], 'osds': [{'osd': 0, 'uuid': '1b007d1d-6c6b-4a68-9e68-2c5fd7d26977', 'up': 0, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 8, 'up_thru': 317, 'down_at': 370, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6801', 'nonce': 3938795530}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6802', 'nonce': 3938795530}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6804', 'nonce': 3938795530}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6803', 'nonce': 3938795530}]}, 'public_addr': '172.21.15.137:6801/3938795530', 'cluster_addr': '172.21.15.137:6802/3938795530', 'heartbeat_back_addr': '172.21.15.137:6804/3938795530', 'heartbeat_front_addr': '172.21.15.137:6803/3938795530', 'state': ['exists']}, {'osd': 1, 'uuid': '8651068e-694f-4bd4-9917-5c55670aca5d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 13, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6805', 'nonce': 2743268361}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6806', 'nonce': 2743268361}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6808', 'nonce': 2743268361}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6807', 'nonce': 2743268361}]}, 'public_addr': '172.21.15.137:6805/2743268361', 'cluster_addr': '172.21.15.137:6806/2743268361', 'heartbeat_back_addr': '172.21.15.137:6808/2743268361', 'heartbeat_front_addr': '172.21.15.137:6807/2743268361', 'state': ['exists', 'up']}, {'osd': 2, 'uuid': '2a22f0f7-c610-4995-98a7-d4785efb1e5d', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 0.3052520751953125, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 18, 'up_thru': 475, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6809', 'nonce': 1500397557}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6810', 'nonce': 1500397557}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6812', 'nonce': 1500397557}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6811', 'nonce': 1500397557}]}, 'public_addr': '172.21.15.137:6809/1500397557', 'cluster_addr': '172.21.15.137:6810/1500397557', 'heartbeat_back_addr': '172.21.15.137:6812/1500397557', 'heartbeat_front_addr': '172.21.15.137:6811/1500397557', 'state': ['exists', 'up']}, {'osd': 3, 'uuid': '73f21606-38be-4578-a86b-bb417217c0e0', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 25, 'up_thru': 243, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6813', 'nonce': 617155021}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6814', 'nonce': 617155021}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6816', 'nonce': 617155021}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.137:6815', 'nonce': 617155021}]}, 'public_addr': '172.21.15.137:6813/617155021', 'cluster_addr': '172.21.15.137:6814/617155021', 'heartbeat_back_addr': '172.21.15.137:6816/617155021', 'heartbeat_front_addr': '172.21.15.137:6815/617155021', 'state': ['exists', 'up']}, {'osd': 4, 'uuid': 'e5f6762b-65f5-4dc7-8e54-9087149ae6ae', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 31, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6800', 'nonce': 70360958}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6801', 'nonce': 70360958}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6803', 'nonce': 70360958}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6802', 'nonce': 70360958}]}, 'public_addr': '172.21.15.165:6800/70360958', 'cluster_addr': '172.21.15.165:6801/70360958', 'heartbeat_back_addr': '172.21.15.165:6803/70360958', 'heartbeat_front_addr': '172.21.15.165:6802/70360958', 'state': ['exists', 'up']}, {'osd': 5, 'uuid': 'f7f8c456-b77f-4435-a860-202e64dda8dd', 'up': 1, 'in': 1, 'weight': 1, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 41, 'up_thru': 478, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6804', 'nonce': 1835779380}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6805', 'nonce': 1835779380}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6807', 'nonce': 1835779380}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6806', 'nonce': 1835779380}]}, 'public_addr': '172.21.15.165:6804/1835779380', 'cluster_addr': '172.21.15.165:6805/1835779380', 'heartbeat_back_addr': '172.21.15.165:6807/1835779380', 'heartbeat_front_addr': '172.21.15.165:6806/1835779380', 'state': ['exists', 'up']}, {'osd': 6, 'uuid': 'b8b3df16-73fa-41a5-a93f-117c599d488b', 'up': 1, 'in': 0, 'weight': 0, 'primary_affinity': 1, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 49, 'up_thru': 94, 'down_at': 0, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6808', 'nonce': 3820354208}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6809', 'nonce': 3820354208}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6811', 'nonce': 3820354208}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6810', 'nonce': 3820354208}]}, 'public_addr': '172.21.15.165:6808/3820354208', 'cluster_addr': '172.21.15.165:6809/3820354208', 'heartbeat_back_addr': '172.21.15.165:6811/3820354208', 'heartbeat_front_addr': '172.21.15.165:6810/3820354208', 'state': ['exists', 'up']}, {'osd': 7, 'uuid': 'ddf234e4-1494-40bd-a31b-333b01717c3f', 'up': 0, 'in': 1, 'weight': 0.9804229736328125, 'primary_affinity': 0.2433013916015625, 'last_clean_begin': 0, 'last_clean_end': 0, 'up_from': 59, 'up_thru': 323, 'down_at': 339, 'lost_at': 0, 'public_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6812', 'nonce': 1933524273}]}, 'cluster_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6813', 'nonce': 1933524273}]}, 'heartbeat_back_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6815', 'nonce': 1933524273}]}, 'heartbeat_front_addrs': {'addrvec': [{'type': 'v1', 'addr': '172.21.15.165:6814', 'nonce': 1933524273}]}, 'public_addr': '172.21.15.165:6812/1933524273', 'cluster_addr': '172.21.15.165:6813/1933524273', 'heartbeat_back_addr': '172.21.15.165:6815/1933524273', 'heartbeat_front_addr': '172.21.15.165:6814/1933524273', 'state': ['exists']}], 'osd_xinfo': [{'osd': 0, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:28:33.169954+0000', 'dead_epoch': 369}, {'osd': 1, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:31:21.021760+0000', 'dead_epoch': 0}, {'osd': 2, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:30:40.190958+0000', 'dead_epoch': 0}, {'osd': 3, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:30:39.895217+0000', 'dead_epoch': 0}, {'osd': 4, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:30:10.561114+0000', 'dead_epoch': 0}, {'osd': 5, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:30:17.716787+0000', 'dead_epoch': 0}, {'osd': 6, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 65536, 'last_purged_snaps_scrub': '2024-09-21T08:30:22.018855+0000', 'dead_epoch': 0}, {'osd': 7, 'down_stamp': '0.000000', 'laggy_probability': 0, 'laggy_interval': 0, 'features': 4540701547738038271, 'old_weight': 0, 'last_purged_snaps_scrub': '2024-09-21T08:28:07.349774+0000', 'dead_epoch': 338}], 'pg_upmap': [{'pgid': '7.10', 'osds': [5, 3, 7]}, {'pgid': '7.1d', 'osds': [4, 3, 1]}], 'pg_upmap_items': [{'pgid': '7.c', 'mappings': [{'from': 7, 'to': 5}]}], 'pg_upmap_primaries': [], 'pg_temp': [{'pgid': '1.4', 'osds': [5, 4, 1]}, {'pgid': '1.6', 'osds': [4, 2, 5]}, {'pgid': '1.7', 'osds': [5, 2, 1]}, {'pgid': '1.9', 'osds': [2, 5, 1]}, {'pgid': '3.1', 'osds': [4, 5, 1]}, {'pgid': '3.6', 'osds': [1, 4, 5]}, {'pgid': '3.7', 'osds': [5, 4, 1]}, {'pgid': '3.8', 'osds': [1, 4, 5]}, {'pgid': '3.9', 'osds': [4, 2, 1]}, {'pgid': '3.b', 'osds': [4, 1, 2]}, {'pgid': '3.d', 'osds': [5, 4, 2]}, {'pgid': '3.e', 'osds': [4, 1, 5]}, {'pgid': '3.f', 'osds': [4, 1, 5]}, {'pgid': '3.11', 'osds': [4, 2, 5]}, {'pgid': '3.13', 'osds': [4, 2, 1]}, {'pgid': '3.14', 'osds': [4, 2, 5]}, {'pgid': '3.15', 'osds': [4, 2, 5]}, {'pgid': '3.19', 'osds': [1, 4, 2]}, {'pgid': '3.1b', 'osds': [4, 1, 2]}, {'pgid': '4.1', 'osds': [4, 5, 2]}, {'pgid': '4.5', 'osds': [4, 5, 1]}, {'pgid': '4.8', 'osds': [5, 4, 1]}, {'pgid': '4.9', 'osds': [4, 1, 2]}, {'pgid': '4.c', 'osds': [4, 1, 2]}, {'pgid': '4.f', 'osds': [1, 4, 5]}, {'pgid': '5.1', 'osds': [4, 5, 2]}, {'pgid': '5.5', 'osds': [1, 4, 2]}, {'pgid': '5.9', 'osds': [4, 1, 2]}, {'pgid': '5.c', 'osds': [1, 4, 2]}, {'pgid': '5.10', 'osds': [4, 5, 2]}, {'pgid': '5.11', 'osds': [4, 1, 5]}, {'pgid': '5.1a', 'osds': [4, 1, 5]}, {'pgid': '5.1b', 'osds': [5, 4, 1]}, {'pgid': '5.1c', 'osds': [4, 2, 1]}, {'pgid': '5.1f', 'osds': [4, 2, 1]}, {'pgid': '6.2', 'osds': [4, 2, 5]}, {'pgid': '6.6', 'osds': [4, 5, 1]}, {'pgid': '6.b', 'osds': [1, 4, 5]}, {'pgid': '6.12', 'osds': [4, 2, 1]}, {'pgid': '6.14', 'osds': [2, 4, 5]}, {'pgid': '6.15', 'osds': [4, 5, 1]}, {'pgid': '6.16', 'osds': [4, 5, 1]}, {'pgid': '6.1e', 'osds': [4, 5, 2]}, {'pgid': '7.1', 'osds': [4, 1, 5]}, {'pgid': '7.3', 'osds': [2, 4, 1]}, {'pgid': '7.7', 'osds': [1, 4, 2]}, {'pgid': '7.8', 'osds': [4, 2, 5]}, {'pgid': '7.9', 'osds': [1, 4, 2]}, {'pgid': '7.f', 'osds': [4, 5, 2]}, {'pgid': '7.16', 'osds': [4, 2, 5]}, {'pgid': '7.1a', 'osds': [4, 5, 1]}, {'pgid': '7.21', 'osds': [5, 4, 1]}, {'pgid': '7.22', 'osds': [4, 1, 5]}, {'pgid': '7.25', 'osds': [4, 5, 1]}, {'pgid': '7.27', 'osds': [1, 4, 2]}, {'pgid': '7.28', 'osds': [5, 4, 2]}, {'pgid': '7.29', 'osds': [5, 1, 4]}, {'pgid': '7.2a', 'osds': [1, 4, 2]}, {'pgid': '7.2b', 'osds': [1, 2, 5]}, {'pgid': '7.2c', 'osds': [1, 2, 5]}], 'primary_temp': [], 'blocklist': {'172.21.15.137:0/1478868457': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/1738156348': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/2643910317': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/705906958': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/3936900881': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/223603733': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:0/1718301516': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:6800/2580000250': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:0/3166961425': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/700535732': '2024-09-22T08:18:48.673163+0000', '172.21.15.137:6800/1900990534': '2024-09-22T08:19:12.726560+0000', '172.21.15.137:6800/1825289826': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/3010106803': '2024-09-22T08:23:59.682278+0000', '172.21.15.137:0/3442439903': '2024-09-22T08:23:59.682278+0000'}, 'range_blocklist': {}, 'erasure_code_profiles': {'default': {'crush-failure-domain': 'osd', 'k': '2', 'm': '1', 'plugin': 'jerasure', 'technique': 'reed_sol_van'}}, 'removed_snaps_queue': [{'pool': 7, 'snaps': [{'begin': 105, 'length': 1}, {'begin': 110, 'length': 22}, {'begin': 133, 'length': 13}, {'begin': 147, 'length': 8}, {'begin': 156, 'length': 9}, {'begin': 166, 'length': 9}, {'begin': 177, 'length': 1}, {'begin': 180, 'length': 4}, {'begin': 186, 'length': 4}, {'begin': 191, 'length': 1}, {'begin': 195, 'length': 2}, {'begin': 198, 'length': 2}, {'begin': 205, 'length': 1}]}], 'new_removed_snaps': [{'pool': 7, 'snaps': [{'begin': 196, 'length': 1}]}], 'new_purged_snaps': [], 'crush_node_flags': {}, 'device_class_flags': {}, 'stretch_mode': {'stretch_mode_enabled': False, 'stretch_bucket_count': 0, 'degraded_stretch_mode': 0, 'recovering_stretch_mode': 0, 'stretch_mode_bucket': 0}} 2024-09-21T08:31:23.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:31:23.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:23.844 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:23.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:23.845 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:23.919 INFO:tasks.rados.rados.0.smithi137.stdout:1947: finishing rollback tid 2 to smithi13779221-16 2024-09-21T08:31:23.919 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 504 (ObjNum 468 snap 126 seq_num 468) dirty exists 2024-09-21T08:31:23.919 INFO:tasks.rados.rados.0.smithi137.stdout:1949: finishing write tid 1 to smithi13779221-41 2024-09-21T08:31:23.919 INFO:tasks.rados.rados.0.smithi137.stdout:1949: finishing write tid 2 to smithi13779221-41 2024-09-21T08:31:23.919 INFO:tasks.rados.rados.0.smithi137.stdout:1946: expect (ObjNum 678 snap 205 seq_num 678) 2024-09-21T08:31:23.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:23.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:23.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:23.968 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:23.986 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:23 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:23.986 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:23 smithi137 ceph-mon[29634]: pgmap v638: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 580 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 12 op/s; 238/1458 objects degraded (16.324%); 114/1458 objects misplaced (7.819%) 2024-09-21T08:31:23.986 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:23 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:23.986 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:23 smithi137 ceph-mon[41044]: pgmap v638: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 580 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 12 op/s; 238/1458 objects degraded (16.324%); 114/1458 objects misplaced (7.819%) 2024-09-21T08:31:23.992 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:23 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:23 smithi165 ceph-mon[32893]: pgmap v638: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 580 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 12 op/s; 238/1458 objects degraded (16.324%); 114/1458 objects misplaced (7.819%) 2024-09-21T08:31:24.042 INFO:tasks.rados.rados.0.smithi137.stdout:1949: finishing write tid 3 to smithi13779221-41 2024-09-21T08:31:24.043 INFO:tasks.rados.rados.0.smithi137.stdout:1949: finishing write tid 4 to smithi13779221-41 2024-09-21T08:31:24.043 INFO:tasks.rados.rados.0.smithi137.stdout:1949: finishing write tid 5 to smithi13779221-41 2024-09-21T08:31:24.043 INFO:tasks.rados.rados.0.smithi137.stdout:1949: oid 41 updating version 0 to 406 2024-09-21T08:31:24.043 INFO:tasks.rados.rados.0.smithi137.stdout:1949: oid 41 updating version 406 to 409 2024-09-21T08:31:24.043 INFO:tasks.rados.rados.0.smithi137.stdout:1949: oid 41 version 409 is already newer than 408 2024-09-21T08:31:24.043 INFO:tasks.rados.rados.0.smithi137.stdout:1949: oid 41 version 409 is already newer than 407 2024-09-21T08:31:24.043 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 409 (ObjNum 691 snap 211 seq_num 691) dirty exists 2024-09-21T08:31:24.043 INFO:tasks.rados.rados.0.smithi137.stdout:1949: left oid 41 (ObjNum 691 snap 211 seq_num 691) 2024-09-21T08:31:24.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:24.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:24.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:24.090 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:24.191 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:24.212 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:24.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:24.213 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:24.314 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:24.339 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:24.340 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:24.340 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:24.441 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:24.453 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:24.453 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:24.466 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:24.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:24.467 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:24.484 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:31:24.484 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:31:24.484 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:31:24.484 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:31:24.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:24.593 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:24.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:24.594 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:24.680 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:31:24.681 INFO:tasks.rados.rados.0.smithi137.stdout:1945: done (6 left) 2024-09-21T08:31:24.681 INFO:tasks.rados.rados.0.smithi137.stdout:1946: done (5 left) 2024-09-21T08:31:24.681 INFO:tasks.rados.rados.0.smithi137.stdout:1947: done (4 left) 2024-09-21T08:31:24.681 INFO:tasks.rados.rados.0.smithi137.stdout:1948: done (3 left) 2024-09-21T08:31:24.682 INFO:tasks.rados.rados.0.smithi137.stdout:1949: done (2 left) 2024-09-21T08:31:24.682 INFO:tasks.rados.rados.0.smithi137.stdout:1950: done (1 left) 2024-09-21T08:31:24.682 INFO:tasks.rados.rados.0.smithi137.stdout:1951: done (0 left) 2024-09-21T08:31:24.682 INFO:tasks.rados.rados.0.smithi137.stdout:1952: rollback oid 31 current snap is 212 2024-09-21T08:31:24.682 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 31 to 201 2024-09-21T08:31:24.682 INFO:tasks.rados.rados.0.smithi137.stdout:1953: snap_create 2024-09-21T08:31:24.690 INFO:tasks.rados.rados.0.smithi137.stdout:1952: finishing rollback tid 0 to smithi13779221-31 2024-09-21T08:31:24.692 INFO:tasks.rados.rados.0.smithi137.stdout:1952: finishing rollback tid 1 to smithi13779221-31 2024-09-21T08:31:24.692 INFO:tasks.rados.rados.0.smithi137.stdout:1952: finishing rollback tid 2 to smithi13779221-31 2024-09-21T08:31:24.692 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 403 (ObjNum 664 snap 199 seq_num 664) dirty exists 2024-09-21T08:31:24.695 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:24.719 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:24.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:24.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:24.820 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:24.847 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:24.847 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:24.848 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:24.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:24.970 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:24.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:24.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:24.979 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:24 smithi137 ceph-mon[29634]: osdmap e490: 8 total, 6 up, 5 in 2024-09-21T08:31:24.979 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:24 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2203298890' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:24.980 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:24 smithi137 ceph-mon[29634]: from='client.45893 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:24.980 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:24 smithi137 ceph-mon[41044]: osdmap e490: 8 total, 6 up, 5 in 2024-09-21T08:31:24.980 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:24 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2203298890' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:24.980 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:24 smithi137 ceph-mon[41044]: from='client.45893 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:24 smithi165 ceph-mon[32893]: osdmap e490: 8 total, 6 up, 5 in 2024-09-21T08:31:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:24 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2203298890' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:24 smithi165 ceph-mon[32893]: from='client.45893 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:25.072 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:25.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:25.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:25.095 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:25.161 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:25.169 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:25.169 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:25.186 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:25.186 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:25.196 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:25.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:25.202 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:25.217 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:25.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:25.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:25.219 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:25.219 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:25.236 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:25.236 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:25.253 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:25.253 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:25.253 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:25.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:25.346 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:25.346 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:25.347 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:25.447 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:25.469 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:25.470 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:25.470 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:25.571 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:25.593 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:25.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:25.594 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:25.685 INFO:tasks.rados.rados.0.smithi137.stdout:1952: done (1 left) 2024-09-21T08:31:25.685 INFO:tasks.rados.rados.0.smithi137.stdout:1953: done (0 left) 2024-09-21T08:31:25.685 INFO:tasks.rados.rados.0.smithi137.stdout:1954: write oid 29 current snap is 213 2024-09-21T08:31:25.685 INFO:tasks.rados.rados.0.smithi137.stdout:1954: seq_num 692 ranges {430444=660431,1677657=695237,2970358=569527} 2024-09-21T08:31:25.689 INFO:tasks.rados.rados.0.smithi137.stdout:1954: writing smithi13779221-29 from 430444 to 1090875 tid 1 2024-09-21T08:31:25.695 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:25.696 INFO:tasks.rados.rados.0.smithi137.stdout:1954: writing smithi13779221-29 from 1677657 to 2372894 tid 2 2024-09-21T08:31:25.704 INFO:tasks.rados.rados.0.smithi137.stdout:1954: writing smithi13779221-29 from 2970358 to 3539885 tid 3 2024-09-21T08:31:25.704 INFO:tasks.rados.rados.0.smithi137.stdout:1955: write oid 32 current snap is 213 2024-09-21T08:31:25.704 INFO:tasks.rados.rados.0.smithi137.stdout:1954: finishing write tid 1 to smithi13779221-29 2024-09-21T08:31:25.704 INFO:tasks.rados.rados.0.smithi137.stdout:1954: finishing write tid 2 to smithi13779221-29 2024-09-21T08:31:25.704 INFO:tasks.rados.rados.0.smithi137.stdout:1955: seq_num 693 ranges {447251=735680,1736139=438895,2597781=780799} 2024-09-21T08:31:25.709 INFO:tasks.rados.rados.0.smithi137.stdout:1955: writing smithi13779221-32 from 447251 to 1182931 tid 1 2024-09-21T08:31:25.715 INFO:tasks.rados.rados.0.smithi137.stdout:1955: writing smithi13779221-32 from 1736139 to 2175034 tid 2 2024-09-21T08:31:25.722 INFO:tasks.rados.rados.0.smithi137.stdout:1955: writing smithi13779221-32 from 2597781 to 3378580 tid 3 2024-09-21T08:31:25.722 INFO:tasks.rados.rados.0.smithi137.stdout:1956: write (excl) oid 8 current snap is 213 2024-09-21T08:31:25.722 INFO:tasks.rados.rados.0.smithi137.stdout:1954: finishing write tid 3 to smithi13779221-29 2024-09-21T08:31:25.722 INFO:tasks.rados.rados.0.smithi137.stdout:1956: seq_num 694 ranges {464058=410929,1394621=698906,2625204=592071,3656107=7951} 2024-09-21T08:31:25.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:25.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:25.726 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:25.727 INFO:tasks.rados.rados.0.smithi137.stdout:1956: writing smithi13779221-8 from 464058 to 874987 tid 1 2024-09-21T08:31:25.734 INFO:tasks.rados.rados.0.smithi137.stdout:1956: writing smithi13779221-8 from 1394621 to 2093527 tid 2 2024-09-21T08:31:25.740 INFO:tasks.rados.rados.0.smithi137.stdout:1956: writing smithi13779221-8 from 2625204 to 3217275 tid 3 2024-09-21T08:31:25.743 INFO:tasks.rados.rados.0.smithi137.stdout:1956: writing smithi13779221-8 from 3656107 to 3664058 tid 4 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1957: copy_from oid 10 from oid 42 current snap is 213 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1954: finishing write tid 4 to smithi13779221-29 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1954: finishing write tid 5 to smithi13779221-29 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1954: oid 29 updating version 0 to 806 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1954: oid 29 version 806 is already newer than 805 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1954: oid 29 updating version 806 to 807 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1954: oid 29 updating version 807 to 808 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 808 (ObjNum 692 snap 213 seq_num 692) dirty exists 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1954: left oid 29 (ObjNum 692 snap 213 seq_num 692) 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1955: finishing write tid 1 to smithi13779221-32 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1955: finishing write tid 2 to smithi13779221-32 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1954: done (3 left) 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:1958: rollback oid 15 current snap is 213 2024-09-21T08:31:25.744 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 15 to 192 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1955: finishing write tid 3 to smithi13779221-32 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1955: finishing write tid 4 to smithi13779221-32 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1955: finishing write tid 5 to smithi13779221-32 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1955: oid 32 updating version 0 to 535 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1955: oid 32 version 535 is already newer than 534 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1955: oid 32 version 535 is already newer than 533 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1955: oid 32 version 535 is already newer than 532 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 535 (ObjNum 693 snap 213 seq_num 693) dirty exists 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1955: left oid 32 (ObjNum 693 snap 213 seq_num 693) 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1956: finishing write tid 1 to smithi13779221-8 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1956: finishing write tid 2 to smithi13779221-8 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1955: done (3 left) 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1959: copy_from oid 40 from oid 31 current snap is 213 2024-09-21T08:31:25.745 INFO:tasks.rados.rados.0.smithi137.stdout:1960: rollback oid 50 current snap is 213 2024-09-21T08:31:25.746 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 50 to 209 2024-09-21T08:31:25.746 INFO:tasks.rados.rados.0.smithi137.stdout:1961: read oid 11 snap -1 2024-09-21T08:31:25.746 INFO:tasks.rados.rados.0.smithi137.stdout:1961: expect (ObjNum 623 snap 184 seq_num 623) 2024-09-21T08:31:25.746 INFO:tasks.rados.rados.0.smithi137.stdout:1962: write oid 22 current snap is 213 2024-09-21T08:31:25.746 INFO:tasks.rados.rados.0.smithi137.stdout:1962: seq_num 695 ranges {480865=486178,1453103=442564,2652627=519696} 2024-09-21T08:31:25.748 INFO:tasks.rados.rados.0.smithi137.stdout:1962: writing smithi13779221-22 from 480865 to 967043 tid 1 2024-09-21T08:31:25.754 INFO:tasks.rados.rados.0.smithi137.stdout:1962: writing smithi13779221-22 from 1453103 to 1895667 tid 2 2024-09-21T08:31:25.762 INFO:tasks.rados.rados.0.smithi137.stdout:1962: writing smithi13779221-22 from 2652627 to 3172323 tid 3 2024-09-21T08:31:25.762 INFO:tasks.rados.rados.0.smithi137.stdout:1963: write (excl) oid 23 current snap is 213 2024-09-21T08:31:25.763 INFO:tasks.rados.rados.0.smithi137.stdout:1956: finishing write tid 3 to smithi13779221-8 2024-09-21T08:31:25.763 INFO:tasks.rados.rados.0.smithi137.stdout:1956: finishing write tid 4 to smithi13779221-8 2024-09-21T08:31:25.763 INFO:tasks.rados.rados.0.smithi137.stdout:1963: seq_num 696 ranges {497672=561427,1795232=702575,2963697=730968} 2024-09-21T08:31:25.768 INFO:tasks.rados.rados.0.smithi137.stdout:1963: writing smithi13779221-23 from 497672 to 1059099 tid 1 2024-09-21T08:31:25.776 INFO:tasks.rados.rados.0.smithi137.stdout:1963: writing smithi13779221-23 from 1795232 to 2497807 tid 2 2024-09-21T08:31:25.783 INFO:tasks.rados.rados.0.smithi137.stdout:1963: writing smithi13779221-23 from 2963697 to 3694665 tid 3 2024-09-21T08:31:25.783 INFO:tasks.rados.rados.0.smithi137.stdout:1964: delete oid 5 current snap is 213 2024-09-21T08:31:25.783 INFO:tasks.rados.rados.0.smithi137.stdout:1958: finishing rollback tid 0 to smithi13779221-15 2024-09-21T08:31:25.783 INFO:tasks.rados.rados.0.smithi137.stdout:1960: finishing rollback tid 0 to smithi13779221-50 2024-09-21T08:31:25.783 INFO:tasks.rados.rados.0.smithi137.stdout:1956: finishing write tid 5 to smithi13779221-8 2024-09-21T08:31:25.783 INFO:tasks.rados.rados.0.smithi137.stdout:1958: finishing rollback tid 1 to smithi13779221-15 2024-09-21T08:31:25.783 INFO:tasks.rados.rados.0.smithi137.stdout:1960: finishing rollback tid 1 to smithi13779221-50 2024-09-21T08:31:25.783 INFO:tasks.rados.rados.0.smithi137.stdout:1958: finishing rollback tid 2 to smithi13779221-15 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 338 (ObjNum 651 snap 192 seq_num 651) dirty exists 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:1960: finishing rollback tid 2 to smithi13779221-50 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 532 (ObjNum 660 snap 197 seq_num 660) dirty exists 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:1956: finishing write tid 6 to smithi13779221-8 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:1956: oid 8 updating version 0 to 447 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:1956: oid 8 updating version 447 to 450 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:1956: oid 8 version 450 is already newer than 448 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:1956: oid 8 version 450 is already newer than 446 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:1956: oid 8 version 450 is already newer than 449 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 450 (ObjNum 694 snap 213 seq_num 694) dirty exists 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:1956: left oid 8 (ObjNum 694 snap 213 seq_num 694) 2024-09-21T08:31:25.784 INFO:tasks.rados.rados.0.smithi137.stdout:1961: expect (ObjNum 623 snap 184 seq_num 623) 2024-09-21T08:31:25.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:25.847 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:25.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:25.848 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:25.868 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:25 smithi137 ceph-mon[29634]: pgmap v641: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 580 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 12 op/s; 238/1458 objects degraded (16.324%); 114/1458 objects misplaced (7.819%) 2024-09-21T08:31:25.869 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:25 smithi137 ceph-mon[29634]: from='client.45893 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:31:25.869 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:25 smithi137 ceph-mon[29634]: osdmap e491: 8 total, 6 up, 5 in 2024-09-21T08:31:25.869 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:25 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/182860358' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:25.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:25 smithi137 ceph-mon[41044]: pgmap v641: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 580 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 12 op/s; 238/1458 objects degraded (16.324%); 114/1458 objects misplaced (7.819%) 2024-09-21T08:31:25.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:25 smithi137 ceph-mon[41044]: from='client.45893 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:31:25.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:25 smithi137 ceph-mon[41044]: osdmap e491: 8 total, 6 up, 5 in 2024-09-21T08:31:25.869 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:25 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/182860358' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:25.921 INFO:tasks.rados.rados.0.smithi137.stdout:1962: finishing write tid 1 to smithi13779221-22 2024-09-21T08:31:25.921 INFO:tasks.rados.rados.0.smithi137.stdout:1962: finishing write tid 2 to smithi13779221-22 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1957: finishing copy_from racing read to smithi13779221-10 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1957: finishing copy_from to smithi13779221-10 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 381 (ObjNum 656 snap 196 seq_num 656) dirty exists 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1959: finishing copy_from racing read to smithi13779221-40 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1956: done (8 left) 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1957: done (7 left) 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1958: done (6 left) 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1960: done (5 left) 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1961: done (4 left) 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1964: done (3 left) 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1965: rollback oid 35 current snap is 213 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 35 to 211 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:1959: finishing copy_from to smithi13779221-40 2024-09-21T08:31:25.922 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 403 (ObjNum 664 snap 199 seq_num 664) dirty exists 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1962: finishing write tid 3 to smithi13779221-22 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1962: finishing write tid 4 to smithi13779221-22 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1962: finishing write tid 5 to smithi13779221-22 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1962: oid 22 updating version 0 to 405 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1962: oid 22 updating version 405 to 407 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1962: oid 22 version 407 is already newer than 406 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1962: oid 22 version 407 is already newer than 404 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 407 (ObjNum 695 snap 213 seq_num 695) dirty exists 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1962: left oid 22 (ObjNum 695 snap 213 seq_num 695) 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1963: finishing write tid 1 to smithi13779221-23 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1963: finishing write tid 2 to smithi13779221-23 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1963: finishing write tid 3 to smithi13779221-23 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1963: finishing write tid 4 to smithi13779221-23 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1963: finishing write tid 5 to smithi13779221-23 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1963: oid 23 updating version 0 to 208 2024-09-21T08:31:25.923 INFO:tasks.rados.rados.0.smithi137.stdout:1963: oid 23 version 208 is already newer than 207 2024-09-21T08:31:25.924 INFO:tasks.rados.rados.0.smithi137.stdout:1963: oid 23 version 208 is already newer than 205 2024-09-21T08:31:25.924 INFO:tasks.rados.rados.0.smithi137.stdout:1963: oid 23 version 208 is already newer than 206 2024-09-21T08:31:25.924 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 208 (ObjNum 696 snap 213 seq_num 696) dirty exists 2024-09-21T08:31:25.924 INFO:tasks.rados.rados.0.smithi137.stdout:1963: left oid 23 (ObjNum 696 snap 213 seq_num 696) 2024-09-21T08:31:25.924 INFO:tasks.rados.rados.0.smithi137.stdout:1959: done (3 left) 2024-09-21T08:31:25.924 INFO:tasks.rados.rados.0.smithi137.stdout:1962: done (2 left) 2024-09-21T08:31:25.924 INFO:tasks.rados.rados.0.smithi137.stdout:1963: done (1 left) 2024-09-21T08:31:25.924 INFO:tasks.rados.rados.0.smithi137.stdout:1966: write oid 10 current snap is 213 2024-09-21T08:31:25.924 INFO:tasks.rados.rados.0.smithi137.stdout:1966: seq_num 698 ranges {531286=711925,1912196=706244,3018543=712743} 2024-09-21T08:31:25.927 INFO:tasks.rados.rados.0.smithi137.stdout:1966: writing smithi13779221-10 from 531286 to 1243211 tid 1 2024-09-21T08:31:25.935 INFO:tasks.rados.rados.0.smithi137.stdout:1966: writing smithi13779221-10 from 1912196 to 2618440 tid 2 2024-09-21T08:31:25.941 INFO:tasks.rados.rados.0.smithi137.stdout:1966: writing smithi13779221-10 from 3018543 to 3731286 tid 3 2024-09-21T08:31:25.941 INFO:tasks.rados.rados.0.smithi137.stdout:1967: rollback oid 33 current snap is 213 2024-09-21T08:31:25.941 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 33 to 210 2024-09-21T08:31:25.941 INFO:tasks.rados.rados.0.smithi137.stdout:1965: finishing rollback tid 1 to smithi13779221-35 2024-09-21T08:31:25.942 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 735 (ObjNum 379559136 snap 32534 seq_num 304) dirty dne 2024-09-21T08:31:25.942 INFO:tasks.rados.rados.0.smithi137.stdout:1966: finishing write tid 1 to smithi13779221-10 2024-09-21T08:31:25.942 INFO:tasks.rados.rados.0.smithi137.stdout:1966: finishing write tid 2 to smithi13779221-10 2024-09-21T08:31:25.942 INFO:tasks.rados.rados.0.smithi137.stdout:1965: done (2 left) 2024-09-21T08:31:25.942 INFO:tasks.rados.rados.0.smithi137.stdout:1968: snap_create 2024-09-21T08:31:25.944 INFO:tasks.rados.rados.0.smithi137.stdout:1967: finishing rollback tid 1 to smithi13779221-33 2024-09-21T08:31:25.944 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 357 (ObjNum -533893408 snap 22031 seq_num 304) dirty dne 2024-09-21T08:31:25.944 INFO:tasks.rados.rados.0.smithi137.stdout:1966: finishing write tid 3 to smithi13779221-10 2024-09-21T08:31:25.944 INFO:tasks.rados.rados.0.smithi137.stdout:1966: finishing write tid 4 to smithi13779221-10 2024-09-21T08:31:25.944 INFO:tasks.rados.rados.0.smithi137.stdout:1966: finishing write tid 5 to smithi13779221-10 2024-09-21T08:31:25.944 INFO:tasks.rados.rados.0.smithi137.stdout:1966: oid 10 updating version 0 to 536 2024-09-21T08:31:25.944 INFO:tasks.rados.rados.0.smithi137.stdout:1966: oid 10 version 536 is already newer than 535 2024-09-21T08:31:25.945 INFO:tasks.rados.rados.0.smithi137.stdout:1966: oid 10 version 536 is already newer than 534 2024-09-21T08:31:25.945 INFO:tasks.rados.rados.0.smithi137.stdout:1966: oid 10 version 536 is already newer than 533 2024-09-21T08:31:25.945 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 536 (ObjNum 698 snap 213 seq_num 698) dirty exists 2024-09-21T08:31:25.945 INFO:tasks.rados.rados.0.smithi137.stdout:1966: left oid 10 (ObjNum 698 snap 213 seq_num 698) 2024-09-21T08:31:25.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:25.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:25.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:25.971 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:25 smithi165 ceph-mon[32893]: pgmap v641: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 580 MiB data, 1.4 GiB used, 446 GiB / 447 GiB avail; 1.9 MiB/s rd, 3.1 MiB/s wr, 12 op/s; 238/1458 objects degraded (16.324%); 114/1458 objects misplaced (7.819%) 2024-09-21T08:31:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:25 smithi165 ceph-mon[32893]: from='client.45893 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:31:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:25 smithi165 ceph-mon[32893]: osdmap e491: 8 total, 6 up, 5 in 2024-09-21T08:31:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:25 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/182860358' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:26.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:26.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:26.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:26.095 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:26.195 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:26.219 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:26.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:26.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:26.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:26.342 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:26.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:26.343 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:26.415 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:31:26.417 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:31:26.444 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:26.469 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:26.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:26.470 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:26.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:26.591 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:26.592 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:26.592 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:26.688 INFO:tasks.rados.rados.0.smithi137.stdout:1966: done (2 left) 2024-09-21T08:31:26.689 INFO:tasks.rados.rados.0.smithi137.stdout:1967: done (1 left) 2024-09-21T08:31:26.689 INFO:tasks.rados.rados.0.smithi137.stdout:1968: done (0 left) 2024-09-21T08:31:26.689 INFO:tasks.rados.rados.0.smithi137.stdout:1969: read oid 26 snap -1 2024-09-21T08:31:26.689 INFO:tasks.rados.rados.0.smithi137.stdout:1969: expect deleted 2024-09-21T08:31:26.689 INFO:tasks.rados.rados.0.smithi137.stdout:1970: read oid 15 snap 200 2024-09-21T08:31:26.689 INFO:tasks.rados.rados.0.smithi137.stdout:1970: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:26.690 INFO:tasks.rados.rados.0.smithi137.stdout:1971: snap_create 2024-09-21T08:31:26.693 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:26.700 INFO:tasks.rados.rados.0.smithi137.stdout:1970: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:26.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:26.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:26.715 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:26.816 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:26.824 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T08:31:26.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:26.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:26.841 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:26.942 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:26.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:26.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:26.967 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:26.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:26 smithi165 ceph-mon[32893]: 7.25 scrub starts 2024-09-21T08:31:26.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:26 smithi165 ceph-mon[32893]: 7.25 scrub ok 2024-09-21T08:31:26.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:26 smithi165 ceph-mon[32893]: 4.b scrub starts 2024-09-21T08:31:26.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:26 smithi165 ceph-mon[32893]: 4.b scrub ok 2024-09-21T08:31:26.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:26 smithi165 ceph-mon[32893]: osdmap e492: 8 total, 6 up, 5 in 2024-09-21T08:31:26.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:26 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:26.977 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:26 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 238/1458 objects degraded (16.324%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:27.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[29634]: 7.25 scrub starts 2024-09-21T08:31:27.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[29634]: 7.25 scrub ok 2024-09-21T08:31:27.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[29634]: 4.b scrub starts 2024-09-21T08:31:27.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[29634]: 4.b scrub ok 2024-09-21T08:31:27.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[29634]: osdmap e492: 8 total, 6 up, 5 in 2024-09-21T08:31:27.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:27.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 238/1458 objects degraded (16.324%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:27.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[41044]: 7.25 scrub starts 2024-09-21T08:31:27.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[41044]: 7.25 scrub ok 2024-09-21T08:31:27.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[41044]: 4.b scrub starts 2024-09-21T08:31:27.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[41044]: 4.b scrub ok 2024-09-21T08:31:27.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[41044]: osdmap e492: 8 total, 6 up, 5 in 2024-09-21T08:31:27.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:27.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:26 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 238/1458 objects degraded (16.324%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:27.068 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:27.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:27.091 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:27.091 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:27.192 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:27.213 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:27.214 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:27.214 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:27.315 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:27.337 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:27.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:27.338 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:27.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:27.464 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:27.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:27.465 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:27.489 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":644,"stamp":"2024-09-21T08:31:26.153760+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":639500058,"num_objects":501,"num_object_clones":254,"num_object_copies":1503,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":246,"num_objects_misplaced":120,"num_objects_unfound":0,"num_objects_dirty":501,"num_whiteouts":15,"num_read":3151,"num_read_kb":2164102,"num_write":1879,"num_write_kb":571526,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1435,"num_bytes_recovered":379386763,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13125,"ondisk_log_size":13125,"up":444,"acting":504,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":634,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":1544080,"kb_used_data":1307608,"kb_used_omap":7,"kb_used_meta":236344,"kb_avail":467181680,"statfs":{"total":479975178240,"available":478394040320,"internally_reserved":0,"allocated":1338990592,"data_stored":1943942696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":7970,"internal_metadata":242016478},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":31550558,"num_objects":15,"num_object_clones":15,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":25,"num_read_kb":20058,"num_write":23,"num_write_kb":11331,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-5,"num_bytes_recovered":-926989,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001167"},"pg_stats":[{"pgid":"7.2d","version":"455'154","reported_seq":743,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.694307+0000","last_change":"2024-09-21T08:31:09.702836+0000","last_active":"2024-09-21T08:31:25.694307+0000","last_peered":"2024-09-21T08:31:25.694307+0000","last_clean":"2024-09-21T08:28:52.661631+0000","last_became_active":"2024-09-21T08:31:09.702836+0000","last_became_peered":"2024-09-21T08:31:09.702836+0000","last_unstale":"2024-09-21T08:31:25.694307+0000","last_undegraded":"2024-09-21T08:31:09.693888+0000","last_fullsized":"2024-09-21T08:31:09.693762+0000","mapping_epoch":475,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":6,"last_scrub":"294'93","last_scrub_stamp":"2024-09-21T08:28:11.373358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:11.373358+0000","objects_scrubbed":3,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":77,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13974387,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":16,"num_read_kb":8938,"num_write":21,"num_write_kb":5598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3285818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2c","version":"492'290","reported_seq":2341,"reported_epoch":492,"state":"active+clean+remapped+snaptrim","last_fresh":"2024-09-21T08:31:25.683662+0000","last_change":"2024-09-21T08:31:24.748980+0000","last_active":"2024-09-21T08:31:25.683662+0000","last_peered":"2024-09-21T08:31:25.683662+0000","last_clean":"2024-09-21T08:31:25.683662+0000","last_became_active":"2024-09-21T08:31:10.705371+0000","last_became_peered":"2024-09-21T08:31:10.705371+0000","last_unstale":"2024-09-21T08:31:25.683662+0000","last_undegraded":"2024-09-21T08:31:25.683662+0000","last_fullsized":"2024-09-21T08:31:25.683662+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"446'254","last_scrub_stamp":"2024-09-21T08:30:46.778996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:46.778996+0000","objects_scrubbed":4,"log_size":290,"log_dups_size":0,"ondisk_log_size":290,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.13259996700000001,"stat_sum":{"num_bytes":13230439,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":6,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":30,"num_read_kb":29087,"num_write":27,"num_write_kb":12392,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3117748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"7.2b","version":"489'819","reported_seq":2984,"reported_epoch":492,"state":"active+clean+remapped+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.738406+0000","last_change":"2024-09-21T08:31:25.738560+0000","last_active":"2024-09-21T08:31:25.738406+0000","last_peered":"2024-09-21T08:31:25.738406+0000","last_clean":"2024-09-21T08:31:25.738406+0000","last_became_active":"2024-09-21T08:31:10.703193+0000","last_became_peered":"2024-09-21T08:31:10.703193+0000","last_unstale":"2024-09-21T08:31:25.738406+0000","last_undegraded":"2024-09-21T08:31:25.738406+0000","last_fullsized":"2024-09-21T08:31:25.738406+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"376'577","last_scrub_stamp":"2024-09-21T08:29:32.265609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.265609+0000","objects_scrubbed":10,"log_size":819,"log_dups_size":0,"ondisk_log_size":819,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":5,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.054830785999999999,"stat_sum":{"num_bytes":33595299,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":14,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":85,"num_read_kb":81629,"num_write":64,"num_write_kb":25982,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":2192636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2,5],"avail_no_missing":["1","2","5"],"object_location_counts":[{"shards":"1,2,5","objects":14}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"3"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"7.2a","version":"472'122","reported_seq":1341,"reported_epoch":492,"state":"active+clean+remapped+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.737483+0000","last_change":"2024-09-21T08:31:25.737645+0000","last_active":"2024-09-21T08:31:25.737483+0000","last_peered":"2024-09-21T08:31:25.737483+0000","last_clean":"2024-09-21T08:31:25.737483+0000","last_became_active":"2024-09-21T08:31:12.610911+0000","last_became_peered":"2024-09-21T08:31:12.610911+0000","last_unstale":"2024-09-21T08:31:25.737483+0000","last_undegraded":"2024-09-21T08:31:25.737483+0000","last_fullsized":"2024-09-21T08:31:25.737483+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":6,"last_scrub":"433'114","last_scrub_stamp":"2024-09-21T08:30:41.677928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:41.677928+0000","objects_scrubbed":2,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.054854864000000003,"stat_sum":{"num_bytes":3703394,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":8,"num_read_kb":2367,"num_write":12,"num_write_kb":2252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":591936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c7","length":"1"}]},{"pgid":"7.29","version":"471'381","reported_seq":2461,"reported_epoch":492,"state":"active+clean+remapped+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.692838+0000","last_change":"2024-09-21T08:31:22.733609+0000","last_active":"2024-09-21T08:31:25.692838+0000","last_peered":"2024-09-21T08:31:25.692838+0000","last_clean":"2024-09-21T08:31:25.692838+0000","last_became_active":"2024-09-21T08:31:12.612074+0000","last_became_peered":"2024-09-21T08:31:12.612074+0000","last_unstale":"2024-09-21T08:31:25.692838+0000","last_undegraded":"2024-09-21T08:31:25.692838+0000","last_fullsized":"2024-09-21T08:31:25.692838+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":381,"log_dups_size":0,"ondisk_log_size":381,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":69,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.13514094400000001,"stat_sum":{"num_bytes":18109166,"num_objects":9,"num_object_clones":7,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":9,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":1,"num_read":92,"num_read_kb":96436,"num_write":60,"num_write_kb":19850,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":66,"num_bytes_recovered":30130114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1,4],"avail_no_missing":["5","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":9}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"7"}]},{"pgid":"7.28","version":"492'409","reported_seq":2385,"reported_epoch":492,"state":"active+clean+remapped+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.746864+0000","last_change":"2024-09-21T08:31:24.736626+0000","last_active":"2024-09-21T08:31:25.746864+0000","last_peered":"2024-09-21T08:31:25.746864+0000","last_clean":"2024-09-21T08:31:25.746864+0000","last_became_active":"2024-09-21T08:31:10.697830+0000","last_became_peered":"2024-09-21T08:31:10.697830+0000","last_unstale":"2024-09-21T08:31:25.746864+0000","last_undegraded":"2024-09-21T08:31:25.746864+0000","last_fullsized":"2024-09-21T08:31:25.746864+0000","mapping_epoch":476,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":477,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'178","last_scrub_stamp":"2024-09-21T08:26:28.217842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.217842+0000","objects_scrubbed":4,"log_size":409,"log_dups_size":0,"ondisk_log_size":409,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.053420202,"stat_sum":{"num_bytes":18048563,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":77,"num_read_kb":92492,"num_write":61,"num_write_kb":24532,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":22413758,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[{"shards":"2,4,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"7.27","version":"492'200","reported_seq":2239,"reported_epoch":492,"state":"active+clean+remapped+snaptrim","last_fresh":"2024-09-21T08:31:25.749233+0000","last_change":"2024-09-21T08:31:25.738524+0000","last_active":"2024-09-21T08:31:25.749233+0000","last_peered":"2024-09-21T08:31:25.749233+0000","last_clean":"2024-09-21T08:31:25.749233+0000","last_became_active":"2024-09-21T08:31:09.703538+0000","last_became_peered":"2024-09-21T08:31:09.703538+0000","last_unstale":"2024-09-21T08:31:25.749233+0000","last_undegraded":"2024-09-21T08:31:25.749233+0000","last_fullsized":"2024-09-21T08:31:25.749233+0000","mapping_epoch":475,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":476,"parent":"0.0","parent_split_bits":6,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":200,"log_dups_size":0,"ondisk_log_size":200,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":60,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.065141395000000005,"stat_sum":{"num_bytes":11388276,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":44,"num_read_kb":39360,"num_write":35,"num_write_kb":18091,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":44,"num_bytes_recovered":27270248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"10"}]},{"pgid":"7.26","version":"453'161","reported_seq":1952,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.694165+0000","last_change":"2024-09-21T08:31:07.692261+0000","last_active":"2024-09-21T08:31:25.694165+0000","last_peered":"2024-09-21T08:31:25.694165+0000","last_clean":"2024-09-21T08:28:51.554645+0000","last_became_active":"2024-09-21T08:31:07.692261+0000","last_became_peered":"2024-09-21T08:31:07.692261+0000","last_unstale":"2024-09-21T08:31:25.694165+0000","last_undegraded":"2024-09-21T08:31:07.688120+0000","last_fullsized":"2024-09-21T08:31:07.687739+0000","mapping_epoch":473,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":6,"last_scrub":"294'138","last_scrub_stamp":"2024-09-21T08:28:41.855754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:41.855754+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":68,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":6518339,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":31642,"num_write":14,"num_write_kb":4831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":4641445,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6e","length":"3"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"7.25","version":"487'327","reported_seq":2360,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684549+0000","last_change":"2024-09-21T08:31:24.990208+0000","last_active":"2024-09-21T08:31:25.684549+0000","last_peered":"2024-09-21T08:31:25.684549+0000","last_clean":"2024-09-21T08:31:25.684549+0000","last_became_active":"2024-09-21T08:31:12.611260+0000","last_became_peered":"2024-09-21T08:31:12.611260+0000","last_unstale":"2024-09-21T08:31:25.684549+0000","last_undegraded":"2024-09-21T08:31:25.684549+0000","last_fullsized":"2024-09-21T08:31:25.684549+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":6,"last_scrub":"487'327","last_scrub_stamp":"2024-09-21T08:31:24.989978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:24.989978+0000","objects_scrubbed":4,"log_size":327,"log_dups_size":0,"ondisk_log_size":327,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:39.794292+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0034316440000000002,"stat_sum":{"num_bytes":10847650,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4,5","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"7.24","version":"491'169","reported_seq":2202,"reported_epoch":492,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:31:25.693280+0000","last_change":"2024-09-21T08:31:24.736634+0000","last_active":"2024-09-21T08:31:25.693280+0000","last_peered":"2024-09-21T08:31:25.693280+0000","last_clean":"2024-09-21T08:31:25.693280+0000","last_became_active":"2024-09-21T08:31:12.614215+0000","last_became_peered":"2024-09-21T08:31:12.614215+0000","last_unstale":"2024-09-21T08:31:25.693280+0000","last_undegraded":"2024-09-21T08:31:25.693280+0000","last_fullsized":"2024-09-21T08:31:25.693280+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":6,"last_scrub":"445'141","last_scrub_stamp":"2024-09-21T08:30:40.016003+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:30:40.016003+0000","objects_scrubbed":4,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":4,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0.13333845499999999,"stat_sum":{"num_bytes":1289429,"num_objects":2,"num_object_clones":1,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":1,"num_read":20,"num_read_kb":23877,"num_write":10,"num_write_kb":5304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":16048647,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"7.23","version":"179'19","reported_seq":975,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.686004+0000","last_change":"2024-09-21T08:31:23.735479+0000","last_active":"2024-09-21T08:31:25.686004+0000","last_peered":"2024-09-21T08:31:25.686004+0000","last_clean":"2024-09-21T08:31:25.686004+0000","last_became_active":"2024-09-21T08:30:56.626989+0000","last_became_peered":"2024-09-21T08:30:56.626989+0000","last_unstale":"2024-09-21T08:31:25.686004+0000","last_undegraded":"2024-09-21T08:31:25.686004+0000","last_fullsized":"2024-09-21T08:31:25.686004+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:28:10.421606+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.421606+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.054340468000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"7.22","version":"257'264","reported_seq":2351,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.682764+0000","last_change":"2024-09-21T08:31:21.600755+0000","last_active":"2024-09-21T08:31:25.682764+0000","last_peered":"2024-09-21T08:31:25.682764+0000","last_clean":"2024-09-21T08:31:25.682764+0000","last_became_active":"2024-09-21T08:30:59.603168+0000","last_became_peered":"2024-09-21T08:30:59.603168+0000","last_unstale":"2024-09-21T08:31:25.682764+0000","last_undegraded":"2024-09-21T08:31:25.682764+0000","last_fullsized":"2024-09-21T08:31:25.682764+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":6,"last_scrub":"257'264","last_scrub_stamp":"2024-09-21T08:27:48.727600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:48.727600+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.003019014,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"7.21","version":"179'43","reported_seq":1798,"reported_epoch":492,"state":"active+clean+remapped+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.693130+0000","last_change":"2024-09-21T08:31:22.734058+0000","last_active":"2024-09-21T08:31:25.693130+0000","last_peered":"2024-09-21T08:31:25.693130+0000","last_clean":"2024-09-21T08:31:25.693130+0000","last_became_active":"2024-09-21T08:30:57.612821+0000","last_became_peered":"2024-09-21T08:30:57.612821+0000","last_unstale":"2024-09-21T08:31:25.693130+0000","last_undegraded":"2024-09-21T08:31:25.693130+0000","last_fullsized":"2024-09-21T08:31:25.693130+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:36.934372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.934372+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.13355373500000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"1"},{"start":"c6","length":"2"}]},{"pgid":"7.20","version":"487'177","reported_seq":1926,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.683875+0000","last_change":"2024-09-21T08:28:53.605282+0000","last_active":"2024-09-21T08:31:25.683875+0000","last_peered":"2024-09-21T08:31:25.683875+0000","last_clean":"2024-09-21T08:28:51.585842+0000","last_became_active":"2024-09-21T08:28:53.605282+0000","last_became_peered":"2024-09-21T08:28:53.605282+0000","last_unstale":"2024-09-21T08:31:25.683875+0000","last_undegraded":"2024-09-21T08:28:53.580783+0000","last_fullsized":"2024-09-21T08:28:53.580670+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":6,"last_scrub":"251'134","last_scrub_stamp":"2024-09-21T08:27:49.318143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.318143+0000","objects_scrubbed":1,"log_size":177,"log_dups_size":0,"ondisk_log_size":177,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":72,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.076740219999999998,"stat_sum":{"num_bytes":15552680,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":29,"num_read_kb":17524,"num_write":34,"num_write_kb":7693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":1848493,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6e","length":"3"},{"start":"73","length":"2"}]},{"pgid":"7.1b","version":"488'807","reported_seq":3110,"reported_epoch":492,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.697298+0000","last_change":"2024-09-21T08:31:23.652626+0000","last_active":"2024-09-21T08:31:25.697298+0000","last_peered":"2024-09-21T08:31:25.697298+0000","last_clean":"2024-09-21T08:31:25.697298+0000","last_became_active":"2024-09-21T08:30:59.601175+0000","last_became_peered":"2024-09-21T08:30:59.601175+0000","last_unstale":"2024-09-21T08:31:25.697298+0000","last_undegraded":"2024-09-21T08:31:25.697298+0000","last_fullsized":"2024-09-21T08:31:25.697298+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'499","last_scrub_stamp":"2024-09-21T08:28:09.328624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.328624+0000","objects_scrubbed":5,"log_size":807,"log_dups_size":0,"ondisk_log_size":807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":10,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.049575615,"stat_sum":{"num_bytes":49200448,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":1,"num_read":153,"num_read_kb":167713,"num_write":100,"num_write_kb":39317,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":61029229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"}]},{"pgid":"3.1f","version":"0'0","reported_seq":372,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.694561+0000","last_change":"2024-09-21T08:30:56.618803+0000","last_active":"2024-09-21T08:31:25.694561+0000","last_peered":"2024-09-21T08:31:25.694561+0000","last_clean":"2024-09-21T08:31:25.694561+0000","last_became_active":"2024-09-21T08:30:56.617989+0000","last_became_peered":"2024-09-21T08:30:56.617989+0000","last_unstale":"2024-09-21T08:31:25.694561+0000","last_undegraded":"2024-09-21T08:31:25.694561+0000","last_fullsized":"2024-09-21T08:31:25.694561+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":79,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.682630+0000","last_change":"2024-09-21T08:30:56.662007+0000","last_active":"2024-09-21T08:31:25.682630+0000","last_peered":"2024-09-21T08:31:25.682630+0000","last_clean":"2024-09-21T08:31:25.682630+0000","last_became_active":"2024-09-21T08:30:56.628842+0000","last_became_peered":"2024-09-21T08:30:56.628842+0000","last_unstale":"2024-09-21T08:31:25.682630+0000","last_undegraded":"2024-09-21T08:31:25.682630+0000","last_fullsized":"2024-09-21T08:31:25.682630+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:44.058465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:44.058465+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:05.271406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":425,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.683650+0000","last_change":"2024-09-21T08:28:53.594621+0000","last_active":"2024-09-21T08:31:25.683650+0000","last_peered":"2024-09-21T08:31:25.683650+0000","last_clean":"2024-09-21T08:28:51.584740+0000","last_became_active":"2024-09-21T08:28:53.594621+0000","last_became_peered":"2024-09-21T08:28:53.594621+0000","last_unstale":"2024-09-21T08:31:25.683650+0000","last_undegraded":"2024-09-21T08:31:25.683650+0000","last_fullsized":"2024-09-21T08:28:53.576425+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.424482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.424482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":69,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683500+0000","last_change":"2024-09-21T08:30:56.631377+0000","last_active":"2024-09-21T08:31:25.683500+0000","last_peered":"2024-09-21T08:31:25.683500+0000","last_clean":"2024-09-21T08:31:25.683500+0000","last_became_active":"2024-09-21T08:30:56.630218+0000","last_became_peered":"2024-09-21T08:30:56.630218+0000","last_unstale":"2024-09-21T08:31:25.683500+0000","last_undegraded":"2024-09-21T08:31:25.683500+0000","last_fullsized":"2024-09-21T08:31:25.683500+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.464969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.464969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"248'68","reported_seq":379,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.682829+0000","last_change":"2024-09-21T08:31:21.600973+0000","last_active":"2024-09-21T08:31:25.682829+0000","last_peered":"2024-09-21T08:31:25.682829+0000","last_clean":"2024-09-21T08:31:25.682829+0000","last_became_active":"2024-09-21T08:30:57.616295+0000","last_became_peered":"2024-09-21T08:30:57.616295+0000","last_unstale":"2024-09-21T08:31:25.682829+0000","last_undegraded":"2024-09-21T08:31:25.682829+0000","last_fullsized":"2024-09-21T08:31:25.682829+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'68","last_scrub_stamp":"2024-09-21T08:27:35.919031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:35.919031+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0030612249999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":782,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.684354+0000","last_change":"2024-09-21T08:28:53.606497+0000","last_active":"2024-09-21T08:31:25.684354+0000","last_peered":"2024-09-21T08:31:25.684354+0000","last_clean":"2024-09-21T08:28:51.585547+0000","last_became_active":"2024-09-21T08:28:53.606497+0000","last_became_peered":"2024-09-21T08:28:53.606497+0000","last_unstale":"2024-09-21T08:31:25.684354+0000","last_undegraded":"2024-09-21T08:31:25.684354+0000","last_fullsized":"2024-09-21T08:28:53.581183+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.804294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.804294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":709,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.685418+0000","last_change":"2024-09-21T08:28:53.602981+0000","last_active":"2024-09-21T08:31:25.685418+0000","last_peered":"2024-09-21T08:31:25.685418+0000","last_clean":"2024-09-21T08:28:51.587203+0000","last_became_active":"2024-09-21T08:28:53.602981+0000","last_became_peered":"2024-09-21T08:28:53.602981+0000","last_unstale":"2024-09-21T08:31:25.685418+0000","last_undegraded":"2024-09-21T08:31:25.685418+0000","last_fullsized":"2024-09-21T08:28:53.581112+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.146798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.146798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":705,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.693431+0000","last_change":"2024-09-21T08:29:00.355702+0000","last_active":"2024-09-21T08:31:25.693431+0000","last_peered":"2024-09-21T08:31:25.693431+0000","last_clean":"2024-09-21T08:31:25.693431+0000","last_became_active":"2024-09-21T08:28:29.620169+0000","last_became_peered":"2024-09-21T08:28:29.620169+0000","last_unstale":"2024-09-21T08:31:25.693431+0000","last_undegraded":"2024-09-21T08:31:25.693431+0000","last_fullsized":"2024-09-21T08:31:25.693431+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:00.355591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:00.355591+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":69,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.682095+0000","last_change":"2024-09-21T08:30:56.628272+0000","last_active":"2024-09-21T08:31:25.682095+0000","last_peered":"2024-09-21T08:31:25.682095+0000","last_clean":"2024-09-21T08:31:25.682095+0000","last_became_active":"2024-09-21T08:30:56.627964+0000","last_became_peered":"2024-09-21T08:30:56.627964+0000","last_unstale":"2024-09-21T08:31:25.682095+0000","last_undegraded":"2024-09-21T08:31:25.682095+0000","last_fullsized":"2024-09-21T08:31:25.682095+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.340817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.340817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"489'213","reported_seq":1642,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.695220+0000","last_change":"2024-09-21T08:28:53.587606+0000","last_active":"2024-09-21T08:31:25.695220+0000","last_peered":"2024-09-21T08:31:25.695220+0000","last_clean":"2024-09-21T08:28:52.589310+0000","last_became_active":"2024-09-21T08:28:53.587606+0000","last_became_peered":"2024-09-21T08:28:53.587606+0000","last_unstale":"2024-09-21T08:31:25.695220+0000","last_undegraded":"2024-09-21T08:28:53.574680+0000","last_fullsized":"2024-09-21T08:28:53.574555+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"266'188","last_scrub_stamp":"2024-09-21T08:27:49.719124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.719124+0000","objects_scrubbed":1,"log_size":213,"log_dups_size":0,"ondisk_log_size":213,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":71,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.034770314,"stat_sum":{"num_bytes":14452344,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":27,"num_read_kb":25355,"num_write":23,"num_write_kb":13257,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6e","length":"3"},{"start":"73","length":"3"}]},{"pgid":"3.1d","version":"0'0","reported_seq":919,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.697506+0000","last_change":"2024-09-21T08:30:56.626716+0000","last_active":"2024-09-21T08:31:25.697506+0000","last_peered":"2024-09-21T08:31:25.697506+0000","last_clean":"2024-09-21T08:31:25.697506+0000","last_became_active":"2024-09-21T08:30:56.625894+0000","last_became_peered":"2024-09-21T08:30:56.625894+0000","last_unstale":"2024-09-21T08:31:25.697506+0000","last_undegraded":"2024-09-21T08:31:25.697506+0000","last_fullsized":"2024-09-21T08:31:25.697506+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.780545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.780545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":79,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684314+0000","last_change":"2024-09-21T08:30:56.648653+0000","last_active":"2024-09-21T08:31:25.684314+0000","last_peered":"2024-09-21T08:31:25.684314+0000","last_clean":"2024-09-21T08:31:25.684314+0000","last_became_active":"2024-09-21T08:30:56.621427+0000","last_became_peered":"2024-09-21T08:30:56.621427+0000","last_unstale":"2024-09-21T08:31:25.684314+0000","last_undegraded":"2024-09-21T08:31:25.684314+0000","last_fullsized":"2024-09-21T08:31:25.684314+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:26.781541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:26.781541+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:43.277471+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":877,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.693391+0000","last_change":"2024-09-21T08:30:57.625041+0000","last_active":"2024-09-21T08:31:25.693391+0000","last_peered":"2024-09-21T08:31:25.693391+0000","last_clean":"2024-09-21T08:31:25.693391+0000","last_became_active":"2024-09-21T08:30:57.624588+0000","last_became_peered":"2024-09-21T08:30:57.624588+0000","last_unstale":"2024-09-21T08:31:25.693391+0000","last_undegraded":"2024-09-21T08:31:25.693391+0000","last_fullsized":"2024-09-21T08:31:25.693391+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:22.581159+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:22.581159+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":828,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.682450+0000","last_change":"2024-09-21T08:28:53.586342+0000","last_active":"2024-09-21T08:31:25.682450+0000","last_peered":"2024-09-21T08:31:25.682450+0000","last_clean":"2024-09-21T08:28:51.584564+0000","last_became_active":"2024-09-21T08:28:53.586342+0000","last_became_peered":"2024-09-21T08:28:53.586342+0000","last_unstale":"2024-09-21T08:31:25.682450+0000","last_undegraded":"2024-09-21T08:31:25.682450+0000","last_fullsized":"2024-09-21T08:28:53.575506+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.291226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.291226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"492'281","reported_seq":2083,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.748900+0000","last_change":"2024-09-21T08:28:53.587953+0000","last_active":"2024-09-21T08:31:25.748900+0000","last_peered":"2024-09-21T08:31:25.748900+0000","last_clean":"2024-09-21T08:28:51.550342+0000","last_became_active":"2024-09-21T08:28:53.587953+0000","last_became_peered":"2024-09-21T08:28:53.587953+0000","last_unstale":"2024-09-21T08:31:25.748900+0000","last_undegraded":"2024-09-21T08:28:53.574363+0000","last_fullsized":"2024-09-21T08:28:53.574234+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":5,"last_scrub":"319'218","last_scrub_stamp":"2024-09-21T08:28:32.867460+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:32.867460+0000","objects_scrubbed":4,"log_size":281,"log_dups_size":0,"ondisk_log_size":281,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":68,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0027063920000000002,"stat_sum":{"num_bytes":36338311,"num_objects":16,"num_object_clones":14,"num_object_copies":48,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":16,"num_whiteouts":0,"num_read":62,"num_read_kb":68207,"num_write":75,"num_write_kb":36516,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":7346792,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":16}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6e","length":"3"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"3.1c","version":"0'0","reported_seq":904,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.694818+0000","last_change":"2024-09-21T08:30:56.627917+0000","last_active":"2024-09-21T08:31:25.694818+0000","last_peered":"2024-09-21T08:31:25.694818+0000","last_clean":"2024-09-21T08:31:25.694818+0000","last_became_active":"2024-09-21T08:30:56.627614+0000","last_became_peered":"2024-09-21T08:30:56.627614+0000","last_unstale":"2024-09-21T08:31:25.694818+0000","last_undegraded":"2024-09-21T08:31:25.694818+0000","last_fullsized":"2024-09-21T08:31:25.694818+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:56.698267+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:56.698267+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":75,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683531+0000","last_change":"2024-09-21T08:30:56.659860+0000","last_active":"2024-09-21T08:31:25.683531+0000","last_peered":"2024-09-21T08:31:25.683531+0000","last_clean":"2024-09-21T08:31:25.683531+0000","last_became_active":"2024-09-21T08:30:56.627915+0000","last_became_peered":"2024-09-21T08:30:56.627915+0000","last_unstale":"2024-09-21T08:31:25.683531+0000","last_undegraded":"2024-09-21T08:31:25.683531+0000","last_fullsized":"2024-09-21T08:31:25.683531+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:27:26.356987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:26.356987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683153+0000","last_change":"2024-09-21T08:30:57.619332+0000","last_active":"2024-09-21T08:31:25.683153+0000","last_peered":"2024-09-21T08:31:25.683153+0000","last_clean":"2024-09-21T08:31:25.683153+0000","last_became_active":"2024-09-21T08:30:57.617614+0000","last_became_peered":"2024-09-21T08:30:57.617614+0000","last_unstale":"2024-09-21T08:31:25.683153+0000","last_undegraded":"2024-09-21T08:31:25.683153+0000","last_fullsized":"2024-09-21T08:31:25.683153+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.850373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":918,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.694766+0000","last_change":"2024-09-21T08:30:04.034656+0000","last_active":"2024-09-21T08:31:25.694766+0000","last_peered":"2024-09-21T08:31:25.694766+0000","last_clean":"2024-09-21T08:31:25.694766+0000","last_became_active":"2024-09-21T08:28:04.705450+0000","last_became_peered":"2024-09-21T08:28:04.705450+0000","last_unstale":"2024-09-21T08:31:25.694766+0000","last_undegraded":"2024-09-21T08:31:25.694766+0000","last_fullsized":"2024-09-21T08:31:25.694766+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:30:04.034575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:04.034575+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1f","version":"247'32","reported_seq":385,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684785+0000","last_change":"2024-09-21T08:31:21.600314+0000","last_active":"2024-09-21T08:31:25.684785+0000","last_peered":"2024-09-21T08:31:25.684785+0000","last_clean":"2024-09-21T08:31:25.684785+0000","last_became_active":"2024-09-21T08:30:56.617686+0000","last_became_peered":"2024-09-21T08:30:56.617686+0000","last_unstale":"2024-09-21T08:31:25.684785+0000","last_undegraded":"2024-09-21T08:31:25.684785+0000","last_fullsized":"2024-09-21T08:31:25.684785+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:27:51.078531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.078531+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0029834409999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"6.1e","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684456+0000","last_change":"2024-09-21T08:30:57.620780+0000","last_active":"2024-09-21T08:31:25.684456+0000","last_peered":"2024-09-21T08:31:25.684456+0000","last_clean":"2024-09-21T08:31:25.684456+0000","last_became_active":"2024-09-21T08:30:57.617563+0000","last_became_peered":"2024-09-21T08:30:57.617563+0000","last_unstale":"2024-09-21T08:31:25.684456+0000","last_undegraded":"2024-09-21T08:31:25.684456+0000","last_fullsized":"2024-09-21T08:31:25.684456+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.855448+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.855448+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683109+0000","last_change":"2024-09-21T08:30:57.614029+0000","last_active":"2024-09-21T08:31:25.683109+0000","last_peered":"2024-09-21T08:31:25.683109+0000","last_clean":"2024-09-21T08:31:25.683109+0000","last_became_active":"2024-09-21T08:30:57.611889+0000","last_became_peered":"2024-09-21T08:30:57.611889+0000","last_unstale":"2024-09-21T08:31:25.683109+0000","last_undegraded":"2024-09-21T08:31:25.683109+0000","last_fullsized":"2024-09-21T08:31:25.683109+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:54.870454+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.870454+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":962,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684391+0000","last_change":"2024-09-21T08:30:56.660342+0000","last_active":"2024-09-21T08:31:25.684391+0000","last_peered":"2024-09-21T08:31:25.684391+0000","last_clean":"2024-09-21T08:31:25.684391+0000","last_became_active":"2024-09-21T08:30:56.630563+0000","last_became_peered":"2024-09-21T08:30:56.630563+0000","last_unstale":"2024-09-21T08:31:25.684391+0000","last_undegraded":"2024-09-21T08:31:25.684391+0000","last_fullsized":"2024-09-21T08:31:25.684391+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:37.352879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:37.352879+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":429,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.686573+0000","last_change":"2024-09-21T08:30:56.628090+0000","last_active":"2024-09-21T08:31:25.686573+0000","last_peered":"2024-09-21T08:31:25.686573+0000","last_clean":"2024-09-21T08:31:25.686573+0000","last_became_active":"2024-09-21T08:30:56.627667+0000","last_became_peered":"2024-09-21T08:30:56.627667+0000","last_unstale":"2024-09-21T08:31:25.686573+0000","last_undegraded":"2024-09-21T08:31:25.686573+0000","last_fullsized":"2024-09-21T08:31:25.686573+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:30.310628+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.310628+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"490'329","reported_seq":2300,"reported_epoch":492,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.684297+0000","last_change":"2024-09-21T08:31:24.748301+0000","last_active":"2024-09-21T08:31:25.684297+0000","last_peered":"2024-09-21T08:31:25.684297+0000","last_clean":"2024-09-21T08:31:25.684297+0000","last_became_active":"2024-09-21T08:30:59.606410+0000","last_became_peered":"2024-09-21T08:30:59.606410+0000","last_unstale":"2024-09-21T08:31:25.684297+0000","last_undegraded":"2024-09-21T08:31:25.684297+0000","last_fullsized":"2024-09-21T08:31:25.684297+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'236","last_scrub_stamp":"2024-09-21T08:28:09.464279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.464279+0000","objects_scrubbed":2,"log_size":329,"log_dups_size":0,"ondisk_log_size":329,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":57,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.068256654999999999,"stat_sum":{"num_bytes":42687371,"num_objects":19,"num_object_clones":16,"num_object_copies":57,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":19,"num_whiteouts":2,"num_read":92,"num_read_kb":94552,"num_write":74,"num_write_kb":25690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":36880118,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"13"}]},{"pgid":"6.1f","version":"0'0","reported_seq":774,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.696328+0000","last_change":"2024-09-21T08:30:37.732703+0000","last_active":"2024-09-21T08:31:25.696328+0000","last_peered":"2024-09-21T08:31:25.696328+0000","last_clean":"2024-09-21T08:31:25.696328+0000","last_became_active":"2024-09-21T08:28:04.725392+0000","last_became_peered":"2024-09-21T08:28:04.725392+0000","last_unstale":"2024-09-21T08:31:25.696328+0000","last_undegraded":"2024-09-21T08:31:25.696328+0000","last_fullsized":"2024-09-21T08:31:25.696328+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:37.732599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:37.732599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:56.617806+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":69,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683185+0000","last_change":"2024-09-21T08:30:56.630745+0000","last_active":"2024-09-21T08:31:25.683185+0000","last_peered":"2024-09-21T08:31:25.683185+0000","last_clean":"2024-09-21T08:31:25.683185+0000","last_became_active":"2024-09-21T08:30:56.630058+0000","last_became_peered":"2024-09-21T08:30:56.630058+0000","last_unstale":"2024-09-21T08:31:25.683185+0000","last_undegraded":"2024-09-21T08:31:25.683185+0000","last_fullsized":"2024-09-21T08:31:25.683185+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:33.237691+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.237691+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":452,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.685324+0000","last_change":"2024-09-21T08:30:56.656296+0000","last_active":"2024-09-21T08:31:25.685324+0000","last_peered":"2024-09-21T08:31:25.685324+0000","last_clean":"2024-09-21T08:31:25.685324+0000","last_became_active":"2024-09-21T08:30:56.626214+0000","last_became_peered":"2024-09-21T08:30:56.626214+0000","last_unstale":"2024-09-21T08:31:25.685324+0000","last_undegraded":"2024-09-21T08:31:25.685324+0000","last_fullsized":"2024-09-21T08:31:25.685324+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:29:15.349124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:15.349124+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684159+0000","last_change":"2024-09-21T08:30:57.611316+0000","last_active":"2024-09-21T08:31:25.684159+0000","last_peered":"2024-09-21T08:31:25.684159+0000","last_clean":"2024-09-21T08:31:25.684159+0000","last_became_active":"2024-09-21T08:30:57.610910+0000","last_became_peered":"2024-09-21T08:30:57.610910+0000","last_unstale":"2024-09-21T08:31:25.684159+0000","last_undegraded":"2024-09-21T08:31:25.684159+0000","last_fullsized":"2024-09-21T08:31:25.684159+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:09.116412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.116412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1d","version":"248'84","reported_seq":695,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.686310+0000","last_change":"2024-09-21T08:28:53.595944+0000","last_active":"2024-09-21T08:31:25.686310+0000","last_peered":"2024-09-21T08:31:25.686310+0000","last_clean":"2024-09-21T08:28:51.587189+0000","last_became_active":"2024-09-21T08:28:53.595944+0000","last_became_peered":"2024-09-21T08:28:53.595944+0000","last_unstale":"2024-09-21T08:31:25.686310+0000","last_undegraded":"2024-09-21T08:31:25.686310+0000","last_fullsized":"2024-09-21T08:28:53.578716+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'84","last_scrub_stamp":"2024-09-21T08:28:12.352559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.352559+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":73,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.072813132000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6e","length":"3"},{"start":"73","length":"1"}]},{"pgid":"6.1c","version":"73'1","reported_seq":410,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.695099+0000","last_change":"2024-09-21T08:28:53.589802+0000","last_active":"2024-09-21T08:31:25.695099+0000","last_peered":"2024-09-21T08:31:25.695099+0000","last_clean":"2024-09-21T08:28:51.557351+0000","last_became_active":"2024-09-21T08:28:53.589802+0000","last_became_peered":"2024-09-21T08:28:53.589802+0000","last_unstale":"2024-09-21T08:31:25.695099+0000","last_undegraded":"2024-09-21T08:28:53.577740+0000","last_fullsized":"2024-09-21T08:28:53.577630+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:27:14.826047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.826047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"198'2","reported_seq":430,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683808+0000","last_change":"2024-09-21T08:30:57.634005+0000","last_active":"2024-09-21T08:31:25.683808+0000","last_peered":"2024-09-21T08:31:25.683808+0000","last_clean":"2024-09-21T08:31:25.683808+0000","last_became_active":"2024-09-21T08:30:57.611269+0000","last_became_peered":"2024-09-21T08:30:57.611269+0000","last_unstale":"2024-09-21T08:31:25.683808+0000","last_undegraded":"2024-09-21T08:31:25.683808+0000","last_fullsized":"2024-09-21T08:31:25.683808+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'2","last_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_deep_scrub":"198'2","last_deep_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_clean_scrub_stamp":"2024-09-21T08:26:32.819313+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":922,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.685922+0000","last_change":"2024-09-21T08:28:53.597832+0000","last_active":"2024-09-21T08:31:25.685922+0000","last_peered":"2024-09-21T08:31:25.685922+0000","last_clean":"2024-09-21T08:28:51.587327+0000","last_became_active":"2024-09-21T08:28:53.597832+0000","last_became_peered":"2024-09-21T08:28:53.597832+0000","last_unstale":"2024-09-21T08:31:25.685922+0000","last_undegraded":"2024-09-21T08:28:53.582409+0000","last_fullsized":"2024-09-21T08:28:53.582308+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:23:31.530408+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:30.136593+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"464'19","reported_seq":80,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.685577+0000","last_change":"2024-09-21T08:30:57.629872+0000","last_active":"2024-09-21T08:31:25.685577+0000","last_peered":"2024-09-21T08:31:25.685577+0000","last_clean":"2024-09-21T08:31:25.685577+0000","last_became_active":"2024-09-21T08:30:57.613278+0000","last_became_peered":"2024-09-21T08:30:57.613278+0000","last_unstale":"2024-09-21T08:31:25.685577+0000","last_undegraded":"2024-09-21T08:31:25.685577+0000","last_fullsized":"2024-09-21T08:31:25.685577+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"291'15","last_scrub_stamp":"2024-09-21T08:28:14.153130+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.153130+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'129","reported_seq":1152,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.692690+0000","last_change":"2024-09-21T08:31:07.783495+0000","last_active":"2024-09-21T08:31:25.692690+0000","last_peered":"2024-09-21T08:31:25.692690+0000","last_clean":"2024-09-21T08:31:25.692690+0000","last_became_active":"2024-09-21T08:30:59.600508+0000","last_became_peered":"2024-09-21T08:30:59.600508+0000","last_unstale":"2024-09-21T08:31:25.692690+0000","last_undegraded":"2024-09-21T08:31:25.692690+0000","last_fullsized":"2024-09-21T08:31:25.692690+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:07.783361+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:07.783361+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:11.166076+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"492'274","reported_seq":2363,"reported_epoch":492,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.746907+0000","last_change":"2024-09-21T08:31:22.733235+0000","last_active":"2024-09-21T08:31:25.746907+0000","last_peered":"2024-09-21T08:31:25.746907+0000","last_clean":"2024-09-21T08:31:25.746907+0000","last_became_active":"2024-09-21T08:31:07.701482+0000","last_became_peered":"2024-09-21T08:31:07.701482+0000","last_unstale":"2024-09-21T08:31:25.746907+0000","last_undegraded":"2024-09-21T08:31:25.746907+0000","last_fullsized":"2024-09-21T08:31:25.746907+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'254","last_scrub_stamp":"2024-09-21T08:30:46.778996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:46.778996+0000","objects_scrubbed":4,"log_size":274,"log_dups_size":0,"ondisk_log_size":274,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.129970325,"stat_sum":{"num_bytes":6505157,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":53,"num_read_kb":66714,"num_write":18,"num_write_kb":6019,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3117748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.f","version":"73'15","reported_seq":454,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.686300+0000","last_change":"2024-09-21T08:30:57.648167+0000","last_active":"2024-09-21T08:31:25.686300+0000","last_peered":"2024-09-21T08:31:25.686300+0000","last_clean":"2024-09-21T08:31:25.686300+0000","last_became_active":"2024-09-21T08:30:57.617493+0000","last_became_peered":"2024-09-21T08:30:57.617493+0000","last_unstale":"2024-09-21T08:31:25.686300+0000","last_undegraded":"2024-09-21T08:31:25.686300+0000","last_fullsized":"2024-09-21T08:31:25.686300+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:26:46.777115+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.777115+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":422,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683339+0000","last_change":"2024-09-21T08:30:57.611521+0000","last_active":"2024-09-21T08:31:25.683339+0000","last_peered":"2024-09-21T08:31:25.683339+0000","last_clean":"2024-09-21T08:31:25.683339+0000","last_became_active":"2024-09-21T08:30:57.611080+0000","last_became_peered":"2024-09-21T08:30:57.611080+0000","last_unstale":"2024-09-21T08:31:25.683339+0000","last_undegraded":"2024-09-21T08:31:25.683339+0000","last_fullsized":"2024-09-21T08:31:25.683339+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:45.434055+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:45.434055+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"463'12","reported_seq":80,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684683+0000","last_change":"2024-09-21T08:30:56.622801+0000","last_active":"2024-09-21T08:31:25.684683+0000","last_peered":"2024-09-21T08:31:25.684683+0000","last_clean":"2024-09-21T08:31:25.684683+0000","last_became_active":"2024-09-21T08:30:56.607770+0000","last_became_peered":"2024-09-21T08:30:56.607770+0000","last_unstale":"2024-09-21T08:31:25.684683+0000","last_undegraded":"2024-09-21T08:31:25.684683+0000","last_fullsized":"2024-09-21T08:31:25.684683+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:26:44.741275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.741275+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":893,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.697718+0000","last_change":"2024-09-21T08:30:21.800509+0000","last_active":"2024-09-21T08:31:25.697718+0000","last_peered":"2024-09-21T08:31:25.697718+0000","last_clean":"2024-09-21T08:31:25.697718+0000","last_became_active":"2024-09-21T08:28:29.619967+0000","last_became_peered":"2024-09-21T08:28:29.619967+0000","last_unstale":"2024-09-21T08:31:25.697718+0000","last_undegraded":"2024-09-21T08:31:25.697718+0000","last_fullsized":"2024-09-21T08:31:25.697718+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:21.800427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:21.800427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:50.788583+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'129","reported_seq":851,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684455+0000","last_change":"2024-09-21T08:31:05.031131+0000","last_active":"2024-09-21T08:31:25.684455+0000","last_peered":"2024-09-21T08:31:25.684455+0000","last_clean":"2024-09-21T08:31:25.684455+0000","last_became_active":"2024-09-21T08:30:59.600613+0000","last_became_peered":"2024-09-21T08:30:59.600613+0000","last_unstale":"2024-09-21T08:31:25.684455+0000","last_undegraded":"2024-09-21T08:31:25.684455+0000","last_fullsized":"2024-09-21T08:31:25.684455+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:05.031027+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.031027+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:14.223000+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"179'19","reported_seq":1917,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.694043+0000","last_change":"2024-09-21T08:31:21.656577+0000","last_active":"2024-09-21T08:31:25.694043+0000","last_peered":"2024-09-21T08:31:25.694043+0000","last_clean":"2024-09-21T08:31:25.694043+0000","last_became_active":"2024-09-21T08:30:57.613435+0000","last_became_peered":"2024-09-21T08:30:57.613435+0000","last_unstale":"2024-09-21T08:31:25.694043+0000","last_undegraded":"2024-09-21T08:31:25.694043+0000","last_fullsized":"2024-09-21T08:31:25.694043+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:31:05.695533+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.695533+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:06.815191+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0037926290000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4,1],"avail_no_missing":["2","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"4.0","version":"73'18","reported_seq":851,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.682945+0000","last_change":"2024-09-21T08:28:53.593785+0000","last_active":"2024-09-21T08:31:25.682945+0000","last_peered":"2024-09-21T08:31:25.682945+0000","last_clean":"2024-09-21T08:28:51.584787+0000","last_became_active":"2024-09-21T08:28:53.593785+0000","last_became_peered":"2024-09-21T08:28:53.593785+0000","last_unstale":"2024-09-21T08:31:25.682945+0000","last_undegraded":"2024-09-21T08:28:53.574919+0000","last_fullsized":"2024-09-21T08:28:53.574229+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-09-21T08:26:54.003173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:54.003173+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":630,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.694069+0000","last_change":"2024-09-21T08:30:57.623808+0000","last_active":"2024-09-21T08:31:25.694069+0000","last_peered":"2024-09-21T08:31:25.694069+0000","last_clean":"2024-09-21T08:31:25.694069+0000","last_became_active":"2024-09-21T08:30:57.622744+0000","last_became_peered":"2024-09-21T08:30:57.622744+0000","last_unstale":"2024-09-21T08:31:25.694069+0000","last_undegraded":"2024-09-21T08:31:25.694069+0000","last_fullsized":"2024-09-21T08:31:25.694069+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:50.648599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:50.648599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683282+0000","last_change":"2024-09-21T08:30:57.625683+0000","last_active":"2024-09-21T08:31:25.683282+0000","last_peered":"2024-09-21T08:31:25.683282+0000","last_clean":"2024-09-21T08:31:25.683282+0000","last_became_active":"2024-09-21T08:30:57.619764+0000","last_became_peered":"2024-09-21T08:30:57.619764+0000","last_unstale":"2024-09-21T08:31:25.683282+0000","last_undegraded":"2024-09-21T08:31:25.683282+0000","last_fullsized":"2024-09-21T08:31:25.683282+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:32.813973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:32.813973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683552+0000","last_change":"2024-09-21T08:30:57.619543+0000","last_active":"2024-09-21T08:31:25.683552+0000","last_peered":"2024-09-21T08:31:25.683552+0000","last_clean":"2024-09-21T08:31:25.683552+0000","last_became_active":"2024-09-21T08:30:57.617414+0000","last_became_peered":"2024-09-21T08:30:57.617414+0000","last_unstale":"2024-09-21T08:31:25.683552+0000","last_undegraded":"2024-09-21T08:31:25.683552+0000","last_fullsized":"2024-09-21T08:31:25.683552+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:16.341564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.341564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"475'130","reported_seq":1157,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.693126+0000","last_change":"2024-09-21T08:31:08.776346+0000","last_active":"2024-09-21T08:31:25.693126+0000","last_peered":"2024-09-21T08:31:25.693126+0000","last_clean":"2024-09-21T08:31:25.693126+0000","last_became_active":"2024-09-21T08:30:59.603597+0000","last_became_peered":"2024-09-21T08:30:59.603597+0000","last_unstale":"2024-09-21T08:31:25.693126+0000","last_undegraded":"2024-09-21T08:31:25.693126+0000","last_fullsized":"2024-09-21T08:31:25.693126+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"475'130","last_scrub_stamp":"2024-09-21T08:31:08.776209+0000","last_deep_scrub":"475'130","last_deep_scrub_stamp":"2024-09-21T08:31:08.776209+0000","last_clean_scrub_stamp":"2024-09-21T08:31:08.776209+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:31.995186+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"492'354","reported_seq":2158,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.758360+0000","last_change":"2024-09-21T08:28:53.605477+0000","last_active":"2024-09-21T08:31:25.758360+0000","last_peered":"2024-09-21T08:31:25.758360+0000","last_clean":"2024-09-21T08:28:52.666434+0000","last_became_active":"2024-09-21T08:28:53.605477+0000","last_became_peered":"2024-09-21T08:28:53.605477+0000","last_unstale":"2024-09-21T08:31:25.758360+0000","last_undegraded":"2024-09-21T08:28:53.586515+0000","last_fullsized":"2024-09-21T08:28:53.586400+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'273","last_scrub_stamp":"2024-09-21T08:27:50.318620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.318620+0000","objects_scrubbed":3,"log_size":354,"log_dups_size":0,"ondisk_log_size":354,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":74,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.073725586999999995,"stat_sum":{"num_bytes":23268488,"num_objects":14,"num_object_clones":12,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":0,"num_read":92,"num_read_kb":111672,"num_write":43,"num_write_kb":17434,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":3845252,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":14}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6e","length":"3"}]},{"pgid":"4.1","version":"239'21","reported_seq":84,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683193+0000","last_change":"2024-09-21T08:30:57.699489+0000","last_active":"2024-09-21T08:31:25.683193+0000","last_peered":"2024-09-21T08:31:25.683193+0000","last_clean":"2024-09-21T08:31:25.683193+0000","last_became_active":"2024-09-21T08:30:57.619835+0000","last_became_peered":"2024-09-21T08:30:57.619835+0000","last_unstale":"2024-09-21T08:31:25.683193+0000","last_undegraded":"2024-09-21T08:31:25.683193+0000","last_fullsized":"2024-09-21T08:31:25.683193+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21","last_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_deep_scrub":"239'21","last_deep_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.969642+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[{"shards":"2,4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"475'2","reported_seq":878,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.685947+0000","last_change":"2024-09-21T08:31:09.109630+0000","last_active":"2024-09-21T08:31:25.685947+0000","last_peered":"2024-09-21T08:31:25.685947+0000","last_clean":"2024-09-21T08:31:25.685947+0000","last_became_active":"2024-09-21T08:30:57.621298+0000","last_became_peered":"2024-09-21T08:30:57.621298+0000","last_unstale":"2024-09-21T08:31:25.685947+0000","last_undegraded":"2024-09-21T08:31:25.685947+0000","last_fullsized":"2024-09-21T08:31:25.685947+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'2","last_scrub_stamp":"2024-09-21T08:31:09.109498+0000","last_deep_scrub":"475'2","last_deep_scrub_stamp":"2024-09-21T08:31:09.109498+0000","last_clean_scrub_stamp":"2024-09-21T08:31:09.109498+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:35.790903+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[{"shards":"1,4,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"463'17","reported_seq":487,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683046+0000","last_change":"2024-09-21T08:30:56.651846+0000","last_active":"2024-09-21T08:31:25.683046+0000","last_peered":"2024-09-21T08:31:25.683046+0000","last_clean":"2024-09-21T08:31:25.683046+0000","last_became_active":"2024-09-21T08:30:56.630142+0000","last_became_peered":"2024-09-21T08:30:56.630142+0000","last_unstale":"2024-09-21T08:31:25.683046+0000","last_undegraded":"2024-09-21T08:31:25.683046+0000","last_fullsized":"2024-09-21T08:31:25.683046+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"317'16","last_scrub_stamp":"2024-09-21T08:30:43.041103+0000","last_deep_scrub":"263'14","last_deep_scrub_stamp":"2024-09-21T08:27:36.998538+0000","last_clean_scrub_stamp":"2024-09-21T08:30:43.041103+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:07.558107+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":872,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.693493+0000","last_change":"2024-09-21T08:28:53.591306+0000","last_active":"2024-09-21T08:31:25.693493+0000","last_peered":"2024-09-21T08:31:25.693493+0000","last_clean":"2024-09-21T08:28:51.554109+0000","last_became_active":"2024-09-21T08:28:53.591306+0000","last_became_peered":"2024-09-21T08:28:53.591306+0000","last_unstale":"2024-09-21T08:31:25.693493+0000","last_undegraded":"2024-09-21T08:31:25.693493+0000","last_fullsized":"2024-09-21T08:28:53.580235+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.344185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.344185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"76'129","reported_seq":1143,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.693763+0000","last_change":"2024-09-21T08:31:05.800297+0000","last_active":"2024-09-21T08:31:25.693763+0000","last_peered":"2024-09-21T08:31:25.693763+0000","last_clean":"2024-09-21T08:31:25.693763+0000","last_became_active":"2024-09-21T08:30:54.612724+0000","last_became_peered":"2024-09-21T08:30:54.612724+0000","last_unstale":"2024-09-21T08:31:25.693763+0000","last_undegraded":"2024-09-21T08:31:25.693763+0000","last_fullsized":"2024-09-21T08:31:25.693763+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:05.800069+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.800069+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:29.047839+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2,1],"avail_no_missing":["5","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"179'43","reported_seq":387,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.682689+0000","last_change":"2024-09-21T08:31:21.596981+0000","last_active":"2024-09-21T08:31:25.682689+0000","last_peered":"2024-09-21T08:31:25.682689+0000","last_clean":"2024-09-21T08:31:25.682689+0000","last_became_active":"2024-09-21T08:30:57.622886+0000","last_became_peered":"2024-09-21T08:30:57.622886+0000","last_unstale":"2024-09-21T08:31:25.682689+0000","last_undegraded":"2024-09-21T08:31:25.682689+0000","last_fullsized":"2024-09-21T08:31:25.682689+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:50.158232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.158232+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0018703109999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"4.2","version":"73'10","reported_seq":470,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684653+0000","last_change":"2024-09-21T08:30:56.663150+0000","last_active":"2024-09-21T08:31:25.684653+0000","last_peered":"2024-09-21T08:31:25.684653+0000","last_clean":"2024-09-21T08:31:25.684653+0000","last_became_active":"2024-09-21T08:30:56.639664+0000","last_became_peered":"2024-09-21T08:30:56.639664+0000","last_unstale":"2024-09-21T08:31:25.684653+0000","last_undegraded":"2024-09-21T08:31:25.684653+0000","last_fullsized":"2024-09-21T08:31:25.684653+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:29:58.300806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.300806+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":920,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.696374+0000","last_change":"2024-09-21T08:28:53.588911+0000","last_active":"2024-09-21T08:31:25.696374+0000","last_peered":"2024-09-21T08:31:25.696374+0000","last_clean":"2024-09-21T08:28:51.554689+0000","last_became_active":"2024-09-21T08:28:53.588911+0000","last_became_peered":"2024-09-21T08:28:53.588911+0000","last_unstale":"2024-09-21T08:31:25.696374+0000","last_undegraded":"2024-09-21T08:28:53.576074+0000","last_fullsized":"2024-09-21T08:28:53.575948+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":242,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:16.003225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.003225+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"340'16","reported_seq":692,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.695097+0000","last_change":"2024-09-21T08:28:53.594258+0000","last_active":"2024-09-21T08:31:25.695097+0000","last_peered":"2024-09-21T08:31:25.695097+0000","last_clean":"2024-09-21T08:28:51.556539+0000","last_became_active":"2024-09-21T08:28:53.594258+0000","last_became_peered":"2024-09-21T08:28:53.594258+0000","last_unstale":"2024-09-21T08:31:25.695097+0000","last_undegraded":"2024-09-21T08:28:53.574644+0000","last_fullsized":"2024-09-21T08:28:53.574523+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'13","last_scrub_stamp":"2024-09-21T08:27:27.874952+0000","last_deep_scrub":"176'11","last_deep_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_clean_scrub_stamp":"2024-09-21T08:27:27.874952+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":716,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.692900+0000","last_change":"2024-09-21T08:30:47.733247+0000","last_active":"2024-09-21T08:31:25.692900+0000","last_peered":"2024-09-21T08:31:25.692900+0000","last_clean":"2024-09-21T08:31:25.692900+0000","last_became_active":"2024-09-21T08:28:37.565624+0000","last_became_peered":"2024-09-21T08:28:37.565624+0000","last_unstale":"2024-09-21T08:31:25.692900+0000","last_undegraded":"2024-09-21T08:31:25.692900+0000","last_fullsized":"2024-09-21T08:31:25.692900+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:47.733148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:47.733148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:00.075734+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"76'129","reported_seq":851,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683669+0000","last_change":"2024-09-21T08:31:06.980561+0000","last_active":"2024-09-21T08:31:25.683669+0000","last_peered":"2024-09-21T08:31:25.683669+0000","last_clean":"2024-09-21T08:31:25.683669+0000","last_became_active":"2024-09-21T08:30:59.602487+0000","last_became_peered":"2024-09-21T08:30:59.602487+0000","last_unstale":"2024-09-21T08:31:25.683669+0000","last_undegraded":"2024-09-21T08:31:25.683669+0000","last_fullsized":"2024-09-21T08:31:25.683669+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:06.980429+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:06.980429+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:10.248475+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"489'237","reported_seq":2396,"reported_epoch":492,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.693772+0000","last_change":"2024-09-21T08:31:25.654607+0000","last_active":"2024-09-21T08:31:25.693772+0000","last_peered":"2024-09-21T08:31:25.693772+0000","last_clean":"2024-09-21T08:31:25.693772+0000","last_became_active":"2024-09-21T08:28:07.549600+0000","last_became_peered":"2024-09-21T08:28:07.549600+0000","last_unstale":"2024-09-21T08:31:25.693772+0000","last_undegraded":"2024-09-21T08:31:25.693772+0000","last_fullsized":"2024-09-21T08:31:25.693772+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'166","last_scrub_stamp":"2024-09-21T08:29:32.253921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.253921+0000","objects_scrubbed":3,"log_size":237,"log_dups_size":0,"ondisk_log_size":237,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.967473731,"stat_sum":{"num_bytes":14474612,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":68,"num_read_kb":84674,"num_write":34,"num_write_kb":12048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1484609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.3","version":"73'19","reported_seq":438,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.696197+0000","last_change":"2024-09-21T08:28:53.593536+0000","last_active":"2024-09-21T08:31:25.696197+0000","last_peered":"2024-09-21T08:31:25.696197+0000","last_clean":"2024-09-21T08:28:51.557090+0000","last_became_active":"2024-09-21T08:28:53.593536+0000","last_became_peered":"2024-09-21T08:28:53.593536+0000","last_unstale":"2024-09-21T08:31:25.696197+0000","last_undegraded":"2024-09-21T08:28:53.574981+0000","last_fullsized":"2024-09-21T08:28:53.574872+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:27:55.260127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:55.260127+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":8}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":422,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.685201+0000","last_change":"2024-09-21T08:30:46.074689+0000","last_active":"2024-09-21T08:31:25.685201+0000","last_peered":"2024-09-21T08:31:25.685201+0000","last_clean":"2024-09-21T08:31:25.685201+0000","last_became_active":"2024-09-21T08:28:04.723998+0000","last_became_peered":"2024-09-21T08:28:04.723998+0000","last_unstale":"2024-09-21T08:31:25.685201+0000","last_undegraded":"2024-09-21T08:31:25.685201+0000","last_fullsized":"2024-09-21T08:31:25.685201+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:46.074613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:30:46.074613+0000","last_clean_scrub_stamp":"2024-09-21T08:30:46.074613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:51.291896+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":368,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.695079+0000","last_change":"2024-09-21T08:28:53.588949+0000","last_active":"2024-09-21T08:31:25.695079+0000","last_peered":"2024-09-21T08:31:25.695079+0000","last_clean":"2024-09-21T08:28:51.557587+0000","last_became_active":"2024-09-21T08:28:53.588949+0000","last_became_peered":"2024-09-21T08:28:53.588949+0000","last_unstale":"2024-09-21T08:31:25.695079+0000","last_undegraded":"2024-09-21T08:31:25.695079+0000","last_fullsized":"2024-09-21T08:28:53.575963+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:39.339126+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:39.339126+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":417,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.683408+0000","last_change":"2024-09-21T08:28:53.600582+0000","last_active":"2024-09-21T08:31:25.683408+0000","last_peered":"2024-09-21T08:31:25.683408+0000","last_clean":"2024-09-21T08:28:51.584621+0000","last_became_active":"2024-09-21T08:28:53.600582+0000","last_became_peered":"2024-09-21T08:28:53.600582+0000","last_unstale":"2024-09-21T08:31:25.683408+0000","last_undegraded":"2024-09-21T08:31:25.683408+0000","last_fullsized":"2024-09-21T08:28:53.576129+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:46.263984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.263984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'129","reported_seq":1151,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.693292+0000","last_change":"2024-09-21T08:31:06.754191+0000","last_active":"2024-09-21T08:31:25.693292+0000","last_peered":"2024-09-21T08:31:25.693292+0000","last_clean":"2024-09-21T08:31:25.693292+0000","last_became_active":"2024-09-21T08:30:59.600493+0000","last_became_peered":"2024-09-21T08:30:59.600493+0000","last_unstale":"2024-09-21T08:31:25.693292+0000","last_undegraded":"2024-09-21T08:31:25.693292+0000","last_fullsized":"2024-09-21T08:31:25.693292+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:06.754121+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:06.754121+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:33.188102+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"492'205","reported_seq":2256,"reported_epoch":492,"state":"active+clean+remapped+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.737878+0000","last_change":"2024-09-21T08:31:25.738055+0000","last_active":"2024-09-21T08:31:25.737878+0000","last_peered":"2024-09-21T08:31:25.737878+0000","last_clean":"2024-09-21T08:31:25.737878+0000","last_became_active":"2024-09-21T08:31:07.700357+0000","last_became_peered":"2024-09-21T08:31:07.700357+0000","last_unstale":"2024-09-21T08:31:25.737878+0000","last_undegraded":"2024-09-21T08:31:25.737878+0000","last_fullsized":"2024-09-21T08:31:25.737878+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":205,"log_dups_size":0,"ondisk_log_size":205,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":57,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.052799059000000002,"stat_sum":{"num_bytes":15969711,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":37,"num_read_kb":29704,"num_write":40,"num_write_kb":20001,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":43,"num_bytes_recovered":27270248,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"13"}]},{"pgid":"4.4","version":"340'35","reported_seq":481,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.684767+0000","last_change":"2024-09-21T08:28:53.603157+0000","last_active":"2024-09-21T08:31:25.684767+0000","last_peered":"2024-09-21T08:31:25.684767+0000","last_clean":"2024-09-21T08:28:51.586037+0000","last_became_active":"2024-09-21T08:28:53.603157+0000","last_became_peered":"2024-09-21T08:28:53.603157+0000","last_unstale":"2024-09-21T08:31:25.684767+0000","last_undegraded":"2024-09-21T08:28:53.579248+0000","last_fullsized":"2024-09-21T08:28:53.579125+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'32","last_scrub_stamp":"2024-09-21T08:27:11.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:11.407384+0000","objects_scrubbed":10,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":69,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683098+0000","last_change":"2024-09-21T08:30:56.627449+0000","last_active":"2024-09-21T08:31:25.683098+0000","last_peered":"2024-09-21T08:31:25.683098+0000","last_clean":"2024-09-21T08:31:25.683098+0000","last_became_active":"2024-09-21T08:30:56.627283+0000","last_became_peered":"2024-09-21T08:30:56.627283+0000","last_unstale":"2024-09-21T08:31:25.683098+0000","last_undegraded":"2024-09-21T08:31:25.683098+0000","last_fullsized":"2024-09-21T08:31:25.683098+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:14.370256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.370256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":944,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.694278+0000","last_change":"2024-09-21T08:28:53.594367+0000","last_active":"2024-09-21T08:31:25.694278+0000","last_peered":"2024-09-21T08:31:25.694278+0000","last_clean":"2024-09-21T08:28:51.557303+0000","last_became_active":"2024-09-21T08:28:53.594367+0000","last_became_peered":"2024-09-21T08:28:53.594367+0000","last_unstale":"2024-09-21T08:31:25.694278+0000","last_undegraded":"2024-09-21T08:28:53.574945+0000","last_fullsized":"2024-09-21T08:28:53.574836+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":862,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.685990+0000","last_change":"2024-09-21T08:30:57.609064+0000","last_active":"2024-09-21T08:31:25.685990+0000","last_peered":"2024-09-21T08:31:25.685990+0000","last_clean":"2024-09-21T08:31:25.685990+0000","last_became_active":"2024-09-21T08:30:57.608038+0000","last_became_peered":"2024-09-21T08:30:57.608038+0000","last_unstale":"2024-09-21T08:31:25.685990+0000","last_undegraded":"2024-09-21T08:31:25.685990+0000","last_fullsized":"2024-09-21T08:31:25.685990+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:45.237804+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:45.237804+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684940+0000","last_change":"2024-09-21T08:30:57.620758+0000","last_active":"2024-09-21T08:31:25.684940+0000","last_peered":"2024-09-21T08:31:25.684940+0000","last_clean":"2024-09-21T08:31:25.684940+0000","last_became_active":"2024-09-21T08:30:57.618058+0000","last_became_peered":"2024-09-21T08:30:57.618058+0000","last_unstale":"2024-09-21T08:31:25.684940+0000","last_undegraded":"2024-09-21T08:31:25.684940+0000","last_fullsized":"2024-09-21T08:31:25.684940+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.844729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.844729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"76'129","reported_seq":1142,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.682445+0000","last_change":"2024-09-21T08:31:06.002480+0000","last_active":"2024-09-21T08:31:25.682445+0000","last_peered":"2024-09-21T08:31:25.682445+0000","last_clean":"2024-09-21T08:31:25.682445+0000","last_became_active":"2024-09-21T08:30:59.600899+0000","last_became_peered":"2024-09-21T08:30:59.600899+0000","last_unstale":"2024-09-21T08:31:25.682445+0000","last_undegraded":"2024-09-21T08:31:25.682445+0000","last_fullsized":"2024-09-21T08:31:25.682445+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:06.002406+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:06.002406+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:15.083339+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"472'161","reported_seq":2196,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684440+0000","last_change":"2024-09-21T08:31:21.599554+0000","last_active":"2024-09-21T08:31:25.684440+0000","last_peered":"2024-09-21T08:31:25.684440+0000","last_clean":"2024-09-21T08:31:25.684440+0000","last_became_active":"2024-09-21T08:31:07.697236+0000","last_became_peered":"2024-09-21T08:31:07.697236+0000","last_unstale":"2024-09-21T08:31:25.684440+0000","last_undegraded":"2024-09-21T08:31:25.684440+0000","last_fullsized":"2024-09-21T08:31:25.684440+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'161","last_scrub_stamp":"2024-09-21T08:31:09.054511+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:31:09.054511+0000","objects_scrubbed":0,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:10.019778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0024518970000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"4.7","version":"73'13","reported_seq":455,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.686499+0000","last_change":"2024-09-21T08:28:53.595533+0000","last_active":"2024-09-21T08:31:25.686499+0000","last_peered":"2024-09-21T08:31:25.686499+0000","last_clean":"2024-09-21T08:28:51.587779+0000","last_became_active":"2024-09-21T08:28:53.595533+0000","last_became_peered":"2024-09-21T08:28:53.595533+0000","last_unstale":"2024-09-21T08:31:25.686499+0000","last_undegraded":"2024-09-21T08:28:53.580029+0000","last_fullsized":"2024-09-21T08:28:53.579372+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-09-21T08:28:16.386950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.386950+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":428,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683248+0000","last_change":"2024-09-21T08:29:56.368212+0000","last_active":"2024-09-21T08:31:25.683248+0000","last_peered":"2024-09-21T08:31:25.683248+0000","last_clean":"2024-09-21T08:31:25.683248+0000","last_became_active":"2024-09-21T08:28:29.629320+0000","last_became_peered":"2024-09-21T08:28:29.629320+0000","last_unstale":"2024-09-21T08:31:25.683248+0000","last_undegraded":"2024-09-21T08:31:25.683248+0000","last_fullsized":"2024-09-21T08:31:25.683248+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:56.368144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:29:56.368144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":896,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684700+0000","last_change":"2024-09-21T08:30:56.650473+0000","last_active":"2024-09-21T08:31:25.684700+0000","last_peered":"2024-09-21T08:31:25.684700+0000","last_clean":"2024-09-21T08:31:25.684700+0000","last_became_active":"2024-09-21T08:30:56.629748+0000","last_became_peered":"2024-09-21T08:30:56.629748+0000","last_unstale":"2024-09-21T08:31:25.684700+0000","last_undegraded":"2024-09-21T08:31:25.684700+0000","last_fullsized":"2024-09-21T08:31:25.684700+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:59.262463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:27:59.262463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":873,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.694434+0000","last_change":"2024-09-21T08:28:53.598122+0000","last_active":"2024-09-21T08:31:25.694434+0000","last_peered":"2024-09-21T08:31:25.694434+0000","last_clean":"2024-09-21T08:28:51.556093+0000","last_became_active":"2024-09-21T08:28:53.598122+0000","last_became_peered":"2024-09-21T08:28:53.598122+0000","last_unstale":"2024-09-21T08:31:25.694434+0000","last_undegraded":"2024-09-21T08:31:25.694434+0000","last_fullsized":"2024-09-21T08:28:53.584774+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":70,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.374076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.374076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":771,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.695028+0000","last_change":"2024-09-21T08:28:53.587481+0000","last_active":"2024-09-21T08:31:25.695028+0000","last_peered":"2024-09-21T08:31:25.695028+0000","last_clean":"2024-09-21T08:28:51.554685+0000","last_became_active":"2024-09-21T08:28:53.587481+0000","last_became_peered":"2024-09-21T08:28:53.587481+0000","last_unstale":"2024-09-21T08:31:25.695028+0000","last_undegraded":"2024-09-21T08:31:25.695028+0000","last_fullsized":"2024-09-21T08:28:53.575943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:08.831386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:08.831386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"76'129","reported_seq":1160,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.686003+0000","last_change":"2024-09-21T08:31:05.106966+0000","last_active":"2024-09-21T08:31:25.686003+0000","last_peered":"2024-09-21T08:31:25.686003+0000","last_clean":"2024-09-21T08:31:25.686003+0000","last_became_active":"2024-09-21T08:30:59.605942+0000","last_became_peered":"2024-09-21T08:30:59.605942+0000","last_unstale":"2024-09-21T08:31:25.686003+0000","last_undegraded":"2024-09-21T08:31:25.686003+0000","last_fullsized":"2024-09-21T08:31:25.686003+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:05.106891+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.106891+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:15.121805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"462'307","reported_seq":2046,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.685398+0000","last_change":"2024-09-21T08:31:07.701911+0000","last_active":"2024-09-21T08:31:25.685398+0000","last_peered":"2024-09-21T08:31:25.685398+0000","last_clean":"2024-09-21T08:28:52.661593+0000","last_became_active":"2024-09-21T08:31:07.701911+0000","last_became_peered":"2024-09-21T08:31:07.701911+0000","last_unstale":"2024-09-21T08:31:25.685398+0000","last_undegraded":"2024-09-21T08:31:07.695987+0000","last_fullsized":"2024-09-21T08:31:07.695856+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'245","last_scrub_stamp":"2024-09-21T08:26:30.864040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.864040+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":72,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":1.075799959,"stat_sum":{"num_bytes":4186220,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":25509,"num_write":17,"num_write_kb":5036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":926989,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":2}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6e","length":"3"},{"start":"74","length":"2"}]},{"pgid":"4.6","version":"73'12","reported_seq":883,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.686116+0000","last_change":"2024-09-21T08:28:53.603118+0000","last_active":"2024-09-21T08:31:25.686116+0000","last_peered":"2024-09-21T08:31:25.686116+0000","last_clean":"2024-09-21T08:28:51.587443+0000","last_became_active":"2024-09-21T08:28:53.603118+0000","last_became_peered":"2024-09-21T08:28:53.603118+0000","last_unstale":"2024-09-21T08:31:25.686116+0000","last_undegraded":"2024-09-21T08:28:53.582699+0000","last_fullsized":"2024-09-21T08:28:53.582597+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:26:07.384640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:07.384640+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":6}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684492+0000","last_change":"2024-09-21T08:30:57.626533+0000","last_active":"2024-09-21T08:31:25.684492+0000","last_peered":"2024-09-21T08:31:25.684492+0000","last_clean":"2024-09-21T08:31:25.684492+0000","last_became_active":"2024-09-21T08:30:57.621632+0000","last_became_peered":"2024-09-21T08:30:57.621632+0000","last_unstale":"2024-09-21T08:31:25.684492+0000","last_undegraded":"2024-09-21T08:31:25.684492+0000","last_fullsized":"2024-09-21T08:31:25.684492+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.917891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.917891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1375,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.682655+0000","last_change":"2024-09-21T08:28:53.594845+0000","last_active":"2024-09-21T08:31:25.682655+0000","last_peered":"2024-09-21T08:31:25.682655+0000","last_clean":"2024-09-21T08:28:52.670294+0000","last_became_active":"2024-09-21T08:28:53.594845+0000","last_became_peered":"2024-09-21T08:28:53.594845+0000","last_unstale":"2024-09-21T08:31:25.682655+0000","last_undegraded":"2024-09-21T08:28:53.577170+0000","last_fullsized":"2024-09-21T08:28:53.575736+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:30.136615+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":452,"num_read_kb":447,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":890,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.694700+0000","last_change":"2024-09-21T08:30:56.627295+0000","last_active":"2024-09-21T08:31:25.694700+0000","last_peered":"2024-09-21T08:31:25.694700+0000","last_clean":"2024-09-21T08:31:25.694700+0000","last_became_active":"2024-09-21T08:30:56.626598+0000","last_became_peered":"2024-09-21T08:30:56.626598+0000","last_unstale":"2024-09-21T08:31:25.694700+0000","last_undegraded":"2024-09-21T08:31:25.694700+0000","last_fullsized":"2024-09-21T08:31:25.694700+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.812789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.812789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":428,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.685106+0000","last_change":"2024-09-21T08:30:56.638995+0000","last_active":"2024-09-21T08:31:25.685106+0000","last_peered":"2024-09-21T08:31:25.685106+0000","last_clean":"2024-09-21T08:31:25.685106+0000","last_became_active":"2024-09-21T08:30:56.638712+0000","last_became_peered":"2024-09-21T08:30:56.638712+0000","last_unstale":"2024-09-21T08:31:25.685106+0000","last_undegraded":"2024-09-21T08:31:25.685106+0000","last_fullsized":"2024-09-21T08:31:25.685106+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:17.424515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:17.424515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"478'163","reported_seq":1954,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.695234+0000","last_change":"2024-09-21T08:31:07.692343+0000","last_active":"2024-09-21T08:31:25.695234+0000","last_peered":"2024-09-21T08:31:25.695234+0000","last_clean":"2024-09-21T08:28:51.554645+0000","last_became_active":"2024-09-21T08:31:07.692343+0000","last_became_peered":"2024-09-21T08:31:07.692343+0000","last_unstale":"2024-09-21T08:31:25.695234+0000","last_undegraded":"2024-09-21T08:31:07.688919+0000","last_fullsized":"2024-09-21T08:31:07.688518+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'138","last_scrub_stamp":"2024-09-21T08:28:41.855754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:41.855754+0000","objects_scrubbed":1,"log_size":163,"log_dups_size":0,"ondisk_log_size":163,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":68,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0027851600000000001,"stat_sum":{"num_bytes":6518338,"num_objects":4,"num_object_clones":4,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":26,"num_read_kb":31642,"num_write":15,"num_write_kb":4830,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":4641444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6e","length":"3"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"4.5","version":"464'24","reported_seq":94,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684890+0000","last_change":"2024-09-21T08:30:57.660855+0000","last_active":"2024-09-21T08:31:25.684890+0000","last_peered":"2024-09-21T08:31:25.684890+0000","last_clean":"2024-09-21T08:31:25.684890+0000","last_became_active":"2024-09-21T08:30:57.616133+0000","last_became_peered":"2024-09-21T08:30:57.616133+0000","last_unstale":"2024-09-21T08:31:25.684890+0000","last_undegraded":"2024-09-21T08:31:25.684890+0000","last_fullsized":"2024-09-21T08:31:25.684890+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-21T08:28:13.956509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.956509+0000","objects_scrubbed":8,"log_size":24,"log_dups_size":0,"ondisk_log_size":24,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":8,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4,5","objects":8}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":788,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.696586+0000","last_change":"2024-09-21T08:30:56.629168+0000","last_active":"2024-09-21T08:31:25.696586+0000","last_peered":"2024-09-21T08:31:25.696586+0000","last_clean":"2024-09-21T08:31:25.696586+0000","last_became_active":"2024-09-21T08:30:56.628829+0000","last_became_peered":"2024-09-21T08:30:56.628829+0000","last_unstale":"2024-09-21T08:31:25.696586+0000","last_undegraded":"2024-09-21T08:31:25.696586+0000","last_fullsized":"2024-09-21T08:31:25.696586+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.045353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.045353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":1135,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.682577+0000","last_change":"2024-09-21T08:30:49.608935+0000","last_active":"2024-09-21T08:31:25.682577+0000","last_peered":"2024-09-21T08:31:25.682577+0000","last_clean":"2024-09-21T08:28:51.584594+0000","last_became_active":"2024-09-21T08:30:49.608935+0000","last_became_peered":"2024-09-21T08:30:49.608935+0000","last_unstale":"2024-09-21T08:31:25.682577+0000","last_undegraded":"2024-09-21T08:31:25.682577+0000","last_fullsized":"2024-09-21T08:30:49.602844+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:25:35.708634+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:25:35.708634+0000","objects_scrubbed":2,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:30.136454+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":53669,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":322018,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":351,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.695025+0000","last_change":"2024-09-21T08:28:53.589976+0000","last_active":"2024-09-21T08:31:25.695025+0000","last_peered":"2024-09-21T08:31:25.695025+0000","last_clean":"2024-09-21T08:28:51.557022+0000","last_became_active":"2024-09-21T08:28:53.589976+0000","last_became_peered":"2024-09-21T08:28:53.589976+0000","last_unstale":"2024-09-21T08:31:25.695025+0000","last_undegraded":"2024-09-21T08:31:25.695025+0000","last_fullsized":"2024-09-21T08:28:53.576557+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":904,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.695364+0000","last_change":"2024-09-21T08:30:56.627506+0000","last_active":"2024-09-21T08:31:25.695364+0000","last_peered":"2024-09-21T08:31:25.695364+0000","last_clean":"2024-09-21T08:31:25.695364+0000","last_became_active":"2024-09-21T08:30:56.627007+0000","last_became_peered":"2024-09-21T08:30:56.627007+0000","last_unstale":"2024-09-21T08:31:25.695364+0000","last_undegraded":"2024-09-21T08:31:25.695364+0000","last_fullsized":"2024-09-21T08:31:25.695364+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:48.726133+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:48.726133+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"481'158","reported_seq":807,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.686349+0000","last_change":"2024-09-21T08:31:07.701999+0000","last_active":"2024-09-21T08:31:25.686349+0000","last_peered":"2024-09-21T08:31:25.686349+0000","last_clean":"2024-09-21T08:28:52.661631+0000","last_became_active":"2024-09-21T08:31:07.701999+0000","last_became_peered":"2024-09-21T08:31:07.701999+0000","last_unstale":"2024-09-21T08:31:25.686349+0000","last_undegraded":"2024-09-21T08:31:07.697349+0000","last_fullsized":"2024-09-21T08:31:07.697220+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'93","last_scrub_stamp":"2024-09-21T08:28:11.373358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:11.373358+0000","objects_scrubbed":3,"log_size":158,"log_dups_size":0,"ondisk_log_size":158,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":77,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.070749670000000001,"stat_sum":{"num_bytes":17352725,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":16,"num_read_kb":8938,"num_write":25,"num_write_kb":7172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3285818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":7}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.e","version":"73'11","reported_seq":81,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683708+0000","last_change":"2024-09-21T08:30:56.656178+0000","last_active":"2024-09-21T08:31:25.683708+0000","last_peered":"2024-09-21T08:31:25.683708+0000","last_clean":"2024-09-21T08:31:25.683708+0000","last_became_active":"2024-09-21T08:30:56.626093+0000","last_became_peered":"2024-09-21T08:30:56.626093+0000","last_unstale":"2024-09-21T08:31:25.683708+0000","last_undegraded":"2024-09-21T08:31:25.683708+0000","last_fullsized":"2024-09-21T08:31:25.683708+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:28:14.416729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.416729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683771+0000","last_change":"2024-09-21T08:30:57.630426+0000","last_active":"2024-09-21T08:31:25.683771+0000","last_peered":"2024-09-21T08:31:25.683771+0000","last_clean":"2024-09-21T08:31:25.683771+0000","last_became_active":"2024-09-21T08:30:57.626173+0000","last_became_peered":"2024-09-21T08:30:57.626173+0000","last_unstale":"2024-09-21T08:31:25.683771+0000","last_undegraded":"2024-09-21T08:31:25.683771+0000","last_fullsized":"2024-09-21T08:31:25.683771+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:28.347474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:28.347474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":913,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.695918+0000","last_change":"2024-09-21T08:30:56.619286+0000","last_active":"2024-09-21T08:31:25.695918+0000","last_peered":"2024-09-21T08:31:25.695918+0000","last_clean":"2024-09-21T08:31:25.695918+0000","last_became_active":"2024-09-21T08:30:56.618484+0000","last_became_peered":"2024-09-21T08:30:56.618484+0000","last_unstale":"2024-09-21T08:31:25.695918+0000","last_undegraded":"2024-09-21T08:31:25.695918+0000","last_fullsized":"2024-09-21T08:31:25.695918+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:55.150058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:32.961381+0000","last_clean_scrub_stamp":"2024-09-21T08:29:55.150058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":760,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.695359+0000","last_change":"2024-09-21T08:28:53.601979+0000","last_active":"2024-09-21T08:31:25.695359+0000","last_peered":"2024-09-21T08:31:25.695359+0000","last_clean":"2024-09-21T08:28:51.557371+0000","last_became_active":"2024-09-21T08:28:53.601979+0000","last_became_peered":"2024-09-21T08:28:53.601979+0000","last_unstale":"2024-09-21T08:31:25.695359+0000","last_undegraded":"2024-09-21T08:31:25.695359+0000","last_fullsized":"2024-09-21T08:28:53.577942+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.977956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"76'129","reported_seq":1156,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.692427+0000","last_change":"2024-09-21T08:31:09.810905+0000","last_active":"2024-09-21T08:31:25.692427+0000","last_peered":"2024-09-21T08:31:25.692427+0000","last_clean":"2024-09-21T08:31:25.692427+0000","last_became_active":"2024-09-21T08:30:59.604289+0000","last_became_peered":"2024-09-21T08:30:59.604289+0000","last_unstale":"2024-09-21T08:31:25.692427+0000","last_undegraded":"2024-09-21T08:31:25.692427+0000","last_fullsized":"2024-09-21T08:31:25.692427+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:09.810830+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:09.810830+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:32.652921+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"256'199","reported_seq":2226,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.694135+0000","last_change":"2024-09-21T08:31:21.656673+0000","last_active":"2024-09-21T08:31:25.694135+0000","last_peered":"2024-09-21T08:31:25.694135+0000","last_clean":"2024-09-21T08:31:25.694135+0000","last_became_active":"2024-09-21T08:28:32.558440+0000","last_became_peered":"2024-09-21T08:28:32.558440+0000","last_unstale":"2024-09-21T08:31:25.694135+0000","last_undegraded":"2024-09-21T08:31:25.694135+0000","last_fullsized":"2024-09-21T08:31:25.694135+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'199","last_scrub_stamp":"2024-09-21T08:30:17.742629+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:17.742629+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:35.507957+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00092334399999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"4.d","version":"345'25","reported_seq":87,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.685660+0000","last_change":"2024-09-21T08:30:56.655936+0000","last_active":"2024-09-21T08:31:25.685660+0000","last_peered":"2024-09-21T08:31:25.685660+0000","last_clean":"2024-09-21T08:31:25.685660+0000","last_became_active":"2024-09-21T08:30:56.617615+0000","last_became_peered":"2024-09-21T08:30:56.617615+0000","last_unstale":"2024-09-21T08:31:25.685660+0000","last_undegraded":"2024-09-21T08:31:25.685660+0000","last_fullsized":"2024-09-21T08:31:25.685660+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'25","last_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_deep_scrub":"345'25","last_deep_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.241931+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":69,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684099+0000","last_change":"2024-09-21T08:30:56.631627+0000","last_active":"2024-09-21T08:31:25.684099+0000","last_peered":"2024-09-21T08:31:25.684099+0000","last_clean":"2024-09-21T08:31:25.684099+0000","last_became_active":"2024-09-21T08:30:56.630646+0000","last_became_peered":"2024-09-21T08:30:56.630646+0000","last_unstale":"2024-09-21T08:31:25.684099+0000","last_undegraded":"2024-09-21T08:31:25.684099+0000","last_fullsized":"2024-09-21T08:31:25.684099+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:30.909902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.909902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":436,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684515+0000","last_change":"2024-09-21T08:31:08.138364+0000","last_active":"2024-09-21T08:31:25.684515+0000","last_peered":"2024-09-21T08:31:25.684515+0000","last_clean":"2024-09-21T08:31:25.684515+0000","last_became_active":"2024-09-21T08:30:57.608826+0000","last_became_peered":"2024-09-21T08:30:57.608826+0000","last_unstale":"2024-09-21T08:31:25.684515+0000","last_undegraded":"2024-09-21T08:31:25.684515+0000","last_fullsized":"2024-09-21T08:31:25.684515+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:08.138148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:08.138148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:08.640949+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":69,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.685475+0000","last_change":"2024-09-21T08:30:56.617373+0000","last_active":"2024-09-21T08:31:25.685475+0000","last_peered":"2024-09-21T08:31:25.685475+0000","last_clean":"2024-09-21T08:31:25.685475+0000","last_became_active":"2024-09-21T08:30:56.617090+0000","last_became_peered":"2024-09-21T08:30:56.617090+0000","last_unstale":"2024-09-21T08:31:25.685475+0000","last_undegraded":"2024-09-21T08:31:25.685475+0000","last_fullsized":"2024-09-21T08:31:25.685475+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:00.276774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:00.276774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"76'129","reported_seq":845,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.691967+0000","last_change":"2024-09-21T08:31:06.742515+0000","last_active":"2024-09-21T08:31:25.691967+0000","last_peered":"2024-09-21T08:31:25.691967+0000","last_clean":"2024-09-21T08:31:25.691967+0000","last_became_active":"2024-09-21T08:30:54.607659+0000","last_became_peered":"2024-09-21T08:30:54.607659+0000","last_unstale":"2024-09-21T08:31:25.691967+0000","last_undegraded":"2024-09-21T08:31:25.691967+0000","last_fullsized":"2024-09-21T08:31:25.691967+0000","mapping_epoch":460,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":461,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:06.742324+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:06.742324+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:32.446152+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5,1],"avail_no_missing":["2","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"481'110","reported_seq":447,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684532+0000","last_change":"2024-09-21T08:31:21.600173+0000","last_active":"2024-09-21T08:31:25.684532+0000","last_peered":"2024-09-21T08:31:25.684532+0000","last_clean":"2024-09-21T08:31:25.684532+0000","last_became_active":"2024-09-21T08:30:57.622924+0000","last_became_peered":"2024-09-21T08:30:57.622924+0000","last_unstale":"2024-09-21T08:31:25.684532+0000","last_undegraded":"2024-09-21T08:31:25.684532+0000","last_fullsized":"2024-09-21T08:31:25.684532+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:28:10.372149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.372149+0000","objects_scrubbed":1,"log_size":110,"log_dups_size":0,"ondisk_log_size":110,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0030577209999999998,"stat_sum":{"num_bytes":13171734,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":29,"num_read_kb":28274,"num_write":39,"num_write_kb":16470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22,"num_bytes_recovered":23448304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[{"shards":"2,4,5","objects":7}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"4.c","version":"73'10","reported_seq":80,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684037+0000","last_change":"2024-09-21T08:30:57.643080+0000","last_active":"2024-09-21T08:31:25.684037+0000","last_peered":"2024-09-21T08:31:25.684037+0000","last_clean":"2024-09-21T08:31:25.684037+0000","last_became_active":"2024-09-21T08:30:57.611720+0000","last_became_peered":"2024-09-21T08:30:57.611720+0000","last_unstale":"2024-09-21T08:31:25.684037+0000","last_undegraded":"2024-09-21T08:31:25.684037+0000","last_fullsized":"2024-09-21T08:31:25.684037+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:26:33.823950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.823950+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.682853+0000","last_change":"2024-09-21T08:30:57.609872+0000","last_active":"2024-09-21T08:31:25.682853+0000","last_peered":"2024-09-21T08:31:25.682853+0000","last_clean":"2024-09-21T08:31:25.682853+0000","last_became_active":"2024-09-21T08:30:57.609402+0000","last_became_peered":"2024-09-21T08:30:57.609402+0000","last_unstale":"2024-09-21T08:31:25.682853+0000","last_undegraded":"2024-09-21T08:31:25.682853+0000","last_fullsized":"2024-09-21T08:31:25.682853+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:54.216528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:54.216528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"340'10","reported_seq":409,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.697609+0000","last_change":"2024-09-21T08:28:53.589819+0000","last_active":"2024-09-21T08:31:25.697609+0000","last_peered":"2024-09-21T08:31:25.697609+0000","last_clean":"2024-09-21T08:28:51.555429+0000","last_became_active":"2024-09-21T08:28:53.589819+0000","last_became_peered":"2024-09-21T08:28:53.589819+0000","last_unstale":"2024-09-21T08:31:25.697609+0000","last_undegraded":"2024-09-21T08:28:53.578694+0000","last_fullsized":"2024-09-21T08:28:53.577943+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:27:51.312273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.312273+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":1}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":69,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684063+0000","last_change":"2024-09-21T08:30:56.627132+0000","last_active":"2024-09-21T08:31:25.684063+0000","last_peered":"2024-09-21T08:31:25.684063+0000","last_clean":"2024-09-21T08:31:25.684063+0000","last_became_active":"2024-09-21T08:30:56.626799+0000","last_became_peered":"2024-09-21T08:30:56.626799+0000","last_unstale":"2024-09-21T08:31:25.684063+0000","last_undegraded":"2024-09-21T08:31:25.684063+0000","last_fullsized":"2024-09-21T08:31:25.684063+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:40.458054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:40.458054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"472'386","reported_seq":2391,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.682467+0000","last_change":"2024-09-21T08:31:21.595945+0000","last_active":"2024-09-21T08:31:25.682467+0000","last_peered":"2024-09-21T08:31:25.682467+0000","last_clean":"2024-09-21T08:31:25.682467+0000","last_became_active":"2024-09-21T08:31:07.695713+0000","last_became_peered":"2024-09-21T08:31:07.695713+0000","last_unstale":"2024-09-21T08:31:25.682467+0000","last_undegraded":"2024-09-21T08:31:25.682467+0000","last_fullsized":"2024-09-21T08:31:25.682467+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'386","last_scrub_stamp":"2024-09-21T08:31:08.026519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:08.026519+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:20.554821+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.001011947,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"4.b","version":"73'9","reported_seq":960,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683891+0000","last_change":"2024-09-21T08:31:25.139313+0000","last_active":"2024-09-21T08:31:25.683891+0000","last_peered":"2024-09-21T08:31:25.683891+0000","last_clean":"2024-09-21T08:31:25.683891+0000","last_became_active":"2024-09-21T08:30:56.626474+0000","last_became_peered":"2024-09-21T08:30:56.626474+0000","last_unstale":"2024-09-21T08:31:25.683891+0000","last_undegraded":"2024-09-21T08:31:25.683891+0000","last_fullsized":"2024-09-21T08:31:25.683891+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:31:25.139208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:31:25.139208+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:52.945547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":921,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.694430+0000","last_change":"2024-09-21T08:28:59.403264+0000","last_active":"2024-09-21T08:31:25.694430+0000","last_peered":"2024-09-21T08:31:25.694430+0000","last_clean":"2024-09-21T08:31:25.694430+0000","last_became_active":"2024-09-21T08:28:29.609526+0000","last_became_peered":"2024-09-21T08:28:29.609526+0000","last_unstale":"2024-09-21T08:31:25.694430+0000","last_undegraded":"2024-09-21T08:31:25.694430+0000","last_fullsized":"2024-09-21T08:31:25.694430+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:59.403194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.403194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":69,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683355+0000","last_change":"2024-09-21T08:30:56.631993+0000","last_active":"2024-09-21T08:31:25.683355+0000","last_peered":"2024-09-21T08:31:25.683355+0000","last_clean":"2024-09-21T08:31:25.683355+0000","last_became_active":"2024-09-21T08:30:56.630411+0000","last_became_peered":"2024-09-21T08:30:56.630411+0000","last_unstale":"2024-09-21T08:31:25.683355+0000","last_undegraded":"2024-09-21T08:31:25.683355+0000","last_fullsized":"2024-09-21T08:31:25.683355+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:23.989969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:23.989969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":374,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.692577+0000","last_change":"2024-09-21T08:28:53.591119+0000","last_active":"2024-09-21T08:31:25.692577+0000","last_peered":"2024-09-21T08:31:25.692577+0000","last_clean":"2024-09-21T08:28:51.552940+0000","last_became_active":"2024-09-21T08:28:53.591119+0000","last_became_peered":"2024-09-21T08:28:53.591119+0000","last_unstale":"2024-09-21T08:31:25.692577+0000","last_undegraded":"2024-09-21T08:31:25.692577+0000","last_fullsized":"2024-09-21T08:28:53.575921+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:36.308161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:36.308161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":["2","1"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"489'390","reported_seq":2520,"reported_epoch":492,"state":"active+clean+remapped+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.685426+0000","last_change":"2024-09-21T08:31:24.749011+0000","last_active":"2024-09-21T08:31:25.685426+0000","last_peered":"2024-09-21T08:31:25.685426+0000","last_clean":"2024-09-21T08:31:25.685426+0000","last_became_active":"2024-09-21T08:31:07.699160+0000","last_became_peered":"2024-09-21T08:31:07.699160+0000","last_unstale":"2024-09-21T08:31:25.685426+0000","last_undegraded":"2024-09-21T08:31:25.685426+0000","last_fullsized":"2024-09-21T08:31:25.685426+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":390,"log_dups_size":0,"ondisk_log_size":390,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":59,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.067869772999999994,"stat_sum":{"num_bytes":20138883,"num_objects":10,"num_object_clones":8,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":10,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":1,"num_read":93,"num_read_kb":96436,"num_write":65,"num_write_kb":21300,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":28545483,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,2],"avail_no_missing":["1","2","4"],"object_location_counts":[{"shards":"1,2,4","objects":10}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"11"}]},{"pgid":"4.a","version":"73'19","reported_seq":437,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.684257+0000","last_change":"2024-09-21T08:28:53.600149+0000","last_active":"2024-09-21T08:31:25.684257+0000","last_peered":"2024-09-21T08:31:25.684257+0000","last_clean":"2024-09-21T08:28:51.585660+0000","last_became_active":"2024-09-21T08:28:53.600149+0000","last_became_peered":"2024-09-21T08:28:53.600149+0000","last_unstale":"2024-09-21T08:31:25.684257+0000","last_undegraded":"2024-09-21T08:28:53.586423+0000","last_fullsized":"2024-09-21T08:28:53.586235+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[{"shards":"1,5","objects":9}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":788,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.696633+0000","last_change":"2024-09-21T08:30:57.613094+0000","last_active":"2024-09-21T08:31:25.696633+0000","last_peered":"2024-09-21T08:31:25.696633+0000","last_clean":"2024-09-21T08:31:25.696633+0000","last_became_active":"2024-09-21T08:30:57.612659+0000","last_became_peered":"2024-09-21T08:30:57.612659+0000","last_unstale":"2024-09-21T08:31:25.696633+0000","last_undegraded":"2024-09-21T08:31:25.696633+0000","last_fullsized":"2024-09-21T08:31:25.696633+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.901202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.901202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,2],"avail_no_missing":["5","2","4"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":871,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.693329+0000","last_change":"2024-09-21T08:28:53.592593+0000","last_active":"2024-09-21T08:31:25.693329+0000","last_peered":"2024-09-21T08:31:25.693329+0000","last_clean":"2024-09-21T08:28:51.554029+0000","last_became_active":"2024-09-21T08:28:53.592593+0000","last_became_peered":"2024-09-21T08:28:53.592593+0000","last_unstale":"2024-09-21T08:31:25.693329+0000","last_undegraded":"2024-09-21T08:31:25.693329+0000","last_fullsized":"2024-09-21T08:28:53.577826+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:15.362045+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:15.362045+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":367,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.682194+0000","last_change":"2024-09-21T08:28:53.586960+0000","last_active":"2024-09-21T08:31:25.682194+0000","last_peered":"2024-09-21T08:31:25.682194+0000","last_clean":"2024-09-21T08:28:51.584413+0000","last_became_active":"2024-09-21T08:28:53.586960+0000","last_became_peered":"2024-09-21T08:28:53.586960+0000","last_unstale":"2024-09-21T08:31:25.682194+0000","last_undegraded":"2024-09-21T08:31:25.682194+0000","last_fullsized":"2024-09-21T08:28:53.575166+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:30.136629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"491'124","reported_seq":1347,"reported_epoch":492,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:31:25.695852+0000","last_change":"2024-09-21T08:31:25.654666+0000","last_active":"2024-09-21T08:31:25.695852+0000","last_peered":"2024-09-21T08:31:25.695852+0000","last_clean":"2024-09-21T08:31:25.695852+0000","last_became_active":"2024-09-21T08:31:07.701495+0000","last_became_peered":"2024-09-21T08:31:07.701495+0000","last_unstale":"2024-09-21T08:31:25.695852+0000","last_undegraded":"2024-09-21T08:31:25.695852+0000","last_fullsized":"2024-09-21T08:31:25.695852+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"433'114","last_scrub_stamp":"2024-09-21T08:30:41.677928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:41.677928+0000","objects_scrubbed":2,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":3,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.132729289,"stat_sum":{"num_bytes":3703393,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":2366,"num_write":12,"num_write_kb":2252,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":591936,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"1"},{"start":"c7","length":"1"}]},{"pgid":"4.9","version":"73'12","reported_seq":80,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683617+0000","last_change":"2024-09-21T08:30:57.948055+0000","last_active":"2024-09-21T08:31:25.683617+0000","last_peered":"2024-09-21T08:31:25.683617+0000","last_clean":"2024-09-21T08:31:25.683617+0000","last_became_active":"2024-09-21T08:30:57.624413+0000","last_became_peered":"2024-09-21T08:30:57.624413+0000","last_unstale":"2024-09-21T08:31:25.683617+0000","last_undegraded":"2024-09-21T08:31:25.683617+0000","last_fullsized":"2024-09-21T08:31:25.683617+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:25:41.463820+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.463820+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":5}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.685546+0000","last_change":"2024-09-21T08:30:57.621571+0000","last_active":"2024-09-21T08:31:25.685546+0000","last_peered":"2024-09-21T08:31:25.685546+0000","last_clean":"2024-09-21T08:31:25.685546+0000","last_became_active":"2024-09-21T08:30:57.619377+0000","last_became_peered":"2024-09-21T08:30:57.619377+0000","last_unstale":"2024-09-21T08:31:25.685546+0000","last_undegraded":"2024-09-21T08:31:25.685546+0000","last_fullsized":"2024-09-21T08:31:25.685546+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:51.769044+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:51.769044+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":885,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.682841+0000","last_change":"2024-09-21T08:29:59.267735+0000","last_active":"2024-09-21T08:31:25.682841+0000","last_peered":"2024-09-21T08:31:25.682841+0000","last_clean":"2024-09-21T08:31:25.682841+0000","last_became_active":"2024-09-21T08:28:37.566192+0000","last_became_peered":"2024-09-21T08:28:37.566192+0000","last_unstale":"2024-09-21T08:31:25.682841+0000","last_undegraded":"2024-09-21T08:31:25.682841+0000","last_fullsized":"2024-09-21T08:31:25.682841+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:59.267668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:29:59.267668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":781,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684039+0000","last_change":"2024-09-21T08:30:57.612738+0000","last_active":"2024-09-21T08:31:25.684039+0000","last_peered":"2024-09-21T08:31:25.684039+0000","last_clean":"2024-09-21T08:31:25.684039+0000","last_became_active":"2024-09-21T08:30:57.611217+0000","last_became_peered":"2024-09-21T08:30:57.611217+0000","last_unstale":"2024-09-21T08:31:25.684039+0000","last_undegraded":"2024-09-21T08:31:25.684039+0000","last_fullsized":"2024-09-21T08:31:25.684039+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:25:25.598341+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4,5],"avail_no_missing":["1","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"492'808","reported_seq":2983,"reported_epoch":492,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:31:25.737047+0000","last_change":"2024-09-21T08:31:25.737096+0000","last_active":"2024-09-21T08:31:25.737047+0000","last_peered":"2024-09-21T08:31:25.737047+0000","last_clean":"2024-09-21T08:31:25.737047+0000","last_became_active":"2024-09-21T08:31:07.705607+0000","last_became_peered":"2024-09-21T08:31:07.705607+0000","last_unstale":"2024-09-21T08:31:25.737047+0000","last_undegraded":"2024-09-21T08:31:25.737047+0000","last_fullsized":"2024-09-21T08:31:25.737047+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'577","last_scrub_stamp":"2024-09-21T08:29:32.265609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.265609+0000","objects_scrubbed":10,"log_size":808,"log_dups_size":0,"ondisk_log_size":808,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":2,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.050786471,"stat_sum":{"num_bytes":26367483,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":83,"num_read_kb":81629,"num_write":59,"num_write_kb":22667,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":2192635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c7","length":"1"}]},{"pgid":"4.8","version":"73'15","reported_seq":1026,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.693193+0000","last_change":"2024-09-21T08:30:57.696703+0000","last_active":"2024-09-21T08:31:25.693193+0000","last_peered":"2024-09-21T08:31:25.693193+0000","last_clean":"2024-09-21T08:31:25.693193+0000","last_became_active":"2024-09-21T08:30:57.622063+0000","last_became_peered":"2024-09-21T08:30:57.622063+0000","last_unstale":"2024-09-21T08:31:25.693193+0000","last_undegraded":"2024-09-21T08:31:25.693193+0000","last_fullsized":"2024-09-21T08:31:25.693193+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:24:46.673359+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.673359+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":7,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4,1],"avail_no_missing":["5","1","4"],"object_location_counts":[{"shards":"1,4,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"476'4","reported_seq":84,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684027+0000","last_change":"2024-09-21T08:31:10.077645+0000","last_active":"2024-09-21T08:31:25.684027+0000","last_peered":"2024-09-21T08:31:25.684027+0000","last_clean":"2024-09-21T08:31:25.684027+0000","last_became_active":"2024-09-21T08:30:57.620311+0000","last_became_peered":"2024-09-21T08:30:57.620311+0000","last_unstale":"2024-09-21T08:31:25.684027+0000","last_undegraded":"2024-09-21T08:31:25.684027+0000","last_fullsized":"2024-09-21T08:31:25.684027+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'4","last_scrub_stamp":"2024-09-21T08:31:10.077516+0000","last_deep_scrub":"476'4","last_deep_scrub_stamp":"2024-09-21T08:31:10.077516+0000","last_clean_scrub_stamp":"2024-09-21T08:31:10.077516+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:15.439829+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[{"shards":"1,4,5","objects":2}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"464'18","reported_seq":80,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.682631+0000","last_change":"2024-09-21T08:30:57.632693+0000","last_active":"2024-09-21T08:31:25.682631+0000","last_peered":"2024-09-21T08:31:25.682631+0000","last_clean":"2024-09-21T08:31:25.682631+0000","last_became_active":"2024-09-21T08:30:57.613452+0000","last_became_peered":"2024-09-21T08:30:57.613452+0000","last_unstale":"2024-09-21T08:31:25.682631+0000","last_undegraded":"2024-09-21T08:31:25.682631+0000","last_fullsized":"2024-09-21T08:31:25.682631+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"103'10","last_scrub_stamp":"2024-09-21T08:25:42.708458+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:42.708458+0000","objects_scrubbed":1,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,2],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":879,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.696164+0000","last_change":"2024-09-21T08:30:50.793864+0000","last_active":"2024-09-21T08:31:25.696164+0000","last_peered":"2024-09-21T08:31:25.696164+0000","last_clean":"2024-09-21T08:31:25.696164+0000","last_became_active":"2024-09-21T08:28:29.619471+0000","last_became_peered":"2024-09-21T08:28:29.619471+0000","last_unstale":"2024-09-21T08:31:25.696164+0000","last_undegraded":"2024-09-21T08:31:25.696164+0000","last_fullsized":"2024-09-21T08:31:25.696164+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:50.793789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:50.793789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:08.828948+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"180'50","reported_seq":1680,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.694449+0000","last_change":"2024-09-21T08:28:53.592066+0000","last_active":"2024-09-21T08:31:25.694449+0000","last_peered":"2024-09-21T08:31:25.694449+0000","last_clean":"2024-09-21T08:28:51.555673+0000","last_became_active":"2024-09-21T08:28:53.592066+0000","last_became_peered":"2024-09-21T08:28:53.592066+0000","last_unstale":"2024-09-21T08:31:25.694449+0000","last_undegraded":"2024-09-21T08:31:25.694449+0000","last_fullsized":"2024-09-21T08:28:53.577992+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:26:50.014780+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:50.014780+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":68,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0020276460000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":["2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"6e","length":"3"},{"start":"73","length":"3"},{"start":"79","length":"3"}]},{"pgid":"3.10","version":"0'0","reported_seq":670,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.693088+0000","last_change":"2024-09-21T08:29:39.177460+0000","last_active":"2024-09-21T08:31:25.693088+0000","last_peered":"2024-09-21T08:31:25.693088+0000","last_clean":"2024-09-21T08:31:25.693088+0000","last_became_active":"2024-09-21T08:28:29.610987+0000","last_became_peered":"2024-09-21T08:28:29.610987+0000","last_unstale":"2024-09-21T08:31:25.693088+0000","last_undegraded":"2024-09-21T08:31:25.693088+0000","last_fullsized":"2024-09-21T08:31:25.693088+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.177382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":822,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.685978+0000","last_change":"2024-09-21T08:28:53.602197+0000","last_active":"2024-09-21T08:31:25.685978+0000","last_peered":"2024-09-21T08:31:25.685978+0000","last_clean":"2024-09-21T08:28:51.587601+0000","last_became_active":"2024-09-21T08:28:53.602197+0000","last_became_peered":"2024-09-21T08:28:53.602197+0000","last_unstale":"2024-09-21T08:31:25.685978+0000","last_undegraded":"2024-09-21T08:28:53.578880+0000","last_fullsized":"2024-09-21T08:28:53.578772+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-09-21T08:27:30.806522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.806522+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":3}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":902,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.695381+0000","last_change":"2024-09-21T08:28:53.600847+0000","last_active":"2024-09-21T08:31:25.695381+0000","last_peered":"2024-09-21T08:31:25.695381+0000","last_clean":"2024-09-21T08:28:51.557075+0000","last_became_active":"2024-09-21T08:28:53.600847+0000","last_became_peered":"2024-09-21T08:28:53.600847+0000","last_unstale":"2024-09-21T08:31:25.695381+0000","last_undegraded":"2024-09-21T08:31:25.695381+0000","last_fullsized":"2024-09-21T08:28:53.578503+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.795325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.685510+0000","last_change":"2024-09-21T08:30:57.616327+0000","last_active":"2024-09-21T08:31:25.685510+0000","last_peered":"2024-09-21T08:31:25.685510+0000","last_clean":"2024-09-21T08:31:25.685510+0000","last_became_active":"2024-09-21T08:30:57.615385+0000","last_became_peered":"2024-09-21T08:30:57.615385+0000","last_unstale":"2024-09-21T08:31:25.685510+0000","last_undegraded":"2024-09-21T08:31:25.685510+0000","last_fullsized":"2024-09-21T08:31:25.685510+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:10.851510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.851510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.15","version":"462'448","reported_seq":2340,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.687194+0000","last_change":"2024-09-21T08:28:53.603910+0000","last_active":"2024-09-21T08:31:25.687194+0000","last_peered":"2024-09-21T08:31:25.687194+0000","last_clean":"2024-09-21T08:28:51.587746+0000","last_became_active":"2024-09-21T08:28:53.603910+0000","last_became_peered":"2024-09-21T08:28:53.603910+0000","last_unstale":"2024-09-21T08:31:25.687194+0000","last_undegraded":"2024-09-21T08:28:53.582931+0000","last_fullsized":"2024-09-21T08:28:53.582809+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'262","last_scrub_stamp":"2024-09-21T08:26:29.875830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.875830+0000","objects_scrubbed":3,"log_size":448,"log_dups_size":0,"ondisk_log_size":448,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":74,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.074129519000000005,"stat_sum":{"num_bytes":42555630,"num_objects":21,"num_object_clones":17,"num_object_copies":63,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":21,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":2,"num_read":170,"num_read_kb":185534,"num_write":90,"num_write_kb":42335,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":8623856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":21}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6e","length":"3"}]},{"pgid":"4.16","version":"73'9","reported_seq":374,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.694190+0000","last_change":"2024-09-21T08:28:53.589261+0000","last_active":"2024-09-21T08:31:25.694190+0000","last_peered":"2024-09-21T08:31:25.694190+0000","last_clean":"2024-09-21T08:28:51.555305+0000","last_became_active":"2024-09-21T08:28:53.589261+0000","last_became_peered":"2024-09-21T08:28:53.589261+0000","last_unstale":"2024-09-21T08:31:25.694190+0000","last_undegraded":"2024-09-21T08:28:53.576380+0000","last_fullsized":"2024-09-21T08:28:53.576277+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:10.183035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.183035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.682940+0000","last_change":"2024-09-21T08:30:57.621029+0000","last_active":"2024-09-21T08:31:25.682940+0000","last_peered":"2024-09-21T08:31:25.682940+0000","last_clean":"2024-09-21T08:31:25.682940+0000","last_became_active":"2024-09-21T08:30:57.619257+0000","last_became_peered":"2024-09-21T08:30:57.619257+0000","last_unstale":"2024-09-21T08:31:25.682940+0000","last_undegraded":"2024-09-21T08:31:25.682940+0000","last_fullsized":"2024-09-21T08:31:25.682940+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:46.719545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:46.719545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":423,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.686662+0000","last_change":"2024-09-21T08:28:53.602669+0000","last_active":"2024-09-21T08:31:25.686662+0000","last_peered":"2024-09-21T08:31:25.686662+0000","last_clean":"2024-09-21T08:28:51.587829+0000","last_became_active":"2024-09-21T08:28:53.602669+0000","last_became_peered":"2024-09-21T08:28:53.602669+0000","last_unstale":"2024-09-21T08:31:25.686662+0000","last_undegraded":"2024-09-21T08:31:25.686662+0000","last_fullsized":"2024-09-21T08:28:53.580175+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:05.814753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.814753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":894,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.694757+0000","last_change":"2024-09-21T08:31:07.730422+0000","last_active":"2024-09-21T08:31:25.694757+0000","last_peered":"2024-09-21T08:31:25.694757+0000","last_clean":"2024-09-21T08:31:25.694757+0000","last_became_active":"2024-09-21T08:30:57.613069+0000","last_became_peered":"2024-09-21T08:30:57.613069+0000","last_unstale":"2024-09-21T08:31:25.694757+0000","last_undegraded":"2024-09-21T08:31:25.694757+0000","last_fullsized":"2024-09-21T08:31:25.694757+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:07.730202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:31:07.730202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:24.187329+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4,5],"avail_no_missing":["2","4","5"],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"487'202","reported_seq":2234,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.685528+0000","last_change":"2024-09-21T08:31:21.601320+0000","last_active":"2024-09-21T08:31:25.685528+0000","last_peered":"2024-09-21T08:31:25.685528+0000","last_clean":"2024-09-21T08:31:25.685528+0000","last_became_active":"2024-09-21T08:30:59.600861+0000","last_became_peered":"2024-09-21T08:30:59.600861+0000","last_unstale":"2024-09-21T08:31:25.685528+0000","last_undegraded":"2024-09-21T08:31:25.685528+0000","last_fullsized":"2024-09-21T08:31:25.685528+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'102","last_scrub_stamp":"2024-09-21T08:26:33.255256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.255256+0000","objects_scrubbed":1,"log_size":202,"log_dups_size":0,"ondisk_log_size":202,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.0033634630000000001,"stat_sum":{"num_bytes":9899595,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":4,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":98,"num_read_kb":141433,"num_write":40,"num_write_kb":9384,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27,"num_bytes_recovered":16651842,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[{"shards":"2,4,5","objects":4}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"69","length":"1"},{"start":"6e","length":"16"},{"start":"85","length":"d"},{"start":"93","length":"8"},{"start":"9c","length":"9"},{"start":"a6","length":"9"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"4"},{"start":"bf","length":"1"},{"start":"c3","length":"2"},{"start":"c6","length":"2"},{"start":"cd","length":"1"}]},{"pgid":"4.15","version":"73'9","reported_seq":966,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.696598+0000","last_change":"2024-09-21T08:28:53.601176+0000","last_active":"2024-09-21T08:31:25.696598+0000","last_peered":"2024-09-21T08:31:25.696598+0000","last_clean":"2024-09-21T08:28:51.554870+0000","last_became_active":"2024-09-21T08:28:53.601176+0000","last_became_peered":"2024-09-21T08:28:53.601176+0000","last_unstale":"2024-09-21T08:31:25.696598+0000","last_undegraded":"2024-09-21T08:28:53.577554+0000","last_fullsized":"2024-09-21T08:28:53.576455+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:28:14.535334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.535334+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":4}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":850,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.682808+0000","last_change":"2024-09-21T08:28:53.605117+0000","last_active":"2024-09-21T08:31:25.682808+0000","last_peered":"2024-09-21T08:31:25.682808+0000","last_clean":"2024-09-21T08:28:51.584984+0000","last_became_active":"2024-09-21T08:28:53.605117+0000","last_became_peered":"2024-09-21T08:28:53.605117+0000","last_unstale":"2024-09-21T08:31:25.682808+0000","last_undegraded":"2024-09-21T08:31:25.682808+0000","last_fullsized":"2024-09-21T08:28:53.584510+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.361096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.361096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"340'16","reported_seq":882,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.684513+0000","last_change":"2024-09-21T08:28:53.600719+0000","last_active":"2024-09-21T08:31:25.684513+0000","last_peered":"2024-09-21T08:31:25.684513+0000","last_clean":"2024-09-21T08:28:51.585913+0000","last_became_active":"2024-09-21T08:28:53.600719+0000","last_became_peered":"2024-09-21T08:28:53.600719+0000","last_unstale":"2024-09-21T08:31:25.684513+0000","last_undegraded":"2024-09-21T08:28:53.576377+0000","last_fullsized":"2024-09-21T08:28:53.576220+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:30.136600+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":1}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":69,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684444+0000","last_change":"2024-09-21T08:30:56.611027+0000","last_active":"2024-09-21T08:31:25.684444+0000","last_peered":"2024-09-21T08:31:25.684444+0000","last_clean":"2024-09-21T08:31:25.684444+0000","last_became_active":"2024-09-21T08:30:56.610628+0000","last_became_peered":"2024-09-21T08:30:56.610628+0000","last_unstale":"2024-09-21T08:31:25.684444+0000","last_undegraded":"2024-09-21T08:31:25.684444+0000","last_fullsized":"2024-09-21T08:31:25.684444+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.364388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.364388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"179'63","reported_seq":1234,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.694885+0000","last_change":"2024-09-21T08:28:53.590100+0000","last_active":"2024-09-21T08:31:25.694885+0000","last_peered":"2024-09-21T08:31:25.694885+0000","last_clean":"2024-09-21T08:28:51.556223+0000","last_became_active":"2024-09-21T08:28:53.590100+0000","last_became_peered":"2024-09-21T08:28:53.590100+0000","last_unstale":"2024-09-21T08:31:25.694885+0000","last_undegraded":"2024-09-21T08:31:25.694885+0000","last_fullsized":"2024-09-21T08:28:53.579666+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'63","last_scrub_stamp":"2024-09-21T08:28:13.511512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.511512+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":72,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.010228017000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6e","length":"3"},{"start":"74","length":"2"}]},{"pgid":"4.14","version":"104'15","reported_seq":429,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.684873+0000","last_change":"2024-09-21T08:28:53.602117+0000","last_active":"2024-09-21T08:31:25.684873+0000","last_peered":"2024-09-21T08:31:25.684873+0000","last_clean":"2024-09-21T08:28:51.587113+0000","last_became_active":"2024-09-21T08:28:53.602117+0000","last_became_peered":"2024-09-21T08:28:53.602117+0000","last_unstale":"2024-09-21T08:31:25.684873+0000","last_undegraded":"2024-09-21T08:28:53.578695+0000","last_fullsized":"2024-09-21T08:28:53.578596+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:24:57.624718+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:30.136622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":5}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":72,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683216+0000","last_change":"2024-09-21T08:30:57.691227+0000","last_active":"2024-09-21T08:31:25.683216+0000","last_peered":"2024-09-21T08:31:25.683216+0000","last_clean":"2024-09-21T08:31:25.683216+0000","last_became_active":"2024-09-21T08:30:57.616000+0000","last_became_peered":"2024-09-21T08:30:57.616000+0000","last_unstale":"2024-09-21T08:31:25.683216+0000","last_undegraded":"2024-09-21T08:31:25.683216+0000","last_fullsized":"2024-09-21T08:31:25.683216+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:28:16.875544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.875544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":2979,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[{"shards":"1,2,4","objects":1}],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.15","version":"463'15","reported_seq":1031,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.697603+0000","last_change":"2024-09-21T08:30:56.635835+0000","last_active":"2024-09-21T08:31:25.697603+0000","last_peered":"2024-09-21T08:31:25.697603+0000","last_clean":"2024-09-21T08:31:25.697603+0000","last_became_active":"2024-09-21T08:30:56.626363+0000","last_became_peered":"2024-09-21T08:30:56.626363+0000","last_unstale":"2024-09-21T08:31:25.697603+0000","last_undegraded":"2024-09-21T08:31:25.697603+0000","last_fullsized":"2024-09-21T08:31:25.697603+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684715+0000","last_change":"2024-09-21T08:30:57.620328+0000","last_active":"2024-09-21T08:31:25.684715+0000","last_peered":"2024-09-21T08:31:25.684715+0000","last_clean":"2024-09-21T08:31:25.684715+0000","last_became_active":"2024-09-21T08:30:57.617666+0000","last_became_peered":"2024-09-21T08:30:57.617666+0000","last_unstale":"2024-09-21T08:31:25.684715+0000","last_undegraded":"2024-09-21T08:31:25.684715+0000","last_fullsized":"2024-09-21T08:31:25.684715+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:48.251366+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:48.251366+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,1],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"490'165","reported_seq":1061,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.685048+0000","last_change":"2024-09-21T08:28:53.604012+0000","last_active":"2024-09-21T08:31:25.685048+0000","last_peered":"2024-09-21T08:31:25.685048+0000","last_clean":"2024-09-21T08:28:53.029949+0000","last_became_active":"2024-09-21T08:28:53.604012+0000","last_became_peered":"2024-09-21T08:28:53.604012+0000","last_unstale":"2024-09-21T08:31:25.685048+0000","last_undegraded":"2024-09-21T08:28:53.580392+0000","last_fullsized":"2024-09-21T08:28:53.580253+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'94","last_scrub_stamp":"2024-09-21T08:26:27.208234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.208234+0000","objects_scrubbed":1,"log_size":165,"log_dups_size":0,"ondisk_log_size":165,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":74,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.075698227000000007,"stat_sum":{"num_bytes":16202879,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":22,"num_read_kb":13076,"num_write":40,"num_write_kb":17037,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":10770304,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":8}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6e","length":"3"}]},{"pgid":"4.13","version":"73'11","reported_seq":81,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.682988+0000","last_change":"2024-09-21T08:30:56.661267+0000","last_active":"2024-09-21T08:31:25.682988+0000","last_peered":"2024-09-21T08:31:25.682988+0000","last_clean":"2024-09-21T08:31:25.682988+0000","last_became_active":"2024-09-21T08:30:56.626643+0000","last_became_peered":"2024-09-21T08:30:56.626643+0000","last_unstale":"2024-09-21T08:31:25.682988+0000","last_undegraded":"2024-09-21T08:31:25.682988+0000","last_fullsized":"2024-09-21T08:31:25.682988+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:30:25.153729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:25.153729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:35.550050+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.684374+0000","last_change":"2024-09-21T08:30:57.617620+0000","last_active":"2024-09-21T08:31:25.684374+0000","last_peered":"2024-09-21T08:31:25.684374+0000","last_clean":"2024-09-21T08:31:25.684374+0000","last_became_active":"2024-09-21T08:30:57.616182+0000","last_became_peered":"2024-09-21T08:30:57.616182+0000","last_unstale":"2024-09-21T08:31:25.684374+0000","last_undegraded":"2024-09-21T08:31:25.684374+0000","last_fullsized":"2024-09-21T08:31:25.684374+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:06.889127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:06.889127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":417,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.684748+0000","last_change":"2024-09-21T08:28:53.600540+0000","last_active":"2024-09-21T08:31:25.684748+0000","last_peered":"2024-09-21T08:31:25.684748+0000","last_clean":"2024-09-21T08:28:51.585918+0000","last_became_active":"2024-09-21T08:28:53.600540+0000","last_became_peered":"2024-09-21T08:28:53.600540+0000","last_unstale":"2024-09-21T08:31:25.684748+0000","last_undegraded":"2024-09-21T08:31:25.684748+0000","last_fullsized":"2024-09-21T08:28:53.586917+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:23:33.544062+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:30.136584+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":["1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":626,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.695169+0000","last_change":"2024-09-21T08:30:56.618534+0000","last_active":"2024-09-21T08:31:25.695169+0000","last_peered":"2024-09-21T08:31:25.695169+0000","last_clean":"2024-09-21T08:31:25.695169+0000","last_became_active":"2024-09-21T08:30:56.618134+0000","last_became_peered":"2024-09-21T08:30:56.618134+0000","last_unstale":"2024-09-21T08:31:25.695169+0000","last_undegraded":"2024-09-21T08:31:25.695169+0000","last_fullsized":"2024-09-21T08:31:25.695169+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:23.816669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:23.816669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:31.339821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"455'203","reported_seq":1815,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.684446+0000","last_change":"2024-09-21T08:28:53.605541+0000","last_active":"2024-09-21T08:31:25.684446+0000","last_peered":"2024-09-21T08:31:25.684446+0000","last_clean":"2024-09-21T08:28:52.665793+0000","last_became_active":"2024-09-21T08:28:53.605541+0000","last_became_peered":"2024-09-21T08:28:53.605541+0000","last_unstale":"2024-09-21T08:31:25.684446+0000","last_undegraded":"2024-09-21T08:28:53.581187+0000","last_fullsized":"2024-09-21T08:28:53.581055+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":299,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'44","last_scrub_stamp":"2024-09-21T08:26:27.882963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.882963+0000","objects_scrubbed":1,"log_size":203,"log_dups_size":0,"ondisk_log_size":203,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":73,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0806039080000001,"stat_sum":{"num_bytes":44372956,"num_objects":23,"num_object_clones":21,"num_object_copies":69,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":23,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":1,"num_read":85,"num_read_kb":82593,"num_write":99,"num_write_kb":40253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17,"num_bytes_recovered":11110430,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[{"shards":"1,2","objects":23}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"6e","length":"3"},{"start":"73","length":"1"}]},{"pgid":"4.12","version":"73'9","reported_seq":448,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.684501+0000","last_change":"2024-09-21T08:30:42.016780+0000","last_active":"2024-09-21T08:31:25.684501+0000","last_peered":"2024-09-21T08:31:25.684501+0000","last_clean":"2024-09-21T08:31:25.684501+0000","last_became_active":"2024-09-21T08:28:29.636404+0000","last_became_peered":"2024-09-21T08:28:29.636404+0000","last_unstale":"2024-09-21T08:31:25.684501+0000","last_undegraded":"2024-09-21T08:31:25.684501+0000","last_fullsized":"2024-09-21T08:31:25.684501+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:42.016697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:42.016697+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:01.081137+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.685742+0000","last_change":"2024-09-21T08:30:57.617367+0000","last_active":"2024-09-21T08:31:25.685742+0000","last_peered":"2024-09-21T08:31:25.685742+0000","last_clean":"2024-09-21T08:31:25.685742+0000","last_became_active":"2024-09-21T08:30:57.615854+0000","last_became_peered":"2024-09-21T08:30:57.615854+0000","last_unstale":"2024-09-21T08:31:25.685742+0000","last_undegraded":"2024-09-21T08:31:25.685742+0000","last_fullsized":"2024-09-21T08:31:25.685742+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:49.790900+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:49.790900+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,5],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":735,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683193+0000","last_change":"2024-09-21T08:30:56.626544+0000","last_active":"2024-09-21T08:31:25.683193+0000","last_peered":"2024-09-21T08:31:25.683193+0000","last_clean":"2024-09-21T08:31:25.683193+0000","last_became_active":"2024-09-21T08:30:56.626197+0000","last_became_peered":"2024-09-21T08:30:56.626197+0000","last_unstale":"2024-09-21T08:31:25.683193+0000","last_undegraded":"2024-09-21T08:31:25.683193+0000","last_fullsized":"2024-09-21T08:31:25.683193+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:13.940546+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:13.940546+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":840,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.693615+0000","last_change":"2024-09-21T08:30:56.627439+0000","last_active":"2024-09-21T08:31:25.693615+0000","last_peered":"2024-09-21T08:31:25.693615+0000","last_clean":"2024-09-21T08:31:25.693615+0000","last_became_active":"2024-09-21T08:30:56.627130+0000","last_became_peered":"2024-09-21T08:30:56.627130+0000","last_unstale":"2024-09-21T08:31:25.693615+0000","last_undegraded":"2024-09-21T08:31:25.693615+0000","last_fullsized":"2024-09-21T08:31:25.693615+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:02.065082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:02.065082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.12","version":"492'421","reported_seq":2308,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.746908+0000","last_change":"2024-09-21T08:28:53.587108+0000","last_active":"2024-09-21T08:31:25.746908+0000","last_peered":"2024-09-21T08:31:25.746908+0000","last_clean":"2024-09-21T08:28:52.589762+0000","last_became_active":"2024-09-21T08:28:53.587108+0000","last_became_peered":"2024-09-21T08:28:53.587108+0000","last_unstale":"2024-09-21T08:31:25.746908+0000","last_undegraded":"2024-09-21T08:28:53.574106+0000","last_fullsized":"2024-09-21T08:28:53.573924+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'200","last_scrub_stamp":"2024-09-21T08:26:27.417028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.417028+0000","objects_scrubbed":3,"log_size":421,"log_dups_size":0,"ondisk_log_size":421,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":72,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.027231871000000001,"stat_sum":{"num_bytes":48824463,"num_objects":24,"num_object_clones":20,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":24,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":2,"num_read":182,"num_read_kb":217976,"num_write":110,"num_write_kb":50678,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":8981699,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":24}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6e","length":"3"},{"start":"74","length":"2"}]},{"pgid":"4.11","version":"73'11","reported_seq":457,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.695325+0000","last_change":"2024-09-21T08:28:53.589901+0000","last_active":"2024-09-21T08:31:25.695325+0000","last_peered":"2024-09-21T08:31:25.695325+0000","last_clean":"2024-09-21T08:28:51.555481+0000","last_became_active":"2024-09-21T08:28:53.589901+0000","last_became_peered":"2024-09-21T08:28:53.589901+0000","last_unstale":"2024-09-21T08:31:25.695325+0000","last_undegraded":"2024-09-21T08:28:53.577098+0000","last_fullsized":"2024-09-21T08:28:53.576477+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:52.673377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:52.673377+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":898,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.695153+0000","last_change":"2024-09-21T08:28:53.600628+0000","last_active":"2024-09-21T08:31:25.695153+0000","last_peered":"2024-09-21T08:31:25.695153+0000","last_clean":"2024-09-21T08:28:51.555529+0000","last_became_active":"2024-09-21T08:28:53.600628+0000","last_became_peered":"2024-09-21T08:28:53.600628+0000","last_unstale":"2024-09-21T08:31:25.695153+0000","last_undegraded":"2024-09-21T08:31:25.695153+0000","last_fullsized":"2024-09-21T08:28:53.577666+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.682218+0000","last_change":"2024-09-21T08:30:57.618001+0000","last_active":"2024-09-21T08:31:25.682218+0000","last_peered":"2024-09-21T08:31:25.682218+0000","last_clean":"2024-09-21T08:31:25.682218+0000","last_became_active":"2024-09-21T08:30:57.617550+0000","last_became_peered":"2024-09-21T08:30:57.617550+0000","last_unstale":"2024-09-21T08:31:25.682218+0000","last_undegraded":"2024-09-21T08:31:25.682218+0000","last_fullsized":"2024-09-21T08:31:25.682218+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5,2],"avail_no_missing":["4","2","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":69,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.682855+0000","last_change":"2024-09-21T08:30:56.608178+0000","last_active":"2024-09-21T08:31:25.682855+0000","last_peered":"2024-09-21T08:31:25.682855+0000","last_clean":"2024-09-21T08:31:25.682855+0000","last_became_active":"2024-09-21T08:30:56.607833+0000","last_became_peered":"2024-09-21T08:30:56.607833+0000","last_unstale":"2024-09-21T08:31:25.682855+0000","last_undegraded":"2024-09-21T08:31:25.682855+0000","last_fullsized":"2024-09-21T08:31:25.682855+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:58.167184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.167184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.13","version":"489'73","reported_seq":1791,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.695181+0000","last_change":"2024-09-21T08:28:53.588606+0000","last_active":"2024-09-21T08:31:25.695181+0000","last_peered":"2024-09-21T08:31:25.695181+0000","last_clean":"2024-09-21T08:28:52.586222+0000","last_became_active":"2024-09-21T08:28:53.588606+0000","last_became_peered":"2024-09-21T08:28:53.588606+0000","last_unstale":"2024-09-21T08:31:25.695181+0000","last_undegraded":"2024-09-21T08:28:53.574192+0000","last_fullsized":"2024-09-21T08:28:53.574061+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'22","last_scrub_stamp":"2024-09-21T08:26:29.353795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.353795+0000","objects_scrubbed":0,"log_size":73,"log_dups_size":0,"ondisk_log_size":73,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":72,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.019195647,"stat_sum":{"num_bytes":16432255,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":57,"num_read_kb":78730,"num_write":36,"num_write_kb":17862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":5630444,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":["5","2"],"object_location_counts":[{"shards":"2,5","objects":7}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6e","length":"3"},{"start":"74","length":"2"}]},{"pgid":"4.10","version":"73'4","reported_seq":440,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.683135+0000","last_change":"2024-09-21T08:30:56.659256+0000","last_active":"2024-09-21T08:31:25.683135+0000","last_peered":"2024-09-21T08:31:25.683135+0000","last_clean":"2024-09-21T08:31:25.683135+0000","last_became_active":"2024-09-21T08:30:56.638177+0000","last_became_peered":"2024-09-21T08:30:56.638177+0000","last_unstale":"2024-09-21T08:31:25.683135+0000","last_undegraded":"2024-09-21T08:31:25.683135+0000","last_fullsized":"2024-09-21T08:31:25.683135+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-09-21T08:29:39.416397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.416397+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":866,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.694411+0000","last_change":"2024-09-21T08:30:24.826266+0000","last_active":"2024-09-21T08:31:25.694411+0000","last_peered":"2024-09-21T08:31:25.694411+0000","last_clean":"2024-09-21T08:31:25.694411+0000","last_became_active":"2024-09-21T08:28:29.611592+0000","last_became_peered":"2024-09-21T08:28:29.611592+0000","last_unstale":"2024-09-21T08:31:25.694411+0000","last_undegraded":"2024-09-21T08:31:25.694411+0000","last_fullsized":"2024-09-21T08:31:25.694411+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:24.826198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:30:24.826198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:28.562894+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.682375+0000","last_change":"2024-09-21T08:30:57.621325+0000","last_active":"2024-09-21T08:31:25.682375+0000","last_peered":"2024-09-21T08:31:25.682375+0000","last_clean":"2024-09-21T08:31:25.682375+0000","last_became_active":"2024-09-21T08:30:57.618219+0000","last_became_peered":"2024-09-21T08:30:57.618219+0000","last_unstale":"2024-09-21T08:31:25.682375+0000","last_undegraded":"2024-09-21T08:31:25.682375+0000","last_fullsized":"2024-09-21T08:31:25.682375+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:46.607860+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:24:46.607860+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1,5],"avail_no_missing":["4","1","5"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":68,"reported_epoch":492,"state":"active+clean+remapped","last_fresh":"2024-09-21T08:31:25.683737+0000","last_change":"2024-09-21T08:30:57.611382+0000","last_active":"2024-09-21T08:31:25.683737+0000","last_peered":"2024-09-21T08:31:25.683737+0000","last_clean":"2024-09-21T08:31:25.683737+0000","last_became_active":"2024-09-21T08:30:57.610902+0000","last_became_peered":"2024-09-21T08:30:57.610902+0000","last_unstale":"2024-09-21T08:31:25.683737+0000","last_undegraded":"2024-09-21T08:31:25.683737+0000","last_fullsized":"2024-09-21T08:31:25.683737+0000","mapping_epoch":463,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":464,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.736816+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.736816+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2,1],"avail_no_missing":["4","1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"461'222","reported_seq":2031,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.694568+0000","last_change":"2024-09-21T08:28:53.600314+0000","last_active":"2024-09-21T08:31:25.694568+0000","last_peered":"2024-09-21T08:31:25.694568+0000","last_clean":"2024-09-21T08:28:51.556402+0000","last_became_active":"2024-09-21T08:28:53.600314+0000","last_became_peered":"2024-09-21T08:28:53.600314+0000","last_unstale":"2024-09-21T08:31:25.694568+0000","last_undegraded":"2024-09-21T08:28:53.575998+0000","last_fullsized":"2024-09-21T08:28:53.575875+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'167","last_scrub_stamp":"2024-09-21T08:27:49.143452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.143452+0000","objects_scrubbed":0,"log_size":222,"log_dups_size":0,"ondisk_log_size":222,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":72,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.015256377999999999,"stat_sum":{"num_bytes":19969264,"num_objects":10,"num_object_clones":9,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":10,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":76,"num_read_kb":97092,"num_write":35,"num_write_kb":17179,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1129140,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":10}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"6e","length":"3"},{"start":"74","length":"2"}]},{"pgid":"6.1d","version":"0'0","reported_seq":428,"reported_epoch":492,"state":"active+clean","last_fresh":"2024-09-21T08:31:25.685388+0000","last_change":"2024-09-21T08:30:56.627685+0000","last_active":"2024-09-21T08:31:25.685388+0000","last_peered":"2024-09-21T08:31:25.685388+0000","last_clean":"2024-09-21T08:31:25.685388+0000","last_became_active":"2024-09-21T08:30:56.627251+0000","last_became_peered":"2024-09-21T08:30:56.627251+0000","last_unstale":"2024-09-21T08:31:25.685388+0000","last_undegraded":"2024-09-21T08:31:25.685388+0000","last_fullsized":"2024-09-21T08:31:25.685388+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:28.069450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:28.069450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:51.381406+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":738,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.682391+0000","last_change":"2024-09-21T08:28:53.585838+0000","last_active":"2024-09-21T08:31:25.682391+0000","last_peered":"2024-09-21T08:31:25.682391+0000","last_clean":"2024-09-21T08:28:51.584499+0000","last_became_active":"2024-09-21T08:28:53.585838+0000","last_became_peered":"2024-09-21T08:28:53.585838+0000","last_unstale":"2024-09-21T08:31:25.682391+0000","last_undegraded":"2024-09-21T08:31:25.682391+0000","last_fullsized":"2024-09-21T08:28:53.575428+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:31:30.136607+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"73'11","reported_seq":933,"reported_epoch":492,"state":"active+undersized+degraded","last_fresh":"2024-09-21T08:31:25.694922+0000","last_change":"2024-09-21T08:28:53.603028+0000","last_active":"2024-09-21T08:31:25.694922+0000","last_peered":"2024-09-21T08:31:25.694922+0000","last_clean":"2024-09-21T08:28:51.556942+0000","last_became_active":"2024-09-21T08:28:53.603028+0000","last_became_peered":"2024-09-21T08:28:53.603028+0000","last_unstale":"2024-09-21T08:31:25.694922+0000","last_undegraded":"2024-09-21T08:28:53.581121+0000","last_fullsized":"2024-09-21T08:28:53.581005+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:24:54.650419+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.650419+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":["5","1"],"object_location_counts":[{"shards":"1,5","objects":5}],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":316,"reported_epoch":492,"state":"active+undersized","last_fresh":"2024-09-21T08:31:25.683807+0000","last_change":"2024-09-21T08:28:53.601908+0000","last_active":"2024-09-21T08:31:25.683807+0000","last_peered":"2024-09-21T08:31:25.683807+0000","last_clean":"2024-09-21T08:23:33.544062+0000","last_became_active":"2024-09-21T08:28:53.601908+0000","last_became_peered":"2024-09-21T08:28:53.601908+0000","last_unstale":"2024-09-21T08:31:25.683807+0000","last_undegraded":"2024-09-21T08:31:25.683807+0000","last_fullsized":"2024-09-21T08:28:53.577043+0000","mapping_epoch":339,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.872291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.872291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":["1","2"],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":46,"stat_sum":{"num_bytes":638849882,"num_objects":302,"num_object_clones":254,"num_object_copies":906,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":162,"num_objects_misplaced":73,"num_objects_unfound":0,"num_objects_dirty":302,"num_whiteouts":15,"num_read":1980,"num_read_kb":2163171,"num_write":1427,"num_write_kb":571262,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":768,"num_bytes_recovered":379045163,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1516871680,"data_stored":2171168505,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11134,"ondisk_log_size":11134,"up":103,"acting":120,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":79,"acting":87,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3,"num_objects_misplaced":2,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":123,"ondisk_log_size":123,"up":74,"acting":84,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":77,"num_objects_misplaced":39,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":676,"num_read_kb":433,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":592,"num_bytes_recovered":12388,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":626688,"data_stored":16766,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":428,"ondisk_log_size":428,"up":77,"acting":83,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1,"num_objects_misplaced":5,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":5459,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":8622,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":76,"acting":91,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":452,"num_read_kb":447,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":7,"acting":7,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":644037,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":1,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":19,"num_read_kb":27,"num_write":16,"num_write_kb":219,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":322018,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":3563520,"data_stored":3542208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1420,"ondisk_log_size":1420,"up":28,"acting":32,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":59,"seq":253403070533,"num_pgs":130,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":228312,"kb_used_data":174916,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93516840,"statfs":{"total":95995035648,"available":95761244160,"internally_reserved":0,"allocated":179113984,"data_stored":227784827,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":54655434},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397610,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659245,"num_pgs":136,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":344184,"kb_used_data":299104,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93400968,"statfs":{"total":95995035648,"available":95642591232,"internally_reserved":0,"allocated":306282496,"data_stored":449419288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,2,3,4,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986291,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":213644,"kb_used_data":174132,"kb_used_omap":1,"kb_used_meta":39486,"kb_avail":93531508,"statfs":{"total":95995035648,"available":95776264192,"internally_reserved":0,"allocated":178311168,"data_stored":230812579,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":40434117},"hb_peers":[1,2,3,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182519,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411453,"num_pgs":121,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":418820,"kb_used_data":373736,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93326332,"statfs":{"total":95995035648,"available":95566163968,"internally_reserved":0,"allocated":382705664,"data_stored":596567210,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[1,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574977,"num_pgs":141,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":339120,"kb_used_data":285720,"kb_used_omap":1,"kb_used_meta":53374,"kb_avail":93406032,"statfs":{"total":95995035648,"available":95647776768,"internally_reserved":0,"allocated":292577280,"data_stored":439358792,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":54655429},"hb_peers":[2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":0,"seq":34359738477,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1177,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":3372,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1568,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":1694,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":2712,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74776576,"data_stored":86348592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":298676224,"data_stored":449773286,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":383873024,"data_stored":602732668,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70848512,"data_stored":84468350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":177811456,"data_stored":233790076,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":306655232,"data_stored":459142808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":176136192,"data_stored":227042310,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:31:27.543 INFO:tasks.thrashosds.thrasher:Setting 1.7 to [2, 4, 0, 7, 6, 5] 2024-09-21T08:31:27.543 INFO:tasks.thrashosds.thrasher:cmd ['osd', 'pg-upmap-items', '1.7', '2', '4', '0', '7', '6', '5'] 2024-09-21T08:31:27.543 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd pg-upmap-items 1.7 2 4 0 7 6 5 2024-09-21T08:31:27.566 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:27.589 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:27.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:27.590 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:27.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:31:27.691 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:27.696 INFO:tasks.rados.rados.0.smithi137.stdout:1969: done (2 left) 2024-09-21T08:31:27.696 INFO:tasks.rados.rados.0.smithi137.stdout:1970: done (1 left) 2024-09-21T08:31:27.696 INFO:tasks.rados.rados.0.smithi137.stdout:1971: done (0 left) 2024-09-21T08:31:27.697 INFO:tasks.rados.rados.0.smithi137.stdout:1972: snap_remove snap 202 2024-09-21T08:31:27.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:27.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:27.718 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:27.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:27 smithi137 ceph-mon[29634]: pgmap v644: 188 pgs: 2 active+clean+snaptrim, 7 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 610 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 11 op/s; 246/1503 objects degraded (16.367%); 120/1503 objects misplaced (7.984%) 2024-09-21T08:31:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:27 smithi137 ceph-mon[29634]: from='client.45905 v1:172.21.15.137:0/2333959998' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:31:27.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:27 smithi137 ceph-mon[29634]: osdmap e493: 8 total, 6 up, 5 in 2024-09-21T08:31:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:27 smithi137 ceph-mon[41044]: pgmap v644: 188 pgs: 2 active+clean+snaptrim, 7 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 610 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 11 op/s; 246/1503 objects degraded (16.367%); 120/1503 objects misplaced (7.984%) 2024-09-21T08:31:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:27 smithi137 ceph-mon[41044]: from='client.45905 v1:172.21.15.137:0/2333959998' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:31:27.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:27 smithi137 ceph-mon[41044]: osdmap e493: 8 total, 6 up, 5 in 2024-09-21T08:31:27.819 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:27.841 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:27.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:27.842 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:27.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:27.964 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:27.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:27.965 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:27 smithi165 ceph-mon[32893]: pgmap v644: 188 pgs: 2 active+clean+snaptrim, 7 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 610 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 11 op/s; 246/1503 objects degraded (16.367%); 120/1503 objects misplaced (7.984%) 2024-09-21T08:31:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:27 smithi165 ceph-mon[32893]: from='client.45905 v1:172.21.15.137:0/2333959998' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:31:28.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:27 smithi165 ceph-mon[32893]: osdmap e493: 8 total, 6 up, 5 in 2024-09-21T08:31:28.066 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:28.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:28.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:28.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:28.151 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:28.151 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:28.177 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:28.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:28.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:28.213 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:28.213 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:28.315 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:28.337 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:28.338 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:28.338 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:28.439 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:28.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:28.462 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:28.462 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:28.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:28.587 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:28.588 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:28.588 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:28.689 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:28.704 INFO:tasks.rados.rados.0.smithi137.stdout:1972: done (0 left) 2024-09-21T08:31:28.704 INFO:tasks.rados.rados.0.smithi137.stdout:1973: rollback oid 47 current snap is 215 2024-09-21T08:31:28.704 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 47 to 130 2024-09-21T08:31:28.704 INFO:tasks.rados.rados.0.smithi137.stdout:1974: write (excl) oid 30 current snap is 215 2024-09-21T08:31:28.704 INFO:tasks.rados.rados.0.smithi137.stdout:1974: seq_num 699 ranges {548093=787174,1970678=449902,3045966=681137} 2024-09-21T08:31:28.712 INFO:tasks.rados.rados.0.smithi137.stdout:1974: writing smithi13779221-30 from 548093 to 1335267 tid 1 2024-09-21T08:31:28.717 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:28.718 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:28.718 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:28.718 INFO:tasks.rados.rados.0.smithi137.stdout:1974: writing smithi13779221-30 from 1970678 to 2420580 tid 2 2024-09-21T08:31:28.726 INFO:tasks.rados.rados.0.smithi137.stdout:1974: writing smithi13779221-30 from 3045966 to 3727103 tid 3 2024-09-21T08:31:28.726 INFO:tasks.rados.rados.0.smithi137.stdout:1975: delete oid 20 current snap is 215 2024-09-21T08:31:28.726 INFO:tasks.rados.rados.0.smithi137.stdout:1973: finishing rollback tid 1 to smithi13779221-47 2024-09-21T08:31:28.726 INFO:tasks.rados.rados.0.smithi137.stdout:1973: finishing rollback tid 2 to smithi13779221-47 2024-09-21T08:31:28.726 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 450 (ObjNum 184 snap 37 seq_num 184) dirty exists 2024-09-21T08:31:28.726 INFO:tasks.rados.rados.0.smithi137.stdout:1974: finishing write tid 1 to smithi13779221-30 2024-09-21T08:31:28.726 INFO:tasks.rados.rados.0.smithi137.stdout:1974: finishing write tid 2 to smithi13779221-30 2024-09-21T08:31:28.727 INFO:tasks.rados.rados.0.smithi137.stdout:1973: done (2 left) 2024-09-21T08:31:28.727 INFO:tasks.rados.rados.0.smithi137.stdout:1975: done (1 left) 2024-09-21T08:31:28.727 INFO:tasks.rados.rados.0.smithi137.stdout:1976: rollback oid 44 current snap is 215 2024-09-21T08:31:28.727 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 44 to 174 2024-09-21T08:31:28.728 INFO:tasks.rados.rados.0.smithi137.stdout:1977: snap_create 2024-09-21T08:31:28.729 INFO:tasks.rados.rados.0.smithi137.stdout:1974: finishing write tid 3 to smithi13779221-30 2024-09-21T08:31:28.729 INFO:tasks.rados.rados.0.smithi137.stdout:1974: finishing write tid 4 to smithi13779221-30 2024-09-21T08:31:28.729 INFO:tasks.rados.rados.0.smithi137.stdout:1974: finishing write tid 5 to smithi13779221-30 2024-09-21T08:31:28.729 INFO:tasks.rados.rados.0.smithi137.stdout:1974: oid 30 updating version 0 to 810 2024-09-21T08:31:28.729 INFO:tasks.rados.rados.0.smithi137.stdout:1974: oid 30 updating version 810 to 812 2024-09-21T08:31:28.729 INFO:tasks.rados.rados.0.smithi137.stdout:1974: oid 30 updating version 812 to 813 2024-09-21T08:31:28.730 INFO:tasks.rados.rados.0.smithi137.stdout:1974: oid 30 version 813 is already newer than 811 2024-09-21T08:31:28.730 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 813 (ObjNum 699 snap 215 seq_num 699) dirty exists 2024-09-21T08:31:28.730 INFO:tasks.rados.rados.0.smithi137.stdout:1974: left oid 30 (ObjNum 699 snap 215 seq_num 699) 2024-09-21T08:31:28.731 INFO:tasks.rados.rados.0.smithi137.stdout:1976: finishing rollback tid 0 to smithi13779221-44 2024-09-21T08:31:28.731 INFO:tasks.rados.rados.0.smithi137.stdout:1976: finishing rollback tid 1 to smithi13779221-44 2024-09-21T08:31:28.732 INFO:tasks.rados.rados.0.smithi137.stdout:1976: finishing rollback tid 2 to smithi13779221-44 2024-09-21T08:31:28.732 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 741 (ObjNum 471 snap 128 seq_num 471) dirty exists 2024-09-21T08:31:28.753 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:28.754 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:28.818 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:28.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:28.842 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:28.842 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:28.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:28.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:28.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:28.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:28.976 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:28 smithi165 ceph-mon[32893]: 7.e scrub starts 2024-09-21T08:31:29.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:28 smithi165 ceph-mon[32893]: 7.e scrub ok 2024-09-21T08:31:29.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:28 smithi165 ceph-mon[32893]: osdmap e494: 8 total, 6 up, 5 in 2024-09-21T08:31:29.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:28 smithi165 ceph-mon[32893]: pgmap v647: 188 pgs: 2 active+clean+snaptrim, 7 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 610 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 11 op/s; 246/1503 objects degraded (16.367%); 120/1503 objects misplaced (7.984%) 2024-09-21T08:31:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:28 smithi137 ceph-mon[41044]: 7.e scrub starts 2024-09-21T08:31:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:28 smithi137 ceph-mon[41044]: 7.e scrub ok 2024-09-21T08:31:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:28 smithi137 ceph-mon[41044]: osdmap e494: 8 total, 6 up, 5 in 2024-09-21T08:31:29.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:28 smithi137 ceph-mon[41044]: pgmap v647: 188 pgs: 2 active+clean+snaptrim, 7 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 610 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 11 op/s; 246/1503 objects degraded (16.367%); 120/1503 objects misplaced (7.984%) 2024-09-21T08:31:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:28 smithi137 ceph-mon[29634]: 7.e scrub starts 2024-09-21T08:31:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:28 smithi137 ceph-mon[29634]: 7.e scrub ok 2024-09-21T08:31:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:28 smithi137 ceph-mon[29634]: osdmap e494: 8 total, 6 up, 5 in 2024-09-21T08:31:29.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:28 smithi137 ceph-mon[29634]: pgmap v647: 188 pgs: 2 active+clean+snaptrim, 7 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 610 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 4.9 MiB/s rd, 2.8 MiB/s wr, 11 op/s; 246/1503 objects degraded (16.367%); 120/1503 objects misplaced (7.984%) 2024-09-21T08:31:29.067 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:29.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:29.089 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:29.089 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:29.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:29.211 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:29.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:29.212 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:29.313 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:29.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:29.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:29.337 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:29.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:29.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:29.461 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:29.461 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:29.479 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T08:31:29.533 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:29.562 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:29.583 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:29.583 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:29.584 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:29.685 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:29.699 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:29 smithi137 ceph-mon[29634]: 5.11 scrub starts 2024-09-21T08:31:29.699 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:29 smithi137 ceph-mon[29634]: 5.11 scrub ok 2024-09-21T08:31:29.699 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:29 smithi137 ceph-mon[29634]: 7.2a scrub starts 2024-09-21T08:31:29.708 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:29.709 INFO:tasks.rados.rados.0.smithi137.stdout:1974: done (2 left) 2024-09-21T08:31:29.709 INFO:tasks.rados.rados.0.smithi137.stdout:1976: done (1 left) 2024-09-21T08:31:29.709 INFO:tasks.rados.rados.0.smithi137.stdout:1977: done (0 left) 2024-09-21T08:31:29.709 INFO:tasks.rados.rados.0.smithi137.stdout:1978: copy_from oid 27 from oid 4 current snap is 216 2024-09-21T08:31:29.709 INFO:tasks.rados.rados.0.smithi137.stdout:1979: write oid 16 current snap is 216 2024-09-21T08:31:29.709 INFO:tasks.rados.rados.0.smithi137.stdout:1979: seq_num 701 ranges {581707=654025,1687642=453571,2700812=703681} 2024-09-21T08:31:29.710 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:29.710 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:29.715 INFO:tasks.rados.rados.0.smithi137.stdout:1979: writing smithi13779221-16 from 581707 to 1235732 tid 1 2024-09-21T08:31:29.721 INFO:tasks.rados.rados.0.smithi137.stdout:1979: writing smithi13779221-16 from 1687642 to 2141213 tid 2 2024-09-21T08:31:29.729 INFO:tasks.rados.rados.0.smithi137.stdout:1979: writing smithi13779221-16 from 2700812 to 3404493 tid 3 2024-09-21T08:31:29.729 INFO:tasks.rados.rados.0.smithi137.stdout:1980: read oid 18 snap -1 2024-09-21T08:31:29.729 INFO:tasks.rados.rados.0.smithi137.stdout:1980: expect (ObjNum 670 snap 203 seq_num 670) 2024-09-21T08:31:29.729 INFO:tasks.rados.rados.0.smithi137.stdout:1981: read oid 43 snap -1 2024-09-21T08:31:29.729 INFO:tasks.rados.rados.0.smithi137.stdout:1981: expect deleted 2024-09-21T08:31:29.729 INFO:tasks.rados.rados.0.smithi137.stdout:1982: read oid 29 snap 177 2024-09-21T08:31:29.729 INFO:tasks.rados.rados.0.smithi137.stdout:1982: expect (ObjNum 567 snap 168 seq_num 567) 2024-09-21T08:31:29.729 INFO:tasks.rados.rados.0.smithi137.stdout:1983: read oid 15 snap -1 2024-09-21T08:31:29.729 INFO:tasks.rados.rados.0.smithi137.stdout:1983: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:29.730 INFO:tasks.rados.rados.0.smithi137.stdout:1984: snap_create 2024-09-21T08:31:29.730 INFO:tasks.rados.rados.0.smithi137.stdout:1978: finishing copy_from to smithi13779221-27 2024-09-21T08:31:29.730 INFO:tasks.rados.rados.0.smithi137.stdout:1978: got expected ENOENT (src dne) 2024-09-21T08:31:29.730 INFO:tasks.rados.rados.0.smithi137.stdout:1978: finishing copy_from racing read to smithi13779221-27 2024-09-21T08:31:29.730 INFO:tasks.rados.rados.0.smithi137.stdout:1979: finishing write tid 1 to smithi13779221-16 2024-09-21T08:31:29.730 INFO:tasks.rados.rados.0.smithi137.stdout:1979: finishing write tid 2 to smithi13779221-16 2024-09-21T08:31:29.731 INFO:tasks.rados.rados.0.smithi137.stdout:1979: finishing write tid 3 to smithi13779221-16 2024-09-21T08:31:29.732 INFO:tasks.rados.rados.0.smithi137.stdout:1979: finishing write tid 4 to smithi13779221-16 2024-09-21T08:31:29.736 INFO:tasks.rados.rados.0.smithi137.stdout:1982: expect (ObjNum 567 snap 168 seq_num 567) 2024-09-21T08:31:29.811 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:29.835 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:29.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:29.836 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:29.908 INFO:tasks.rados.rados.0.smithi137.stdout:1979: finishing write tid 5 to smithi13779221-16 2024-09-21T08:31:29.908 INFO:tasks.rados.rados.0.smithi137.stdout:1979: oid 16 updating version 0 to 508 2024-09-21T08:31:29.908 INFO:tasks.rados.rados.0.smithi137.stdout:1979: oid 16 version 508 is already newer than 507 2024-09-21T08:31:29.908 INFO:tasks.rados.rados.0.smithi137.stdout:1979: oid 16 version 508 is already newer than 506 2024-09-21T08:31:29.908 INFO:tasks.rados.rados.0.smithi137.stdout:1979: oid 16 version 508 is already newer than 505 2024-09-21T08:31:29.908 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 508 (ObjNum 701 snap 216 seq_num 701) dirty exists 2024-09-21T08:31:29.908 INFO:tasks.rados.rados.0.smithi137.stdout:1979: left oid 16 (ObjNum 701 snap 216 seq_num 701) 2024-09-21T08:31:29.908 INFO:tasks.rados.rados.0.smithi137.stdout:1980: expect (ObjNum 670 snap 203 seq_num 670) 2024-09-21T08:31:29.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:29.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:29.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:29.960 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:30.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:29 smithi165 ceph-mon[32893]: 5.11 scrub starts 2024-09-21T08:31:30.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:29 smithi165 ceph-mon[32893]: 5.11 scrub ok 2024-09-21T08:31:30.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:29 smithi165 ceph-mon[32893]: 7.2a scrub starts 2024-09-21T08:31:30.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:29 smithi165 ceph-mon[32893]: 7.2a scrub ok 2024-09-21T08:31:30.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:29 smithi165 ceph-mon[32893]: osdmap e495: 8 total, 6 up, 5 in 2024-09-21T08:31:30.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:29 smithi137 ceph-mon[29634]: 7.2a scrub ok 2024-09-21T08:31:30.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:29 smithi137 ceph-mon[29634]: osdmap e495: 8 total, 6 up, 5 in 2024-09-21T08:31:30.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:29 smithi137 ceph-mon[41044]: 5.11 scrub starts 2024-09-21T08:31:30.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:29 smithi137 ceph-mon[41044]: 5.11 scrub ok 2024-09-21T08:31:30.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:29 smithi137 ceph-mon[41044]: 7.2a scrub starts 2024-09-21T08:31:30.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:29 smithi137 ceph-mon[41044]: 7.2a scrub ok 2024-09-21T08:31:30.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:29 smithi137 ceph-mon[41044]: osdmap e495: 8 total, 6 up, 5 in 2024-09-21T08:31:30.061 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:30.084 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:30.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:30.085 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:30.111 INFO:tasks.rados.rados.0.smithi137.stdout:1983: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:30.153 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:31:30.186 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:30.229 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:30.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:30.230 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:30.332 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:30.344 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:31:30.152+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:31:30.356 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:30.357 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:30.357 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:30.458 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:30.481 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:30.482 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:30.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:30.583 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:30.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:30.605 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:30.606 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:30.619 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:30.661 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:30.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:30.714 INFO:teuthology.orchestra.run.smithi137.stderr:set 1.7 pg_upmap_items mapping to [2->4,0->7,6->5] 2024-09-21T08:31:30.715 INFO:tasks.rados.rados.0.smithi137.stdout:1978: done (6 left) 2024-09-21T08:31:30.715 INFO:tasks.rados.rados.0.smithi137.stdout:1979: done (5 left) 2024-09-21T08:31:30.715 INFO:tasks.rados.rados.0.smithi137.stdout:1980: done (4 left) 2024-09-21T08:31:30.715 INFO:tasks.rados.rados.0.smithi137.stdout:1981: done (3 left) 2024-09-21T08:31:30.715 INFO:tasks.rados.rados.0.smithi137.stdout:1982: done (2 left) 2024-09-21T08:31:30.715 INFO:tasks.rados.rados.0.smithi137.stdout:1983: done (1 left) 2024-09-21T08:31:30.715 INFO:tasks.rados.rados.0.smithi137.stdout:1984: done (0 left) 2024-09-21T08:31:30.715 INFO:tasks.rados.rados.0.smithi137.stdout:1985: delete oid 8 current snap is 217 2024-09-21T08:31:30.730 INFO:tasks.rados.rados.0.smithi137.stdout:1985: done (0 left) 2024-09-21T08:31:30.730 INFO:tasks.rados.rados.0.smithi137.stdout:1986: rollback oid 41 current snap is 217 2024-09-21T08:31:30.730 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 41 to 183 2024-09-21T08:31:30.730 INFO:tasks.rados.rados.0.smithi137.stdout:1987: delete oid 46 current snap is 217 2024-09-21T08:31:30.734 INFO:tasks.rados.rados.0.smithi137.stdout:1986: finishing rollback tid 0 to smithi13779221-41 2024-09-21T08:31:30.734 INFO:tasks.rados.rados.0.smithi137.stdout:1987: done (1 left) 2024-09-21T08:31:30.734 INFO:tasks.rados.rados.0.smithi137.stdout:1988: read oid 6 snap -1 2024-09-21T08:31:30.734 INFO:tasks.rados.rados.0.smithi137.stdout:1988: expect deleted 2024-09-21T08:31:30.734 INFO:tasks.rados.rados.0.smithi137.stdout:1989: write oid 11 current snap is 217 2024-09-21T08:31:30.734 INFO:tasks.rados.rados.0.smithi137.stdout:1989: seq_num 704 ranges {632128=479772,1746735=600898,3066728=653850} 2024-09-21T08:31:30.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:30.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:30.736 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:30.740 INFO:tasks.rados.rados.0.smithi137.stdout:1989: writing smithi13779221-11 from 632128 to 1111900 tid 1 2024-09-21T08:31:30.747 INFO:tasks.rados.rados.0.smithi137.stdout:1989: writing smithi13779221-11 from 1746735 to 2347633 tid 2 2024-09-21T08:31:30.755 INFO:tasks.rados.rados.0.smithi137.stdout:1989: writing smithi13779221-11 from 3066728 to 3720578 tid 3 2024-09-21T08:31:30.755 INFO:tasks.rados.rados.0.smithi137.stdout:1990: snap_remove snap 211 2024-09-21T08:31:30.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:30.858 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:30.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:30.859 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:30.960 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:30.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:30.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:30.983 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: 6.b scrub starts 2024-09-21T08:31:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: 6.b scrub ok 2024-09-21T08:31:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: osdmap e496: 8 total, 6 up, 5 in 2024-09-21T08:31:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:31:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: pgmap v650: 188 pgs: 2 active+clean+snaptrim, 7 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 610 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 246/1503 objects degraded (16.367%); 120/1503 objects misplaced (7.984%) 2024-09-21T08:31:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1704011233' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [2, 4, 0, 7, 6, 5]}]: dispatch 2024-09-21T08:31:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: from='client.45911 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [2, 4, 0, 7, 6, 5]}]: dispatch 2024-09-21T08:31:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:31:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:31:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:31.017 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:31:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:31:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: 6.b scrub starts 2024-09-21T08:31:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: 6.b scrub ok 2024-09-21T08:31:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: osdmap e496: 8 total, 6 up, 5 in 2024-09-21T08:31:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:31:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: pgmap v650: 188 pgs: 2 active+clean+snaptrim, 7 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 610 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 246/1503 objects degraded (16.367%); 120/1503 objects misplaced (7.984%) 2024-09-21T08:31:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1704011233' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [2, 4, 0, 7, 6, 5]}]: dispatch 2024-09-21T08:31:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: from='client.45911 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [2, 4, 0, 7, 6, 5]}]: dispatch 2024-09-21T08:31:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:31:31.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: 6.b scrub starts 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: 6.b scrub ok 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: osdmap e496: 8 total, 6 up, 5 in 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: pgmap v650: 188 pgs: 2 active+clean+snaptrim, 7 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 61 active+clean; 610 MiB data, 1.5 GiB used, 446 GiB / 447 GiB avail; 246/1503 objects degraded (16.367%); 120/1503 objects misplaced (7.984%) 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1704011233' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [2, 4, 0, 7, 6, 5]}]: dispatch 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: from='client.45911 ' entity='client.admin' cmd=[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [2, 4, 0, 7, 6, 5]}]: dispatch 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:31:31.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:31:31.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:31.084 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:31.106 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:31.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:31.107 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:31.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:31.229 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:31.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:31.230 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:31.289 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:31.290 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:31.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:31.352 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:31.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:31.353 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:31.454 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:31.477 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:31.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:31.478 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:31.578 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:31.594 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:31:31.601 INFO:tasks.rados.rados.0.smithi137.stdout:1990: done (3 left) 2024-09-21T08:31:31.601 INFO:tasks.rados.rados.0.smithi137.stdout:1991: snap_create 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1986: finishing rollback tid 1 to smithi13779221-41 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 411 (ObjNum 289 snap 0 seq_num 1) dirty dne 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: finishing write tid 1 to smithi13779221-11 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: finishing write tid 2 to smithi13779221-11 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: finishing write tid 3 to smithi13779221-11 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: finishing write tid 4 to smithi13779221-11 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: finishing write tid 5 to smithi13779221-11 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: oid 11 updating version 0 to 534 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: oid 11 version 534 is already newer than 533 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: oid 11 updating version 534 to 535 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: oid 11 version 535 is already newer than 532 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 535 (ObjNum 704 snap 217 seq_num 704) dirty exists 2024-09-21T08:31:31.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: left oid 11 (ObjNum 704 snap 217 seq_num 704) 2024-09-21T08:31:31.604 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:31.605 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:31.606 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:31.706 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:31.732 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:31.733 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:31.733 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:31.833 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:31.855 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:31.855 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:31.855 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:31.956 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:31.977 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:31.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:31.978 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:31.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:31 smithi165 ceph-mon[32893]: from='client.45911 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [2, 4, 0, 7, 6, 5]}]': finished 2024-09-21T08:31:31.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:31 smithi165 ceph-mon[32893]: osdmap e497: 8 total, 6 up, 5 in 2024-09-21T08:31:31.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:31 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2464533775' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:31.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:31 smithi165 ceph-mon[32893]: 7.24 scrub starts 2024-09-21T08:31:31.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:31 smithi165 ceph-mon[32893]: 7.24 scrub ok 2024-09-21T08:31:31.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:31 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 246/1503 objects degraded (16.367%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:31.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:31 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:31:31.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:31 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2464533775' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:31.983 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:31 smithi165 ceph-mon[32893]: osdmap e498: 8 total, 6 up, 5 in 2024-09-21T08:31:32.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[29634]: from='client.45911 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [2, 4, 0, 7, 6, 5]}]': finished 2024-09-21T08:31:32.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[29634]: osdmap e497: 8 total, 6 up, 5 in 2024-09-21T08:31:32.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2464533775' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:32.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[29634]: 7.24 scrub starts 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[29634]: 7.24 scrub ok 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 246/1503 objects degraded (16.367%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2464533775' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[29634]: osdmap e498: 8 total, 6 up, 5 in 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[41044]: from='client.45911 ' entity='client.admin' cmd='[{"prefix": "osd pg-upmap-items", "pgid": "1.7", "id": [2, 4, 0, 7, 6, 5]}]': finished 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[41044]: osdmap e497: 8 total, 6 up, 5 in 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2464533775' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[41044]: 7.24 scrub starts 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[41044]: 7.24 scrub ok 2024-09-21T08:31:32.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 246/1503 objects degraded (16.367%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:32.004 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:31:32.004 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2464533775' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:32.004 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:31 smithi137 ceph-mon[41044]: osdmap e498: 8 total, 6 up, 5 in 2024-09-21T08:31:32.079 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:32.103 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:32.104 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:32.104 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:32.204 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:32.228 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:32.229 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:32.229 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:32.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:31:31 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:31:32.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:32.355 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:32.355 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:32.356 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:32.456 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:32.481 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:32.481 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:32.482 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:32.582 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:32.601 INFO:tasks.rados.rados.0.smithi137.stdout:1986: done (3 left) 2024-09-21T08:31:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1988: done (2 left) 2024-09-21T08:31:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1989: done (1 left) 2024-09-21T08:31:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1991: done (0 left) 2024-09-21T08:31:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1992: read oid 39 snap 212 2024-09-21T08:31:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1992: expect (ObjNum 666 snap 200 seq_num 666) 2024-09-21T08:31:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1993: read oid 44 snap 216 2024-09-21T08:31:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1993: expect (ObjNum 471 snap 128 seq_num 471) 2024-09-21T08:31:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1994: rollback oid 6 current snap is 218 2024-09-21T08:31:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 6 to 200 2024-09-21T08:31:32.602 INFO:tasks.rados.rados.0.smithi137.stdout:1995: delete oid 28 current snap is 218 2024-09-21T08:31:32.610 INFO:tasks.rados.rados.0.smithi137.stdout:1994: finishing rollback tid 1 to smithi13779221-6 2024-09-21T08:31:32.610 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 401 (ObjNum 33 snap 0 seq_num 2161705991) dirty dne 2024-09-21T08:31:32.611 INFO:tasks.rados.rados.0.smithi137.stdout:1993: expect (ObjNum 471 snap 128 seq_num 471) 2024-09-21T08:31:32.612 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:32.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:32.613 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:32.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:32.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:32.735 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:32.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:32.745 INFO:tasks.rados.rados.0.smithi137.stdout:1993: done (3 left) 2024-09-21T08:31:32.745 INFO:tasks.rados.rados.0.smithi137.stdout:1994: done (2 left) 2024-09-21T08:31:32.745 INFO:tasks.rados.rados.0.smithi137.stdout:1995: done (1 left) 2024-09-21T08:31:32.745 INFO:tasks.rados.rados.0.smithi137.stdout:1996: rollback oid 7 current snap is 218 2024-09-21T08:31:32.745 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 7 to 216 2024-09-21T08:31:32.746 INFO:tasks.rados.rados.0.smithi137.stdout:1992: expect (ObjNum 666 snap 200 seq_num 666) 2024-09-21T08:31:32.836 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:32.861 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:32.862 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:32.862 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:32.917 INFO:tasks.rados.rados.0.smithi137.stdout:1996: finishing rollback tid 0 to smithi13779221-7 2024-09-21T08:31:32.917 INFO:tasks.rados.rados.0.smithi137.stdout:1996: finishing rollback tid 1 to smithi13779221-7 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:1992: done (1 left) 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:1997: read oid 11 snap -1 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:1997: expect (ObjNum 704 snap 217 seq_num 704) 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:1998: copy_from oid 32 from oid 21 current snap is 218 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:1996: finishing rollback tid 2 to smithi13779221-7 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 827 (ObjNum 670 snap 203 seq_num 670) dirty exists 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:1996: done (2 left) 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:1999: rollback oid 34 current snap is 218 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 34 to 207 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:2000: copy_from oid 50 from oid 8 current snap is 218 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:2001: copy_from oid 12 from oid 49 current snap is 218 2024-09-21T08:31:32.918 INFO:tasks.rados.rados.0.smithi137.stdout:2002: snap_remove snap 198 2024-09-21T08:31:32.963 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:32.986 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:32.987 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:32.987 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:33.088 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:33.112 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:33.112 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:33.113 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:33.213 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:33.237 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:33.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:33.237 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:33.253 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T08:31:33.338 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:33.359 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:33.360 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:33.360 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:33.461 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:33.486 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:33.487 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:33.487 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:33.588 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:33.606 INFO:tasks.rados.rados.0.smithi137.stdout:2002: done (5 left) 2024-09-21T08:31:33.607 INFO:tasks.rados.rados.0.smithi137.stdout:2003: delete oid 40 current snap is 218 2024-09-21T08:31:33.607 INFO:tasks.rados.rados.0.smithi137.stdout:2000: finishing copy_from to smithi13779221-50 2024-09-21T08:31:33.607 INFO:tasks.rados.rados.0.smithi137.stdout:2000: got expected ENOENT (src dne) 2024-09-21T08:31:33.607 INFO:tasks.rados.rados.0.smithi137.stdout:2000: finishing copy_from racing read to smithi13779221-50 2024-09-21T08:31:33.607 INFO:tasks.rados.rados.0.smithi137.stdout:1999: finishing rollback tid 0 to smithi13779221-34 2024-09-21T08:31:33.607 INFO:tasks.rados.rados.0.smithi137.stdout:1997: expect (ObjNum 704 snap 217 seq_num 704) 2024-09-21T08:31:33.615 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:33.616 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:33.616 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:33.718 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:33.739 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:33.740 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:33.740 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:33.841 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:33.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: 6.7 scrub starts 2024-09-21T08:31:33.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: 6.7 scrub ok 2024-09-21T08:31:33.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:33.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: 3.8 scrub starts 2024-09-21T08:31:33.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: 3.8 scrub ok 2024-09-21T08:31:33.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: pgmap v653: 188 pgs: 2 active+clean+snaptrim, 1 remapped+peering, 5 active+clean+remapped+snaptrim_wait, 52 active+clean+remapped, 2 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 64 active+clean; 621 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 16 op/s; 248/1521 objects degraded (16.305%); 123/1521 objects misplaced (8.087%) 2024-09-21T08:31:33.846 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: osdmap e499: 8 total, 6 up, 5 in 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: 3.1b scrub starts 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[29634]: 3.1b scrub ok 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: 6.7 scrub starts 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: 6.7 scrub ok 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: 3.8 scrub starts 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: 3.8 scrub ok 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: pgmap v653: 188 pgs: 2 active+clean+snaptrim, 1 remapped+peering, 5 active+clean+remapped+snaptrim_wait, 52 active+clean+remapped, 2 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 64 active+clean; 621 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 16 op/s; 248/1521 objects degraded (16.305%); 123/1521 objects misplaced (8.087%) 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:31:33.847 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: osdmap e499: 8 total, 6 up, 5 in 2024-09-21T08:31:33.848 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: 3.1b scrub starts 2024-09-21T08:31:33.848 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:33 smithi137 ceph-mon[41044]: 3.1b scrub ok 2024-09-21T08:31:33.855 INFO:tasks.rados.rados.0.smithi137.stdout:1999: finishing rollback tid 1 to smithi13779221-34 2024-09-21T08:31:33.855 INFO:tasks.rados.rados.0.smithi137.stdout:1999: finishing rollback tid 2 to smithi13779221-34 2024-09-21T08:31:33.855 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 455 (ObjNum 620 snap 182 seq_num 620) dirty exists 2024-09-21T08:31:33.855 INFO:tasks.rados.rados.0.smithi137.stdout:1998: finishing copy_from racing read to smithi13779221-32 2024-09-21T08:31:33.855 INFO:tasks.rados.rados.0.smithi137.stdout:1997: done (5 left) 2024-09-21T08:31:33.856 INFO:tasks.rados.rados.0.smithi137.stdout:1999: done (4 left) 2024-09-21T08:31:33.856 INFO:tasks.rados.rados.0.smithi137.stdout:2000: done (3 left) 2024-09-21T08:31:33.856 INFO:tasks.rados.rados.0.smithi137.stdout:2003: done (2 left) 2024-09-21T08:31:33.856 INFO:tasks.rados.rados.0.smithi137.stdout:2004: snap_remove snap 206 2024-09-21T08:31:33.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:33.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:33.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:33.886 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:33.886 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:33.886 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:33.895 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:33.896 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:33.914 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:33.914 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:33.921 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:33.932 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:33.932 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:33.950 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:33.950 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:33.951 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:33.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:33.994 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:33.994 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:33.994 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:34.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: 6.7 scrub starts 2024-09-21T08:31:34.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: 6.7 scrub ok 2024-09-21T08:31:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: 3.8 scrub starts 2024-09-21T08:31:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: 3.8 scrub ok 2024-09-21T08:31:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: pgmap v653: 188 pgs: 2 active+clean+snaptrim, 1 remapped+peering, 5 active+clean+remapped+snaptrim_wait, 52 active+clean+remapped, 2 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 64 active+clean; 621 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 16 op/s; 248/1521 objects degraded (16.305%); 123/1521 objects misplaced (8.087%) 2024-09-21T08:31:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:31:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:31:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: osdmap e499: 8 total, 6 up, 5 in 2024-09-21T08:31:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: 3.1b scrub starts 2024-09-21T08:31:34.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:33 smithi165 ceph-mon[32893]: 3.1b scrub ok 2024-09-21T08:31:34.096 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:34.117 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:34.118 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:34.118 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:34.219 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:34.240 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:34.241 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:34.241 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:34.342 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:34.363 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:34.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:34.365 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:34.465 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:34.492 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:34.492 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:34.492 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:34.593 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:34.616 INFO:tasks.rados.rados.0.smithi137.stdout:2004: done (2 left) 2024-09-21T08:31:34.616 INFO:tasks.rados.rados.0.smithi137.stdout:2005: delete oid 11 current snap is 218 2024-09-21T08:31:34.616 INFO:tasks.rados.rados.0.smithi137.stdout:2001: finishing copy_from racing read to smithi13779221-12 2024-09-21T08:31:34.616 INFO:tasks.rados.rados.0.smithi137.stdout:1998: finishing copy_from to smithi13779221-32 2024-09-21T08:31:34.616 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 519 (ObjNum 677 snap 205 seq_num 677) dirty exists 2024-09-21T08:31:34.616 INFO:tasks.rados.rados.0.smithi137.stdout:2001: finishing copy_from to smithi13779221-12 2024-09-21T08:31:34.616 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 444 (ObjNum 560 snap 167 seq_num 560) dirty exists 2024-09-21T08:31:34.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:34.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:34.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:34.626 INFO:tasks.rados.rados.0.smithi137.stdout:1998: done (2 left) 2024-09-21T08:31:34.626 INFO:tasks.rados.rados.0.smithi137.stdout:2001: done (1 left) 2024-09-21T08:31:34.626 INFO:tasks.rados.rados.0.smithi137.stdout:2005: done (0 left) 2024-09-21T08:31:34.626 INFO:tasks.rados.rados.0.smithi137.stdout:2006: write oid 20 current snap is 218 2024-09-21T08:31:34.626 INFO:tasks.rados.rados.0.smithi137.stdout:2006: seq_num 708 ranges {699356=497121,1980663=608236,3176420=415291} 2024-09-21T08:31:34.633 INFO:tasks.rados.rados.0.smithi137.stdout:2006: writing smithi13779221-20 from 699356 to 1196477 tid 1 2024-09-21T08:31:34.640 INFO:tasks.rados.rados.0.smithi137.stdout:2006: writing smithi13779221-20 from 1980663 to 2588899 tid 2 2024-09-21T08:31:34.647 INFO:tasks.rados.rados.0.smithi137.stdout:2006: writing smithi13779221-20 from 3176420 to 3591711 tid 3 2024-09-21T08:31:34.647 INFO:tasks.rados.rados.0.smithi137.stdout:2007: read oid 42 snap -1 2024-09-21T08:31:34.647 INFO:tasks.rados.rados.0.smithi137.stdout:2007: expect (ObjNum 656 snap 196 seq_num 656) 2024-09-21T08:31:34.647 INFO:tasks.rados.rados.0.smithi137.stdout:2008: rollback oid 4 current snap is 218 2024-09-21T08:31:34.647 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 4 to 192 2024-09-21T08:31:34.647 INFO:tasks.rados.rados.0.smithi137.stdout:2009: rollback oid 10 current snap is 218 2024-09-21T08:31:34.648 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 10 to 207 2024-09-21T08:31:34.648 INFO:tasks.rados.rados.0.smithi137.stdout:2006: finishing write tid 1 to smithi13779221-20 2024-09-21T08:31:34.648 INFO:tasks.rados.rados.0.smithi137.stdout:2010: read oid 49 snap -1 2024-09-21T08:31:34.648 INFO:tasks.rados.rados.0.smithi137.stdout:2010: expect (ObjNum 560 snap 167 seq_num 560) 2024-09-21T08:31:34.648 INFO:tasks.rados.rados.0.smithi137.stdout:2011: copy_from oid 30 from oid 9 current snap is 218 2024-09-21T08:31:34.648 INFO:tasks.rados.rados.0.smithi137.stdout:2006: finishing write tid 2 to smithi13779221-20 2024-09-21T08:31:34.648 INFO:tasks.rados.rados.0.smithi137.stdout:2012: delete oid 19 current snap is 218 2024-09-21T08:31:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:2011: finishing copy_from to smithi13779221-30 2024-09-21T08:31:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:2011: got expected ENOENT (src dne) 2024-09-21T08:31:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:2011: finishing copy_from racing read to smithi13779221-30 2024-09-21T08:31:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:2009: finishing rollback tid 0 to smithi13779221-10 2024-09-21T08:31:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:2009: finishing rollback tid 1 to smithi13779221-10 2024-09-21T08:31:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 538 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-21T08:31:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:2008: finishing rollback tid 1 to smithi13779221-4 2024-09-21T08:31:34.649 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 440 (ObjNum 273 snap 0 seq_num 0) dirty dne 2024-09-21T08:31:34.650 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:31:34.653 INFO:tasks.rados.rados.0.smithi137.stdout:2006: finishing write tid 3 to smithi13779221-20 2024-09-21T08:31:34.654 INFO:tasks.rados.rados.0.smithi137.stdout:2006: finishing write tid 4 to smithi13779221-20 2024-09-21T08:31:34.654 INFO:tasks.rados.rados.0.smithi137.stdout:2006: finishing write tid 5 to smithi13779221-20 2024-09-21T08:31:34.654 INFO:tasks.rados.rados.0.smithi137.stdout:2006: oid 20 updating version 0 to 457 2024-09-21T08:31:34.654 INFO:tasks.rados.rados.0.smithi137.stdout:2006: oid 20 version 457 is already newer than 456 2024-09-21T08:31:34.654 INFO:tasks.rados.rados.0.smithi137.stdout:2006: oid 20 updating version 457 to 459 2024-09-21T08:31:34.654 INFO:tasks.rados.rados.0.smithi137.stdout:2006: oid 20 version 459 is already newer than 458 2024-09-21T08:31:34.654 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 459 (ObjNum 708 snap 218 seq_num 708) dirty exists 2024-09-21T08:31:34.654 INFO:tasks.rados.rados.0.smithi137.stdout:2006: left oid 20 (ObjNum 708 snap 218 seq_num 708) 2024-09-21T08:31:34.654 INFO:tasks.rados.rados.0.smithi137.stdout:2010: expect (ObjNum 560 snap 167 seq_num 560) 2024-09-21T08:31:34.675 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[29634]: 7.a scrub starts 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[29634]: 7.a scrub ok 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[29634]: 3.19 scrub starts 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[29634]: 3.19 scrub ok 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[29634]: osdmap e500: 8 total, 6 up, 5 in 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3886532159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[41044]: 7.a scrub starts 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[41044]: 7.a scrub ok 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[41044]: 3.19 scrub starts 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[41044]: 3.19 scrub ok 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[41044]: osdmap e500: 8 total, 6 up, 5 in 2024-09-21T08:31:34.676 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3886532159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:34.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:34.745 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:34.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:34.745 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:34.766 INFO:tasks.rados.rados.0.smithi137.stdout:2006: done (6 left) 2024-09-21T08:31:34.766 INFO:tasks.rados.rados.0.smithi137.stdout:2008: done (5 left) 2024-09-21T08:31:34.766 INFO:tasks.rados.rados.0.smithi137.stdout:2009: done (4 left) 2024-09-21T08:31:34.767 INFO:tasks.rados.rados.0.smithi137.stdout:2010: done (3 left) 2024-09-21T08:31:34.767 INFO:tasks.rados.rados.0.smithi137.stdout:2011: done (2 left) 2024-09-21T08:31:34.767 INFO:tasks.rados.rados.0.smithi137.stdout:2012: done (1 left) 2024-09-21T08:31:34.767 INFO:tasks.rados.rados.0.smithi137.stdout:2013: write oid 32 current snap is 218 2024-09-21T08:31:34.767 INFO:tasks.rados.rados.0.smithi137.stdout:2007: expect (ObjNum 656 snap 196 seq_num 656) 2024-09-21T08:31:34.846 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:34.849 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:34.850 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:34.870 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:34.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:34.871 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:34.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:34.994 INFO:tasks.rados.rados.0.smithi137.stdout:2013: seq_num 710 ranges {732970=647619,2097627=611905,3231266=437835} 2024-09-21T08:31:34.995 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:34.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:34.995 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:35.001 INFO:tasks.rados.rados.0.smithi137.stdout:2013: writing smithi13779221-32 from 732970 to 1380589 tid 1 2024-09-21T08:31:35.009 INFO:tasks.rados.rados.0.smithi137.stdout:2013: writing smithi13779221-32 from 2097627 to 2709532 tid 2 2024-09-21T08:31:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:34 smithi165 ceph-mon[32893]: 7.a scrub starts 2024-09-21T08:31:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:34 smithi165 ceph-mon[32893]: 7.a scrub ok 2024-09-21T08:31:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:34 smithi165 ceph-mon[32893]: 3.19 scrub starts 2024-09-21T08:31:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:34 smithi165 ceph-mon[32893]: 3.19 scrub ok 2024-09-21T08:31:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:34 smithi165 ceph-mon[32893]: osdmap e500: 8 total, 6 up, 5 in 2024-09-21T08:31:35.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3886532159' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:35.015 INFO:tasks.rados.rados.0.smithi137.stdout:2013: writing smithi13779221-32 from 3231266 to 3669101 tid 3 2024-09-21T08:31:35.015 INFO:tasks.rados.rados.0.smithi137.stdout:2007: done (1 left) 2024-09-21T08:31:35.015 INFO:tasks.rados.rados.0.smithi137.stdout:2014: copy_from oid 20 from oid 10 current snap is 218 2024-09-21T08:31:35.015 INFO:tasks.rados.rados.0.smithi137.stdout:2013: finishing write tid 1 to smithi13779221-32 2024-09-21T08:31:35.015 INFO:tasks.rados.rados.0.smithi137.stdout:2015: write oid 24 current snap is 218 2024-09-21T08:31:35.015 INFO:tasks.rados.rados.0.smithi137.stdout:2013: finishing write tid 2 to smithi13779221-32 2024-09-21T08:31:35.015 INFO:tasks.rados.rados.0.smithi137.stdout:2015: seq_num 711 ranges {749777=722868,2156109=471916,3258689=649107} 2024-09-21T08:31:35.023 INFO:tasks.rados.rados.0.smithi137.stdout:2015: writing smithi13779221-24 from 749777 to 1472645 tid 1 2024-09-21T08:31:35.029 INFO:tasks.rados.rados.0.smithi137.stdout:2015: writing smithi13779221-24 from 2156109 to 2628025 tid 2 2024-09-21T08:31:35.037 INFO:tasks.rados.rados.0.smithi137.stdout:2015: writing smithi13779221-24 from 3258689 to 3907796 tid 3 2024-09-21T08:31:35.037 INFO:tasks.rados.rados.0.smithi137.stdout:2016: snap_create 2024-09-21T08:31:35.037 INFO:tasks.rados.rados.0.smithi137.stdout:2014: finishing copy_from to smithi13779221-20 2024-09-21T08:31:35.037 INFO:tasks.rados.rados.0.smithi137.stdout:2014: got expected ENOENT (src dne) 2024-09-21T08:31:35.037 INFO:tasks.rados.rados.0.smithi137.stdout:2014: finishing copy_from racing read to smithi13779221-20 2024-09-21T08:31:35.037 INFO:tasks.rados.rados.0.smithi137.stdout:2013: finishing write tid 3 to smithi13779221-32 2024-09-21T08:31:35.037 INFO:tasks.rados.rados.0.smithi137.stdout:2013: finishing write tid 4 to smithi13779221-32 2024-09-21T08:31:35.037 INFO:tasks.rados.rados.0.smithi137.stdout:2013: finishing write tid 5 to smithi13779221-32 2024-09-21T08:31:35.037 INFO:tasks.rados.rados.0.smithi137.stdout:2013: oid 32 updating version 0 to 537 2024-09-21T08:31:35.037 INFO:tasks.rados.rados.0.smithi137.stdout:2013: oid 32 version 537 is already newer than 536 2024-09-21T08:31:35.038 INFO:tasks.rados.rados.0.smithi137.stdout:2013: oid 32 updating version 537 to 538 2024-09-21T08:31:35.038 INFO:tasks.rados.rados.0.smithi137.stdout:2013: oid 32 updating version 538 to 539 2024-09-21T08:31:35.038 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 539 (ObjNum 710 snap 218 seq_num 710) dirty exists 2024-09-21T08:31:35.038 INFO:tasks.rados.rados.0.smithi137.stdout:2013: left oid 32 (ObjNum 710 snap 218 seq_num 710) 2024-09-21T08:31:35.038 INFO:tasks.rados.rados.0.smithi137.stdout:2015: finishing write tid 1 to smithi13779221-24 2024-09-21T08:31:35.038 INFO:tasks.rados.rados.0.smithi137.stdout:2015: finishing write tid 2 to smithi13779221-24 2024-09-21T08:31:35.039 INFO:tasks.rados.rados.0.smithi137.stdout:2015: finishing write tid 3 to smithi13779221-24 2024-09-21T08:31:35.039 INFO:tasks.rados.rados.0.smithi137.stdout:2015: finishing write tid 4 to smithi13779221-24 2024-09-21T08:31:35.040 INFO:tasks.rados.rados.0.smithi137.stdout:2015: finishing write tid 5 to smithi13779221-24 2024-09-21T08:31:35.040 INFO:tasks.rados.rados.0.smithi137.stdout:2015: oid 24 updating version 0 to 734 2024-09-21T08:31:35.040 INFO:tasks.rados.rados.0.smithi137.stdout:2015: oid 24 updating version 734 to 737 2024-09-21T08:31:35.040 INFO:tasks.rados.rados.0.smithi137.stdout:2015: oid 24 version 737 is already newer than 735 2024-09-21T08:31:35.040 INFO:tasks.rados.rados.0.smithi137.stdout:2015: oid 24 version 737 is already newer than 736 2024-09-21T08:31:35.040 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 737 (ObjNum 711 snap 218 seq_num 711) dirty exists 2024-09-21T08:31:35.040 INFO:tasks.rados.rados.0.smithi137.stdout:2015: left oid 24 (ObjNum 711 snap 218 seq_num 711) 2024-09-21T08:31:35.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:35.118 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:35.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:35.120 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:35.220 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:35.242 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:35.243 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:35.243 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:35.287 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:31:35.287 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:31:35.287 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:31:35.288 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:31:35.344 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:35.366 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:35.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:35.367 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:35.467 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:35.493 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:35.494 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:35.494 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:35.595 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:35.622 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:35.623 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:35.623 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:35.625 INFO:tasks.rados.rados.0.smithi137.stdout:2013: done (3 left) 2024-09-21T08:31:35.625 INFO:tasks.rados.rados.0.smithi137.stdout:2014: done (2 left) 2024-09-21T08:31:35.625 INFO:tasks.rados.rados.0.smithi137.stdout:2015: done (1 left) 2024-09-21T08:31:35.625 INFO:tasks.rados.rados.0.smithi137.stdout:2016: done (0 left) 2024-09-21T08:31:35.625 INFO:tasks.rados.rados.0.smithi137.stdout:2017: rollback oid 17 current snap is 219 2024-09-21T08:31:35.625 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 17 to 214 2024-09-21T08:31:35.625 INFO:tasks.rados.rados.0.smithi137.stdout:2018: rollback oid 32 current snap is 219 2024-09-21T08:31:35.625 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 32 to 163 2024-09-21T08:31:35.625 INFO:tasks.rados.rados.0.smithi137.stdout:2019: snap_remove snap 204 2024-09-21T08:31:35.724 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:35.745 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:35.746 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:35.746 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:35.847 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:35.872 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:35.873 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:35.873 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:35.974 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:35.995 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:35.996 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:35.996 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:36.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:35 smithi165 ceph-mon[32893]: pgmap v656: 188 pgs: 2 active+clean+snaptrim, 1 remapped+peering, 5 active+clean+remapped+snaptrim_wait, 52 active+clean+remapped, 2 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 64 active+clean; 621 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 16 op/s; 248/1521 objects degraded (16.305%); 123/1521 objects misplaced (8.087%) 2024-09-21T08:31:36.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:35 smithi165 ceph-mon[32893]: osdmap e501: 8 total, 6 up, 5 in 2024-09-21T08:31:36.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:35 smithi165 ceph-mon[32893]: 4.9 scrub starts 2024-09-21T08:31:36.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:35 smithi165 ceph-mon[32893]: 4.9 scrub ok 2024-09-21T08:31:36.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:35 smithi137 ceph-mon[41044]: pgmap v656: 188 pgs: 2 active+clean+snaptrim, 1 remapped+peering, 5 active+clean+remapped+snaptrim_wait, 52 active+clean+remapped, 2 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 64 active+clean; 621 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 16 op/s; 248/1521 objects degraded (16.305%); 123/1521 objects misplaced (8.087%) 2024-09-21T08:31:36.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:35 smithi137 ceph-mon[41044]: osdmap e501: 8 total, 6 up, 5 in 2024-09-21T08:31:36.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:35 smithi137 ceph-mon[41044]: 4.9 scrub starts 2024-09-21T08:31:36.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:35 smithi137 ceph-mon[41044]: 4.9 scrub ok 2024-09-21T08:31:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:35 smithi137 ceph-mon[29634]: pgmap v656: 188 pgs: 2 active+clean+snaptrim, 1 remapped+peering, 5 active+clean+remapped+snaptrim_wait, 52 active+clean+remapped, 2 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 64 active+clean; 621 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 2.0 MiB/s wr, 16 op/s; 248/1521 objects degraded (16.305%); 123/1521 objects misplaced (8.087%) 2024-09-21T08:31:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:35 smithi137 ceph-mon[29634]: osdmap e501: 8 total, 6 up, 5 in 2024-09-21T08:31:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:35 smithi137 ceph-mon[29634]: 4.9 scrub starts 2024-09-21T08:31:36.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:35 smithi137 ceph-mon[29634]: 4.9 scrub ok 2024-09-21T08:31:36.059 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T08:31:36.097 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:36.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:36.123 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:36.123 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:36.143 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:36.224 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:36.252 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:36.252 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:36.253 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:36.354 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:36.379 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:36.380 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:36.380 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:36.481 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:36.502 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:36.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:36.503 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:36.604 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:36.625 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:36.632 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:36.632 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:36.633 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:36.633 INFO:tasks.rados.rados.0.smithi137.stdout:2019: done (2 left) 2024-09-21T08:31:36.633 INFO:tasks.rados.rados.0.smithi137.stdout:2020: delete oid 26 current snap is 219 2024-09-21T08:31:36.633 INFO:tasks.rados.rados.0.smithi137.stdout:2017: finishing rollback tid 0 to smithi13779221-17 2024-09-21T08:31:36.633 INFO:tasks.rados.rados.0.smithi137.stdout:2018: finishing rollback tid 0 to smithi13779221-32 2024-09-21T08:31:36.633 INFO:tasks.rados.rados.0.smithi137.stdout:2018: finishing rollback tid 1 to smithi13779221-32 2024-09-21T08:31:36.633 INFO:tasks.rados.rados.0.smithi137.stdout:2017: finishing rollback tid 1 to smithi13779221-17 2024-09-21T08:31:36.633 INFO:tasks.rados.rados.0.smithi137.stdout:2018: finishing rollback tid 2 to smithi13779221-32 2024-09-21T08:31:36.634 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 542 (ObjNum 428 snap 108 seq_num 428) dirty exists 2024-09-21T08:31:36.634 INFO:tasks.rados.rados.0.smithi137.stdout:2017: finishing rollback tid 2 to smithi13779221-17 2024-09-21T08:31:36.634 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 744 (ObjNum 690 snap 210 seq_num 690) dirty exists 2024-09-21T08:31:36.636 INFO:tasks.rados.rados.0.smithi137.stdout:2017: done (2 left) 2024-09-21T08:31:36.636 INFO:tasks.rados.rados.0.smithi137.stdout:2018: done (1 left) 2024-09-21T08:31:36.636 INFO:tasks.rados.rados.0.smithi137.stdout:2020: done (0 left) 2024-09-21T08:31:36.636 INFO:tasks.rados.rados.0.smithi137.stdout:2021: snap_create 2024-09-21T08:31:36.733 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1, 4] out_osds: [6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:31:36.733 INFO:tasks.thrashosds.thrasher:choose_action: min_in 4 min_out 0 min_live 2 min_dead 0 chance_down 0.40 2024-09-21T08:31:36.733 INFO:tasks.thrashosds.thrasher:Reweighting osd 4 to 0.7231158690613188 2024-09-21T08:31:36.734 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd reweight 4 0.7231158690613188 2024-09-21T08:31:36.734 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:36.750 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[29634]: 3.1c scrub starts 2024-09-21T08:31:36.750 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[29634]: 3.1c scrub ok 2024-09-21T08:31:36.750 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[29634]: 5.1d scrub starts 2024-09-21T08:31:36.750 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[29634]: 5.1d scrub ok 2024-09-21T08:31:36.750 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[29634]: osdmap e502: 8 total, 6 up, 5 in 2024-09-21T08:31:36.750 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[29634]: 6.1e scrub starts 2024-09-21T08:31:36.750 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[29634]: 6.1e scrub ok 2024-09-21T08:31:36.750 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 248/1521 objects degraded (16.305%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:36.750 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[41044]: 3.1c scrub starts 2024-09-21T08:31:36.751 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[41044]: 3.1c scrub ok 2024-09-21T08:31:36.751 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[41044]: 5.1d scrub starts 2024-09-21T08:31:36.751 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[41044]: 5.1d scrub ok 2024-09-21T08:31:36.751 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[41044]: osdmap e502: 8 total, 6 up, 5 in 2024-09-21T08:31:36.751 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[41044]: 6.1e scrub starts 2024-09-21T08:31:36.751 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[41044]: 6.1e scrub ok 2024-09-21T08:31:36.751 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:36 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 248/1521 objects degraded (16.305%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:36.760 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:36.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:36.761 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:36.851 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:31:36.862 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:36.917 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:36.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:36.919 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:36 smithi165 ceph-mon[32893]: 3.1c scrub starts 2024-09-21T08:31:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:36 smithi165 ceph-mon[32893]: 3.1c scrub ok 2024-09-21T08:31:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:36 smithi165 ceph-mon[32893]: 5.1d scrub starts 2024-09-21T08:31:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:36 smithi165 ceph-mon[32893]: 5.1d scrub ok 2024-09-21T08:31:37.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:36 smithi165 ceph-mon[32893]: osdmap e502: 8 total, 6 up, 5 in 2024-09-21T08:31:37.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:36 smithi165 ceph-mon[32893]: 6.1e scrub starts 2024-09-21T08:31:37.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:36 smithi165 ceph-mon[32893]: 6.1e scrub ok 2024-09-21T08:31:37.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:36 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 248/1521 objects degraded (16.305%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:37.019 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:37.043 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:37.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:37.044 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:37.134 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:31:37.135 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:31:37.135 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:31:37.135 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:31:37.135 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:31:37.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:37.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:37.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:37.167 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:37.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:37.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:37.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:37.290 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:37.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:37.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:37.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:37.417 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:37.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:37.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:37.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:37.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:37.635 INFO:tasks.rados.rados.0.smithi137.stdout:2021: done (0 left) 2024-09-21T08:31:37.635 INFO:tasks.rados.rados.0.smithi137.stdout:2022: read oid 41 snap 195 2024-09-21T08:31:37.635 INFO:tasks.rados.rados.0.smithi137.stdout:2022: expect deleted 2024-09-21T08:31:37.635 INFO:tasks.rados.rados.0.smithi137.stdout:2023: snap_remove snap 205 2024-09-21T08:31:37.637 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:31:37.641 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:37.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:37.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:37.664 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:37.764 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:37.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[29634]: 4.f scrub starts 2024-09-21T08:31:37.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[29634]: 4.f scrub ok 2024-09-21T08:31:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[29634]: pgmap v659: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 645 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1005 KiB/s wr, 6 op/s; 251/1560 objects degraded (16.090%); 129/1560 objects misplaced (8.269%) 2024-09-21T08:31:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[29634]: osdmap e503: 8 total, 6 up, 5 in 2024-09-21T08:31:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/385045124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:37.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[29634]: from='client.45929 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:37.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[41044]: 4.f scrub starts 2024-09-21T08:31:37.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[41044]: 4.f scrub ok 2024-09-21T08:31:37.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[41044]: pgmap v659: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 645 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1005 KiB/s wr, 6 op/s; 251/1560 objects degraded (16.090%); 129/1560 objects misplaced (8.269%) 2024-09-21T08:31:37.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[41044]: osdmap e503: 8 total, 6 up, 5 in 2024-09-21T08:31:37.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/385045124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:37.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:37 smithi137 ceph-mon[41044]: from='client.45929 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:37.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:37.786 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:37.786 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:37.887 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:37.908 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:37.909 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:37.909 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:38.010 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:37 smithi165 ceph-mon[32893]: 4.f scrub starts 2024-09-21T08:31:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:37 smithi165 ceph-mon[32893]: 4.f scrub ok 2024-09-21T08:31:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:37 smithi165 ceph-mon[32893]: pgmap v659: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 645 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1005 KiB/s wr, 6 op/s; 251/1560 objects degraded (16.090%); 129/1560 objects misplaced (8.269%) 2024-09-21T08:31:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:37 smithi165 ceph-mon[32893]: osdmap e503: 8 total, 6 up, 5 in 2024-09-21T08:31:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/385045124' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:38.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:37 smithi165 ceph-mon[32893]: from='client.45929 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:38.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:38.037 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:38.037 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:38.138 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:38.159 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:38.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:38.160 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:38.260 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:38.283 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T08:31:38.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:38.284 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:38.284 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:38.385 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:38.405 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:38.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:38.406 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:38.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:38.533 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:38.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:38.534 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:38.635 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:38.638 INFO:tasks.rados.rados.0.smithi137.stdout:2023: done (1 left) 2024-09-21T08:31:38.638 INFO:tasks.rados.rados.0.smithi137.stdout:2024: read oid 47 snap -1 2024-09-21T08:31:38.638 INFO:tasks.rados.rados.0.smithi137.stdout:2024: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:31:38.638 INFO:tasks.rados.rados.0.smithi137.stdout:2025: write oid 38 current snap is 220 2024-09-21T08:31:38.638 INFO:tasks.rados.rados.0.smithi137.stdout:2025: seq_num 713 ranges {783391=473366,1756720=475585,2797182=788004} 2024-09-21T08:31:38.644 INFO:tasks.rados.rados.0.smithi137.stdout:2025: writing smithi13779221-38 from 783391 to 1256757 tid 1 2024-09-21T08:31:38.650 INFO:tasks.rados.rados.0.smithi137.stdout:2025: writing smithi13779221-38 from 1756720 to 2232305 tid 2 2024-09-21T08:31:38.658 INFO:tasks.rados.rados.0.smithi137.stdout:2025: writing smithi13779221-38 from 2797182 to 3585186 tid 3 2024-09-21T08:31:38.658 INFO:tasks.rados.rados.0.smithi137.stdout:2026: snap_create 2024-09-21T08:31:38.658 INFO:tasks.rados.rados.0.smithi137.stdout:2024: expect (ObjNum 184 snap 37 seq_num 184) 2024-09-21T08:31:38.661 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:38.661 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:38.661 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:38.749 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:38.762 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:38.790 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:38.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:38.791 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:38.792 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:38.872 INFO:tasks.rados.rados.0.smithi137.stdout:2025: finishing write tid 1 to smithi13779221-38 2024-09-21T08:31:38.873 INFO:tasks.rados.rados.0.smithi137.stdout:2025: finishing write tid 2 to smithi13779221-38 2024-09-21T08:31:38.873 INFO:tasks.rados.rados.0.smithi137.stdout:2025: finishing write tid 3 to smithi13779221-38 2024-09-21T08:31:38.873 INFO:tasks.rados.rados.0.smithi137.stdout:2025: finishing write tid 4 to smithi13779221-38 2024-09-21T08:31:38.873 INFO:tasks.rados.rados.0.smithi137.stdout:2025: finishing write tid 5 to smithi13779221-38 2024-09-21T08:31:38.873 INFO:tasks.rados.rados.0.smithi137.stdout:2025: oid 38 updating version 0 to 822 2024-09-21T08:31:38.873 INFO:tasks.rados.rados.0.smithi137.stdout:2025: oid 38 version 822 is already newer than 821 2024-09-21T08:31:38.873 INFO:tasks.rados.rados.0.smithi137.stdout:2025: oid 38 updating version 822 to 823 2024-09-21T08:31:38.873 INFO:tasks.rados.rados.0.smithi137.stdout:2025: oid 38 updating version 823 to 824 2024-09-21T08:31:38.873 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 824 (ObjNum 713 snap 220 seq_num 713) dirty exists 2024-09-21T08:31:38.873 INFO:tasks.rados.rados.0.smithi137.stdout:2025: left oid 38 (ObjNum 713 snap 220 seq_num 713) 2024-09-21T08:31:38.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:38 smithi137 ceph-mon[29634]: 5.5 scrub starts 2024-09-21T08:31:38.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:38 smithi137 ceph-mon[29634]: 5.5 scrub ok 2024-09-21T08:31:38.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:38 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:38.882 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:38 smithi137 ceph-mon[29634]: from='client.45929 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:31:38.883 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:38 smithi137 ceph-mon[29634]: osdmap e504: 8 total, 6 up, 5 in 2024-09-21T08:31:38.883 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:38 smithi137 ceph-mon[41044]: 5.5 scrub starts 2024-09-21T08:31:38.883 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:38 smithi137 ceph-mon[41044]: 5.5 scrub ok 2024-09-21T08:31:38.883 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:38 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:38.883 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:38 smithi137 ceph-mon[41044]: from='client.45929 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:31:38.883 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:38 smithi137 ceph-mon[41044]: osdmap e504: 8 total, 6 up, 5 in 2024-09-21T08:31:38.892 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:38.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:38.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:38.917 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:38 smithi165 ceph-mon[32893]: 5.5 scrub starts 2024-09-21T08:31:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:38 smithi165 ceph-mon[32893]: 5.5 scrub ok 2024-09-21T08:31:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:38 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:38 smithi165 ceph-mon[32893]: from='client.45929 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:31:39.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:38 smithi165 ceph-mon[32893]: osdmap e504: 8 total, 6 up, 5 in 2024-09-21T08:31:39.018 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:39.039 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:39.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:39.040 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:39.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:39.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:39.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:39.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:39.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:39.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:39.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:39.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:39.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:39.416 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:39.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:39.417 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:39.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:39.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:39.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:39.544 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:39.645 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:39.654 INFO:tasks.rados.rados.0.smithi137.stdout:2022: done (3 left) 2024-09-21T08:31:39.654 INFO:tasks.rados.rados.0.smithi137.stdout:2024: done (2 left) 2024-09-21T08:31:39.654 INFO:tasks.rados.rados.0.smithi137.stdout:2025: done (1 left) 2024-09-21T08:31:39.655 INFO:tasks.rados.rados.0.smithi137.stdout:2026: done (0 left) 2024-09-21T08:31:39.655 INFO:tasks.rados.rados.0.smithi137.stdout:2027: rollback oid 8 current snap is 221 2024-09-21T08:31:39.655 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 8 to 217 2024-09-21T08:31:39.655 INFO:tasks.rados.rados.0.smithi137.stdout:2028: snap_create 2024-09-21T08:31:39.659 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.4 to 0.723116 (b91e) 2024-09-21T08:31:39.664 INFO:tasks.rados.rados.0.smithi137.stdout:2027: finishing rollback tid 1 to smithi13779221-8 2024-09-21T08:31:39.664 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 451 (ObjNum 353 snap 0 seq_num 3761330640) dirty dne 2024-09-21T08:31:39.675 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:39.676 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:39.676 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:39.701 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:39.721 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:39.721 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:39.737 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:39.737 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:39.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[29634]: pgmap v662: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 645 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1005 KiB/s wr, 6 op/s; 251/1560 objects degraded (16.090%); 129/1560 objects misplaced (8.269%) 2024-09-21T08:31:39.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[29634]: osdmap e505: 8 total, 6 up, 5 in 2024-09-21T08:31:39.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:31:39.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:31:39.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1168840284' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.7231158690613188}]: dispatch 2024-09-21T08:31:39.738 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[29634]: from='client.45935 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.7231158690613188}]: dispatch 2024-09-21T08:31:39.738 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[41044]: pgmap v662: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 645 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1005 KiB/s wr, 6 op/s; 251/1560 objects degraded (16.090%); 129/1560 objects misplaced (8.269%) 2024-09-21T08:31:39.739 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[41044]: osdmap e505: 8 total, 6 up, 5 in 2024-09-21T08:31:39.739 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:31:39.739 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:31:39.739 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1168840284' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.7231158690613188}]: dispatch 2024-09-21T08:31:39.739 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:39 smithi137 ceph-mon[41044]: from='client.45935 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.7231158690613188}]: dispatch 2024-09-21T08:31:39.753 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:39.753 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:39.771 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:39.772 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:39.777 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:39.788 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:39.788 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:39.800 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:39.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:39.801 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:39.804 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:39.805 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:39.805 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:39.902 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:39.924 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:39.924 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:39.925 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:39 smithi165 ceph-mon[32893]: pgmap v662: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 645 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1005 KiB/s wr, 6 op/s; 251/1560 objects degraded (16.090%); 129/1560 objects misplaced (8.269%) 2024-09-21T08:31:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:39 smithi165 ceph-mon[32893]: osdmap e505: 8 total, 6 up, 5 in 2024-09-21T08:31:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:39 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:31:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:39 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:31:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:39 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1168840284' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.7231158690613188}]: dispatch 2024-09-21T08:31:40.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:39 smithi165 ceph-mon[32893]: from='client.45935 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 0.7231158690613188}]: dispatch 2024-09-21T08:31:40.025 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:40.047 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:40.047 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:40.048 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:40.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:40.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:40.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:40.171 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:40.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:40.298 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:40.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:40.439 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:40.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:40.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:40.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:40.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:40.653 INFO:tasks.rados.rados.0.smithi137.stdout:2027: done (1 left) 2024-09-21T08:31:40.654 INFO:tasks.rados.rados.0.smithi137.stdout:2028: done (0 left) 2024-09-21T08:31:40.654 INFO:tasks.rados.rados.0.smithi137.stdout:2029: write oid 27 current snap is 222 2024-09-21T08:31:40.654 INFO:tasks.rados.rados.0.smithi137.stdout:2029: seq_num 714 ranges {400198=548615,1415202=584996} 2024-09-21T08:31:40.659 INFO:tasks.rados.rados.0.smithi137.stdout:2029: writing smithi13779221-27 from 400198 to 948813 tid 1 2024-09-21T08:31:40.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:40.666 INFO:tasks.rados.rados.0.smithi137.stdout:2029: writing smithi13779221-27 from 1415202 to 2000198 tid 2 2024-09-21T08:31:40.666 INFO:tasks.rados.rados.0.smithi137.stdout:2030: read oid 15 snap -1 2024-09-21T08:31:40.666 INFO:tasks.rados.rados.0.smithi137.stdout:2030: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:40.667 INFO:tasks.rados.rados.0.smithi137.stdout:2031: write oid 44 current snap is 222 2024-09-21T08:31:40.667 INFO:tasks.rados.rados.0.smithi137.stdout:2029: finishing write tid 1 to smithi13779221-27 2024-09-21T08:31:40.667 INFO:tasks.rados.rados.0.smithi137.stdout:2031: seq_num 715 ranges {417005=740217,1590037=426968} 2024-09-21T08:31:40.673 INFO:tasks.rados.rados.0.smithi137.stdout:2031: writing smithi13779221-44 from 417005 to 1157222 tid 1 2024-09-21T08:31:40.678 INFO:tasks.rados.rados.0.smithi137.stdout:2031: writing smithi13779221-44 from 1590037 to 2017005 tid 2 2024-09-21T08:31:40.678 INFO:tasks.rados.rados.0.smithi137.stdout:2032: rollback oid 6 current snap is 222 2024-09-21T08:31:40.678 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 6 to 216 2024-09-21T08:31:40.679 INFO:tasks.rados.rados.0.smithi137.stdout:2029: finishing write tid 2 to smithi13779221-27 2024-09-21T08:31:40.679 INFO:tasks.rados.rados.0.smithi137.stdout:2029: finishing write tid 3 to smithi13779221-27 2024-09-21T08:31:40.679 INFO:tasks.rados.rados.0.smithi137.stdout:2029: finishing write tid 4 to smithi13779221-27 2024-09-21T08:31:40.679 INFO:tasks.rados.rados.0.smithi137.stdout:2029: oid 27 updating version 0 to 395 2024-09-21T08:31:40.679 INFO:tasks.rados.rados.0.smithi137.stdout:2029: oid 27 version 395 is already newer than 394 2024-09-21T08:31:40.679 INFO:tasks.rados.rados.0.smithi137.stdout:2029: oid 27 updating version 395 to 396 2024-09-21T08:31:40.679 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 27 v 396 (ObjNum 714 snap 222 seq_num 714) dirty exists 2024-09-21T08:31:40.679 INFO:tasks.rados.rados.0.smithi137.stdout:2029: left oid 27 (ObjNum 714 snap 222 seq_num 714) 2024-09-21T08:31:40.679 INFO:tasks.rados.rados.0.smithi137.stdout:2030: expect (ObjNum 651 snap 192 seq_num 651) 2024-09-21T08:31:40.689 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:40.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:40.690 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:40.781 INFO:tasks.rados.rados.0.smithi137.stdout:2032: finishing rollback tid 1 to smithi13779221-6 2024-09-21T08:31:40.782 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 6 v 401 (ObjNum 129 snap 0 seq_num 2161677176) dirty dne 2024-09-21T08:31:40.782 INFO:tasks.rados.rados.0.smithi137.stdout:2029: done (3 left) 2024-09-21T08:31:40.782 INFO:tasks.rados.rados.0.smithi137.stdout:2030: done (2 left) 2024-09-21T08:31:40.782 INFO:tasks.rados.rados.0.smithi137.stdout:2032: done (1 left) 2024-09-21T08:31:40.782 INFO:tasks.rados.rados.0.smithi137.stdout:2033: read oid 33 snap -1 2024-09-21T08:31:40.782 INFO:tasks.rados.rados.0.smithi137.stdout:2033: expect deleted 2024-09-21T08:31:40.782 INFO:tasks.rados.rados.0.smithi137.stdout:2034: read oid 24 snap 216 2024-09-21T08:31:40.782 INFO:tasks.rados.rados.0.smithi137.stdout:2034: expect deleted 2024-09-21T08:31:40.782 INFO:tasks.rados.rados.0.smithi137.stdout:2035: snap_remove snap 220 2024-09-21T08:31:40.784 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:31:40.791 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:40.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:40.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:40.815 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:40.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:40.937 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:40.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:40.938 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:40 smithi165 ceph-mon[32893]: from='client.45935 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.7231158690613188}]': finished 2024-09-21T08:31:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:40 smithi165 ceph-mon[32893]: osdmap e506: 8 total, 6 up, 5 in 2024-09-21T08:31:41.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:40 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/273808965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:40 smithi137 ceph-mon[29634]: from='client.45935 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.7231158690613188}]': finished 2024-09-21T08:31:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:40 smithi137 ceph-mon[29634]: osdmap e506: 8 total, 6 up, 5 in 2024-09-21T08:31:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:40 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/273808965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:41.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:31:40 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:31:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:31:41.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:40 smithi137 ceph-mon[41044]: from='client.45935 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 0.7231158690613188}]': finished 2024-09-21T08:31:41.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:40 smithi137 ceph-mon[41044]: osdmap e506: 8 total, 6 up, 5 in 2024-09-21T08:31:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:40 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/273808965' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:41.039 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:41.048 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:41.048 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:41.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:41.064 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:41.064 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:41.091 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:41.124 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:41.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:41.187 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:41.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:41.188 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:41.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:41.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:41.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:41.311 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:41.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:41.438 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:41.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:41.439 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:41.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:41.565 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:41.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:41.566 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:41.601 INFO:tasks.rados.rados.0.smithi137.stdout:2033: done (3 left) 2024-09-21T08:31:41.601 INFO:tasks.rados.rados.0.smithi137.stdout:2034: done (2 left) 2024-09-21T08:31:41.601 INFO:tasks.rados.rados.0.smithi137.stdout:2035: done (1 left) 2024-09-21T08:31:41.601 INFO:tasks.rados.rados.0.smithi137.stdout:2036: write (excl) oid 26 current snap is 222 2024-09-21T08:31:41.601 INFO:tasks.rados.rados.0.smithi137.stdout:2036: seq_num 716 ranges {433812=415466,1648519=385293} 2024-09-21T08:31:41.607 INFO:tasks.rados.rados.0.smithi137.stdout:2036: writing smithi13779221-26 from 433812 to 849278 tid 1 2024-09-21T08:31:41.614 INFO:tasks.rados.rados.0.smithi137.stdout:2036: writing smithi13779221-26 from 1648519 to 2033812 tid 2 2024-09-21T08:31:41.614 INFO:tasks.rados.rados.0.smithi137.stdout:2037: write (excl) oid 29 current snap is 222 2024-09-21T08:31:41.614 INFO:tasks.rados.rados.0.smithi137.stdout:2037: seq_num 717 ranges {450619=490715,1590648=459971} 2024-09-21T08:31:41.620 INFO:tasks.rados.rados.0.smithi137.stdout:2037: writing smithi13779221-29 from 450619 to 941334 tid 1 2024-09-21T08:31:41.627 INFO:tasks.rados.rados.0.smithi137.stdout:2037: writing smithi13779221-29 from 1590648 to 2050619 tid 2 2024-09-21T08:31:41.627 INFO:tasks.rados.rados.0.smithi137.stdout:2038: read oid 34 snap -1 2024-09-21T08:31:41.627 INFO:tasks.rados.rados.0.smithi137.stdout:2038: expect (ObjNum 620 snap 182 seq_num 620) 2024-09-21T08:31:41.627 INFO:tasks.rados.rados.0.smithi137.stdout:2039: copy_from oid 48 from oid 7 current snap is 222 2024-09-21T08:31:41.628 INFO:tasks.rados.rados.0.smithi137.stdout:2040: copy_from oid 15 from oid 27 current snap is 222 2024-09-21T08:31:41.628 INFO:tasks.rados.rados.0.smithi137.stdout:2036: finishing write tid 1 to smithi13779221-26 2024-09-21T08:31:41.628 INFO:tasks.rados.rados.0.smithi137.stdout:2041: read oid 25 snap -1 2024-09-21T08:31:41.628 INFO:tasks.rados.rados.0.smithi137.stdout:2041: expect (ObjNum 688 snap 210 seq_num 688) 2024-09-21T08:31:41.628 INFO:tasks.rados.rados.0.smithi137.stdout:2042: read oid 21 snap -1 2024-09-21T08:31:41.628 INFO:tasks.rados.rados.0.smithi137.stdout:2042: expect (ObjNum 677 snap 205 seq_num 677) 2024-09-21T08:31:41.628 INFO:tasks.rados.rados.0.smithi137.stdout:2043: write (excl) oid 47 current snap is 222 2024-09-21T08:31:41.628 INFO:tasks.rados.rados.0.smithi137.stdout:2043: seq_num 718 ranges {467426=565964,1649130=418296} 2024-09-21T08:31:41.634 INFO:tasks.rados.rados.0.smithi137.stdout:2043: writing smithi13779221-47 from 467426 to 1033390 tid 1 2024-09-21T08:31:41.640 INFO:tasks.rados.rados.0.smithi137.stdout:2043: writing smithi13779221-47 from 1649130 to 2067426 tid 2 2024-09-21T08:31:41.641 INFO:tasks.rados.rados.0.smithi137.stdout:2044: snap_create 2024-09-21T08:31:41.641 INFO:tasks.rados.rados.0.smithi137.stdout:2038: expect (ObjNum 620 snap 182 seq_num 620) 2024-09-21T08:31:41.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:41.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:41.690 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:41.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:41.732 INFO:tasks.rados.rados.0.smithi137.stdout:2036: finishing write tid 2 to smithi13779221-26 2024-09-21T08:31:41.732 INFO:tasks.rados.rados.0.smithi137.stdout:2036: finishing write tid 3 to smithi13779221-26 2024-09-21T08:31:41.732 INFO:tasks.rados.rados.0.smithi137.stdout:2036: finishing write tid 4 to smithi13779221-26 2024-09-21T08:31:41.732 INFO:tasks.rados.rados.0.smithi137.stdout:2036: oid 26 updating version 0 to 413 2024-09-21T08:31:41.732 INFO:tasks.rados.rados.0.smithi137.stdout:2036: oid 26 version 413 is already newer than 412 2024-09-21T08:31:41.732 INFO:tasks.rados.rados.0.smithi137.stdout:2036: oid 26 updating version 413 to 414 2024-09-21T08:31:41.732 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 414 (ObjNum 716 snap 222 seq_num 716) dirty exists 2024-09-21T08:31:41.732 INFO:tasks.rados.rados.0.smithi137.stdout:2036: left oid 26 (ObjNum 716 snap 222 seq_num 716) 2024-09-21T08:31:41.732 INFO:tasks.rados.rados.0.smithi137.stdout:2042: expect (ObjNum 677 snap 205 seq_num 677) 2024-09-21T08:31:41.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:41.803 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:41.803 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:41.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:41.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:41.814 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:41.846 INFO:tasks.rados.rados.0.smithi137.stdout:2037: finishing write tid 1 to smithi13779221-29 2024-09-21T08:31:41.846 INFO:tasks.rados.rados.0.smithi137.stdout:2041: expect (ObjNum 688 snap 210 seq_num 688) 2024-09-21T08:31:41.915 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:41.936 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:41.937 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:41.937 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:41.993 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:41 smithi165 ceph-mon[32893]: pgmap v665: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 645 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 251/1560 objects degraded (16.090%); 129/1560 objects misplaced (8.269%) 2024-09-21T08:31:41.993 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:41 smithi165 ceph-mon[32893]: osdmap e507: 8 total, 6 up, 5 in 2024-09-21T08:31:41.993 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:41 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 251/1560 objects degraded (16.090%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:41.993 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:41 smithi165 ceph-mon[32893]: osdmap e508: 8 total, 6 up, 5 in 2024-09-21T08:31:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:41 smithi137 ceph-mon[29634]: pgmap v665: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 645 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 251/1560 objects degraded (16.090%); 129/1560 objects misplaced (8.269%) 2024-09-21T08:31:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:41 smithi137 ceph-mon[29634]: osdmap e507: 8 total, 6 up, 5 in 2024-09-21T08:31:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:41 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 251/1560 objects degraded (16.090%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:41 smithi137 ceph-mon[29634]: osdmap e508: 8 total, 6 up, 5 in 2024-09-21T08:31:42.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:41 smithi137 ceph-mon[41044]: pgmap v665: 188 pgs: 3 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 51 active+clean+remapped, 5 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 60 active+clean; 645 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 251/1560 objects degraded (16.090%); 129/1560 objects misplaced (8.269%) 2024-09-21T08:31:42.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:41 smithi137 ceph-mon[41044]: osdmap e507: 8 total, 6 up, 5 in 2024-09-21T08:31:42.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:41 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 251/1560 objects degraded (16.090%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:42.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:41 smithi137 ceph-mon[41044]: osdmap e508: 8 total, 6 up, 5 in 2024-09-21T08:31:42.038 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:42.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:42.063 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:42.063 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2040: finishing copy_from racing read to smithi13779221-15 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2037: finishing write tid 2 to smithi13779221-29 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2037: finishing write tid 3 to smithi13779221-29 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2031: finishing write tid 1 to smithi13779221-44 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2037: finishing write tid 4 to smithi13779221-29 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2037: oid 29 updating version 0 to 818 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2037: oid 29 updating version 818 to 819 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2037: oid 29 updating version 819 to 820 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 820 (ObjNum 717 snap 222 seq_num 717) dirty exists 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2037: left oid 29 (ObjNum 717 snap 222 seq_num 717) 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2031: finishing write tid 2 to smithi13779221-44 2024-09-21T08:31:42.111 INFO:tasks.rados.rados.0.smithi137.stdout:2043: finishing write tid 1 to smithi13779221-47 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2031: finishing write tid 3 to smithi13779221-44 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2031: finishing write tid 4 to smithi13779221-44 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2031: oid 44 updating version 0 to 746 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2031: oid 44 version 746 is already newer than 745 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2031: oid 44 updating version 746 to 747 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 747 (ObjNum 715 snap 222 seq_num 715) dirty exists 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2031: left oid 44 (ObjNum 715 snap 222 seq_num 715) 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2043: finishing write tid 2 to smithi13779221-47 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2040: finishing copy_from to smithi13779221-15 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 15 v 396 (ObjNum 714 snap 222 seq_num 714) dirty exists 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2043: finishing write tid 3 to smithi13779221-47 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2043: finishing write tid 4 to smithi13779221-47 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2043: oid 47 updating version 0 to 463 2024-09-21T08:31:42.112 INFO:tasks.rados.rados.0.smithi137.stdout:2043: oid 47 version 463 is already newer than 461 2024-09-21T08:31:42.113 INFO:tasks.rados.rados.0.smithi137.stdout:2043: oid 47 version 463 is already newer than 462 2024-09-21T08:31:42.113 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 47 v 463 (ObjNum 718 snap 222 seq_num 718) dirty exists 2024-09-21T08:31:42.113 INFO:tasks.rados.rados.0.smithi137.stdout:2043: left oid 47 (ObjNum 718 snap 222 seq_num 718) 2024-09-21T08:31:42.113 INFO:tasks.rados.rados.0.smithi137.stdout:2039: finishing copy_from racing read to smithi13779221-48 2024-09-21T08:31:42.113 INFO:tasks.rados.rados.0.smithi137.stdout:2039: finishing copy_from to smithi13779221-48 2024-09-21T08:31:42.113 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 827 (ObjNum 670 snap 203 seq_num 670) dirty exists 2024-09-21T08:31:42.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:42.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:42.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:42.187 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:42.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:31:41 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:31:42.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:42.309 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:42.309 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:42.310 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:42.410 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:42.432 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:42.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:42.433 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:42.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T08:31:42.534 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:42.555 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:42.555 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:42.556 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:42.607 INFO:tasks.rados.rados.0.smithi137.stdout:2031: done (9 left) 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2036: done (8 left) 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2037: done (7 left) 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2038: done (6 left) 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2039: done (5 left) 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2040: done (4 left) 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2041: done (3 left) 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2042: done (2 left) 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2043: done (1 left) 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2044: done (0 left) 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2045: copy_from oid 24 from oid 5 current snap is 223 2024-09-21T08:31:42.608 INFO:tasks.rados.rados.0.smithi137.stdout:2046: snap_create 2024-09-21T08:31:42.616 INFO:tasks.rados.rados.0.smithi137.stdout:2045: finishing copy_from to smithi13779221-24 2024-09-21T08:31:42.616 INFO:tasks.rados.rados.0.smithi137.stdout:2045: got expected ENOENT (src dne) 2024-09-21T08:31:42.616 INFO:tasks.rados.rados.0.smithi137.stdout:2045: finishing copy_from racing read to smithi13779221-24 2024-09-21T08:31:42.656 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:42.677 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:42.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:42.678 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:42.773 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:42.779 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:42.802 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:42.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:42.803 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:42.903 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:42.925 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:42.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:42.926 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:43.027 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:43.049 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:31:43.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:43.051 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:43.051 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:43.151 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:43.172 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:43.173 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:43.173 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:43.274 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:43.275 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:43.275 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:43.296 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:43.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:43.297 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:43.298 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:43.398 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:43.420 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:43.420 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:43.421 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:43.521 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:43.544 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:43.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:43.544 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:43.610 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:31:43.616 INFO:tasks.rados.rados.0.smithi137.stdout:2045: done (1 left) 2024-09-21T08:31:43.616 INFO:tasks.rados.rados.0.smithi137.stdout:2046: done (0 left) 2024-09-21T08:31:43.616 INFO:tasks.rados.rados.0.smithi137.stdout:2047: write oid 38 current snap is 224 2024-09-21T08:31:43.616 INFO:tasks.rados.rados.0.smithi137.stdout:2047: seq_num 719 ranges {484233=641213,1707612=376621} 2024-09-21T08:31:43.623 INFO:tasks.rados.rados.0.smithi137.stdout:2047: writing smithi13779221-38 from 484233 to 1125446 tid 1 2024-09-21T08:31:43.628 INFO:tasks.rados.rados.0.smithi137.stdout:2047: writing smithi13779221-38 from 1707612 to 2084233 tid 2 2024-09-21T08:31:43.628 INFO:tasks.rados.rados.0.smithi137.stdout:2048: snap_remove snap 214 2024-09-21T08:31:43.646 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:43.667 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:43.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:43.668 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:43.722 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:43.722 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[29634]: pgmap v668: 188 pgs: 3 active+clean+snaptrim, 17 remapped+peering, 6 active+clean+remapped+snaptrim_wait, 45 active+clean+remapped, 3 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 649 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 24 op/s; 251/1566 objects degraded (16.028%); 109/1566 objects misplaced (6.960%); 2.2 MiB/s, 7 objects/s recovering 2024-09-21T08:31:43.722 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[29634]: osdmap e509: 8 total, 6 up, 5 in 2024-09-21T08:31:43.722 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[29634]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T08:31:43.722 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:31:43.722 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:31:43.722 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/867638987' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:43.722 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[29634]: from='client.25939 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:43.723 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:43.723 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[41044]: pgmap v668: 188 pgs: 3 active+clean+snaptrim, 17 remapped+peering, 6 active+clean+remapped+snaptrim_wait, 45 active+clean+remapped, 3 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 649 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 24 op/s; 251/1566 objects degraded (16.028%); 109/1566 objects misplaced (6.960%); 2.2 MiB/s, 7 objects/s recovering 2024-09-21T08:31:43.723 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[41044]: osdmap e509: 8 total, 6 up, 5 in 2024-09-21T08:31:43.723 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[41044]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T08:31:43.723 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:31:43.723 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:31:43.723 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/867638987' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:43.723 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:43 smithi137 ceph-mon[41044]: from='client.25939 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:43.770 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:43.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:43.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:43.792 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:43.893 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:43.915 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:43.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:43.916 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:43.929 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:43 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:43.929 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:43 smithi165 ceph-mon[32893]: pgmap v668: 188 pgs: 3 active+clean+snaptrim, 17 remapped+peering, 6 active+clean+remapped+snaptrim_wait, 45 active+clean+remapped, 3 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 649 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 24 op/s; 251/1566 objects degraded (16.028%); 109/1566 objects misplaced (6.960%); 2.2 MiB/s, 7 objects/s recovering 2024-09-21T08:31:43.929 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:43 smithi165 ceph-mon[32893]: osdmap e509: 8 total, 6 up, 5 in 2024-09-21T08:31:43.929 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:43 smithi165 ceph-mon[32893]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-09-21T08:31:43.929 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:43 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:31:43.929 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:43 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:31:43.930 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:43 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/867638987' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:43.930 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:43 smithi165 ceph-mon[32893]: from='client.25939 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:43.955 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T08:31:43.964 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:43.964 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:43.965 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:43.965 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:43.965 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:43.965 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:43.965 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:43.965 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:43.965 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:43.965 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:43.966 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:43.966 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:43.966 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:43.966 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:44.017 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:44.038 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:44.039 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:44.039 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:44.140 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:44.162 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:44.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:44.163 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:44.206 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:44.264 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:44.286 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:44.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:44.287 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:44.388 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:44.409 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:44.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:44.410 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:44.511 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:44.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:44.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:44.533 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:44.620 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:44.620 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:44.632 INFO:tasks.rados.rados.0.smithi137.stdout:2048: done (1 left) 2024-09-21T08:31:44.632 INFO:tasks.rados.rados.0.smithi137.stdout:2049: delete oid 44 current snap is 224 2024-09-21T08:31:44.632 INFO:tasks.rados.rados.0.smithi137.stdout:2047: finishing write tid 1 to smithi13779221-38 2024-09-21T08:31:44.633 INFO:tasks.rados.rados.0.smithi137.stdout:2047: finishing write tid 2 to smithi13779221-38 2024-09-21T08:31:44.633 INFO:tasks.rados.rados.0.smithi137.stdout:2047: finishing write tid 3 to smithi13779221-38 2024-09-21T08:31:44.633 INFO:tasks.rados.rados.0.smithi137.stdout:2047: finishing write tid 4 to smithi13779221-38 2024-09-21T08:31:44.633 INFO:tasks.rados.rados.0.smithi137.stdout:2047: oid 38 updating version 0 to 826 2024-09-21T08:31:44.633 INFO:tasks.rados.rados.0.smithi137.stdout:2047: oid 38 updating version 826 to 827 2024-09-21T08:31:44.633 INFO:tasks.rados.rados.0.smithi137.stdout:2047: oid 38 updating version 827 to 828 2024-09-21T08:31:44.633 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 828 (ObjNum 719 snap 224 seq_num 719) dirty exists 2024-09-21T08:31:44.633 INFO:tasks.rados.rados.0.smithi137.stdout:2047: left oid 38 (ObjNum 719 snap 224 seq_num 719) 2024-09-21T08:31:44.633 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:44.640 INFO:tasks.rados.rados.0.smithi137.stdout:2047: done (1 left) 2024-09-21T08:31:44.640 INFO:tasks.rados.rados.0.smithi137.stdout:2049: done (0 left) 2024-09-21T08:31:44.641 INFO:tasks.rados.rados.0.smithi137.stdout:2050: delete oid 12 current snap is 224 2024-09-21T08:31:44.642 INFO:tasks.rados.rados.0.smithi137.stdout:2050: done (0 left) 2024-09-21T08:31:44.642 INFO:tasks.rados.rados.0.smithi137.stdout:2051: delete oid 38 current snap is 224 2024-09-21T08:31:44.644 INFO:tasks.rados.rados.0.smithi137.stdout:2051: done (0 left) 2024-09-21T08:31:44.644 INFO:tasks.rados.rados.0.smithi137.stdout:2052: snap_create 2024-09-21T08:31:44.657 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:44.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:44.658 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:44.757 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:44.757 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:31:44.757 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:31:44.757 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:31:44.757 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:44.759 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:44.781 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:44.782 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:44.782 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:44.885 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:44.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:44.907 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:44.907 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:45.008 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:44 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:44 smithi165 ceph-mon[32893]: from='client.25939 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:45.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:44 smithi165 ceph-mon[32893]: osdmap e510: 8 total, 6 up, 5 in 2024-09-21T08:31:45.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:44 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1380032560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:45.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:44 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:45.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:44 smithi137 ceph-mon[29634]: from='client.25939 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:45.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:44 smithi137 ceph-mon[29634]: osdmap e510: 8 total, 6 up, 5 in 2024-09-21T08:31:45.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:44 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1380032560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:45.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:44 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:45.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:44 smithi137 ceph-mon[41044]: from='client.25939 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:45.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:44 smithi137 ceph-mon[41044]: osdmap e510: 8 total, 6 up, 5 in 2024-09-21T08:31:45.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:44 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1380032560' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:45.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:45.031 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:45.031 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:45.132 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:45.154 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:45.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:45.155 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:45.256 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:45.278 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:45.279 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:45.279 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:45.359 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:31:45 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:31:45.170+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:31:45.363 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T08:31:45.380 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:45.402 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:45.402 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:45.403 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:45.440 INFO:tasks.thrashosds.thrasher:in_osds: [2, 5, 7, 1, 4] out_osds: [6, 3, 0] dead_osds: [7, 0] live_osds: [1, 2, 3, 4, 5, 6] 2024-09-21T08:31:45.440 INFO:tasks.thrashosds.thrasher:Reviving osd 7 2024-09-21T08:31:45.440 INFO:tasks.cephadm.osd.7:Restarting osd.7 (starting--it wasn't running)... 2024-09-21T08:31:45.441 DEBUG:teuthology.orchestra.run.smithi165:osd.7> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.7.service 2024-09-21T08:31:45.448 DEBUG:teuthology.orchestra.run.smithi165:> sudo systemctl start ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.7 2024-09-21T08:31:45.503 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:45.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:45.561 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:45.561 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:45.623 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:45.630 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:45 smithi165 systemd[1]: Starting Ceph osd.7 for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:31:45.643 INFO:tasks.rados.rados.0.smithi137.stdout:2052: done (0 left) 2024-09-21T08:31:45.644 INFO:tasks.rados.rados.0.smithi137.stdout:2053: rollback oid 46 current snap is 225 2024-09-21T08:31:45.644 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 46 to 183 2024-09-21T08:31:45.644 INFO:tasks.rados.rados.0.smithi137.stdout:2054: read oid 32 snap 221 2024-09-21T08:31:45.644 INFO:tasks.rados.rados.0.smithi137.stdout:2054: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:31:45.644 INFO:tasks.rados.rados.0.smithi137.stdout:2055: rollback oid 5 current snap is 225 2024-09-21T08:31:45.644 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 5 to 182 2024-09-21T08:31:45.644 INFO:tasks.rados.rados.0.smithi137.stdout:2056: rollback oid 23 current snap is 225 2024-09-21T08:31:45.644 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 23 to 210 2024-09-21T08:31:45.644 INFO:tasks.rados.rados.0.smithi137.stdout:2057: snap_create 2024-09-21T08:31:45.646 INFO:tasks.rados.rados.0.smithi137.stdout:2055: finishing rollback tid 1 to smithi13779221-5 2024-09-21T08:31:45.646 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 533 (ObjNum 225 snap 0 seq_num 379559344) dirty dne 2024-09-21T08:31:45.648 INFO:tasks.rados.rados.0.smithi137.stdout:2053: finishing rollback tid 1 to smithi13779221-46 2024-09-21T08:31:45.650 INFO:tasks.rados.rados.0.smithi137.stdout:2054: expect (ObjNum 428 snap 108 seq_num 428) 2024-09-21T08:31:45.662 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:45.685 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:45.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:45.686 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:45.787 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:45.809 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:45.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:45.810 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:45.892 INFO:tasks.rados.rados.0.smithi137.stdout:2053: finishing rollback tid 2 to smithi13779221-46 2024-09-21T08:31:45.892 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 845 (ObjNum 499 snap 140 seq_num 499) dirty exists 2024-09-21T08:31:45.892 INFO:tasks.rados.rados.0.smithi137.stdout:2056: finishing rollback tid 0 to smithi13779221-23 2024-09-21T08:31:45.892 INFO:tasks.rados.rados.0.smithi137.stdout:2056: finishing rollback tid 1 to smithi13779221-23 2024-09-21T08:31:45.892 INFO:tasks.rados.rados.0.smithi137.stdout:2056: finishing rollback tid 2 to smithi13779221-23 2024-09-21T08:31:45.892 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 223 (ObjNum 581 snap 171 seq_num 581) dirty exists 2024-09-21T08:31:45.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:45 smithi165 ceph-mon[32893]: pgmap v671: 188 pgs: 3 active+clean+snaptrim, 17 remapped+peering, 6 active+clean+remapped+snaptrim_wait, 45 active+clean+remapped, 3 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 649 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 24 op/s; 251/1566 objects degraded (16.028%); 109/1566 objects misplaced (6.960%); 2.2 MiB/s, 7 objects/s recovering 2024-09-21T08:31:45.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:45 smithi165 ceph-mon[32893]: osdmap e511: 8 total, 6 up, 5 in 2024-09-21T08:31:45.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:45 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:45.895 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:45 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:45.911 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:45.931 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:45.932 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:45.932 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:45 smithi137 ceph-mon[29634]: pgmap v671: 188 pgs: 3 active+clean+snaptrim, 17 remapped+peering, 6 active+clean+remapped+snaptrim_wait, 45 active+clean+remapped, 3 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 649 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 24 op/s; 251/1566 objects degraded (16.028%); 109/1566 objects misplaced (6.960%); 2.2 MiB/s, 7 objects/s recovering 2024-09-21T08:31:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:45 smithi137 ceph-mon[29634]: osdmap e511: 8 total, 6 up, 5 in 2024-09-21T08:31:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:45 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:46.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:45 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:46.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:45 smithi137 ceph-mon[41044]: pgmap v671: 188 pgs: 3 active+clean+snaptrim, 17 remapped+peering, 6 active+clean+remapped+snaptrim_wait, 45 active+clean+remapped, 3 active+clean+snaptrim_wait, 1 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 649 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 12 MiB/s rd, 2.0 MiB/s wr, 24 op/s; 251/1566 objects degraded (16.028%); 109/1566 objects misplaced (6.960%); 2.2 MiB/s, 7 objects/s recovering 2024-09-21T08:31:46.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:45 smithi137 ceph-mon[41044]: osdmap e511: 8 total, 6 up, 5 in 2024-09-21T08:31:46.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:45 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:31:46.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:45 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:31:46.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:46.054 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:46.055 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:46.055 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:46.156 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:46.162 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 podman[111154]: 2024-09-21 08:31:45.907658251 +0000 UTC m=+0.248861776 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T08:31:46.163 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 podman[111154]: 2024-09-21 08:31:46.030785404 +0000 UTC m=+0.371988958 container create a7d232782db56bfb5886bf43303cf05114742431756db59295b9d9a49e4ea99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=squid-9e2ac70, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T08:31:46.178 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:46.179 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:46.179 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:46.280 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:46.304 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:46.304 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:46.305 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDScrubReserve(3.f,QREQUEST,e:79,from:7,reservation_nonce:3)", 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 434.59332950200002, 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003249558, 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.402 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742332+0000", 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742334+0000", 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6809999999999998e-06 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742342+0000", 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6930000000000007e-06 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742355+0000", 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2556e-05 2024-09-21T08:31:46.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742366+0000", 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0994e-05 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742426+0000", 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0509e-05 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.742436+0000", 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8060000000000006e-06 2024-09-21T08:31:46.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745540+0000", 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031038630000000001 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745553+0000", 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2669e-05 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:31.745581+0000", 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8787000000000001e-05 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.1.0:1138 7.1e e472/465 7:78d05d33:::smithi13779221-41:81 v 472'316, mlcod=468'312)", 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:06.690512+0000", 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 39.645148880999997, 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056824299999999996, 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.407 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690512+0000", 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690512+0000", 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690513+0000", 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2000000000000001e-07 2024-09-21T08:31:46.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690523+0000", 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.064e-05 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690530+0000", 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3450000000000002e-06 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690538+0000", 2024-09-21T08:31:46.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8289999999999997e-06 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690718+0000", 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000179087 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690739+0000", 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1569999999999998e-05 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.696151+0000", 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054117999999999996 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.696195+0000", 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3838999999999998e-05 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:4848 7.28 e477/476 7:15ddb887:::smithi13779221-8:head v 477'387, mlcod=0'0)", 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.708762+0000", 2024-09-21T08:31:46.411 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.626899105, 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011385744999999999, 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.708762+0000", 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.708762+0000", 2024-09-21T08:31:46.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.708763+0000", 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4e-07 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.711674+0000", 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029111330000000002 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.711703+0000", 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8404000000000001e-05 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.711713+0000", 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0875e-05 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.712110+0000", 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039688700000000001 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.414 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.712153+0000", 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2273e-05 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720094+0000", 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0079410149999999992 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720148+0000", 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4218000000000002e-05 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.415 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:4856 7.9 e477/473 7:92b92e8a:::smithi13779221-35:head v 477'383, mlcod=0'0)", 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.713175+0000", 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.622486735999999, 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066575059999999997, 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.416 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713175+0000", 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713175+0000", 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713177+0000", 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4480000000000001e-06 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713187+0000", 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0399e-05 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713207+0000", 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0071000000000001e-05 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713221+0000", 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3508e-05 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713343+0000", 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012158 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713368+0000", 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5168999999999999e-05 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.719777+0000", 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064095189999999998 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.719832+0000", 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4812000000000003e-05 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:4857 7.9 e477/473 7:92b92e8a:::smithi13779221-35:head v 477'384, mlcod=0'0)", 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.714415+0000", 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.621246153999998, 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057016469999999998, 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.420 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714415+0000", 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714415+0000", 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714417+0000", 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1349999999999999e-06 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714427+0000", 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8800000000000003e-06 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714445+0000", 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7861999999999999e-05 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714454+0000", 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0289999999999994e-06 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714519+0000", 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4759999999999997e-05 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714545+0000", 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5819000000000001e-05 2024-09-21T08:31:46.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720079+0000", 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055343359999999999 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720117+0000", 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7826e-05 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.5.0:969 7.28 e477/476 7:17170b1f:::smithi13779221-4:b5 v 477'389, mlcod=0'0)", 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.714652+0000", 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.621009243000003, 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055407950000000003, 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714652+0000", 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.425 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714652+0000", 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714653+0000", 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4720000000000001e-06 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714662+0000", 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7639999999999994e-06 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714676+0000", 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3373000000000001e-05 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714681+0000", 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.772e-06 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714744+0000", 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2284000000000002e-05 2024-09-21T08:31:46.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714773+0000", 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8898999999999999e-05 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720154+0000", 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053813740000000004 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720193+0000", 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8856999999999997e-05 2024-09-21T08:31:46.428 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGRecoveryDelete(7.25 e477,476 7:a6c204bd:::smithi13779221-42:18,135'150)", 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.715829+0000", 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.619832670000001, 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041196369999999998, 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.715829+0000", 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.715829+0000", 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.715830+0000", 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8819999999999999e-06 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.715838+0000", 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.294e-06 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.715863+0000", 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5578000000000001e-05 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.715872+0000", 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0909999999999999e-06 2024-09-21T08:31:46.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.715942+0000", 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9905000000000002e-05 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.715952+0000", 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4930000000000004e-06 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.719948+0000", 2024-09-21T08:31:46.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039963940000000003 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.25 477/476 [PushOp(7:a6c204bd:::smithi13779221-42:7f, version: 334'288, data_included: [474063~922525,1650547~126051], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:a6c204bd:::smithi13779221-42:7f@334'288, size: 3479115, copy_subset: [(0, 3479115)], clone_subset: {}, snapset: cb=[]:{74: [72, 71, 69], 7f: [7f, 7e, 7d, 7c, 7a, 79, 78, 77, 76], a8: [a8, a7, a5, a4, a3, a2, a1, 9e, 9c, 9b, 9a, 99, 98, 94, 93, 92, 91, 90, 8c, 8b, 88, 86, 85, 84], ac: [ac, ab, aa, a9], cb: [cb, ca, c9, c8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1776598, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.732485+0000", 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.603176169000001, 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028355580000000002, 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.732485+0000", 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.732485+0000", 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.732486+0000", 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2619999999999999e-06 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.734394+0000", 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001907281 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.734414+0000", 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0506999999999999e-05 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.734429+0000", 2024-09-21T08:31:46.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5353e-05 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.734522+0000", 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2867000000000004e-05 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.734534+0000", 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1694e-05 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.735321+0000", 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00078659400000000005 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.25 477/476 [PushOp(7:a6c204bd:::smithi13779221-42:7f, version: 334'288, data_included: [1776598~777711,2765651~108412,3003820~162453], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:a6c204bd:::smithi13779221-42:7f@334'288, size: 3479115, copy_subset: [(0, 3479115)], clone_subset: {}, snapset: cb=[]:{74: [72, 71, 69], 7f: [7f, 7e, 7d, 7c, 7a, 79, 78, 77, 76], a8: [a8, a7, a5, a4, a3, a2, a1, 9e, 9c, 9b, 9a, 99, 98, 94, 93, 92, 91, 90, 8c, 8b, 88, 86, 85, 84], ac: [ac, ab, aa, a9], cb: [cb, ca, c9, c8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3166273, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1776598, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.736352+0000", 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.599309503000001, 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026247990000000001, 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.437 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.736352+0000", 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.736352+0000", 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.736352+0000", 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1200000000000002e-07 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.738153+0000", 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001800383 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.738175+0000", 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2235999999999999e-05 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.738200+0000", 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4859e-05 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.738305+0000", 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010494500000000001 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.738317+0000", 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1821000000000001e-05 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.738977+0000", 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00065984300000000002 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.25 477/476 [PushOp(7:a6c204bd:::smithi13779221-42:74, version: 337'292, data_included: [679115~634279,2090487~383101], data_size: 1017380, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:a6c204bd:::smithi13779221-42:74@337'292, size: 2874063, copy_subset: [(679115, 634279), (2090487, 463822)], clone_subset: {7:a6c204bd:::smithi13779221-42:7f: [(0, 679115), (1313394, 777093), (2554309, 319754)]}, snapset: cb=[]:{74: [72, 71, 69], 7f: [7f, 7e, 7d, 7c, 7a, 79, 78, 77, 76], a8: [a8, a7, a5, a4, a3, a2, a1, 9e, 9c, 9b, 9a, 99, 98, 94, 93, 92, 91, 90, 8c, 8b, 88, 86, 85, 84], ac: [ac, ab, aa, a9], cb: [cb, ca, c9, c8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2554309, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.744747+0000", 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 35.59091454, 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033966579999999999, 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.441 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.744747+0000", 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.744747+0000", 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.744752+0000", 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2769999999999996e-06 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.746827+0000", 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020751020000000001 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.746864+0000", 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7283000000000002e-05 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.746890+0000", 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6114000000000001e-05 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.747229+0000", 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033878899999999998 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.747280+0000", 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0727999999999999e-05 2024-09-21T08:31:46.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.748143+0000", 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00086336500000000005 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:4880 7.1b e478/465 7:dd97af7e:::smithi13779221-30:head v 478'785, mlcod=475'781)", 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:11.616312+0000", 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 34.719349571999999, 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068370100000000001, 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.445 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616312+0000", 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616312+0000", 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616314+0000", 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2900000000000001e-06 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616324+0000", 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0409e-05 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616339+0000", 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4309e-05 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616348+0000", 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1519999999999998e-06 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616417+0000", 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9021000000000002e-05 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616446+0000", 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9037999999999999e-05 2024-09-21T08:31:46.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.623098+0000", 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066520060000000002 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.623149+0000", 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0785000000000003e-05 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.449 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4893 7.25 7:a6c204bd:::smithi13779221-42:head [append 0~0] snapc ce=[ce,cc,cb,ca,c9,c8,c6,c5,c4,c2,c1,c0,bf,be,b9,b8,b3,b2,b0,af,a5,a4,9b,92,84] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:13.613471+0000", 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.722189931999999, 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032909150000000002, 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4893 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613471+0000", 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613471+0000", 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613473+0000", 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4330000000000001e-06 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613478+0000", 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0869999999999997e-06 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613494+0000", 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6055000000000001e-05 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613502+0000", 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8029999999999997e-06 2024-09-21T08:31:46.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613601+0000", 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8882000000000006e-05 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613700+0000", 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9147999999999993e-05 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613872+0000", 2024-09-21T08:31:46.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000171836 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.615328+0000", 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001456031 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.615347+0000", 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9289000000000001e-05 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.616548+0000", 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012012329999999999 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.616619+0000", 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1124000000000005e-05 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.616762+0000", 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014299399999999999 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.455 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4894 7.25 7:a6c204bd:::smithi13779221-42:head [rollback c0] snapc ce=[ce,cc,cb,ca,c9,c8,c6,c5,c4,c2,c1,c0,bf,be,b9,b8,b3,b2,b0,af,a5,a4,9b,92,84] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:13.613515+0000", 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 32.722146250000002, 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081910390000000007, 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4894 2024-09-21T08:31:46.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613515+0000", 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613515+0000", 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613516+0000", 2024-09-21T08:31:46.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3799999999999996e-07 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613519+0000", 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1920000000000001e-06 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613530+0000", 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1277000000000001e-05 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613549+0000", 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8819999999999999e-05 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.614509+0000", 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095967999999999997 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.614581+0000", 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2457000000000001e-05 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.614894+0000", 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031301000000000001 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.617391+0000", 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024968049999999999 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:46.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.618189+0000", 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079810899999999997 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.621564+0000", 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033748979999999999 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.621615+0000", 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0534000000000001e-05 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.621706+0000", 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1519000000000002e-05 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4957 7.25 7:a6c204bd:::smithi13779221-42:head [copy-from ver 381] snapc d2=[d2,d1,d0,cf,ce,cc,cb,ca,c9,c8,c5,c4,c2,c1,c0,be,b9,b8,b3,b2,b0,af,a5,9b,92,84] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:20.644293+0000", 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.691368304000001, 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027762614000000001, 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.462 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4957 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644293+0000", 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644293+0000", 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644295+0000", 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3949999999999999e-06 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644304+0000", 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6109999999999994e-06 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.464 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644315+0000", 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1477e-05 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644326+0000", 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0919e-05 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644658+0000", 2024-09-21T08:31:46.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000331742 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644790+0000", 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013230199999999999 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.660702+0000", 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015912017000000001 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:46.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.667882+0000", 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071797939999999998 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.667894+0000", 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1807000000000001e-05 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.671798+0000", 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003903672 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.671922+0000", 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012392499999999999 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.672056+0000", 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000133953 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4958 7.25 7:a6c204bd:::smithi13779221-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e487)", 2024-09-21T08:31:46.468 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:20.644340+0000", 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.69132119, 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020106951000000001, 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4958 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644340+0000", 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644340+0000", 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644341+0000", 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9100000000000003e-07 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644344+0000", 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2590000000000001e-06 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644365+0000", 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1206e-05 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644373+0000", 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6909999999999997e-06 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.645689+0000", 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001316273 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.645739+0000", 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9679999999999999e-05 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.664230+0000", 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018491408000000001 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.664312+0000", 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2021999999999995e-05 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.664447+0000", 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134721 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.473 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:4961 7.1b e487/465 7:dd97af7e:::smithi13779221-30:head v 487'803, mlcod=487'801)", 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:20.665019+0000", 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.670642356999998, 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029643529999999999, 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.665019+0000", 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.665019+0000", 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.665021+0000", 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.877e-06 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.665568+0000", 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00054739299999999997 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.665588+0000", 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0202000000000001e-05 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.665598+0000", 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4830000000000005e-06 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.665659+0000", 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1294e-05 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.665675+0000", 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5877000000000001e-05 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.667950+0000", 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022754479999999998 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.667983+0000", 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2778999999999999e-05 2024-09-21T08:31:46.477 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:5013 7.1e e490/465 7:78d05d33:::smithi13779221-41:head v 490'326, mlcod=476'323)", 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:23.683058+0000", 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 22.652603001999999, 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035899109999999999, 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:23.683058+0000", 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:23.683058+0000", 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:23.683059+0000", 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1099999999999996e-07 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:23.683638+0000", 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00057876100000000001 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:23.683652+0000", 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4296e-05 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:23.683662+0000", 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9229999999999997e-06 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:23.683779+0000", 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000116797 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:23.683806+0000", 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7532e-05 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:23.686591+0000", 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027847509999999998 2024-09-21T08:31:46.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:23.686648+0000", 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.7340000000000003e-05 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:5272 7.16 e512/507 7:6a4067f0:::smithi13779221-23:head v 512'221, mlcod=0'0)", 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:45.645636+0000", 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.69002518599999996, 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052462949999999998, 2024-09-21T08:31:46.482 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645636+0000", 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645636+0000", 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645636+0000", 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.89e-07 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645658+0000", 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1725e-05 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645669+0000", 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0783e-05 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645678+0000", 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6959999999999994e-06 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645752+0000", 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4153000000000006e-05 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645779+0000", 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6744999999999999e-05 2024-09-21T08:31:46.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.650826+0000", 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050476289999999997 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.650882+0000", 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6174999999999997e-05 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:5273 7.16 e512/507 7:6a4067f0:::smithi13779221-23:head v 512'222, mlcod=0'0)", 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:45.649444+0000", 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.68621750599999998, 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028853189999999999, 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649444+0000", 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649444+0000", 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649446+0000", 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.65e-06 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649458+0000", 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1647999999999999e-05 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649471+0000", 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3397999999999999e-05 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649478+0000", 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7340000000000002e-06 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.651345+0000", 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001867201 2024-09-21T08:31:46.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.651374+0000", 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8734e-05 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.652284+0000", 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00090961299999999998 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.652329+0000", 2024-09-21T08:31:46.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5340999999999998e-05 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:5274 7.16 e512/507 7:6a4067f0:::smithi13779221-23:head v 512'223, mlcod=0'0)", 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:45.649489+0000", 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.68617208699999999, 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029778249999999999, 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:46.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649489+0000", 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649489+0000", 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649490+0000", 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5499999999999998e-07 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649501+0000", 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1175000000000001e-05 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649504+0000", 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4300000000000002e-06 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.649507+0000", 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.712e-06 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.651990+0000", 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024834829999999999 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.652030+0000", 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9410999999999999e-05 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.652439+0000", 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00040926999999999999 2024-09-21T08:31:46.494 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:46.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:46.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:46.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.652467+0000", 2024-09-21T08:31:46.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7889e-05 2024-09-21T08:31:46.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.495 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.495 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:46.495 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:46.495 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:46.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:46.496 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 podman[111154]: 2024-09-21 08:31:46.300209512 +0000 UTC m=+0.641413036 container init a7d232782db56bfb5886bf43303cf05114742431756db59295b9d9a49e4ea99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T08:31:46.496 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 podman[111154]: 2024-09-21 08:31:46.305587584 +0000 UTC m=+0.646791107 container start a7d232782db56bfb5886bf43303cf05114742431756db59295b9d9a49e4ea99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-09-21T08:31:46.519 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:46.519 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:46.519 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:46.620 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:46.621 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:31:46.647 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:46.647 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:46.648 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:46.648 INFO:tasks.rados.rados.0.smithi137.stdout:2053: done (4 left) 2024-09-21T08:31:46.648 INFO:tasks.rados.rados.0.smithi137.stdout:2054: done (3 left) 2024-09-21T08:31:46.648 INFO:tasks.rados.rados.0.smithi137.stdout:2055: done (2 left) 2024-09-21T08:31:46.649 INFO:tasks.rados.rados.0.smithi137.stdout:2056: done (1 left) 2024-09-21T08:31:46.649 INFO:tasks.rados.rados.0.smithi137.stdout:2057: done (0 left) 2024-09-21T08:31:46.649 INFO:tasks.rados.rados.0.smithi137.stdout:2058: read oid 48 snap -1 2024-09-21T08:31:46.649 INFO:tasks.rados.rados.0.smithi137.stdout:2058: expect (ObjNum 670 snap 203 seq_num 670) 2024-09-21T08:31:46.649 INFO:tasks.rados.rados.0.smithi137.stdout:2059: write (excl) oid 5 current snap is 226 2024-09-21T08:31:46.649 INFO:tasks.rados.rados.0.smithi137.stdout:2059: seq_num 723 ranges {551461=658562,1941540=209921} 2024-09-21T08:31:46.655 INFO:tasks.rados.rados.0.smithi137.stdout:2059: writing smithi13779221-5 from 551461 to 1210023 tid 1 2024-09-21T08:31:46.661 INFO:tasks.rados.rados.0.smithi137.stdout:2059: writing smithi13779221-5 from 1941540 to 2151461 tid 2 2024-09-21T08:31:46.661 INFO:tasks.rados.rados.0.smithi137.stdout:2060: delete oid 41 current snap is 226 2024-09-21T08:31:46.661 INFO:tasks.rados.rados.0.smithi137.stdout:2058: expect (ObjNum 670 snap 203 seq_num 670) 2024-09-21T08:31:46.749 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:46 smithi165 ceph-mon[32893]: osdmap e512: 8 total, 6 up, 5 in 2024-09-21T08:31:46.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:46 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 251/1566 objects degraded (16.028%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:46.763 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate[111209]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T08:31:46.763 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate[111209]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T08:31:46.771 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:46.772 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:46.772 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:2058: done (2 left) 2024-09-21T08:31:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:2060: done (1 left) 2024-09-21T08:31:46.857 INFO:tasks.rados.rados.0.smithi137.stdout:2061: read oid 2 snap 173 2024-09-21T08:31:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:2061: expect deleted 2024-09-21T08:31:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:2062: copy_from oid 4 from oid 14 current snap is 226 2024-09-21T08:31:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:2059: finishing write tid 1 to smithi13779221-5 2024-09-21T08:31:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:2059: finishing write tid 2 to smithi13779221-5 2024-09-21T08:31:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:2059: finishing write tid 3 to smithi13779221-5 2024-09-21T08:31:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:2059: finishing write tid 4 to smithi13779221-5 2024-09-21T08:31:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:2059: oid 5 updating version 0 to 535 2024-09-21T08:31:46.858 INFO:tasks.rados.rados.0.smithi137.stdout:2059: oid 5 updating version 535 to 537 2024-09-21T08:31:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:2059: oid 5 version 537 is already newer than 536 2024-09-21T08:31:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 537 (ObjNum 723 snap 226 seq_num 723) dirty exists 2024-09-21T08:31:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:2059: left oid 5 (ObjNum 723 snap 226 seq_num 723) 2024-09-21T08:31:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:2059: done (2 left) 2024-09-21T08:31:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:2063: write (excl) oid 39 current snap is 226 2024-09-21T08:31:46.859 INFO:tasks.rados.rados.0.smithi137.stdout:2063: seq_num 725 ranges {585075=409060,1542151=642924} 2024-09-21T08:31:46.862 INFO:tasks.rados.rados.0.smithi137.stdout:2063: writing smithi13779221-39 from 585075 to 994135 tid 1 2024-09-21T08:31:46.869 INFO:tasks.rados.rados.0.smithi137.stdout:2063: writing smithi13779221-39 from 1542151 to 2185075 tid 2 2024-09-21T08:31:46.869 INFO:tasks.rados.rados.0.smithi137.stdout:2064: snap_remove snap 188 2024-09-21T08:31:46.873 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:46.889 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:46.900 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:46.900 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:46.901 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:47.001 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:47.009 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:46 smithi137 ceph-mon[29634]: osdmap e512: 8 total, 6 up, 5 in 2024-09-21T08:31:47.009 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:46 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 251/1566 objects degraded (16.028%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:47.009 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:46 smithi137 ceph-mon[41044]: osdmap e512: 8 total, 6 up, 5 in 2024-09-21T08:31:47.010 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:46 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 251/1566 objects degraded (16.028%), 35 pgs degraded, 54 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:47.024 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:47.024 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:47.025 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:47.099 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:31:47.125 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:47.164 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:47.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:47.165 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:47.263 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 podman[111154]: 2024-09-21 08:31:46.891212449 +0000 UTC m=+1.232415969 container attach a7d232782db56bfb5886bf43303cf05114742431756db59295b9d9a49e4ea99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2) 2024-09-21T08:31:47.264 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 bash[111154]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T08:31:47.264 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 bash[111154]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T08:31:47.264 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate[111209]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T08:31:47.264 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 bash[111154]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T08:31:47.264 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate[111209]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T08:31:47.264 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:46 smithi165 bash[111154]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-7 --no-mon-config --dev /dev/mapper/vg_nvme-lv_1 2024-09-21T08:31:47.265 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:47.289 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:47.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:47.290 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:47.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:47.394 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:47.395 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T08:31:47.413 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:47.413 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:47.416 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:47.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:47.417 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:47.432 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:47.433 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:47.450 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:47.450 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:47.457 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:47.467 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:47.467 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:47.484 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:47.484 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:47.501 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:47.501 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:47.501 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:47.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:47.540 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:47.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:47.541 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:47.640 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate[111209]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T08:31:47.640 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 bash[111154]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_1 2024-09-21T08:31:47.640 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate[111209]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T08:31:47.641 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 bash[111154]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-0 2024-09-21T08:31:47.641 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate[111209]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-21T08:31:47.641 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 bash[111154]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_1 /var/lib/ceph/osd/ceph-7/block 2024-09-21T08:31:47.641 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate[111209]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T08:31:47.641 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 bash[111154]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-09-21T08:31:47.641 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate[111209]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-21T08:31:47.641 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 bash[111154]: --> ceph-volume raw activate successful for osd ID: 7 2024-09-21T08:31:47.641 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 podman[111154]: 2024-09-21 08:31:47.382114318 +0000 UTC m=+1.723317843 container died a7d232782db56bfb5886bf43303cf05114742431756db59295b9d9a49e4ea99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate, org.label-schema.build-date=20240731, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70) 2024-09-21T08:31:47.641 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:47.652 INFO:tasks.rados.rados.0.smithi137.stdout:2064: done (3 left) 2024-09-21T08:31:47.652 INFO:tasks.rados.rados.0.smithi137.stdout:2065: rollback oid 42 current snap is 226 2024-09-21T08:31:47.652 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 42 to 173 2024-09-21T08:31:47.652 INFO:tasks.rados.rados.0.smithi137.stdout:2066: snap_remove snap 225 2024-09-21T08:31:47.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:47.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:47.670 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:47.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:47.793 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:47.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:47.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:47.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:47.916 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:47.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:47.917 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:47.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:47 smithi165 ceph-mon[32893]: pgmap v674: 188 pgs: 1 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 56 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 56 active+clean; 659 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 19 op/s; 254/1590 objects degraded (15.975%); 135/1590 objects misplaced (8.491%) 2024-09-21T08:31:47.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:47 smithi165 ceph-mon[32893]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T08:31:47.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:47 smithi165 ceph-mon[32893]: osdmap e513: 8 total, 6 up, 5 in 2024-09-21T08:31:47.978 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:47 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3439288467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:48.006 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:47 smithi137 ceph-mon[29634]: pgmap v674: 188 pgs: 1 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 56 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 56 active+clean; 659 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 19 op/s; 254/1590 objects degraded (15.975%); 135/1590 objects misplaced (8.491%) 2024-09-21T08:31:48.007 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:47 smithi137 ceph-mon[29634]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T08:31:48.007 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:47 smithi137 ceph-mon[29634]: osdmap e513: 8 total, 6 up, 5 in 2024-09-21T08:31:48.007 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:47 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3439288467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:48.007 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:47 smithi137 ceph-mon[41044]: pgmap v674: 188 pgs: 1 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 56 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 56 active+clean; 659 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 19 op/s; 254/1590 objects degraded (15.975%); 135/1590 objects misplaced (8.491%) 2024-09-21T08:31:48.007 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:47 smithi137 ceph-mon[41044]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-09-21T08:31:48.007 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:47 smithi137 ceph-mon[41044]: osdmap e513: 8 total, 6 up, 5 in 2024-09-21T08:31:48.007 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:47 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3439288467' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:48.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:48.042 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:48.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:48.043 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:48.085 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:48.144 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:48.166 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:48.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:48.167 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:48.263 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:47 smithi165 podman[111431]: 2024-09-21 08:31:47.977824033 +0000 UTC m=+0.664325157 container remove a7d232782db56bfb5886bf43303cf05114742431756db59295b9d9a49e4ea99e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7-activate, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.37.2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD) 2024-09-21T08:31:48.263 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:48 smithi165 podman[111619]: 2024-09-21 08:31:48.187105082 +0000 UTC m=+0.110633202 container create ef3665316fbcc30781db1ae9fe2abd7e91aa93b1c341796f11f808231d41161f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7, org.label-schema.build-date=20240731, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.37.2) 2024-09-21T08:31:48.263 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:48 smithi165 podman[111619]: 2024-09-21 08:31:48.093964078 +0000 UTC m=+0.017492198 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T08:31:48.268 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:48.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:48.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:48.290 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:48.391 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:48.411 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:48.412 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:48.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:48.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:48.414 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:48.470 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T08:31:48.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:48.536 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:48.537 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:48.537 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:48.638 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:48.652 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:48 smithi165 podman[111619]: 2024-09-21 08:31:48.381339624 +0000 UTC m=+0.304867751 container init ef3665316fbcc30781db1ae9fe2abd7e91aa93b1c341796f11f808231d41161f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, GIT_CLEAN=True, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, ceph=True) 2024-09-21T08:31:48.653 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:48 smithi165 podman[111619]: 2024-09-21 08:31:48.385994524 +0000 UTC m=+0.309522645 container start ef3665316fbcc30781db1ae9fe2abd7e91aa93b1c341796f11f808231d41161f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-09-21T08:31:48.653 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:48 smithi165 ceph-osd[111672]: -- 172.21.15.165:0/1736434645 <== mon.1 v1:172.21.15.165:6789/0 6 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (unknown 1363117908 0 0) 0x55a715ce7a40 con 0x55a715e9e000 2024-09-21T08:31:48.653 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:48 smithi165 bash[111619]: ef3665316fbcc30781db1ae9fe2abd7e91aa93b1c341796f11f808231d41161f 2024-09-21T08:31:48.653 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:48 smithi165 systemd[1]: Started Ceph osd.7 for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:31:48.659 INFO:tasks.rados.rados.0.smithi137.stdout:2066: done (4 left) 2024-09-21T08:31:48.659 INFO:tasks.rados.rados.0.smithi137.stdout:2067: snap_remove snap 217 2024-09-21T08:31:48.665 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:48.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:48.666 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:48.767 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:48.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:48.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:48.789 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:48.890 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:48.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:48.912 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:48.912 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:48.913 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:48.914 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:48.914 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:48 smithi165 ceph-mon[32893]: osdmap e514: 8 total, 6 up, 5 in 2024-09-21T08:31:49.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:49.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:48 smithi137 ceph-mon[29634]: osdmap e514: 8 total, 6 up, 5 in 2024-09-21T08:31:49.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:48 smithi137 ceph-mon[41044]: osdmap e514: 8 total, 6 up, 5 in 2024-09-21T08:31:49.037 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:49.037 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:49.038 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:49.138 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:49.160 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:49.161 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:49.161 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:49.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:49.284 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:49.285 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:49.285 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:49.386 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:49.407 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:49.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:49.409 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:49.509 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:49.532 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:49.532 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:49.533 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:49.543 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T08:31:49.558 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:49.634 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:49.654 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:31:49.658 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:49.658 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:49.659 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:2067: done (4 left) 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:2068: copy_from oid 31 from oid 37 current snap is 226 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:2062: finishing copy_from racing read to smithi13779221-4 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:2062: finishing copy_from to smithi13779221-4 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 400 (ObjNum 619 snap 182 seq_num 619) dirty exists 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:2063: finishing write tid 1 to smithi13779221-39 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:2063: finishing write tid 2 to smithi13779221-39 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:2063: finishing write tid 3 to smithi13779221-39 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:2063: finishing write tid 4 to smithi13779221-39 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:2063: oid 39 updating version 0 to 243 2024-09-21T08:31:49.664 INFO:tasks.rados.rados.0.smithi137.stdout:2063: oid 39 updating version 243 to 244 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:2063: oid 39 version 244 is already newer than 242 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 244 (ObjNum 725 snap 226 seq_num 725) dirty exists 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:2063: left oid 39 (ObjNum 725 snap 226 seq_num 725) 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:2065: finishing rollback tid 0 to smithi13779221-42 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:2065: finishing rollback tid 1 to smithi13779221-42 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 701 (ObjNum 65 snap 0 seq_num 3731215952) dirty dne 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:2061: done (4 left) 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:2062: done (3 left) 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:2063: done (2 left) 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:2065: done (1 left) 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:2069: write oid 45 current snap is 226 2024-09-21T08:31:49.665 INFO:tasks.rados.rados.0.smithi137.stdout:2069: seq_num 726 ranges {601882=484309,1600633=524904} 2024-09-21T08:31:49.666 INFO:tasks.rados.rados.0.smithi137.stdout:2069: writing smithi13779221-45 from 601882 to 1086191 tid 1 2024-09-21T08:31:49.673 INFO:tasks.rados.rados.0.smithi137.stdout:2069: writing smithi13779221-45 from 1600633 to 2125537 tid 2 2024-09-21T08:31:49.673 INFO:tasks.rados.rados.0.smithi137.stdout:2070: read oid 36 snap -1 2024-09-21T08:31:49.673 INFO:tasks.rados.rados.0.smithi137.stdout:2070: expect (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:31:49.673 INFO:tasks.rados.rados.0.smithi137.stdout:2071: delete oid 29 current snap is 226 2024-09-21T08:31:49.673 INFO:tasks.rados.rados.0.smithi137.stdout:2069: finishing write tid 1 to smithi13779221-45 2024-09-21T08:31:49.673 INFO:tasks.rados.rados.0.smithi137.stdout:2068: finishing copy_from racing read to smithi13779221-31 2024-09-21T08:31:49.677 INFO:tasks.rados.rados.0.smithi137.stdout:2068: finishing copy_from to smithi13779221-31 2024-09-21T08:31:49.677 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 510 (ObjNum 618 snap 182 seq_num 618) dirty exists 2024-09-21T08:31:49.679 INFO:tasks.rados.rados.0.smithi137.stdout:2069: finishing write tid 2 to smithi13779221-45 2024-09-21T08:31:49.679 INFO:tasks.rados.rados.0.smithi137.stdout:2069: finishing write tid 3 to smithi13779221-45 2024-09-21T08:31:49.680 INFO:tasks.rados.rados.0.smithi137.stdout:2069: finishing write tid 4 to smithi13779221-45 2024-09-21T08:31:49.680 INFO:tasks.rados.rados.0.smithi137.stdout:2069: oid 45 updating version 0 to 453 2024-09-21T08:31:49.680 INFO:tasks.rados.rados.0.smithi137.stdout:2069: oid 45 version 453 is already newer than 452 2024-09-21T08:31:49.680 INFO:tasks.rados.rados.0.smithi137.stdout:2069: oid 45 updating version 453 to 454 2024-09-21T08:31:49.680 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 454 (ObjNum 726 snap 226 seq_num 726) dirty exists 2024-09-21T08:31:49.680 INFO:tasks.rados.rados.0.smithi137.stdout:2069: left oid 45 (ObjNum 726 snap 226 seq_num 726) 2024-09-21T08:31:49.682 INFO:tasks.rados.rados.0.smithi137.stdout:2068: done (3 left) 2024-09-21T08:31:49.682 INFO:tasks.rados.rados.0.smithi137.stdout:2069: done (2 left) 2024-09-21T08:31:49.682 INFO:tasks.rados.rados.0.smithi137.stdout:2071: done (1 left) 2024-09-21T08:31:49.682 INFO:tasks.rados.rados.0.smithi137.stdout:2072: snap_create 2024-09-21T08:31:49.690 INFO:tasks.rados.rados.0.smithi137.stdout:2070: expect (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:31:49.760 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:49.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:49 smithi165 ceph-mon[32893]: pgmap v677: 188 pgs: 1 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 56 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 56 active+clean; 659 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 19 op/s; 254/1590 objects degraded (15.975%); 135/1590 objects misplaced (8.491%) 2024-09-21T08:31:49.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:49 smithi165 ceph-mon[32893]: osdmap e515: 8 total, 6 up, 5 in 2024-09-21T08:31:49.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:49 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3451939030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:49.784 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:49.784 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:49.785 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:49.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:49.907 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:49.907 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:49.908 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:49.987 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:49 smithi137 ceph-mon[29634]: pgmap v677: 188 pgs: 1 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 56 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 56 active+clean; 659 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 19 op/s; 254/1590 objects degraded (15.975%); 135/1590 objects misplaced (8.491%) 2024-09-21T08:31:49.988 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:49 smithi137 ceph-mon[29634]: osdmap e515: 8 total, 6 up, 5 in 2024-09-21T08:31:49.988 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:49 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3451939030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:49.988 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:49 smithi137 ceph-mon[41044]: pgmap v677: 188 pgs: 1 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 56 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 56 active+clean; 659 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 14 MiB/s rd, 2.4 MiB/s wr, 19 op/s; 254/1590 objects degraded (15.975%); 135/1590 objects misplaced (8.491%) 2024-09-21T08:31:49.988 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:49 smithi137 ceph-mon[41044]: osdmap e515: 8 total, 6 up, 5 in 2024-09-21T08:31:49.988 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:49 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3451939030' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:31:50.009 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:50.030 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:50.030 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:50.031 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:50.039 INFO:teuthology.orchestra.run.smithi165.stderr:no valid command found; 10 closest matches: 2024-09-21T08:31:50.039 INFO:teuthology.orchestra.run.smithi165.stderr:0 2024-09-21T08:31:50.039 INFO:teuthology.orchestra.run.smithi165.stderr:1 2024-09-21T08:31:50.039 INFO:teuthology.orchestra.run.smithi165.stderr:2 2024-09-21T08:31:50.039 INFO:teuthology.orchestra.run.smithi165.stderr:abort 2024-09-21T08:31:50.039 INFO:teuthology.orchestra.run.smithi165.stderr:assert 2024-09-21T08:31:50.039 INFO:teuthology.orchestra.run.smithi165.stderr:config diff 2024-09-21T08:31:50.040 INFO:teuthology.orchestra.run.smithi165.stderr:config diff get 2024-09-21T08:31:50.040 INFO:teuthology.orchestra.run.smithi165.stderr:config get 2024-09-21T08:31:50.040 INFO:teuthology.orchestra.run.smithi165.stderr:config help [] 2024-09-21T08:31:50.040 INFO:teuthology.orchestra.run.smithi165.stderr:config set ... 2024-09-21T08:31:50.040 INFO:teuthology.orchestra.run.smithi165.stderr:admin_socket: invalid command 2024-09-21T08:31:50.131 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:50.154 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:50.154 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:50.155 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:50.255 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:50.263 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:50 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7[111647]: 2024-09-21T08:31:50.026+0000 7f01eeffa640 -1 Falling back to public interface 2024-09-21T08:31:50.277 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:50.277 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:50.277 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:50.378 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:50.401 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:50.401 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:50.401 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:50.413 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:31:50.502 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:50.524 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:50.525 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:50.525 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:50.626 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:50.647 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:50.648 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:50.648 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:50.655 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:50.672 INFO:tasks.rados.rados.0.smithi137.stdout:2070: done (1 left) 2024-09-21T08:31:50.672 INFO:tasks.rados.rados.0.smithi137.stdout:2072: done (0 left) 2024-09-21T08:31:50.672 INFO:tasks.rados.rados.0.smithi137.stdout:2073: snap_remove snap 130 2024-09-21T08:31:50.749 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:50.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:50 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:50.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:50 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3451939030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:31:50.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:50 smithi165 ceph-mon[32893]: osdmap e516: 8 total, 6 up, 5 in 2024-09-21T08:31:50.770 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:50.771 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:50.771 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:50.852 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:50.872 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:50.884 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T08:31:50.884 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-7, ['dump_ops_in_flight'] 2024-09-21T08:31:50.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:50.894 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:50.894 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:50.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:50 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:50.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:50 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3451939030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:31:50.925 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:50 smithi137 ceph-mon[29634]: osdmap e516: 8 total, 6 up, 5 in 2024-09-21T08:31:50.926 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:31:50 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:31:50] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:31:50.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:50 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:50.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:50 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3451939030' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:31:50.926 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:50 smithi137 ceph-mon[41044]: osdmap e516: 8 total, 6 up, 5 in 2024-09-21T08:31:50.996 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:51.018 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:51.018 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:51.019 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:51.120 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:51.141 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:51.144 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:51.144 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:51.145 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:51.158 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:51.158 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:51.174 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:51.175 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:51.190 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:51.190 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:51.207 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:51.207 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:51.223 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:51.223 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:51.239 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:51.240 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:51.240 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:51.245 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:51.267 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:51.267 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:51.267 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:51.368 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:51.383 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:51.383 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:31:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:31:51.384 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:31:51.384 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:51.389 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:51.390 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:51.390 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:51.491 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:51.512 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:51.513 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:51.513 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:51.603 INFO:tasks.rados.rados.0.smithi137.stdout:2073: done (0 left) 2024-09-21T08:31:51.603 INFO:tasks.rados.rados.0.smithi137.stdout:2074: snap_create 2024-09-21T08:31:51.614 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:51.637 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:51.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:51.638 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:51.739 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:51.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:51.762 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:51.762 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:51.863 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:51.885 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:51.885 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:51.885 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:51.981 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[29634]: 5.1f scrub starts 2024-09-21T08:31:51.981 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[29634]: 5.1f scrub ok 2024-09-21T08:31:51.981 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[29634]: pgmap v680: 188 pgs: 1 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 56 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 56 active+clean; 659 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 254/1590 objects degraded (15.975%); 135/1590 objects misplaced (8.491%) 2024-09-21T08:31:51.981 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[29634]: osdmap e517: 8 total, 6 up, 5 in 2024-09-21T08:31:51.981 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3265572880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:51.982 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 254/1590 objects degraded (15.975%), 35 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:51.982 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[29634]: osdmap e518: 8 total, 6 up, 5 in 2024-09-21T08:31:51.982 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[41044]: 5.1f scrub starts 2024-09-21T08:31:51.982 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[41044]: 5.1f scrub ok 2024-09-21T08:31:51.982 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[41044]: pgmap v680: 188 pgs: 1 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 56 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 56 active+clean; 659 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 254/1590 objects degraded (15.975%); 135/1590 objects misplaced (8.491%) 2024-09-21T08:31:51.982 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[41044]: osdmap e517: 8 total, 6 up, 5 in 2024-09-21T08:31:51.982 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3265572880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:51.982 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 254/1590 objects degraded (15.975%), 35 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:51.982 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:51 smithi137 ceph-mon[41044]: osdmap e518: 8 total, 6 up, 5 in 2024-09-21T08:31:51.986 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:51.987 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:51 smithi165 ceph-mon[32893]: 5.1f scrub starts 2024-09-21T08:31:51.987 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:51 smithi165 ceph-mon[32893]: 5.1f scrub ok 2024-09-21T08:31:51.987 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:51 smithi165 ceph-mon[32893]: pgmap v680: 188 pgs: 1 active+clean+snaptrim, 8 active+clean+remapped+snaptrim_wait, 56 active+clean+remapped, 4 active+clean+snaptrim_wait, 3 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 56 active+clean; 659 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 254/1590 objects degraded (15.975%); 135/1590 objects misplaced (8.491%) 2024-09-21T08:31:51.987 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:51 smithi165 ceph-mon[32893]: osdmap e517: 8 total, 6 up, 5 in 2024-09-21T08:31:51.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:51 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3265572880' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:51.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:51 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 254/1590 objects degraded (15.975%), 35 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:51.988 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:51 smithi165 ceph-mon[32893]: osdmap e518: 8 total, 6 up, 5 in 2024-09-21T08:31:52.007 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:52.008 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:52.008 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:52.009 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:52.009 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:52.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T08:31:52.109 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:52.130 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:52.131 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:52.131 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:52.232 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:52.253 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:52.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:52.254 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:52.259 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:52.263 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:31:52 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:31:52.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:52.365 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:31:52.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:52.377 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:52.377 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:52.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:52.500 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:52.501 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:52.501 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:52.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:52.607 INFO:tasks.rados.rados.0.smithi137.stdout:2074: done (0 left) 2024-09-21T08:31:52.607 INFO:tasks.rados.rados.0.smithi137.stdout:2075: write oid 7 current snap is 228 2024-09-21T08:31:52.607 INFO:tasks.rados.rados.0.smithi137.stdout:2075: seq_num 728 ranges {635496=634807,1717597=517899} 2024-09-21T08:31:52.613 INFO:tasks.rados.rados.0.smithi137.stdout:2075: writing smithi13779221-7 from 635496 to 1270303 tid 1 2024-09-21T08:31:52.620 INFO:tasks.rados.rados.0.smithi137.stdout:2075: writing smithi13779221-7 from 1717597 to 2235496 tid 2 2024-09-21T08:31:52.621 INFO:tasks.rados.rados.0.smithi137.stdout:2076: read oid 26 snap -1 2024-09-21T08:31:52.621 INFO:tasks.rados.rados.0.smithi137.stdout:2076: expect (ObjNum 716 snap 222 seq_num 716) 2024-09-21T08:31:52.621 INFO:tasks.rados.rados.0.smithi137.stdout:2077: snap_create 2024-09-21T08:31:52.622 INFO:tasks.rados.rados.0.smithi137.stdout:2075: finishing write tid 1 to smithi13779221-7 2024-09-21T08:31:52.625 INFO:tasks.rados.rados.0.smithi137.stdout:2075: finishing write tid 2 to smithi13779221-7 2024-09-21T08:31:52.625 INFO:tasks.rados.rados.0.smithi137.stdout:2076: expect (ObjNum 716 snap 222 seq_num 716) 2024-09-21T08:31:52.631 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:52.635 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:52.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:52.636 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:52.697 INFO:tasks.rados.rados.0.smithi137.stdout:2075: finishing write tid 3 to smithi13779221-7 2024-09-21T08:31:52.697 INFO:tasks.rados.rados.0.smithi137.stdout:2075: finishing write tid 4 to smithi13779221-7 2024-09-21T08:31:52.697 INFO:tasks.rados.rados.0.smithi137.stdout:2075: oid 7 updating version 0 to 852 2024-09-21T08:31:52.697 INFO:tasks.rados.rados.0.smithi137.stdout:2075: oid 7 updating version 852 to 853 2024-09-21T08:31:52.697 INFO:tasks.rados.rados.0.smithi137.stdout:2075: oid 7 version 853 is already newer than 851 2024-09-21T08:31:52.697 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 7 v 853 (ObjNum 728 snap 228 seq_num 728) dirty exists 2024-09-21T08:31:52.697 INFO:tasks.rados.rados.0.smithi137.stdout:2075: left oid 7 (ObjNum 728 snap 228 seq_num 728) 2024-09-21T08:31:52.726 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:52.726 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:31:52.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:31:52.726 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:31:52.726 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.726 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:1083 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94837804037760 gen 1] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e84)", 2024-09-21T08:31:52.726 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:31:52.726 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 434.56386093499998, 2024-09-21T08:31:52.726 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.395092834, 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1083 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.727 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159580+0000", 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159586+0000", 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.48e-06 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159599+0000", 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3699e-05 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.728 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159630+0000", 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0629000000000001e-05 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159647+0000", 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7030000000000001e-05 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159767+0000", 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000119571 2024-09-21T08:31:52.729 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.159781+0000", 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3844e-05 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.549248+0000", 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.38946733 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.730 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for degraded object", 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.549559+0000", 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000311057 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552031+0000", 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024720720000000001 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.552119+0000", 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8261000000000001e-05 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.731 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:24:38.554673+0000", 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0025538610000000002 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1159 7.b 7:df2b42d9:::smithi13779221-36:head [write 642067~696188,stat] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 361.833322555, 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.62427366799999995, 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.732 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1159 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.733 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890119+0000", 2024-09-21T08:31:52.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.734 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890120+0000", 2024-09-21T08:31:52.734 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6950000000000001e-06 2024-09-21T08:31:52.734 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.734 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.734 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890849+0000", 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072891299999999998 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890861+0000", 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1899000000000001e-05 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890868+0000", 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2110000000000001e-06 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.735 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890947+0000", 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8598999999999997e-05 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.890954+0000", 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1180000000000003e-06 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508089+0000", 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61713446299999997 2024-09-21T08:31:52.736 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508263+0000", 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174671 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.508366+0000", 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000102786 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511111+0000", 2024-09-21T08:31:52.737 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0027452710000000001 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.513304+0000", 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0021930560000000001 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514317+0000", 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0010129959999999999 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514355+0000", 2024-09-21T08:31:52.738 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7404000000000003e-05 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514392+0000", 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7586000000000002e-05 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1160 7.b 7:df2b42d9:::smithi13779221-36:head [write 2025476~459397] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 361.82687466599998, 2024-09-21T08:31:52.739 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61820804500000004, 2024-09-21T08:31:52.740 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.740 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.740 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.740 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:52.741 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.741 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.741 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.741 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1160 2024-09-21T08:31:52.741 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.741 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896567+0000", 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.742 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896568+0000", 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2610000000000001e-06 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896958+0000", 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039041199999999997 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896984+0000", 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6115e-05 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.743 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.896989+0000", 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0379999999999999e-06 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.897069+0000", 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9196000000000005e-05 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.897075+0000", 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.28e-06 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.744 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509576+0000", 2024-09-21T08:31:52.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61250132700000004 2024-09-21T08:31:52.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509636+0000", 2024-09-21T08:31:52.745 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9753999999999997e-05 2024-09-21T08:31:52.745 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.745 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.745 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:31:52.745 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.509725+0000", 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9524000000000005e-05 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511333+0000", 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001607619 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.513767+0000", 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024336819999999999 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.746 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514703+0000", 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00093633599999999996 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514736+0000", 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2438999999999999e-05 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514775+0000", 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9062e-05 2024-09-21T08:31:52.747 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1161 7.b 7:df2b42d9:::smithi13779221-36:head [write 2971091~70976] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 361.82399574099998, 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61638660599999995, 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.748 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1161 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.749 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899445+0000", 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899446+0000", 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0800000000000004e-07 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899537+0000", 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0414999999999994e-05 2024-09-21T08:31:52.750 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899544+0000", 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8580000000000002e-06 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899547+0000", 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8970000000000001e-06 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.751 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899597+0000", 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9058999999999999e-05 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899605+0000", 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3990000000000004e-06 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510251+0000", 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61064600599999996 2024-09-21T08:31:52.752 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510324+0000", 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2961999999999997e-05 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510438+0000", 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000113793 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.753 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511337+0000", 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089906399999999998 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514615+0000", 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00327853 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.754 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515770+0000", 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001154715 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515802+0000", 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1956999999999999e-05 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515832+0000", 2024-09-21T08:31:52.755 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0142999999999999e-05 2024-09-21T08:31:52.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.756 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.756 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.756 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.756 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.756 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1162 7.b 7:df2b42d9:::smithi13779221-36:head [setxattr _header (57) in=7b,truncate 3042067] snapc 25=[25] ondisk+write+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:31:52.756 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:31:52.756 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 361.823886673, 2024-09-21T08:31:52.756 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61641711799999999, 2024-09-21T08:31:52.756 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.757 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.757 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.757 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.757 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1162 2024-09-21T08:31:52.757 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.757 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.757 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.757 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899555+0000", 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9100000000000004e-07 2024-09-21T08:31:52.758 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899557+0000", 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3989999999999998e-06 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899561+0000", 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9349999999999996e-06 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899564+0000", 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7700000000000002e-06 2024-09-21T08:31:52.759 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899633+0000", 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9268000000000002e-05 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899637+0000", 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9650000000000002e-06 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510758+0000", 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61112027999999996 2024-09-21T08:31:52.760 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510819+0000", 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0866999999999998e-05 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.510929+0000", 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000110914 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511340+0000", 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041042100000000001 2024-09-21T08:31:52.761 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.514641+0000", 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003301511 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515865+0000", 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0012240319999999999 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515898+0000", 2024-09-21T08:31:52.762 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.3036e-05 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.515972+0000", 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3229e-05 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:1163 7.b 7:df2b42d9:::smithi13779221-36:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e157)", 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 361.82385526000002, 2024-09-21T08:31:52.763 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61652938899999998, 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 1163 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.764 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899586+0000", 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8599999999999998e-07 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899589+0000", 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1299999999999999e-06 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.765 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899592+0000", 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9190000000000001e-06 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899595+0000", 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9639999999999999e-06 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899654+0000", 2024-09-21T08:31:52.766 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.8958e-05 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:50.899658+0000", 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2669999999999998e-06 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511140+0000", 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.61148218399999998 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:31:52.767 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.511235+0000", 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4927999999999996e-05 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516004+0000", 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047691640000000002 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516060+0000", 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5726999999999997e-05 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.768 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:25:51.516115+0000", 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5661999999999997e-05 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2116 7.b 7:df2b42d9:::smithi13779221-36:head [copy-from ver 318] snapc 4a=[4a,49,48,47,45,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e228)", 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 291.20021623399998, 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.030972873000000001, 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.769 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2116 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.770 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523225+0000", 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523228+0000", 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6469999999999998e-06 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523237+0000", 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4590000000000004e-06 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.771 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523251+0000", 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3856000000000001e-05 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523260+0000", 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9500000000000007e-06 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523351+0000", 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0787999999999999e-05 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:31:52.772 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.523359+0000", 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5860000000000005e-06 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.526688+0000", 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033287339999999999 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.526800+0000", 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000111566 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.773 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 0,2", 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.531948+0000", 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0051486199999999996 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.538302+0000", 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063539729999999997 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.553780+0000", 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015477451 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.774 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554118+0000", 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000338549 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554160+0000", 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2218000000000001e-05 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:01.554198+0000", 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7475999999999998e-05 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.775 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2577 7.1c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 174] snapc 59=[59,56] ondisk+write+known_if_redirected+supports_pool_eio e264)", 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 255.16069003699999, 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0145323420000001, 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2577 2024-09-21T08:31:52.776 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562751+0000", 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.777 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562753+0000", 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.745e-06 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562759+0000", 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1330000000000004e-06 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562770+0000", 2024-09-21T08:31:52.778 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0617e-05 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562775+0000", 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4500000000000003e-06 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562831+0000", 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6233999999999997e-05 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.779 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562838+0000", 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7379999999999997e-06 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552540+0000", 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.98970149799999996 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552665+0000", 2024-09-21T08:31:52.780 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012560800000000001 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.562825+0000", 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010159916 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.569448+0000", 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066225670000000002 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.781 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577182+0000", 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0077346680000000001 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577203+0000", 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0619999999999999e-05 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577240+0000", 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6613999999999999e-05 2024-09-21T08:31:52.782 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.577284+0000", 2024-09-21T08:31:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3934000000000003e-05 2024-09-21T08:31:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.783 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:2578 7.1c 7:3cae00e7:::smithi13779221-28:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e264)", 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 255.16064416, 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0041372150000001, 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2578 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.784 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562797+0000", 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562798+0000", 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5000000000000003e-07 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.785 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562800+0000", 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2460000000000002e-06 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562806+0000", 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6620000000000002e-06 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562809+0000", 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8500000000000004e-06 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.786 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562869+0000", 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9892999999999997e-05 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:37.562879+0000", 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0054999999999999e-05 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552788+0000", 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.989908287 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.787 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:31:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.552833+0000", 2024-09-21T08:31:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5427999999999999e-05 2024-09-21T08:31:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566793+0000", 2024-09-21T08:31:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.013959875 2024-09-21T08:31:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.788 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566858+0000", 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4870999999999996e-05 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:27:38.566934+0000", 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6497999999999998e-05 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3198 7.8 7:15ddb887:::smithi13779221-8:head [append 0~0] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:31:52.789 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.713165+0000", 2024-09-21T08:31:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 196.010276519, 2024-09-21T08:31:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85552278699999995, 2024-09-21T08:31:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3198 2024-09-21T08:31:52.790 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713165+0000", 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713165+0000", 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.791 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713169+0000", 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.177e-06 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713182+0000", 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3593e-05 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.792 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713212+0000", 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9292999999999999e-05 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713227+0000", 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5630000000000001e-05 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713409+0000", 2024-09-21T08:31:52.793 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018163700000000001 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713424+0000", 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5366000000000002e-05 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.562792+0000", 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.84936760099999997 2024-09-21T08:31:52.794 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.563075+0000", 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000282945 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.563308+0000", 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023297899999999999 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.795 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566597+0000", 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003289541 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.567690+0000", 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001092906 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568570+0000", 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087925999999999998 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.796 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568631+0000", 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1511000000000001e-05 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568687+0000", 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6348000000000003e-05 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3199 7.8 7:15ddb887:::smithi13779221-8:head [rollback 69] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:31:52.797 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.713255+0000", 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 196.010186086, 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85558638099999995, 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3199 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713255+0000", 2024-09-21T08:31:52.798 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713255+0000", 2024-09-21T08:31:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.799 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713257+0000", 2024-09-21T08:31:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5260000000000001e-06 2024-09-21T08:31:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.800 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713264+0000", 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2599999999999999e-06 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713287+0000", 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3470000000000001e-05 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713302+0000", 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4195999999999999e-05 2024-09-21T08:31:52.801 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713470+0000", 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016798400000000001 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713478+0000", 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3059999999999998e-06 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.564578+0000", 2024-09-21T08:31:52.802 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85110020600000003 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.564669+0000", 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1321000000000002e-05 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.564879+0000", 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00020947899999999999 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566604+0000", 2024-09-21T08:31:52.803 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0017247409999999999 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.567857+0000", 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001253852 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568733+0000", 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087562999999999996 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568793+0000", 2024-09-21T08:31:52.804 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9429000000000003e-05 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568842+0000", 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8980999999999997e-05 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3200 7.8 7:15ddb887:::smithi13779221-8:head [append 0~0] snapc 77=[77,76,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e323)", 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:36.713346+0000", 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 196.010095377, 2024-09-21T08:31:52.805 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85569378200000001, 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3200 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713346+0000", 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.806 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713346+0000", 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713347+0000", 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.626e-06 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713354+0000", 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.567e-06 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.807 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713369+0000", 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5216e-05 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713386+0000", 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6286e-05 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713509+0000", 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012365299999999999 2024-09-21T08:31:52.808 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:36.713520+0000", 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0536e-05 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566084+0000", 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.85256443900000001 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566173+0000", 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8936000000000004e-05 2024-09-21T08:31:52.809 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.566334+0000", 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000160941 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.567127+0000", 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079256800000000001 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568004+0000", 2024-09-21T08:31:52.810 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00087773799999999998 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568895+0000", 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00089024700000000004 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.568975+0000", 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0527999999999999e-05 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.811 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:37.569040+0000", 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4500999999999998e-05 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:3391 7.12 7:4b20cb2f:::smithi13779221-12:head [copy-from ver 108] snapc 81=[81,80,7f,7e,7d,7c,7a,79,78,77,76,72,71,69] ondisk+write+known_if_redirected+supports_pool_eio e336)", 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 183.13723753900001, 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026028484000000001, 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.812 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 3391 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:31:52.813 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586204+0000", 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5499999999999998e-07 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586207+0000", 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.588e-06 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.814 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586216+0000", 2024-09-21T08:31:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1079999999999999e-06 2024-09-21T08:31:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586220+0000", 2024-09-21T08:31:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5319999999999998e-06 2024-09-21T08:31:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.815 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586265+0000", 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.5469999999999997e-05 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.586369+0000", 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103698 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.597860+0000", 2024-09-21T08:31:52.816 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011490887999999999 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.607396+0000", 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0095365120000000005 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.607414+0000", 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8102e-05 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.612155+0000", 2024-09-21T08:31:52.817 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0047404739999999997 2024-09-21T08:31:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.612196+0000", 2024-09-21T08:31:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1007999999999998e-05 2024-09-21T08:31:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.818 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:28:49.612232+0000", 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.6449000000000003e-05 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4034 7.8 7:144c2ec5:::smithi13779221-45:head [copy-from ver 374] snapc a8=[a8,a7,a6,a5,a4,a3,a2,a1,9e,9c,9b,9a,99,98,94,93,92,91,90,8c,8b,88,86,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e400)", 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:29:53.687463+0000", 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 119.03597823, 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.025036505000000001, 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.819 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4034 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687463+0000", 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.820 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687463+0000", 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687464+0000", 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.046e-06 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687468+0000", 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3490000000000002e-06 2024-09-21T08:31:52.821 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687475+0000", 2024-09-21T08:31:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5250000000000002e-06 2024-09-21T08:31:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687481+0000", 2024-09-21T08:31:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9800000000000003e-06 2024-09-21T08:31:52.822 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687529+0000", 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8270999999999999e-05 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.687635+0000", 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00010632999999999999 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 2", 2024-09-21T08:31:52.823 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.704060+0000", 2024-09-21T08:31:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016425006999999998 2024-09-21T08:31:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.708268+0000", 2024-09-21T08:31:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0042073040000000003 2024-09-21T08:31:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.824 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.712346+0000", 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0040779060000000001 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.712418+0000", 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2045000000000006e-05 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:53.712499+0000", 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1742000000000004e-05 2024-09-21T08:31:52.825 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4057 7.1c 7:3cae00e7:::smithi13779221-28:head [copy-from ver 336] snapc a8=[a8,a7,a5,a4,a3,a2,a1,9e,9c,9b,9a,99,98,94,93,92,91,90,8c,8b,88,86,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e401)", 2024-09-21T08:31:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:29:54.711023+0000", 2024-09-21T08:31:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 118.01241847999999, 2024-09-21T08:31:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027355181999999999, 2024-09-21T08:31:52.826 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4057 2024-09-21T08:31:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.827 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711023+0000", 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711023+0000", 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.828 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711023+0000", 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1500000000000004e-07 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711026+0000", 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9450000000000002e-06 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711033+0000", 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7769999999999997e-06 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.829 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711036+0000", 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1219999999999999e-06 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711605+0000", 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00056895899999999996 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.711739+0000", 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013337599999999999 2024-09-21T08:31:52.830 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1", 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.729672+0000", 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.017933359999999999 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.735954+0000", 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062820030000000004 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.738163+0000", 2024-09-21T08:31:52.831 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0022090489999999998 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.738243+0000", 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9549999999999996e-05 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:29:54.738378+0000", 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013532600000000001 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.832 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4753 7.0 7:00f4c0d4:::smithi13779221-18:head [write 642199~644951] snapc c6=[c6,c5,c4,c2,c1,c0,bf,be,bd,bc,ba,b9,b8,b7,b5,b3,b2,b0,af,a5,a4,a3,9b,99,92,84] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:30:55.603685+0000", 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 57.119756009, 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.036118635000000003, 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4753 2024-09-21T08:31:52.833 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.603685+0000", 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.603685+0000", 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.834 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.603690+0000", 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7199999999999997e-06 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.604399+0000", 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00070936199999999995 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.604419+0000", 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9324e-05 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.835 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.604447+0000", 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8209999999999999e-05 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.604538+0000", 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1057000000000006e-05 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op must wait for map", 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.604549+0000", 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1328999999999999e-05 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.836 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.621216+0000", 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.016666800999999998 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.621347+0000", 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000131359 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.621620+0000", 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000272985 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.837 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.627941+0000", 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0063205500000000003 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.631493+0000", 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035525190000000001 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.639659+0000", 2024-09-21T08:31:52.838 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081655789999999992 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.639737+0000", 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7614999999999995e-05 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.639804+0000", 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7224999999999994e-05 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.839 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4754 7.0 7:00f4c0d4:::smithi13779221-18:head [write 1696199~605559] snapc c6=[c6,c5,c4,c2,c1,c0,bf,be,bd,bc,ba,b9,b8,b7,b5,b3,b2,b0,af,a5,a4,a3,9b,99,92,84] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:30:55.614066+0000", 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 57.109375370999999, 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027472956999999999, 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4754 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.840 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614066+0000", 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614066+0000", 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614069+0000", 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2160000000000002e-06 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.841 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614697+0000", 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062755500000000002 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614712+0000", 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5801999999999999e-05 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614727+0000", 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4238e-05 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.842 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.624823+0000", 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010096324 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.624967+0000", 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014410700000000001 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.625279+0000", 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031171399999999999 2024-09-21T08:31:52.843 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.627949+0000", 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026697119999999999 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.631544+0000", 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035952380000000002 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.641254+0000", 2024-09-21T08:31:52.844 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0097102779999999993 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.641479+0000", 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000224868 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.641539+0000", 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9905000000000003e-05 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.845 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4755 7.0 7:00f4c0d4:::smithi13779221-18:head [write 2992318~49881] snapc c6=[c6,c5,c4,c2,c1,c0,bf,be,bd,bc,ba,b9,b8,b7,b5,b3,b2,b0,af,a5,a4,a3,9b,99,92,84] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:30:55.614744+0000", 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 57.108697614999997, 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.026921884, 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4755 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.846 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614744+0000", 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614744+0000", 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614745+0000", 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.356e-06 2024-09-21T08:31:52.847 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614785+0000", 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0049000000000002e-05 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614793+0000", 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3259999999999996e-06 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614800+0000", 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4659999999999996e-06 2024-09-21T08:31:52.848 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.626223+0000", 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011423027000000001 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.626353+0000", 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013007399999999999 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:31:52.849 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.626560+0000", 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000207328 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.627959+0000", 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001399218 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.631588+0000", 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0036289099999999999 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.850 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.641591+0000", 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010002156999999999 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.641631+0000", 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.0723000000000003e-05 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.641665+0000", 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.4249999999999999e-05 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.851 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4756 7.0 7:00f4c0d4:::smithi13779221-18:head [setxattr _header (58) in=7b,truncate 3042199] snapc c6=[c6,c5,c4,c2,c1,c0,bf,be,bd,bc,ba,b9,b8,b7,b5,b3,b2,b0,af,a5,a4,a3,9b,99,92,84] ondisk+write+known_if_redirected+supports_pool_eio e462)", 2024-09-21T08:31:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:30:55.614809+0000", 2024-09-21T08:31:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 57.108632210000003, 2024-09-21T08:31:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027105734999999999, 2024-09-21T08:31:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.852 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4756 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614809+0000", 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.853 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614809+0000", 2024-09-21T08:31:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614810+0000", 2024-09-21T08:31:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.4600000000000003e-07 2024-09-21T08:31:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.854 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614814+0000", 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9439999999999998e-06 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614819+0000", 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.921e-06 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614825+0000", 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.1369999999999999e-06 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.855 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.627093+0000", 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012268472000000001 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.627215+0000", 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000121352 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,2", 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.627424+0000", 2024-09-21T08:31:52.856 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000209414 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.628047+0000", 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000623071 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.631633+0000", 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003585839 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.857 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.641727+0000", 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.010093523 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.641826+0000", 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9538999999999997e-05 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.858 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.641915+0000", 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8677000000000005e-05 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4757 7.0 7:00f4c0d4:::smithi13779221-18:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e462)", 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:30:55.614831+0000", 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 57.108610347999999, 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027418623999999999, 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:31:52.859 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4757 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614831+0000", 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614831+0000", 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:31:52.860 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614831+0000", 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.8800000000000003e-07 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614834+0000", 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.424e-06 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614839+0000", 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5670000000000002e-06 2024-09-21T08:31:52.861 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:31:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.614845+0000", 2024-09-21T08:31:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1970000000000004e-06 2024-09-21T08:31:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.627831+0000", 2024-09-21T08:31:52.862 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.012986643000000001 2024-09-21T08:31:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:31:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.627923+0000", 2024-09-21T08:31:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.2186000000000001e-05 2024-09-21T08:31:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:31:52.863 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.641987+0000", 2024-09-21T08:31:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.014063295999999999 2024-09-21T08:31:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:31:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.642099+0000", 2024-09-21T08:31:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000112691 2024-09-21T08:31:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:31:52.864 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:31:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:31:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:30:55.642249+0000", 2024-09-21T08:31:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00015013200000000001 2024-09-21T08:31:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:31:52.865 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:31:52.865 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:52.866 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:52.867 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:52.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:52.968 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:52.990 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:52.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:52.991 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:53.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:52 smithi165 ceph-mon[32893]: 5.a scrub starts 2024-09-21T08:31:53.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:52 smithi165 ceph-mon[32893]: 5.a scrub ok 2024-09-21T08:31:53.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:52 smithi165 ceph-mon[32893]: osdmap e519: 8 total, 6 up, 5 in 2024-09-21T08:31:53.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:52 smithi137 ceph-mon[29634]: 5.a scrub starts 2024-09-21T08:31:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:52 smithi137 ceph-mon[29634]: 5.a scrub ok 2024-09-21T08:31:53.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:52 smithi137 ceph-mon[29634]: osdmap e519: 8 total, 6 up, 5 in 2024-09-21T08:31:53.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:52 smithi137 ceph-mon[41044]: 5.a scrub starts 2024-09-21T08:31:53.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:52 smithi137 ceph-mon[41044]: 5.a scrub ok 2024-09-21T08:31:53.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:52 smithi137 ceph-mon[41044]: osdmap e519: 8 total, 6 up, 5 in 2024-09-21T08:31:53.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:53.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:53.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:53.115 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:53.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:53.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:53.239 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:53.239 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:53.340 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:53.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:53.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:53.362 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:53.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:53.485 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:53.485 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:53.486 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:53.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T08:31:53.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:53.604 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:31:53.610 INFO:tasks.rados.rados.0.smithi137.stdout:2075: done (2 left) 2024-09-21T08:31:53.610 INFO:tasks.rados.rados.0.smithi137.stdout:2076: done (1 left) 2024-09-21T08:31:53.610 INFO:tasks.rados.rados.0.smithi137.stdout:2077: done (0 left) 2024-09-21T08:31:53.610 INFO:tasks.rados.rados.0.smithi137.stdout:2078: write oid 4 current snap is 229 2024-09-21T08:31:53.610 INFO:tasks.rados.rados.0.smithi137.stdout:2078: seq_num 729 ranges {652303=710056,2059726=192577} 2024-09-21T08:31:53.617 INFO:tasks.rados.rados.0.smithi137.stdout:2078: writing smithi13779221-4 from 652303 to 1362359 tid 1 2024-09-21T08:31:53.622 INFO:tasks.rados.rados.0.smithi137.stdout:2078: writing smithi13779221-4 from 2059726 to 2252303 tid 2 2024-09-21T08:31:53.623 INFO:tasks.rados.rados.0.smithi137.stdout:2079: delete oid 5 current snap is 229 2024-09-21T08:31:53.629 INFO:tasks.rados.rados.0.smithi137.stdout:2078: finishing write tid 1 to smithi13779221-4 2024-09-21T08:31:53.630 INFO:tasks.rados.rados.0.smithi137.stdout:2078: finishing write tid 2 to smithi13779221-4 2024-09-21T08:31:53.631 INFO:tasks.rados.rados.0.smithi137.stdout:2078: finishing write tid 3 to smithi13779221-4 2024-09-21T08:31:53.632 INFO:tasks.rados.rados.0.smithi137.stdout:2078: finishing write tid 4 to smithi13779221-4 2024-09-21T08:31:53.632 INFO:tasks.rados.rados.0.smithi137.stdout:2078: oid 4 updating version 0 to 456 2024-09-21T08:31:53.632 INFO:tasks.rados.rados.0.smithi137.stdout:2078: oid 4 version 456 is already newer than 455 2024-09-21T08:31:53.632 INFO:tasks.rados.rados.0.smithi137.stdout:2078: oid 4 updating version 456 to 457 2024-09-21T08:31:53.632 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 457 (ObjNum 729 snap 229 seq_num 729) dirty exists 2024-09-21T08:31:53.632 INFO:tasks.rados.rados.0.smithi137.stdout:2078: left oid 4 (ObjNum 729 snap 229 seq_num 729) 2024-09-21T08:31:53.633 INFO:tasks.rados.rados.0.smithi137.stdout:2078: done (1 left) 2024-09-21T08:31:53.633 INFO:tasks.rados.rados.0.smithi137.stdout:2079: done (0 left) 2024-09-21T08:31:53.633 INFO:tasks.rados.rados.0.smithi137.stdout:2080: snap_remove snap 228 2024-09-21T08:31:53.640 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:53.641 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:53.641 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:53.742 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:53.764 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:53.765 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:53.765 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:53.831 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:53.865 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:53.887 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:53.887 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:53.887 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:53.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: 5.9 scrub starts 2024-09-21T08:31:53.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: 5.9 scrub ok 2024-09-21T08:31:53.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:53.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: pgmap v683: 188 pgs: 2 active+clean+snaptrim, 10 active+clean+remapped+snaptrim_wait, 55 active+clean+remapped, 6 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 667 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1.8 MiB/s wr, 11 op/s; 255/1602 objects degraded (15.918%); 137/1602 objects misplaced (8.552%) 2024-09-21T08:31:53.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: 3.7 scrub starts 2024-09-21T08:31:53.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: 3.7 scrub ok 2024-09-21T08:31:53.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4271023934' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:53.938 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4271023934' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: osdmap e520: 8 total, 6 up, 5 in 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: from='osd.7 v1:172.21.15.165:6812/69065038' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[41044]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: 5.9 scrub starts 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: 5.9 scrub ok 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: pgmap v683: 188 pgs: 2 active+clean+snaptrim, 10 active+clean+remapped+snaptrim_wait, 55 active+clean+remapped, 6 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 667 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1.8 MiB/s wr, 11 op/s; 255/1602 objects degraded (15.918%); 137/1602 objects misplaced (8.552%) 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: 3.7 scrub starts 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: 3.7 scrub ok 2024-09-21T08:31:53.939 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4271023934' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:53.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:31:53.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4271023934' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:53.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: osdmap e520: 8 total, 6 up, 5 in 2024-09-21T08:31:53.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: from='osd.7 v1:172.21.15.165:6812/69065038' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:31:53.940 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:53 smithi137 ceph-mon[29634]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:31:53.953 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7[111647]: 2024-09-21T08:31:53.657+0000 7f01eeffa640 -1 osd.7 338 log_to_monitors true 2024-09-21T08:31:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: 5.9 scrub starts 2024-09-21T08:31:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: 5.9 scrub ok 2024-09-21T08:31:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:31:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: pgmap v683: 188 pgs: 2 active+clean+snaptrim, 10 active+clean+remapped+snaptrim_wait, 55 active+clean+remapped, 6 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 667 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1.8 MiB/s wr, 11 op/s; 255/1602 objects degraded (15.918%); 137/1602 objects misplaced (8.552%) 2024-09-21T08:31:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: 3.7 scrub starts 2024-09-21T08:31:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: 3.7 scrub ok 2024-09-21T08:31:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4271023934' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:31:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:31:53.953 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4271023934' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:31:53.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: osdmap e520: 8 total, 6 up, 5 in 2024-09-21T08:31:53.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: from='osd.7 v1:172.21.15.165:6812/69065038' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:31:53.954 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:53 smithi165 ceph-mon[32893]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-09-21T08:31:53.988 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:54.009 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:54.010 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:54.010 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:54.011 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:31:54.112 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:54.134 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:54.134 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:54.134 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:54.235 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:54.257 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:54.257 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:54.258 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:54.351 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:54.351 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:31:54.351 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:31:54.351 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:54.359 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:54.380 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:54.381 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:54.381 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:54.475 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:54.482 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:54.504 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:54.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:54.505 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:54.606 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:54.609 INFO:tasks.rados.rados.0.smithi137.stdout:2080: done (0 left) 2024-09-21T08:31:54.609 INFO:tasks.rados.rados.0.smithi137.stdout:2081: write (excl) oid 32 current snap is 229 2024-09-21T08:31:54.609 INFO:tasks.rados.rados.0.smithi137.stdout:2081: seq_num 731 ranges {685917=576907,1893043=392874} 2024-09-21T08:31:54.615 INFO:tasks.rados.rados.0.smithi137.stdout:2081: writing smithi13779221-32 from 685917 to 1262824 tid 1 2024-09-21T08:31:54.621 INFO:tasks.rados.rados.0.smithi137.stdout:2081: writing smithi13779221-32 from 1893043 to 2285917 tid 2 2024-09-21T08:31:54.622 INFO:tasks.rados.rados.0.smithi137.stdout:2082: delete oid 11 current snap is 229 2024-09-21T08:31:54.630 INFO:tasks.rados.rados.0.smithi137.stdout:2081: finishing write tid 1 to smithi13779221-32 2024-09-21T08:31:54.631 INFO:tasks.rados.rados.0.smithi137.stdout:2081: finishing write tid 2 to smithi13779221-32 2024-09-21T08:31:54.632 INFO:tasks.rados.rados.0.smithi137.stdout:2081: finishing write tid 3 to smithi13779221-32 2024-09-21T08:31:54.632 INFO:tasks.rados.rados.0.smithi137.stdout:2081: finishing write tid 4 to smithi13779221-32 2024-09-21T08:31:54.632 INFO:tasks.rados.rados.0.smithi137.stdout:2081: oid 32 updating version 0 to 545 2024-09-21T08:31:54.632 INFO:tasks.rados.rados.0.smithi137.stdout:2081: oid 32 version 545 is already newer than 543 2024-09-21T08:31:54.632 INFO:tasks.rados.rados.0.smithi137.stdout:2081: oid 32 version 545 is already newer than 544 2024-09-21T08:31:54.632 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 545 (ObjNum 731 snap 229 seq_num 731) dirty exists 2024-09-21T08:31:54.632 INFO:tasks.rados.rados.0.smithi137.stdout:2081: left oid 32 (ObjNum 731 snap 229 seq_num 731) 2024-09-21T08:31:54.634 INFO:tasks.rados.rados.0.smithi137.stdout:2081: done (1 left) 2024-09-21T08:31:54.634 INFO:tasks.rados.rados.0.smithi137.stdout:2082: done (0 left) 2024-09-21T08:31:54.634 INFO:tasks.rados.rados.0.smithi137.stdout:2083: snap_create 2024-09-21T08:31:54.637 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:54.638 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:54.638 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:54.739 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:54.761 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:54.762 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:54.762 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:54.863 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:54.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:54.890 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:54.890 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:54.937 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:54 smithi165 ceph-mon[32893]: 3.e scrub starts 2024-09-21T08:31:54.937 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:54 smithi165 ceph-mon[32893]: 3.e scrub ok 2024-09-21T08:31:54.937 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:54 smithi165 ceph-mon[32893]: 4.8 scrub starts 2024-09-21T08:31:54.937 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:54 smithi165 ceph-mon[32893]: 4.8 scrub ok 2024-09-21T08:31:54.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:54 smithi165 ceph-mon[32893]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T08:31:54.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:54 smithi165 ceph-mon[32893]: osdmap e521: 8 total, 6 up, 5 in 2024-09-21T08:31:54.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:54 smithi165 ceph-mon[32893]: from='osd.7 v1:172.21.15.165:6812/69065038' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:31:54.938 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:54 smithi165 ceph-mon[32893]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:31:54.971 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_blocked_ops 2024-09-21T08:31:54.983 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[29634]: 3.e scrub starts 2024-09-21T08:31:54.983 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[29634]: 3.e scrub ok 2024-09-21T08:31:54.983 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[29634]: 4.8 scrub starts 2024-09-21T08:31:54.983 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[29634]: 4.8 scrub ok 2024-09-21T08:31:54.983 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[29634]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T08:31:54.983 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[29634]: osdmap e521: 8 total, 6 up, 5 in 2024-09-21T08:31:54.983 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[29634]: from='osd.7 v1:172.21.15.165:6812/69065038' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:31:54.983 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[29634]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:31:54.984 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[41044]: 3.e scrub starts 2024-09-21T08:31:54.984 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[41044]: 3.e scrub ok 2024-09-21T08:31:54.984 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[41044]: 4.8 scrub starts 2024-09-21T08:31:54.984 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[41044]: 4.8 scrub ok 2024-09-21T08:31:54.984 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[41044]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-09-21T08:31:54.984 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[41044]: osdmap e521: 8 total, 6 up, 5 in 2024-09-21T08:31:54.984 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[41044]: from='osd.7 v1:172.21.15.165:6812/69065038' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:31:54.984 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:54 smithi137 ceph-mon[41044]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi165", "root=default"]}]: dispatch 2024-09-21T08:31:54.991 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:55.013 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:55.013 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:55.013 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:55.016 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:55.016 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:55.032 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:55.032 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:55.040 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:55.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:55.041 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:55.049 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:55.049 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:55.066 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:55.066 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:55.082 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:55.082 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:55.083 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.7 2024-09-21T08:31:55.142 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:55.163 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:55.164 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:55.164 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:55.265 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:55.286 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:55.287 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:55.287 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:55.296 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:55.388 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:55.410 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:55.410 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:55.411 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:55.511 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:55.533 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:55.534 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:55.534 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:55.556 INFO:journalctl@ceph.osd.7.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-7[111647]: 2024-09-21T08:31:55.294+0000 7f01e2d9d640 -1 osd.7 521 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T08:31:55.618 INFO:tasks.rados.rados.0.smithi137.stdout:2083: done (0 left) 2024-09-21T08:31:55.618 INFO:tasks.rados.rados.0.smithi137.stdout:2084: delete oid 33 current snap is 230 2024-09-21T08:31:55.635 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:55.663 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:31:55.664 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:31:55.666 INFO:tasks.rados.rados.0.smithi137.stdout:2084: done (0 left) 2024-09-21T08:31:55.666 INFO:tasks.rados.rados.0.smithi137.stdout:2085: write (excl) oid 38 current snap is 230 2024-09-21T08:31:55.666 INFO:tasks.rados.rados.0.smithi137.stdout:2085: seq_num 734 ranges {736338=402654,1552136=539580} 2024-09-21T08:31:55.672 INFO:tasks.rados.rados.0.smithi137.stdout:2085: writing smithi13779221-38 from 736338 to 1138992 tid 1 2024-09-21T08:31:55.678 INFO:tasks.rados.rados.0.smithi137.stdout:2085: writing smithi13779221-38 from 1552136 to 2091716 tid 2 2024-09-21T08:31:55.678 INFO:tasks.rados.rados.0.smithi137.stdout:2086: snap_remove snap 221 2024-09-21T08:31:55.678 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:55.679 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:55.679 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:55.774 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:55.774 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:55.780 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:55.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:55.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:55.803 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:55.885 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T08:31:55.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:55.925 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:55.925 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:55.926 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-mon[32893]: 3.11 scrub starts 2024-09-21T08:31:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-mon[32893]: 3.11 scrub ok 2024-09-21T08:31:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-mon[32893]: pgmap v686: 188 pgs: 2 active+clean+snaptrim, 10 active+clean+remapped+snaptrim_wait, 55 active+clean+remapped, 6 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 667 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1.8 MiB/s wr, 11 op/s; 255/1602 objects degraded (15.918%); 137/1602 objects misplaced (8.552%) 2024-09-21T08:31:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-mon[32893]: 1.2 scrub starts 2024-09-21T08:31:56.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-mon[32893]: 1.2 scrub ok 2024-09-21T08:31:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/123816125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-mon[32893]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T08:31:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-mon[32893]: osd.7 v1:172.21.15.165:6812/69065038 boot 2024-09-21T08:31:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-mon[32893]: osdmap e522: 8 total, 7 up, 5 in 2024-09-21T08:31:56.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:55 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[41044]: 3.11 scrub starts 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[41044]: 3.11 scrub ok 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[41044]: pgmap v686: 188 pgs: 2 active+clean+snaptrim, 10 active+clean+remapped+snaptrim_wait, 55 active+clean+remapped, 6 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 667 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1.8 MiB/s wr, 11 op/s; 255/1602 objects degraded (15.918%); 137/1602 objects misplaced (8.552%) 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[41044]: 1.2 scrub starts 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[41044]: 1.2 scrub ok 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/123816125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[41044]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[41044]: osd.7 v1:172.21.15.165:6812/69065038 boot 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[41044]: osdmap e522: 8 total, 7 up, 5 in 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:31:56.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[29634]: 3.11 scrub starts 2024-09-21T08:31:56.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[29634]: 3.11 scrub ok 2024-09-21T08:31:56.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[29634]: pgmap v686: 188 pgs: 2 active+clean+snaptrim, 10 active+clean+remapped+snaptrim_wait, 55 active+clean+remapped, 6 active+clean+snaptrim_wait, 2 active+clean+remapped+snaptrim, 25 active+undersized, 35 active+undersized+degraded, 53 active+clean; 667 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 7.3 MiB/s rd, 1.8 MiB/s wr, 11 op/s; 255/1602 objects degraded (15.918%); 137/1602 objects misplaced (8.552%) 2024-09-21T08:31:56.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[29634]: 1.2 scrub starts 2024-09-21T08:31:56.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[29634]: 1.2 scrub ok 2024-09-21T08:31:56.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/123816125' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:31:56.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[29634]: Health check cleared: OSD_DOWN (was: 1 osds down) 2024-09-21T08:31:56.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[29634]: osd.7 v1:172.21.15.165:6812/69065038 boot 2024-09-21T08:31:56.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[29634]: osdmap e522: 8 total, 7 up, 5 in 2024-09-21T08:31:56.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:55 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-09-21T08:31:56.026 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:56.048 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:56.048 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:56.048 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:56.149 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:56.171 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:56.171 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:56.172 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:56.273 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:56.295 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:56.295 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:56.295 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:56.370 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_historic_ops 2024-09-21T08:31:56.396 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:56.435 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:56.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:56.436 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:56.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:31:56.465 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:56.537 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:56.559 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:56.560 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:56.560 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:56.608 INFO:tasks.rados.rados.0.smithi137.stdout:2086: done (1 left) 2024-09-21T08:31:56.608 INFO:tasks.rados.rados.0.smithi137.stdout:2087: write oid 49 current snap is 230 2024-09-21T08:31:56.608 INFO:tasks.rados.rados.0.smithi137.stdout:2087: seq_num 735 ranges {753145=477903,2010618=342527} 2024-09-21T08:31:56.622 INFO:tasks.rados.rados.0.smithi137.stdout:2087: writing smithi13779221-49 from 753145 to 1231048 tid 1 2024-09-21T08:31:56.627 INFO:tasks.rados.rados.0.smithi137.stdout:2087: writing smithi13779221-49 from 2010618 to 2353145 tid 2 2024-09-21T08:31:56.627 INFO:tasks.rados.rados.0.smithi137.stdout:2085: finishing write tid 1 to smithi13779221-38 2024-09-21T08:31:56.627 INFO:tasks.rados.rados.0.smithi137.stdout:2085: finishing write tid 2 to smithi13779221-38 2024-09-21T08:31:56.627 INFO:tasks.rados.rados.0.smithi137.stdout:2085: finishing write tid 3 to smithi13779221-38 2024-09-21T08:31:56.628 INFO:tasks.rados.rados.0.smithi137.stdout:2085: finishing write tid 4 to smithi13779221-38 2024-09-21T08:31:56.628 INFO:tasks.rados.rados.0.smithi137.stdout:2085: oid 38 updating version 0 to 838 2024-09-21T08:31:56.628 INFO:tasks.rados.rados.0.smithi137.stdout:2085: oid 38 version 838 is already newer than 837 2024-09-21T08:31:56.628 INFO:tasks.rados.rados.0.smithi137.stdout:2085: oid 38 version 838 is already newer than 836 2024-09-21T08:31:56.628 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 38 v 838 (ObjNum 734 snap 230 seq_num 734) dirty exists 2024-09-21T08:31:56.628 INFO:tasks.rados.rados.0.smithi137.stdout:2085: left oid 38 (ObjNum 734 snap 230 seq_num 734) 2024-09-21T08:31:56.628 INFO:tasks.rados.rados.0.smithi137.stdout:2085: done (1 left) 2024-09-21T08:31:56.628 INFO:tasks.rados.rados.0.smithi137.stdout:2088: delete oid 11 current snap is 230 2024-09-21T08:31:56.661 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:56.690 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:56.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:56.691 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:56.697 INFO:tasks.rados.rados.0.smithi137.stdout:2087: finishing write tid 1 to smithi13779221-49 2024-09-21T08:31:56.697 INFO:tasks.rados.rados.0.smithi137.stdout:2087: finishing write tid 2 to smithi13779221-49 2024-09-21T08:31:56.697 INFO:tasks.rados.rados.0.smithi137.stdout:2087: finishing write tid 3 to smithi13779221-49 2024-09-21T08:31:56.698 INFO:tasks.rados.rados.0.smithi137.stdout:2087: finishing write tid 4 to smithi13779221-49 2024-09-21T08:31:56.698 INFO:tasks.rados.rados.0.smithi137.stdout:2087: oid 49 updating version 0 to 465 2024-09-21T08:31:56.698 INFO:tasks.rados.rados.0.smithi137.stdout:2087: oid 49 updating version 465 to 467 2024-09-21T08:31:56.698 INFO:tasks.rados.rados.0.smithi137.stdout:2087: oid 49 version 467 is already newer than 466 2024-09-21T08:31:56.698 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 467 (ObjNum 735 snap 230 seq_num 735) dirty exists 2024-09-21T08:31:56.698 INFO:tasks.rados.rados.0.smithi137.stdout:2087: left oid 49 (ObjNum 735 snap 230 seq_num 735) 2024-09-21T08:31:56.724 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:56.792 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:56.814 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:56.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:56.815 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:56.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:56 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:31:56.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:56 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:31:56.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:56 smithi165 ceph-mon[32893]: 5.7 scrub starts 2024-09-21T08:31:56.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:56 smithi165 ceph-mon[32893]: 5.7 scrub ok 2024-09-21T08:31:56.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:56 smithi165 ceph-mon[32893]: Health check update: Degraded data redundancy: 255/1602 objects degraded (15.918%), 35 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:56.907 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:56 smithi165 ceph-mon[32893]: osdmap e523: 8 total, 7 up, 5 in 2024-09-21T08:31:56.917 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:56.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:56.940 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:56.940 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:57.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:31:57.002 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:31:57.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[29634]: 5.7 scrub starts 2024-09-21T08:31:57.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[29634]: 5.7 scrub ok 2024-09-21T08:31:57.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[29634]: Health check update: Degraded data redundancy: 255/1602 objects degraded (15.918%), 35 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:57.003 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[29634]: osdmap e523: 8 total, 7 up, 5 in 2024-09-21T08:31:57.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:31:57.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:31:57.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[41044]: 5.7 scrub starts 2024-09-21T08:31:57.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[41044]: 5.7 scrub ok 2024-09-21T08:31:57.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[41044]: Health check update: Degraded data redundancy: 255/1602 objects degraded (15.918%), 35 pgs degraded, 55 pgs undersized (PG_DEGRADED) 2024-09-21T08:31:57.003 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:56 smithi137 ceph-mon[41044]: osdmap e523: 8 total, 7 up, 5 in 2024-09-21T08:31:57.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:31:57.042 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:57.042 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:31:57.042 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:31:57.042 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:57.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:57.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:57.064 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:31:57.165 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:57.186 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:57.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:57.187 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:57.288 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:57.310 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:57.311 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:57.311 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:57.412 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:57.434 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:57.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:57.435 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:57.536 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:57.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:57.558 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:57.558 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:57.615 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:31:57.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:57.664 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:31:57.677 INFO:tasks.rados.rados.0.smithi137.stdout:2087: done (1 left) 2024-09-21T08:31:57.677 INFO:tasks.rados.rados.0.smithi137.stdout:2088: done (0 left) 2024-09-21T08:31:57.678 INFO:tasks.rados.rados.0.smithi137.stdout:2089: read oid 9 snap -1 2024-09-21T08:31:57.678 INFO:tasks.rados.rados.0.smithi137.stdout:2089: expect deleted 2024-09-21T08:31:57.678 INFO:tasks.rados.rados.0.smithi137.stdout:2090: read oid 47 snap -1 2024-09-21T08:31:57.678 INFO:tasks.rados.rados.0.smithi137.stdout:2090: expect (ObjNum 718 snap 222 seq_num 718) 2024-09-21T08:31:57.678 INFO:tasks.rados.rados.0.smithi137.stdout:2091: read oid 17 snap -1 2024-09-21T08:31:57.678 INFO:tasks.rados.rados.0.smithi137.stdout:2091: expect (ObjNum 690 snap 210 seq_num 690) 2024-09-21T08:31:57.678 INFO:tasks.rados.rados.0.smithi137.stdout:2092: write (excl) oid 42 current snap is 230 2024-09-21T08:31:57.678 INFO:tasks.rados.rados.0.smithi137.stdout:2092: seq_num 737 ranges {786759=628401,2011229=375530} 2024-09-21T08:31:57.681 INFO:tasks.thrashosds.thrasher:Reviving osd 0 2024-09-21T08:31:57.681 INFO:tasks.cephadm.osd.0:Restarting osd.0 (starting--it wasn't running)... 2024-09-21T08:31:57.681 DEBUG:teuthology.orchestra.run.smithi137:osd.0> sudo journalctl -f -n 0 -u ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.0.service 2024-09-21T08:31:57.684 DEBUG:teuthology.orchestra.run.smithi137:> sudo systemctl start ceph-ec8566d6-77f1-11ef-baf6-efdc52797490@osd.0 2024-09-21T08:31:57.685 INFO:tasks.rados.rados.0.smithi137.stdout:2092: writing smithi13779221-42 from 786759 to 1415160 tid 1 2024-09-21T08:31:57.690 INFO:tasks.rados.rados.0.smithi137.stdout:2092: writing smithi13779221-42 from 2011229 to 2386759 tid 2 2024-09-21T08:31:57.690 INFO:tasks.rados.rados.0.smithi137.stdout:2093: snap_remove snap 190 2024-09-21T08:31:57.691 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:57.691 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:57.692 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:57.758 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:57.792 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:57.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:57.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:57.819 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:57.883 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:57 smithi165 ceph-mon[32893]: 3.1a scrub starts 2024-09-21T08:31:57.883 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:57 smithi165 ceph-mon[32893]: 3.1a scrub ok 2024-09-21T08:31:57.884 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:57 smithi165 ceph-mon[32893]: pgmap v689: 188 pgs: 4 active+clean+snaptrim, 67 remapped+peering, 57 peering, 4 active+clean+snaptrim_wait, 1 active+undersized, 2 active+undersized+degraded, 53 active+clean; 642 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 21/1578 objects degraded (1.331%) 2024-09-21T08:31:57.884 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:57 smithi165 ceph-mon[32893]: 5.13 scrub starts 2024-09-21T08:31:57.884 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:57 smithi165 ceph-mon[32893]: 5.13 scrub ok 2024-09-21T08:31:57.884 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:57 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1754908239' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:57.884 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:57 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:57.884 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:57 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/1754908239' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:31:57.884 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:57 smithi165 ceph-mon[32893]: osdmap e524: 8 total, 7 up, 5 in 2024-09-21T08:31:57.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:57.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:57.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:57.943 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:57.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[29634]: 3.1a scrub starts 2024-09-21T08:31:57.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[29634]: 3.1a scrub ok 2024-09-21T08:31:57.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[29634]: pgmap v689: 188 pgs: 4 active+clean+snaptrim, 67 remapped+peering, 57 peering, 4 active+clean+snaptrim_wait, 1 active+undersized, 2 active+undersized+degraded, 53 active+clean; 642 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 21/1578 objects degraded (1.331%) 2024-09-21T08:31:57.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[29634]: 5.13 scrub starts 2024-09-21T08:31:57.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[29634]: 5.13 scrub ok 2024-09-21T08:31:57.996 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1754908239' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:57.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:57.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/1754908239' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:31:57.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[29634]: osdmap e524: 8 total, 7 up, 5 in 2024-09-21T08:31:57.997 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:57 smithi137 systemd[1]: Starting Ceph osd.0 for ec8566d6-77f1-11ef-baf6-efdc52797490... 2024-09-21T08:31:57.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[41044]: 3.1a scrub starts 2024-09-21T08:31:57.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[41044]: 3.1a scrub ok 2024-09-21T08:31:57.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[41044]: pgmap v689: 188 pgs: 4 active+clean+snaptrim, 67 remapped+peering, 57 peering, 4 active+clean+snaptrim_wait, 1 active+undersized, 2 active+undersized+degraded, 53 active+clean; 642 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 21/1578 objects degraded (1.331%) 2024-09-21T08:31:57.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[41044]: 5.13 scrub starts 2024-09-21T08:31:57.997 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[41044]: 5.13 scrub ok 2024-09-21T08:31:57.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1754908239' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:31:57.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:31:57.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/1754908239' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:31:57.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:57 smithi137 ceph-mon[41044]: osdmap e524: 8 total, 7 up, 5 in 2024-09-21T08:31:58.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:58.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:58.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:58.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:58.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:31:58.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:58.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:58.191 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:31:58.249 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:31:58.249 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:31:58.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:31:58.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:58.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:58.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:31:58.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:31:58.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:58.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:58.439 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:31:58.484 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:31:58.540 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:58.561 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:58.562 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:58.562 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:58.624 INFO:tasks.rados.rados.0.smithi137.stdout:2093: done (4 left) 2024-09-21T08:31:58.625 INFO:tasks.rados.rados.0.smithi137.stdout:2094: read oid 6 snap 174 2024-09-21T08:31:58.625 INFO:tasks.rados.rados.0.smithi137.stdout:2094: expect (ObjNum 593 snap 172 seq_num 593) 2024-09-21T08:31:58.625 INFO:tasks.rados.rados.0.smithi137.stdout:2095: snap_create 2024-09-21T08:31:58.625 INFO:tasks.rados.rados.0.smithi137.stdout:2090: expect (ObjNum 718 snap 222 seq_num 718) 2024-09-21T08:31:58.663 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:58.676 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:58 smithi137 podman[156808]: 2024-09-21 08:31:58.572503378 +0000 UTC m=+0.111406569 container create 70ea74f905f56758f94f9efb2744c1b85d537152a41e87b7d1ba1ca276894a5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.37.2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240731, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23) 2024-09-21T08:31:58.676 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:58 smithi137 podman[156808]: 2024-09-21 08:31:58.48201458 +0000 UTC m=+0.020917730 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T08:31:58.685 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:58.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:58.686 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:58.757 INFO:tasks.rados.rados.0.smithi137.stdout:2091: expect (ObjNum 690 snap 210 seq_num 690) 2024-09-21T08:31:58.787 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:58.808 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:58.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:58.809 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:58.834 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:58 smithi165 ceph-mon[32893]: 1.0 scrub starts 2024-09-21T08:31:58.835 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:58 smithi165 ceph-mon[32893]: 1.0 scrub ok 2024-09-21T08:31:58.835 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:58 smithi165 ceph-mon[32893]: pgmap v692: 188 pgs: 4 active+clean+snaptrim, 67 remapped+peering, 57 peering, 4 active+clean+snaptrim_wait, 1 active+undersized, 2 active+undersized+degraded, 53 active+clean; 642 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 21/1578 objects degraded (1.331%) 2024-09-21T08:31:58.835 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:31:58 smithi165 ceph-mon[32893]: osdmap e525: 8 total, 7 up, 5 in 2024-09-21T08:31:58.861 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_ops_in_flight 2024-09-21T08:31:58.911 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:58.965 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:58.966 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:58.966 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:59.005 INFO:tasks.rados.rados.0.smithi137.stdout:2092: finishing write tid 1 to smithi13779221-42 2024-09-21T08:31:59.005 INFO:tasks.rados.rados.0.smithi137.stdout:2092: finishing write tid 2 to smithi13779221-42 2024-09-21T08:31:59.005 INFO:tasks.rados.rados.0.smithi137.stdout:2092: finishing write tid 3 to smithi13779221-42 2024-09-21T08:31:59.005 INFO:tasks.rados.rados.0.smithi137.stdout:2092: finishing write tid 4 to smithi13779221-42 2024-09-21T08:31:59.005 INFO:tasks.rados.rados.0.smithi137.stdout:2092: oid 42 updating version 0 to 704 2024-09-21T08:31:59.005 INFO:tasks.rados.rados.0.smithi137.stdout:2092: oid 42 version 704 is already newer than 703 2024-09-21T08:31:59.006 INFO:tasks.rados.rados.0.smithi137.stdout:2092: oid 42 version 704 is already newer than 702 2024-09-21T08:31:59.006 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 42 v 704 (ObjNum 737 snap 230 seq_num 737) dirty exists 2024-09-21T08:31:59.006 INFO:tasks.rados.rados.0.smithi137.stdout:2092: left oid 42 (ObjNum 737 snap 230 seq_num 737) 2024-09-21T08:31:59.006 INFO:tasks.rados.rados.0.smithi137.stdout:2094: expect (ObjNum 593 snap 172 seq_num 593) 2024-09-21T08:31:59.008 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:58 smithi137 ceph-mon[29634]: 1.0 scrub starts 2024-09-21T08:31:59.009 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:58 smithi137 ceph-mon[29634]: 1.0 scrub ok 2024-09-21T08:31:59.009 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:58 smithi137 ceph-mon[29634]: pgmap v692: 188 pgs: 4 active+clean+snaptrim, 67 remapped+peering, 57 peering, 4 active+clean+snaptrim_wait, 1 active+undersized, 2 active+undersized+degraded, 53 active+clean; 642 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 21/1578 objects degraded (1.331%) 2024-09-21T08:31:59.009 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:31:58 smithi137 ceph-mon[29634]: osdmap e525: 8 total, 7 up, 5 in 2024-09-21T08:31:59.009 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:58 smithi137 ceph-mon[41044]: 1.0 scrub starts 2024-09-21T08:31:59.009 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:58 smithi137 ceph-mon[41044]: 1.0 scrub ok 2024-09-21T08:31:59.009 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:58 smithi137 ceph-mon[41044]: pgmap v692: 188 pgs: 4 active+clean+snaptrim, 67 remapped+peering, 57 peering, 4 active+clean+snaptrim_wait, 1 active+undersized, 2 active+undersized+degraded, 53 active+clean; 642 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 21/1578 objects degraded (1.331%) 2024-09-21T08:31:59.009 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:31:58 smithi137 ceph-mon[41044]: osdmap e525: 8 total, 7 up, 5 in 2024-09-21T08:31:59.067 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:31:59.088 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:59.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:59.089 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:31:59.148 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:31:59.190 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:59.211 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:59.212 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:59.212 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:59.268 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 podman[156808]: 2024-09-21 08:31:59.008491152 +0000 UTC m=+0.547394302 container init 70ea74f905f56758f94f9efb2744c1b85d537152a41e87b7d1ba1ca276894a5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, io.buildah.version=1.37.2, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-09-21T08:31:59.268 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 podman[156808]: 2024-09-21 08:31:59.01343176 +0000 UTC m=+0.552334910 container start 70ea74f905f56758f94f9efb2744c1b85d537152a41e87b7d1ba1ca276894a5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, org.label-schema.license=GPLv2) 2024-09-21T08:31:59.268 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate[156864]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T08:31:59.269 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate[156864]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T08:31:59.269 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 podman[156808]: 2024-09-21 08:31:59.183661538 +0000 UTC m=+0.722564692 container attach 70ea74f905f56758f94f9efb2744c1b85d537152a41e87b7d1ba1ca276894a5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240731, io.buildah.version=1.37.2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.vendor=CentOS) 2024-09-21T08:31:59.269 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 bash[156808]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T08:31:59.269 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 bash[156808]: Running command: /usr/bin/ceph-authtool --gen-print-key 2024-09-21T08:31:59.313 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:59.335 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:59.335 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:59.336 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:59.437 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:59.458 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:59.459 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:59.459 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:31:59.560 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:59.582 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:59.582 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:59.583 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:59.622 INFO:tasks.rados.rados.0.smithi137.stdout:2089: done (5 left) 2024-09-21T08:31:59.623 INFO:tasks.rados.rados.0.smithi137.stdout:2090: done (4 left) 2024-09-21T08:31:59.623 INFO:tasks.rados.rados.0.smithi137.stdout:2091: done (3 left) 2024-09-21T08:31:59.623 INFO:tasks.rados.rados.0.smithi137.stdout:2092: done (2 left) 2024-09-21T08:31:59.623 INFO:tasks.rados.rados.0.smithi137.stdout:2094: done (1 left) 2024-09-21T08:31:59.623 INFO:tasks.rados.rados.0.smithi137.stdout:2095: done (0 left) 2024-09-21T08:31:59.623 INFO:tasks.rados.rados.0.smithi137.stdout:2096: snap_create 2024-09-21T08:31:59.680 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate[156864]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T08:31:59.681 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 bash[156808]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T08:31:59.681 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate[156864]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T08:31:59.681 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 bash[156808]: Running command: /usr/bin/ceph-bluestore-tool prime-osd-dir --path /var/lib/ceph/osd/ceph-0 --no-mon-config --dev /dev/mapper/vg_nvme-lv_4 2024-09-21T08:31:59.683 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:31:59.699 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:31:59.699 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:31:59.699 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:31:59.699 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:31:59.705 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:59.705 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:59.706 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:31:59.764 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:31:59.780 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:31:59.781 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:31:59.797 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:31:59.797 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:31:59.807 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:31:59.813 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:31:59.813 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:31:59.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:31:59.830 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:31:59.830 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:31:59.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:59.831 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:31:59.846 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:31:59.846 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:31:59.863 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:31:59.863 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:31:59.879 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:31:59.879 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_delete_sleep = '' osd_delete_sleep_hdd = '' osd_delete_sleep_hybrid = '' osd_delete_sleep_ssd = '' osd_enable_op_tracker = '' osd_recovery_max_active = '' osd_recovery_max_active_hdd = '' osd_recovery_max_active_ssd = '' osd_recovery_sleep = '' osd_recovery_sleep_hdd = '' osd_recovery_sleep_hybrid = '' osd_recovery_sleep_ssd = '' osd_scrub_sleep = '' osd_snap_trim_sleep = '' osd_snap_trim_sleep_hdd = '' osd_snap_trim_sleep_hybrid = '' osd_snap_trim_sleep_ssd = '' 2024-09-21T08:31:59.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:31:59.950 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate[156864]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T08:31:59.951 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 bash[156808]: Running command: /usr/bin/chown -h ceph:ceph /dev/mapper/vg_nvme-lv_4 2024-09-21T08:31:59.951 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate[156864]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T08:31:59.951 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 bash[156808]: Running command: /usr/bin/chown -R ceph:ceph /dev/dm-3 2024-09-21T08:31:59.951 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate[156864]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T08:31:59.951 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 bash[156808]: Running command: /usr/bin/ln -s /dev/mapper/vg_nvme-lv_4 /var/lib/ceph/osd/ceph-0/block 2024-09-21T08:31:59.951 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate[156864]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T08:31:59.951 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 bash[156808]: Running command: /usr/bin/chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-09-21T08:31:59.951 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate[156864]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T08:31:59.951 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:31:59 smithi137 bash[156808]: --> ceph-volume raw activate successful for osd ID: 0 2024-09-21T08:31:59.956 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:31:59.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:31:59.957 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:00.057 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:00.079 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:00.079 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:00.079 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:00.180 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:00.201 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:00.202 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:00.202 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:00.234 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:32:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: 2024-09-21T08:32:00.185+0000 7f532deac640 -1 mgr get_metadata_python Requested missing service osd.0 2024-09-21T08:32:00.302 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:00.324 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:00.325 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:00.325 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:00.368 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_blocked_ops 2024-09-21T08:32:00.383 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:32:00.425 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:00.447 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:00.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:00.448 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:00.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:00.571 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:00.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:00.572 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:00.611 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:32:00.621 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:32:00.621 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:32:00.629 INFO:tasks.rados.rados.0.smithi137.stdout:2096: done (0 left) 2024-09-21T08:32:00.629 INFO:tasks.rados.rados.0.smithi137.stdout:2097: read oid 14 snap -1 2024-09-21T08:32:00.629 INFO:tasks.rados.rados.0.smithi137.stdout:2097: expect (ObjNum 619 snap 182 seq_num 619) 2024-09-21T08:32:00.629 INFO:tasks.rados.rados.0.smithi137.stdout:2098: read oid 32 snap -1 2024-09-21T08:32:00.629 INFO:tasks.rados.rados.0.smithi137.stdout:2098: expect (ObjNum 731 snap 229 seq_num 731) 2024-09-21T08:32:00.629 INFO:tasks.rados.rados.0.smithi137.stdout:2099: read oid 24 snap 218 2024-09-21T08:32:00.630 INFO:tasks.rados.rados.0.smithi137.stdout:2099: expect (ObjNum 711 snap 218 seq_num 711) 2024-09-21T08:32:00.630 INFO:tasks.rados.rados.0.smithi137.stdout:2100: copy_from oid 29 from oid 36 current snap is 232 2024-09-21T08:32:00.630 INFO:tasks.rados.rados.0.smithi137.stdout:2101: read oid 15 snap -1 2024-09-21T08:32:00.630 INFO:tasks.rados.rados.0.smithi137.stdout:2101: expect (ObjNum 714 snap 222 seq_num 714) 2024-09-21T08:32:00.630 INFO:tasks.rados.rados.0.smithi137.stdout:2102: snap_create 2024-09-21T08:32:00.636 INFO:tasks.rados.rados.0.smithi137.stdout:2097: expect (ObjNum 619 snap 182 seq_num 619) 2024-09-21T08:32:00.673 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:00.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:00.696 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:00.696 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:00.789 INFO:tasks.rados.rados.0.smithi137.stdout:2098: expect (ObjNum 731 snap 229 seq_num 731) 2024-09-21T08:32:00.797 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:00.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:00.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:00.819 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:00.876 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:00 smithi165 ceph-mon[32893]: osdmap e526: 8 total, 7 up, 5 in 2024-09-21T08:32:00.877 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:00 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2804672458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:00.877 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:00 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:00.877 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:00 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:32:00.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:00.922 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:32:00 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:32:00] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:32:00.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:00 smithi137 ceph-mon[29634]: osdmap e526: 8 total, 7 up, 5 in 2024-09-21T08:32:00.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:00 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2804672458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:00.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:00 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:00.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:00 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:32:00.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:00 smithi137 ceph-mon[41044]: osdmap e526: 8 total, 7 up, 5 in 2024-09-21T08:32:00.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:00 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2804672458' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:00.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:00 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:00.923 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:00 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:32:00.923 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:00 smithi137 podman[156808]: 2024-09-21 08:32:00.679932302 +0000 UTC m=+2.218835452 container died 70ea74f905f56758f94f9efb2744c1b85d537152a41e87b7d1ba1ca276894a5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240731, GIT_BRANCH=HEAD, io.buildah.version=1.37.2, GIT_CLEAN=True, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, maintainer=Guillaume Abrioux , RELEASE=squid-9e2ac70) 2024-09-21T08:32:00.923 INFO:tasks.rados.rados.0.smithi137.stdout:2099: expect (ObjNum 711 snap 218 seq_num 711) 2024-09-21T08:32:00.942 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:00.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:00.943 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:01.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:01.047 INFO:tasks.rados.rados.0.smithi137.stdout:2100: finishing copy_from racing read to smithi13779221-29 2024-09-21T08:32:01.047 INFO:tasks.rados.rados.0.smithi137.stdout:2101: expect (ObjNum 714 snap 222 seq_num 714) 2024-09-21T08:32:01.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:01.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:01.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:01.110 INFO:tasks.rados.rados.0.smithi137.stdout:2100: finishing copy_from to smithi13779221-29 2024-09-21T08:32:01.110 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 734 (ObjNum 566 snap 168 seq_num 566) dirty exists 2024-09-21T08:32:01.128 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:32:01.128 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:32:01.129 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:32:01.129 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:32:01.129 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:32:01.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:01.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:01.191 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:01.191 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:01.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:01.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:01.314 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:01.314 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:01.402 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:01.415 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:01.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:01.438 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:01.439 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:01.539 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:01.561 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:01.562 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:01.562 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:01.598 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:01 smithi137 podman[157160]: 2024-09-21 08:32:01.288195418 +0000 UTC m=+1.326980138 container remove 70ea74f905f56758f94f9efb2744c1b85d537152a41e87b7d1ba1ca276894a5f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0-activate, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, io.buildah.version=1.37.2, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-09-21T08:32:01.598 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:01 smithi137 podman[157324]: 2024-09-21 08:32:01.398201282 +0000 UTC m=+0.017408109 image pull cc9251b744a3faafca7c3095f79bfff81873afe823f31a03a11e015a34abcdbd quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5 2024-09-21T08:32:01.598 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:01 smithi137 podman[157324]: 2024-09-21 08:32:01.505813301 +0000 UTC m=+0.125020114 container create 9809fde96f37de0a53f8d6f5efa1b43efc0fff10888612c4da85c57b99b46361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-9e2ac70, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240731, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, CEPH_POINT_RELEASE=) 2024-09-21T08:32:01.608 INFO:tasks.rados.rados.0.smithi137.stdout:2097: done (5 left) 2024-09-21T08:32:01.608 INFO:tasks.rados.rados.0.smithi137.stdout:2098: done (4 left) 2024-09-21T08:32:01.609 INFO:tasks.rados.rados.0.smithi137.stdout:2099: done (3 left) 2024-09-21T08:32:01.609 INFO:tasks.rados.rados.0.smithi137.stdout:2100: done (2 left) 2024-09-21T08:32:01.609 INFO:tasks.rados.rados.0.smithi137.stdout:2101: done (1 left) 2024-09-21T08:32:01.609 INFO:tasks.rados.rados.0.smithi137.stdout:2102: done (0 left) 2024-09-21T08:32:01.609 INFO:tasks.rados.rados.0.smithi137.stdout:2103: write oid 33 current snap is 233 2024-09-21T08:32:01.609 INFO:tasks.rados.rados.0.smithi137.stdout:2103: seq_num 738 ranges {403566=420003,1386064=546918} 2024-09-21T08:32:01.613 INFO:tasks.rados.rados.0.smithi137.stdout:2103: writing smithi13779221-33 from 403566 to 823569 tid 1 2024-09-21T08:32:01.619 INFO:tasks.rados.rados.0.smithi137.stdout:2103: writing smithi13779221-33 from 1386064 to 1932982 tid 2 2024-09-21T08:32:01.619 INFO:tasks.rados.rados.0.smithi137.stdout:2104: rollback oid 4 current snap is 233 2024-09-21T08:32:01.619 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 4 to 182 2024-09-21T08:32:01.620 INFO:tasks.rados.rados.0.smithi137.stdout:2103: finishing write tid 1 to smithi13779221-33 2024-09-21T08:32:01.620 INFO:tasks.rados.rados.0.smithi137.stdout:2105: snap_remove snap 195 2024-09-21T08:32:01.663 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:01.682 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:01 smithi165 ceph-mon[32893]: pgmap v695: 188 pgs: 5 active+clean+snaptrim, 60 remapped+peering, 57 peering, 5 active+clean+snaptrim_wait, 61 active+clean; 622 MiB data, 1.8 GiB used, 445 GiB / 447 GiB avail; 20 MiB/s, 22 objects/s recovering 2024-09-21T08:32:01.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:01 smithi165 ceph-mon[32893]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1578 objects degraded (1.331%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T08:32:01.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:01 smithi165 ceph-mon[32893]: osdmap e527: 8 total, 7 up, 5 in 2024-09-21T08:32:01.683 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:01 smithi165 ceph-mon[32893]: osdmap e528: 8 total, 7 up, 5 in 2024-09-21T08:32:01.686 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:01.687 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:01.687 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:01.710 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.7 dump_historic_ops 2024-09-21T08:32:01.788 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:01.814 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:01.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:01.815 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:01.821 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T08:32:01.884 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:01 smithi137 ceph-mon[29634]: pgmap v695: 188 pgs: 5 active+clean+snaptrim, 60 remapped+peering, 57 peering, 5 active+clean+snaptrim_wait, 61 active+clean; 622 MiB data, 1.8 GiB used, 445 GiB / 447 GiB avail; 20 MiB/s, 22 objects/s recovering 2024-09-21T08:32:01.884 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:01 smithi137 ceph-mon[29634]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1578 objects degraded (1.331%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T08:32:01.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:01 smithi137 ceph-mon[29634]: osdmap e527: 8 total, 7 up, 5 in 2024-09-21T08:32:01.885 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:01 smithi137 ceph-mon[29634]: osdmap e528: 8 total, 7 up, 5 in 2024-09-21T08:32:01.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:01 smithi137 ceph-mon[41044]: pgmap v695: 188 pgs: 5 active+clean+snaptrim, 60 remapped+peering, 57 peering, 5 active+clean+snaptrim_wait, 61 active+clean; 622 MiB data, 1.8 GiB used, 445 GiB / 447 GiB avail; 20 MiB/s, 22 objects/s recovering 2024-09-21T08:32:01.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:01 smithi137 ceph-mon[41044]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 21/1578 objects degraded (1.331%), 2 pgs degraded, 2 pgs undersized) 2024-09-21T08:32:01.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:01 smithi137 ceph-mon[41044]: osdmap e527: 8 total, 7 up, 5 in 2024-09-21T08:32:01.885 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:01 smithi137 ceph-mon[41044]: osdmap e528: 8 total, 7 up, 5 in 2024-09-21T08:32:01.885 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:01 smithi137 podman[157324]: 2024-09-21 08:32:01.716681719 +0000 UTC m=+0.335888540 container init 9809fde96f37de0a53f8d6f5efa1b43efc0fff10888612c4da85c57b99b46361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, org.label-schema.build-date=20240731, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.schema-version=1.0, io.buildah.version=1.37.2, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-09-21T08:32:01.885 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:01 smithi137 podman[157324]: 2024-09-21 08:32:01.721887999 +0000 UTC m=+0.341094813 container start 9809fde96f37de0a53f8d6f5efa1b43efc0fff10888612c4da85c57b99b46361 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5, name=ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.37.2, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=92e6e0d02ffae0030ac4c9454206214b98a73a23, org.label-schema.build-date=20240731, ceph=True, org.label-schema.vendor=CentOS, RELEASE=squid-9e2ac70, GIT_CLEAN=True) 2024-09-21T08:32:01.886 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:01 smithi137 bash[157324]: 9809fde96f37de0a53f8d6f5efa1b43efc0fff10888612c4da85c57b99b46361 2024-09-21T08:32:01.886 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:01 smithi137 systemd[1]: Started Ceph osd.0 for ec8566d6-77f1-11ef-baf6-efdc52797490. 2024-09-21T08:32:01.916 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:01.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:01.939 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:01.939 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:01.997 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:32:02.041 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:02.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:02.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:02.067 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:02.168 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:02.190 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:02.190 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:02.190 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:02.268 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:32:02 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:32:02.291 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:02.312 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:02.313 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:02.313 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:02.313 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:02.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:02.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:02.437 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:02.438 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:02.479 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:32:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:32:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:32:02.479 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.13 523/522 [PushOp(7:c851e2c3:::smithi13779221-25:head, version: 489'73, data_included: [2249223~282405,3260666~302550], data_size: 584955, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:c851e2c3:::smithi13779221-25:head@489'73, size: 3563216, copy_subset: [(763216, 643082), (1843729, 687899), (2890936, 672280)], clone_subset: {7:c851e2c3:::smithi13779221-25:d3: [(0, 763216), (1406298, 437431), (2531628, 359308)]}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3563216, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2249223, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:56.710052+0000", 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.7671657109999996, 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037567849999999999, 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.710052+0000", 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.710052+0000", 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.710053+0000", 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5300000000000003e-07 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.710305+0000", 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00025140399999999999 2024-09-21T08:32:02.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.710317+0000", 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2906e-05 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.710327+0000", 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.6290000000000005e-06 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.710417+0000", 2024-09-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9827000000000006e-05 2024-09-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.710432+0000", 2024-09-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5522000000000001e-05 2024-09-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.713809+0000", 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033766439999999998 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.18 523/522 [PushOp(7:1fd29771:::smithi13779221-22:head, version: 492'284, data_included: [1713624~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:1fd29771:::smithi13779221-22:head@492'284, size: 3680865, copy_subset: [(0, 3680865)], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2762200, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1713624, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:56.723820+0000", 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.7533979500000001, 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032263880000000002, 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.723820+0000", 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.723820+0000", 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.723821+0000", 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.153e-06 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.726160+0000", 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0023389249999999999 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.726172+0000", 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.189e-05 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.726181+0000", 2024-09-21T08:32:02.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9719999999999998e-06 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.726272+0000", 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0869e-05 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.726290+0000", 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8525999999999998e-05 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.727046+0000", 2024-09-21T08:32:02.487 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00075605300000000002 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.18 523/522 [PushOp(7:1fd29771:::smithi13779221-22:86, version: 346'233, data_included: [682483~789314,1944996~259262], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 2, recovery_info: ObjectRecoveryInfo(7:1fd29771:::smithi13779221-22:86@346'233, size: 3882483, copy_subset: [(0, 3882483)], clone_subset: {7:1fd29771:::smithi13779221-22:7e: []}, snapset: d6=[]:{6c: [69], 7e: [7e, 7d, 7c, 78, 77, 76, 72, 71], 86: [86, 85, 84, 83, 82, 81], a3: [a3, a2, a1, 9f, 9e, 9d, 9c, 9b, 9a, 99, 98, 94, 93, 92, 91, 90, 8c, 8b, 88], ce: [ce, cc, cb, ca, c9, c8, c6, c5, c4, c2, c1, c0, bf, be, b9, b8, b3, b2, b0, af], d6: [d6, d5, d4, d3, d2, d1, d0, cf]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2204258, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:56.734438+0000", 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.7427798770000003, 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0031269639999999999, 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.488 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.734438+0000", 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.734438+0000", 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.734440+0000", 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.392e-06 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.489 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.736498+0000", 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0020584660000000001 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.736513+0000", 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5251e-05 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.490 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.736535+0000", 2024-09-21T08:32:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2058000000000001e-05 2024-09-21T08:32:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.736710+0000", 2024-09-21T08:32:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000174463 2024-09-21T08:32:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.491 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.736756+0000", 2024-09-21T08:32:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.6353000000000002e-05 2024-09-21T08:32:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.492 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.737565+0000", 2024-09-21T08:32:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00080898099999999996 2024-09-21T08:32:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.493 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.18 523/522 [PushOp(7:1a1aa5cd:::smithi13779221-31:head, version: 516'286, data_included: [0~1048576], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7:1a1aa5cd:::smithi13779221-31:head@516'286, size: 2386726, copy_subset: [(0, 2386726)], clone_subset: {}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1048576, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T08:32:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:56.750566+0000", 2024-09-21T08:32:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.7266517940000004, 2024-09-21T08:32:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038156990000000001, 2024-09-21T08:32:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.494 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.750566+0000", 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.750566+0000", 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.750567+0000", 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.034e-06 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.495 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.753627+0000", 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003060063 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.753646+0000", 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8910999999999999e-05 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.753662+0000", 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6022e-05 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.496 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.753750+0000", 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8257000000000006e-05 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.753760+0000", 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.9569999999999997e-06 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.754382+0000", 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062145500000000003 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.497 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.6 523/522 [PushOp(7:611b1c4b:::smithi13779221-3:92, version: 367'147, data_included: [2097152~726457], data_size: 726457, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:611b1c4b:::smithi13779221-3:92@367'147, size: 2823609, copy_subset: [(0, 2823609)], clone_subset: {7:611b1c4b:::smithi13779221-3:8f: []}, snapset: cd=[]:{81: [81, 80, 7f, 7e, 7d, 7c, 78, 77, 76, 72, 71, 69], 8f: [8f, 8c, 8b, 8a, 89, 88, 86, 85, 84, 83, 82], 92: [92, 91, 90], a8: [a8, a7, a6, a5, a4, a3, a2, a1, 9e, 9c, 9b, 9a, 99, 98, 94, 93], ad: [ad, ac, ab, a9], c0: [c0, bf, be, bd, bc, ba, b9, b8, b7, b6, b5, b3, b2, b0, af], cd: [cd, cc, cb, ca, c9, c8, c6, c5, c4, c2, c1]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2823609, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2097152, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:56.763703+0000", 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.7135146429999999, 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003773752, 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.498 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.763703+0000", 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.763703+0000", 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.763705+0000", 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5799999999999999e-06 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.499 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.764243+0000", 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00053849200000000001 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.764269+0000", 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5975000000000001e-05 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.764300+0000", 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0491999999999999e-05 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.500 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.764429+0000", 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012900699999999999 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.764442+0000", 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3354e-05 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.767477+0000", 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030348520000000002 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.501 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.502 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.502 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.502 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.503 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.d 523/522 [PushOp(7:b0b32695:::smithi13779221-19:af, version: 423'141, data_included: [2551853~152502,3315201~231175], data_size: 383677, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:b0b32695:::smithi13779221-19:af@423'141, size: 3546376, copy_subset: [(746376, 551554), (2054831, 649524), (3315201, 231175)], clone_subset: {7:b0b32695:::smithi13779221-19:aa: [(0, 746376), (1297930, 756901), (2704355, 610846)]}, snapset: c1=[]:{70: [69], 96: [96, 94, 93, 92, 91, 90], aa: [aa], af: [af, ae, ad, ac, ab], bd: [bd, bc, bb, ba, b9, b8], c1: [c1]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3546376, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2551853, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.503 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:56.803417+0000", 2024-09-21T08:32:02.503 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.6738009519999997, 2024-09-21T08:32:02.503 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038848989999999998, 2024-09-21T08:32:02.503 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.503 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.803417+0000", 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.803417+0000", 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.504 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.803418+0000", 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0499999999999999e-06 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.803582+0000", 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016427100000000001 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.803598+0000", 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5560999999999999e-05 2024-09-21T08:32:02.505 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.803618+0000", 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9624e-05 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.803695+0000", 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7189999999999998e-05 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.803705+0000", 2024-09-21T08:32:02.506 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8109999999999997e-06 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.807302+0000", 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003597392 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.18 523/522 [PushOp(7:1a1aa5cd:::smithi13779221-31:b6, version: 439'258, data_included: [1579829~213478,1957837~835098], data_size: 1048576, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:1a1aa5cd:::smithi13779221-31:b6@439'258, size: 3815288, copy_subset: [(0, 3815288)], clone_subset: {7:1a1aa5cd:::smithi13779221-31:aa: []}, snapset: e3=[]:{73: [72, 71, 69], 96: [96, 95, 94, 93, 92, 91, 90, 8c, 8b, 8a, 89, 88, 86, 85, 84, 7e, 7d], aa: [aa, a9, a8, a7, a5, a4, a3, 9e, 9c, 9b, 9a, 99, 98], b6: [b6, b5, b4, b3, b2, b1, b0, af, ae], c0: [c0, bf, be, bd, bc, ba, b9, b8, b7], c8: [c8, c6, c5, c4, c2, c1], d3: [d3, d2, d1, d0, cf, ce, cc, cb, ca, c9], d5: [d5, d4], e3: [e2, e1, e0, df, dd, dc, da, d9, d7, d6]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2792935, data_complete: false, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1579829, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:56.810132+0000", 2024-09-21T08:32:02.507 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.667086168, 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032074059999999999, 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.810132+0000", 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.810132+0000", 2024-09-21T08:32:02.508 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.810133+0000", 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6600000000000005e-07 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.810476+0000", 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00034325399999999998 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.509 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.810500+0000", 2024-09-21T08:32:02.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3845e-05 2024-09-21T08:32:02.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.510 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.510 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.810529+0000", 2024-09-21T08:32:02.510 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9221e-05 2024-09-21T08:32:02.510 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.510 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.810647+0000", 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000117759 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.810661+0000", 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3822000000000001e-05 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.511 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.813339+0000", 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002678639 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.19 523/522 [PushOp(7:9d4d36a0:::smithi13779221-27:d3, version: 489'212, data_included: [2182907~132716,2770957~523347], data_size: 656063, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:9d4d36a0:::smithi13779221-27:d3@489'212, size: 3294304, copy_subset: [(0, 3294304)], clone_subset: {7:9d4d36a0:::smithi13779221-27:c1: []}, snapset: df=[]:{86: [86, 85, 84, 83, 82, 81, 7f, 7e, 7d], 8a: [8a, 89, 88, 87], aa: [aa, a9, a8, a7, a5, a4, a3, 9e, 9c, 9b, 9a, 99, 98, 94, 93, 92, 91, 90], ad: [ad, ac, ab], c1: [c1, c0, bf, be, bd, bc, ba, b9, b8, b7, b6, b5, b3, b2, b0, af], d3: [d3, d2, d1, d0, cf, ce]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3294304, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2182907, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:56.884996+0000", 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.592221908, 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038947949999999999, 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.512 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.884996+0000", 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.884996+0000", 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.884997+0000", 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.24e-06 2024-09-21T08:32:02.513 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.885184+0000", 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000186808 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.885202+0000", 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8207e-05 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.885218+0000", 2024-09-21T08:32:02.514 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5583e-05 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.885303+0000", 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5206999999999999e-05 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.885319+0000", 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6246e-05 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.515 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.888891+0000", 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035715040000000001 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.2 523/522 [PushOp(7:43ab705f:::smithi13779221-10:ab, version: 411'346, data_included: [2403206~14008], data_size: 14008, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:43ab705f:::smithi13779221-10:ab@411'346, size: 2470695, copy_subset: [(689219, 415737), (1770367, 646847)], clone_subset: {7:43ab705f:::smithi13779221-10:a9: [(0, 689219), (1104956, 665411), (2417214, 53481)]}, snapset: db=[]:{6b: [69], 72: [72, 71], 85: [85, 84, 83, 82, 81, 7f, 7e, 7d, 7c, 78, 77], 93: [93, 92, 91, 90, 8c, 8b, 8a, 89, 88, 86], 9e: [9e, 9d, 9c, 9b, 9a, 99, 98, 96, 94], a9: [a9, a8, a7, a5, a4, a3], ab: [ab, aa], c1: [c1, c0, bf, be, bd, bc, ba, b9, b8, b7, b6, b5, b3, b2, b0, af], db: [db, da, d9, d8, d7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2417214, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2403206, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:56.927781+0000", 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5494373130000003, 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037350159999999999, 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.516 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.927781+0000", 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.927781+0000", 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.517 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.927782+0000", 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.263e-06 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.927813+0000", 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1154e-05 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.927841+0000", 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7855e-05 2024-09-21T08:32:02.518 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.927871+0000", 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0134e-05 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.928041+0000", 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016993499999999999 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.519 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.928054+0000", 2024-09-21T08:32:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2563e-05 2024-09-21T08:32:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.931516+0000", 2024-09-21T08:32:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034621119999999998 2024-09-21T08:32:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.520 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.12 523/522 [PushOp(7:4ccee2c4:::smithi13779221-50:86, version: 346'369, data_included: [2421213~353381,3432022~349619], data_size: 703000, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:4ccee2c4:::smithi13779221-50:86@346'369, size: 3781641, copy_subset: [(581641, 621467), (1994104, 780490), (3092488, 689153)], clone_subset: {7:4ccee2c4:::smithi13779221-50:7d: [(0, 581641), (1203108, 790996), (2774594, 317894)]}, snapset: d6=[]:{75: [72, 71, 69], 7d: [7d, 7c, 7a, 79, 78, 77, 76], 86: [86, 85, 84, 83, 82, 81, 7f, 7e], 8e: [8e, 8c, 8b, 8a, 89, 88, 87], 93: [93, 92, 91, 90], a3: [a3, a2, a1, 9f, 9e, 9d, 9c, 9b, 9a, 99, 98, 94], af: [af, ae, ad, ac, ab, a9, a7, a5, a4], c6: [c6, c5, c4, c2, c1, c0, bf, be, bd, bc, ba, b9, b8, b7, b5, b3, b2, b0], d6: [d6, d5, d4, d3, d2, d1, d0, cf, ce, cc, cb, ca, c9, c8]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3781641, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2421213, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:56.966569+0000", 2024-09-21T08:32:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 5.5106489520000004, 2024-09-21T08:32:02.521 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039937740000000003, 2024-09-21T08:32:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.522 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.966569+0000", 2024-09-21T08:32:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.966569+0000", 2024-09-21T08:32:02.523 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.524 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.524 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.966570+0000", 2024-09-21T08:32:02.524 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0299999999999997e-07 2024-09-21T08:32:02.524 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.524 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.966796+0000", 2024-09-21T08:32:02.525 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00022649599999999999 2024-09-21T08:32:02.525 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.525 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.525 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.525 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.966817+0000", 2024-09-21T08:32:02.526 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0743000000000001e-05 2024-09-21T08:32:02.526 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.526 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.526 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.526 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.966839+0000", 2024-09-21T08:32:02.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1345999999999999e-05 2024-09-21T08:32:02.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.527 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.527 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.527 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.966978+0000", 2024-09-21T08:32:02.527 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000139713 2024-09-21T08:32:02.527 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.528 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.967002+0000", 2024-09-21T08:32:02.528 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3983999999999999e-05 2024-09-21T08:32:02.528 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.528 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.528 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.529 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:56.970563+0000", 2024-09-21T08:32:02.529 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035605889999999999 2024-09-21T08:32:02.529 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.529 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.529 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.529 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.530 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.530 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "pg_update_log_missing(4.5 epoch 524/523 rep_tid 198 entries 524'25 (0'0) error 4:a60f9269:lc::lua_package_allowlist:head by client.44387.0:2057 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-21T08:32:02.530 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:57.655996+0000", 2024-09-21T08:32:02.530 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8212224700000004, 2024-09-21T08:32:02.530 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048934460000000001, 2024-09-21T08:32:02.530 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.530 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.531 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:32:02.531 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.531 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.531 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.531 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.655996+0000", 2024-09-21T08:32:02.531 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.532 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.532 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.532 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.655996+0000", 2024-09-21T08:32:02.532 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.532 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.533 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.655996+0000", 2024-09-21T08:32:02.533 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.5400000000000002e-07 2024-09-21T08:32:02.533 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.533 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.533 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656002+0000", 2024-09-21T08:32:02.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2060000000000004e-06 2024-09-21T08:32:02.534 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.534 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.534 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.534 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656019+0000", 2024-09-21T08:32:02.534 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7265000000000002e-05 2024-09-21T08:32:02.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.535 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.535 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656033+0000", 2024-09-21T08:32:02.535 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4083999999999999e-05 2024-09-21T08:32:02.535 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.535 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656536+0000", 2024-09-21T08:32:02.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00050254 2024-09-21T08:32:02.536 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.536 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.536 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.536 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.660889+0000", 2024-09-21T08:32:02.536 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0043529969999999999 2024-09-21T08:32:02.536 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.536 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.537 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.537 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.537 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.537 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "pg_update_log_missing(4.5 epoch 524/523 rep_tid 199 entries 524'26 (0'0) error 4:a60f9269:lc::lua_package_allowlist:head by client.44387.0:2064 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-21T08:32:02.537 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:57.656052+0000", 2024-09-21T08:32:02.537 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8211655359999996, 2024-09-21T08:32:02.537 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048488250000000002, 2024-09-21T08:32:02.537 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.537 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.537 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656052+0000", 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656052+0000", 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.538 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656053+0000", 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7100000000000002e-07 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656058+0000", 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9439999999999996e-06 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.539 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656066+0000", 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.9799999999999998e-06 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656077+0000", 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1253999999999999e-05 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.656734+0000", 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00065675 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.540 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.660901+0000", 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0041674260000000001 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "pg_update_log_missing(5.1f epoch 524/523 rep_tid 521 entries 524'22 (0'0) error 5:f95f44c2:::notify.0:head by client.44387.0:2055 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:57.661935+0000", 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8152833939999997, 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0062995300000000002, 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.541 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.661935+0000", 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.661935+0000", 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.661935+0000", 2024-09-21T08:32:02.542 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8500000000000001e-07 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.661943+0000", 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8159999999999994e-06 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.661959+0000", 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6257e-05 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.661974+0000", 2024-09-21T08:32:02.543 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4918e-05 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.662040+0000", 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5290999999999995e-05 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.668234+0000", 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061945630000000002 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.544 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.545 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "pg_update_log_missing(5.1f epoch 524/523 rep_tid 522 entries 524'23 (0'0) error 5:f95f44c2:::notify.0:head by client.44387.0:2062 0.000000 0 ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 0'0)", 2024-09-21T08:32:02.545 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:57.662133+0000", 2024-09-21T08:32:02.545 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.8150845269999998, 2024-09-21T08:32:02.545 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0061114150000000002, 2024-09-21T08:32:02.545 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.545 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.545 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:32:02.545 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.545 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.545 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.662133+0000", 2024-09-21T08:32:02.546 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.546 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.546 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.546 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.546 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.662133+0000", 2024-09-21T08:32:02.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.547 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.547 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.662134+0000", 2024-09-21T08:32:02.547 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3099999999999996e-07 2024-09-21T08:32:02.547 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.547 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.662145+0000", 2024-09-21T08:32:02.548 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0389e-05 2024-09-21T08:32:02.548 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.548 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.548 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.548 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.662156+0000", 2024-09-21T08:32:02.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1429e-05 2024-09-21T08:32:02.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.549 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.549 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.549 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.662167+0000", 2024-09-21T08:32:02.549 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0781000000000001e-05 2024-09-21T08:32:02.549 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.550 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.662217+0000", 2024-09-21T08:32:02.550 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0065000000000003e-05 2024-09-21T08:32:02.550 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.550 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.550 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.551 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.668245+0000", 2024-09-21T08:32:02.551 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00602792 2024-09-21T08:32:02.551 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.551 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.551 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.551 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.552 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.552 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:5352 7.25 e524/523 7:a6c204bd:::smithi13779221-42:head v 524'344, mlcod=515'338)", 2024-09-21T08:32:02.552 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:57.694258+0000", 2024-09-21T08:32:02.552 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.78296002, 2024-09-21T08:32:02.552 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003897275, 2024-09-21T08:32:02.552 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.552 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.553 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:02.553 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.553 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.553 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.553 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.694258+0000", 2024-09-21T08:32:02.553 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.554 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.554 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.554 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.694258+0000", 2024-09-21T08:32:02.554 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.554 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.555 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.694260+0000", 2024-09-21T08:32:02.555 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8729999999999999e-06 2024-09-21T08:32:02.555 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.555 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.555 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.694405+0000", 2024-09-21T08:32:02.556 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000145635 2024-09-21T08:32:02.556 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.556 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.556 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.556 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.694416+0000", 2024-09-21T08:32:02.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0655999999999999e-05 2024-09-21T08:32:02.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.557 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.557 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.557 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.694421+0000", 2024-09-21T08:32:02.557 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.1379999999999998e-06 2024-09-21T08:32:02.557 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.558 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.558 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.694828+0000", 2024-09-21T08:32:02.558 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00040670600000000001 2024-09-21T08:32:02.558 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.558 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.559 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.694854+0000", 2024-09-21T08:32:02.559 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5636999999999999e-05 2024-09-21T08:32:02.559 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.559 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.559 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:02.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.698113+0000", 2024-09-21T08:32:02.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0032598219999999999 2024-09-21T08:32:02.560 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.560 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.560 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.560 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.698155+0000", 2024-09-21T08:32:02.560 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1807999999999997e-05 2024-09-21T08:32:02.561 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.561 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.561 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.561 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.561 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.561 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.9 524/523 [PushOp(7:93168146:::smithi13779221-44:b1, version: 427'351, data_included: [2454371~286402,3535053~381044], data_size: 667446, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:93168146:::smithi13779221-44:b1@427'351, size: 3916097, copy_subset: [(0, 3916097)], clone_subset: {}, snapset: e1=[]:{b1: [b1, b0, af, ae, ad, ac, a9, a7, a5, a4, a3, 9c, 9b, 99, 94, 93, 92, 84], b9: [b9, b8, b7, b6, b5, b3, b2], d8: [d8, d7, d6, d5, d4, d3, d2, d1, d0, cf, ce, cb, ca, c9, c8, c5, c2, c1, c0, be], df: [df, dd, dc, db, da, d9], e1: [e1, e0]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3916097, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2454371, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.562 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:57.766815+0000", 2024-09-21T08:32:02.562 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.7104034510000004, 2024-09-21T08:32:02.562 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034169529999999999, 2024-09-21T08:32:02.562 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.562 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.562 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.563 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.563 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.563 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.563 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.766815+0000", 2024-09-21T08:32:02.563 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.564 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.564 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.766815+0000", 2024-09-21T08:32:02.564 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.564 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.564 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.565 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.766816+0000", 2024-09-21T08:32:02.565 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2279999999999999e-06 2024-09-21T08:32:02.565 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.565 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.565 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.767442+0000", 2024-09-21T08:32:02.566 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00062643300000000002 2024-09-21T08:32:02.566 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.566 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.566 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.566 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.767466+0000", 2024-09-21T08:32:02.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4172000000000002e-05 2024-09-21T08:32:02.567 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.567 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.567 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.567 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.767492+0000", 2024-09-21T08:32:02.567 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5372e-05 2024-09-21T08:32:02.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.568 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.568 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.568 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.767583+0000", 2024-09-21T08:32:02.568 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1252e-05 2024-09-21T08:32:02.568 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.767594+0000", 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.096e-05 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.770232+0000", 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0026375359999999998 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.569 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.28 524/523 [PushOp(7:144c2ec5:::smithi13779221-45:head, version: 519'434, data_included: [601882~484309,1600633~524904], data_size: 1009213, omap_header_size: 0, omap_entries_size: 0, attrset_size: 3, recovery_info: ObjectRecoveryInfo(7:144c2ec5:::smithi13779221-45:head@519'434, size: 2201882, copy_subset: [(601882, 484309), (1600633, 524904)], clone_subset: {7:144c2ec5:::smithi13779221-45:e3: [(0, 601882), (1086191, 514442), (2125537, 76345)]}, snapset: 0=[]:{}, object_exist: true), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2125537, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(first, data_recovered_to: 0, data_complete: false, omap_recovered_to: , omap_complete: false, error: false))])", 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:57.806062+0000", 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.6711558909999997, 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0038065429999999999, 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.806062+0000", 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.570 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.806062+0000", 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.806064+0000", 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6360000000000001e-06 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.806290+0000", 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000226226 2024-09-21T08:32:02.571 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.806307+0000", 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7110000000000001e-05 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.806325+0000", 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8323999999999999e-05 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.806411+0000", 2024-09-21T08:32:02.572 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5650999999999997e-05 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.806423+0000", 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1499e-05 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.809869+0000", 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003446097 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.573 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.16 524/523 [PushOp(7:6a4067f0:::smithi13779221-23:e2, version: 512'220, data_included: [2282381~215426,2963697~730968], data_size: 946394, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:6a4067f0:::smithi13779221-23:e2@512'220, size: 3697672, copy_subset: [(497672, 561427), (1795232, 702575), (2963697, 730968)], clone_subset: {7:6a4067f0:::smithi13779221-23:d6: [(0, 497672), (1059099, 736133), (2497807, 465890), (3694665, 3007)]}, snapset: e2=[]:{a6: [a5, 9b, 92, 84], b0: [b0, af], c0: [c0, b9, b8, b3, b2], d6: [d6, d4, d3, d2, d1, cb, ca, c9, c5, c2, c1], e2: [e2, e1, e0, df, dd, dc, db, da, d9, d7]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 3694665, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2282381, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:57.848160+0000", 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.629057564, 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003827076, 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.848160+0000", 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.574 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.848160+0000", 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.848162+0000", 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1519999999999999e-06 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.848395+0000", 2024-09-21T08:32:02.575 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023347600000000001 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.848415+0000", 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9609000000000001e-05 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.848430+0000", 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4917e-05 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.848513+0000", 2024-09-21T08:32:02.576 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.3341e-05 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.848525+0000", 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1885000000000001e-05 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.851988+0000", 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0034626959999999999 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.577 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "MOSDPGPush(7.7 524/523 [PushOp(7:e1536981:::smithi13779221-32:d6, version: 492'201, data_included: [1667318~618566], data_size: 618566, omap_header_size: 0, omap_entries_size: 0, attrset_size: 0, recovery_info: ObjectRecoveryInfo(7:e1536981:::smithi13779221-32:d6@492'201, size: 2285884, copy_subset: [(0, 2285884)], clone_subset: {7:e1536981:::smithi13779221-32:b5: []}, snapset: e6=[]:{96: [92], 97: [97], b5: [b5, b4, b3, b2, b1, b0, af, ae, ad, ac, a7, a5, a4, a3, 9b, 99], d6: [d6, d5, d4, d3, d2, d1, d0, cf, ce, cc, cb, ca, c9, c8, c5, c2, c1, c0, be, b9, b8], db: [db, da, d9, d8, d7], dc: [dc], e6: [e5, e4, e2, e1, e0, df, dd]}, object_exist: false), after_progress: ObjectRecoveryProgress(!first, data_recovered_to: 2285884, data_complete: true, omap_recovered_to: , omap_complete: true, error: false), before_progress: ObjectRecoveryProgress(!first, data_recovered_to: 1667318, data_complete: false, omap_recovered_to: , omap_complete: true, error: false))])", 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:57.865667+0000", 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 4.6115511119999999, 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035871089999999998, 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.865667+0000", 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.578 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.865667+0000", 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.865667+0000", 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6000000000000004e-07 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.866101+0000", 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00043343899999999998 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.579 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.866117+0000", 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.6075000000000001e-05 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.866132+0000", 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4601e-05 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.866217+0000", 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5567000000000003e-05 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.580 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.866228+0000", 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1131000000000001e-05 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:57.869254+0000", 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0030257359999999998 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.581 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:5381 7.28 e528/525 7:17170b1f:::smithi13779221-4:head v 528'445, mlcod=0'0)", 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:32:01.621883+0000", 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.85533527399999998, 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0037147769999999998, 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:01.621883+0000", 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.582 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:01.621883+0000", 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:01.621884+0000", 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.344e-06 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:01.621890+0000", 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3790000000000003e-06 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.583 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:01.621902+0000", 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1474000000000001e-05 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:01.621910+0000", 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.0460000000000006e-06 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:01.621962+0000", 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.2142000000000003e-05 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:02.584 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:01.621985+0000", 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2738000000000002e-05 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:01.624621+0000", 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.002636518 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:01.625598+0000", 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00097613599999999995 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:02.585 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:02.586 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:32:02.586 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:02.608 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:02.609 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:02.609 INFO:tasks.rados.rados.0.smithi137.stdout:2105: done (2 left) 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2106: read oid 5 snap -1 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2106: expect deleted 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2107: snap_create 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2103: finishing write tid 2 to smithi13779221-33 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2104: finishing rollback tid 0 to smithi13779221-4 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2103: finishing write tid 3 to smithi13779221-33 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2103: finishing write tid 4 to smithi13779221-33 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2103: oid 33 updating version 0 to 359 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2103: oid 33 updating version 359 to 360 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2103: oid 33 version 360 is already newer than 358 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 360 (ObjNum 738 snap 233 seq_num 738) dirty exists 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2103: left oid 33 (ObjNum 738 snap 233 seq_num 738) 2024-09-21T08:32:02.610 INFO:tasks.rados.rados.0.smithi137.stdout:2104: finishing rollback tid 1 to smithi13779221-4 2024-09-21T08:32:02.611 INFO:tasks.rados.rados.0.smithi137.stdout:2104: finishing rollback tid 2 to smithi13779221-4 2024-09-21T08:32:02.611 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 4 v 460 (ObjNum 615 snap 180 seq_num 615) dirty exists 2024-09-21T08:32:02.611 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:02.612 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:32:02.622 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:32:02.712 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:02.733 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:02.733 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:02.734 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:02.790 INFO:teuthology.orchestra.run.smithi137.stderr:no valid command found; 10 closest matches: 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:0 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:1 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:2 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:abort 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:assert 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:config diff 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:config diff get 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:config get 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:config help [] 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:config set ... 2024-09-21T08:32:02.791 INFO:teuthology.orchestra.run.smithi137.stderr:admin_socket: invalid command 2024-09-21T08:32:02.835 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:02.857 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:02.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:02.859 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:02.957 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:02 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/439324436' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:02.957 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:02 smithi165 ceph-mon[32893]: from='client.46010 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:02.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:02 smithi165 ceph-mon[32893]: from='client.46010 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:02.958 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:02 smithi165 ceph-mon[32893]: osdmap e529: 8 total, 7 up, 5 in 2024-09-21T08:32:02.959 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:02.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:02.983 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:02.983 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:02 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/439324436' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:02 smithi137 ceph-mon[41044]: from='client.46010 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:02 smithi137 ceph-mon[41044]: from='client.46010 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:03.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:02 smithi137 ceph-mon[41044]: osdmap e529: 8 total, 7 up, 5 in 2024-09-21T08:32:03.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:02 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/439324436' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:03.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:02 smithi137 ceph-mon[29634]: from='client.46010 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:03.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:02 smithi137 ceph-mon[29634]: from='client.46010 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:03.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:02 smithi137 ceph-mon[29634]: osdmap e529: 8 total, 7 up, 5 in 2024-09-21T08:32:03.083 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:03.107 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:03.107 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:03.107 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:03.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:03.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:03.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:03.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:03.331 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:03.353 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:03.354 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:03.354 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:03.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_ops_in_flight 2024-09-21T08:32:03.454 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:03.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:03.478 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:03.478 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:03.479 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-09-21T08:32:03.479 INFO:tasks.cephadm.ceph_manager.ceph:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2024-09-21T08:32:03.579 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:03.602 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:03.603 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:03.603 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:03.610 INFO:tasks.rados.rados.0.smithi137.stdout:2103: done (3 left) 2024-09-21T08:32:03.610 INFO:tasks.rados.rados.0.smithi137.stdout:2104: done (2 left) 2024-09-21T08:32:03.610 INFO:tasks.rados.rados.0.smithi137.stdout:2106: done (1 left) 2024-09-21T08:32:03.611 INFO:tasks.rados.rados.0.smithi137.stdout:2107: done (0 left) 2024-09-21T08:32:03.611 INFO:tasks.rados.rados.0.smithi137.stdout:2108: read oid 49 snap -1 2024-09-21T08:32:03.611 INFO:tasks.rados.rados.0.smithi137.stdout:2108: expect (ObjNum 735 snap 230 seq_num 735) 2024-09-21T08:32:03.611 INFO:tasks.rados.rados.0.smithi137.stdout:2109: write oid 40 current snap is 234 2024-09-21T08:32:03.611 INFO:tasks.rados.rados.0.smithi137.stdout:2109: seq_num 739 ranges {420373=495252,1444546=690576} 2024-09-21T08:32:03.616 INFO:tasks.rados.rados.0.smithi137.stdout:2109: writing smithi13779221-40 from 420373 to 915625 tid 1 2024-09-21T08:32:03.625 INFO:tasks.rados.rados.0.smithi137.stdout:2109: writing smithi13779221-40 from 1444546 to 2135122 tid 2 2024-09-21T08:32:03.625 INFO:tasks.rados.rados.0.smithi137.stdout:2110: write (excl) oid 45 current snap is 234 2024-09-21T08:32:03.625 INFO:tasks.rados.rados.0.smithi137.stdout:2110: seq_num 740 ranges {437180=570501,1503028=550587} 2024-09-21T08:32:03.630 INFO:tasks.rados.rados.0.smithi137.stdout:2110: writing smithi13779221-45 from 437180 to 1007681 tid 1 2024-09-21T08:32:03.637 INFO:tasks.rados.rados.0.smithi137.stdout:2110: writing smithi13779221-45 from 1503028 to 2053615 tid 2 2024-09-21T08:32:03.637 INFO:tasks.rados.rados.0.smithi137.stdout:2111: snap_create 2024-09-21T08:32:03.637 INFO:tasks.rados.rados.0.smithi137.stdout:2108: expect (ObjNum 735 snap 230 seq_num 735) 2024-09-21T08:32:03.704 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:03.725 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:03.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:03.726 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:2109: finishing write tid 1 to smithi13779221-40 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:2109: finishing write tid 2 to smithi13779221-40 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:2109: finishing write tid 3 to smithi13779221-40 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:2109: finishing write tid 4 to smithi13779221-40 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:2109: oid 40 updating version 0 to 406 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:2109: oid 40 updating version 406 to 407 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:2109: oid 40 version 407 is already newer than 405 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 407 (ObjNum 739 snap 234 seq_num 739) dirty exists 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:2109: left oid 40 (ObjNum 739 snap 234 seq_num 739) 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:2110: finishing write tid 1 to smithi13779221-45 2024-09-21T08:32:03.748 INFO:tasks.rados.rados.0.smithi137.stdout:2110: finishing write tid 2 to smithi13779221-45 2024-09-21T08:32:03.749 INFO:tasks.rados.rados.0.smithi137.stdout:2110: finishing write tid 3 to smithi13779221-45 2024-09-21T08:32:03.749 INFO:tasks.rados.rados.0.smithi137.stdout:2110: finishing write tid 4 to smithi13779221-45 2024-09-21T08:32:03.749 INFO:tasks.rados.rados.0.smithi137.stdout:2110: oid 45 updating version 0 to 463 2024-09-21T08:32:03.749 INFO:tasks.rados.rados.0.smithi137.stdout:2110: oid 45 version 463 is already newer than 461 2024-09-21T08:32:03.749 INFO:tasks.rados.rados.0.smithi137.stdout:2110: oid 45 version 463 is already newer than 462 2024-09-21T08:32:03.749 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 45 v 463 (ObjNum 740 snap 234 seq_num 740) dirty exists 2024-09-21T08:32:03.749 INFO:tasks.rados.rados.0.smithi137.stdout:2110: left oid 45 (ObjNum 740 snap 234 seq_num 740) 2024-09-21T08:32:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:03 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:03 smithi165 ceph-mon[32893]: pgmap v698: 188 pgs: 5 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 153 active+clean; 613 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 20 MiB/s rd, 11 op/s; 92 MiB/s, 78 objects/s recovering 2024-09-21T08:32:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:03 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:32:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:03 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:32:03.763 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:03 smithi165 ceph-mon[32893]: osdmap e530: 8 total, 7 up, 5 in 2024-09-21T08:32:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-mon[41044]: pgmap v698: 188 pgs: 5 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 153 active+clean; 613 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 20 MiB/s rd, 11 op/s; 92 MiB/s, 78 objects/s recovering 2024-09-21T08:32:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:32:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:32:03.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-mon[41044]: osdmap e530: 8 total, 7 up, 5 in 2024-09-21T08:32:03.768 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0[157386]: 2024-09-21T08:32:03.387+0000 7f720f24d640 -1 Falling back to public interface 2024-09-21T08:32:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-mon[29634]: pgmap v698: 188 pgs: 5 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 153 active+clean; 613 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 20 MiB/s rd, 11 op/s; 92 MiB/s, 78 objects/s recovering 2024-09-21T08:32:03.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:32:03.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:32:03.769 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:03 smithi137 ceph-mon[29634]: osdmap e530: 8 total, 7 up, 5 in 2024-09-21T08:32:03.827 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:03.849 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:03.849 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:03.849 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:03.950 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:03.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:03.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:03.972 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:04.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:04.097 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:04.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:04.098 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:04.123 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:04.123 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:04.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:04.221 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:04.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:04.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:04.322 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:04.344 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:04.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:04.345 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:04.446 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:04.468 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:04.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:04.469 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:04.570 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:04.592 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:04.593 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:04.593 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:04.636 INFO:tasks.rados.rados.0.smithi137.stdout:2108: done (3 left) 2024-09-21T08:32:04.637 INFO:tasks.rados.rados.0.smithi137.stdout:2109: done (2 left) 2024-09-21T08:32:04.637 INFO:tasks.rados.rados.0.smithi137.stdout:2110: done (1 left) 2024-09-21T08:32:04.637 INFO:tasks.rados.rados.0.smithi137.stdout:2111: done (0 left) 2024-09-21T08:32:04.637 INFO:tasks.rados.rados.0.smithi137.stdout:2112: snap_remove snap 222 2024-09-21T08:32:04.694 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:04.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:04.717 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:04.717 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:04.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:04.843 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:04.843 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:04.844 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:04.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:04 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:32:04.922 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:04 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:32:04.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:04 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:32:04.922 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:04 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:32:04.945 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:04.967 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:04.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:04.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:05.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:04 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:32:05.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:04 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:32:05.030 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:32:05.030 INFO:teuthology.orchestra.run.smithi137.stderr:Error ENXIO: problem getting command descriptions from osd.0 2024-09-21T08:32:05.030 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:32:05.042 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:32:05.042 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:32:05.058 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:32:05.058 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:32:05.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:05.075 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:32:05.075 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:32:05.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:05.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:05.091 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:05.092 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:32:05.092 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:32:05.109 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:32:05.110 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:32:05.126 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:32:05.126 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:32:05.192 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:05.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:05.219 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:05.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:05.249 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:32:05.249 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:32:05.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:05.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:05.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:05.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:05.447 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:05.472 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:05.473 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:05.473 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:05.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:05.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:05.597 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:05.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:05.639 INFO:tasks.rados.rados.0.smithi137.stdout:2112: done (0 left) 2024-09-21T08:32:05.639 INFO:tasks.rados.rados.0.smithi137.stdout:2113: write oid 12 current snap is 235 2024-09-21T08:32:05.640 INFO:tasks.rados.rados.0.smithi137.stdout:2113: seq_num 741 ranges {453987=645750,1845157=608830} 2024-09-21T08:32:05.645 INFO:tasks.rados.rados.0.smithi137.stdout:2113: writing smithi13779221-12 from 453987 to 1099737 tid 1 2024-09-21T08:32:05.653 INFO:tasks.rados.rados.0.smithi137.stdout:2113: writing smithi13779221-12 from 1845157 to 2453987 tid 2 2024-09-21T08:32:05.654 INFO:tasks.rados.rados.0.smithi137.stdout:2114: write oid 40 current snap is 235 2024-09-21T08:32:05.654 INFO:tasks.rados.rados.0.smithi137.stdout:2114: seq_num 742 ranges {470794=720999,1903639=554256} 2024-09-21T08:32:05.660 INFO:tasks.rados.rados.0.smithi137.stdout:2114: writing smithi13779221-40 from 470794 to 1191793 tid 1 2024-09-21T08:32:05.667 INFO:tasks.rados.rados.0.smithi137.stdout:2114: writing smithi13779221-40 from 1903639 to 2457895 tid 2 2024-09-21T08:32:05.667 INFO:tasks.rados.rados.0.smithi137.stdout:2115: snap_remove snap 210 2024-09-21T08:32:05.698 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:05.721 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:05.722 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:05.722 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:05.823 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:05.845 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:05.845 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:05.846 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:05.946 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:05.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:05.968 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:05.968 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:05 smithi165 ceph-mon[32893]: pgmap v701: 188 pgs: 5 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 153 active+clean; 613 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 20 MiB/s rd, 12 op/s; 72 MiB/s, 55 objects/s recovering 2024-09-21T08:32:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:05 smithi165 ceph-mon[32893]: osdmap e531: 8 total, 7 up, 5 in 2024-09-21T08:32:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:05 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/995918499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:06.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:05 smithi165 ceph-mon[32893]: osdmap e532: 8 total, 7 up, 5 in 2024-09-21T08:32:06.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:05 smithi137 ceph-mon[29634]: pgmap v701: 188 pgs: 5 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 153 active+clean; 613 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 20 MiB/s rd, 12 op/s; 72 MiB/s, 55 objects/s recovering 2024-09-21T08:32:06.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:05 smithi137 ceph-mon[29634]: osdmap e531: 8 total, 7 up, 5 in 2024-09-21T08:32:06.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:05 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/995918499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:06.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:05 smithi137 ceph-mon[29634]: osdmap e532: 8 total, 7 up, 5 in 2024-09-21T08:32:06.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:05 smithi137 ceph-mon[41044]: pgmap v701: 188 pgs: 5 active+clean+snaptrim, 30 active+clean+snaptrim_wait, 153 active+clean; 613 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 20 MiB/s rd, 12 op/s; 72 MiB/s, 55 objects/s recovering 2024-09-21T08:32:06.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:05 smithi137 ceph-mon[41044]: osdmap e531: 8 total, 7 up, 5 in 2024-09-21T08:32:06.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:05 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/995918499' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:06.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:05 smithi137 ceph-mon[41044]: osdmap e532: 8 total, 7 up, 5 in 2024-09-21T08:32:06.050 DEBUG:teuthology.orchestra.run:got remote process result: 6 2024-09-21T08:32:06.050 INFO:tasks.thrashosds.thrasher:Failed to tell all osds, ignoring 2024-09-21T08:32:06.070 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:06.091 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:06.092 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:06.093 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:06.123 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:32:06.194 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:06.215 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:06.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:06.216 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:06.317 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:06.338 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:06.339 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:06.340 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:06.385 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:06.440 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:06.462 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:06.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:06.463 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:06.564 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:06.590 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:06.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:06.591 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:06.656 INFO:tasks.rados.rados.0.smithi137.stdout:2115: done (2 left) 2024-09-21T08:32:06.656 INFO:tasks.rados.rados.0.smithi137.stdout:2116: snap_remove snap 229 2024-09-21T08:32:06.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:06.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:06.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:06.715 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:06.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:06 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:06.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:06 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:06.817 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:06.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:06.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:06.842 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:06.943 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:06.967 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:06.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:06.968 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:07.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:06 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:07.069 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:07.094 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:07.094 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:07.095 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:07.145 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:06 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0[157386]: 2024-09-21T08:32:06.934+0000 7f720f24d640 -1 osd.0 369 log_to_monitors true 2024-09-21T08:32:07.195 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:07.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:07.218 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:07.218 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:07.319 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:07.345 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:07.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:07.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:07.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:07.471 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:07.472 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:07.472 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:07.522 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_blocked_ops 2024-09-21T08:32:07.573 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:07.597 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:07.598 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:07.599 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:07.653 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:32:07.659 INFO:tasks.rados.rados.0.smithi137.stdout:2116: done (2 left) 2024-09-21T08:32:07.659 INFO:tasks.rados.rados.0.smithi137.stdout:2117: read oid 20 snap -1 2024-09-21T08:32:07.659 INFO:tasks.rados.rados.0.smithi137.stdout:2117: expect (ObjNum 708 snap 218 seq_num 708) 2024-09-21T08:32:07.660 INFO:tasks.rados.rados.0.smithi137.stdout:2118: write oid 24 current snap is 235 2024-09-21T08:32:07.660 INFO:tasks.rados.rados.0.smithi137.stdout:2118: seq_num 743 ranges {487601=796248,1962121=525480} 2024-09-21T08:32:07.669 INFO:tasks.rados.rados.0.smithi137.stdout:2118: writing smithi13779221-24 from 487601 to 1283849 tid 1 2024-09-21T08:32:07.677 INFO:tasks.rados.rados.0.smithi137.stdout:2118: writing smithi13779221-24 from 1962121 to 2487601 tid 2 2024-09-21T08:32:07.677 INFO:tasks.rados.rados.0.smithi137.stdout:2119: copy_from oid 49 from oid 6 current snap is 235 2024-09-21T08:32:07.677 INFO:tasks.rados.rados.0.smithi137.stdout:2113: finishing write tid 1 to smithi13779221-12 2024-09-21T08:32:07.678 INFO:tasks.rados.rados.0.smithi137.stdout:2120: write oid 14 current snap is 235 2024-09-21T08:32:07.678 INFO:tasks.rados.rados.0.smithi137.stdout:2113: finishing write tid 2 to smithi13779221-12 2024-09-21T08:32:07.678 INFO:tasks.rados.rados.0.smithi137.stdout:2113: finishing write tid 3 to smithi13779221-12 2024-09-21T08:32:07.678 INFO:tasks.rados.rados.0.smithi137.stdout:2120: seq_num 744 ranges {504408=471497,1620603=557925} 2024-09-21T08:32:07.683 INFO:tasks.rados.rados.0.smithi137.stdout:2120: writing smithi13779221-14 from 504408 to 975905 tid 1 2024-09-21T08:32:07.691 INFO:tasks.rados.rados.0.smithi137.stdout:2120: writing smithi13779221-14 from 1620603 to 2178528 tid 2 2024-09-21T08:32:07.691 INFO:tasks.rados.rados.0.smithi137.stdout:2121: rollback oid 2 current snap is 235 2024-09-21T08:32:07.693 INFO:tasks.rados.rados.0.smithi137.stdout:2113: finishing write tid 4 to smithi13779221-12 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2113: oid 12 updating version 0 to 539 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2113: oid 12 updating version 539 to 541 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2113: oid 12 version 541 is already newer than 540 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 12 v 541 (ObjNum 741 snap 235 seq_num 741) dirty exists 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2113: left oid 12 (ObjNum 741 snap 235 seq_num 741) 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2114: finishing write tid 1 to smithi13779221-40 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2114: finishing write tid 2 to smithi13779221-40 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2114: finishing write tid 3 to smithi13779221-40 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2114: finishing write tid 4 to smithi13779221-40 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2114: oid 40 updating version 0 to 409 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2114: oid 40 version 409 is already newer than 408 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:2114: oid 40 updating version 409 to 410 2024-09-21T08:32:07.694 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 410 (ObjNum 742 snap 235 seq_num 742) dirty exists 2024-09-21T08:32:07.695 INFO:tasks.rados.rados.0.smithi137.stdout:2114: left oid 40 (ObjNum 742 snap 235 seq_num 742) 2024-09-21T08:32:07.695 INFO:tasks.rados.rados.0.smithi137.stdout:2117: expect (ObjNum 708 snap 218 seq_num 708) 2024-09-21T08:32:07.699 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:07.723 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:07.724 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:07.724 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:07.768 INFO:journalctl@ceph.osd.0.smithi137.stdout:Sep 21 08:32:07 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd-0[157386]: 2024-09-21T08:32:07.715+0000 7f7206ff8640 -1 osd.0 534 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-21T08:32:07.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:07 smithi137 ceph-mon[29634]: pgmap v704: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 607 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-21T08:32:07.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:07 smithi137 ceph-mon[29634]: osdmap e533: 8 total, 7 up, 5 in 2024-09-21T08:32:07.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:07 smithi137 ceph-mon[29634]: from='osd.0 v1:172.21.15.137:6801/865601427' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T08:32:07.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:07 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3998500860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:32:07.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:07 smithi137 ceph-mon[41044]: pgmap v704: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 607 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-21T08:32:07.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:07 smithi137 ceph-mon[41044]: osdmap e533: 8 total, 7 up, 5 in 2024-09-21T08:32:07.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:07 smithi137 ceph-mon[41044]: from='osd.0 v1:172.21.15.137:6801/865601427' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T08:32:07.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:07 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3998500860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:32:07.825 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:2119: finishing copy_from to smithi13779221-49 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:2119: got expected ENOENT (src dne) 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:2119: finishing copy_from racing read to smithi13779221-49 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:2118: finishing write tid 1 to smithi13779221-24 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:2118: finishing write tid 2 to smithi13779221-24 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:2118: finishing write tid 3 to smithi13779221-24 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:2118: finishing write tid 4 to smithi13779221-24 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:2118: oid 24 updating version 0 to 738 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:2118: oid 24 updating version 738 to 740 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:2118: oid 24 version 740 is already newer than 739 2024-09-21T08:32:07.832 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 740 (ObjNum 743 snap 235 seq_num 743) dirty exists 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2118: left oid 24 (ObjNum 743 snap 235 seq_num 743) 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2120: finishing write tid 1 to smithi13779221-14 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2120: finishing write tid 2 to smithi13779221-14 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2120: finishing write tid 3 to smithi13779221-14 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 2 to 153 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2120: finishing write tid 4 to smithi13779221-14 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2120: oid 14 updating version 0 to 404 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2120: oid 14 version 404 is already newer than 403 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2120: oid 14 version 404 is already newer than 402 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 14 v 404 (ObjNum 744 snap 235 seq_num 744) dirty exists 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2120: left oid 14 (ObjNum 744 snap 235 seq_num 744) 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2113: done (6 left) 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2114: done (5 left) 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2117: done (4 left) 2024-09-21T08:32:07.833 INFO:tasks.rados.rados.0.smithi137.stdout:2118: done (3 left) 2024-09-21T08:32:07.834 INFO:tasks.rados.rados.0.smithi137.stdout:2119: done (2 left) 2024-09-21T08:32:07.834 INFO:tasks.rados.rados.0.smithi137.stdout:2120: done (1 left) 2024-09-21T08:32:07.834 INFO:tasks.rados.rados.0.smithi137.stdout:2122: write oid 8 current snap is 235 2024-09-21T08:32:07.834 INFO:tasks.rados.rados.0.smithi137.stdout:2122: seq_num 745 ranges {521215=546746,1562732=701583} 2024-09-21T08:32:07.837 INFO:tasks.rados.rados.0.smithi137.stdout:2122: writing smithi13779221-8 from 521215 to 1067961 tid 1 2024-09-21T08:32:07.844 INFO:tasks.rados.rados.0.smithi137.stdout:2122: writing smithi13779221-8 from 1562732 to 2264315 tid 2 2024-09-21T08:32:07.844 INFO:tasks.rados.rados.0.smithi137.stdout:2123: copy_from oid 48 from oid 29 current snap is 235 2024-09-21T08:32:07.844 INFO:tasks.rados.rados.0.smithi137.stdout:2121: finishing rollback tid 0 to smithi13779221-2 2024-09-21T08:32:07.844 INFO:tasks.rados.rados.0.smithi137.stdout:2121: finishing rollback tid 1 to smithi13779221-2 2024-09-21T08:32:07.844 INFO:tasks.rados.rados.0.smithi137.stdout:2121: finishing rollback tid 2 to smithi13779221-2 2024-09-21T08:32:07.844 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 2 v 183 (ObjNum 508 snap 142 seq_num 508) dirty exists 2024-09-21T08:32:07.845 INFO:tasks.rados.rados.0.smithi137.stdout:2122: finishing write tid 1 to smithi13779221-8 2024-09-21T08:32:07.845 INFO:tasks.rados.rados.0.smithi137.stdout:2121: done (2 left) 2024-09-21T08:32:07.845 INFO:tasks.rados.rados.0.smithi137.stdout:2124: snap_create 2024-09-21T08:32:07.847 INFO:tasks.rados.rados.0.smithi137.stdout:2122: finishing write tid 2 to smithi13779221-8 2024-09-21T08:32:07.847 INFO:tasks.rados.rados.0.smithi137.stdout:2122: finishing write tid 3 to smithi13779221-8 2024-09-21T08:32:07.847 INFO:tasks.rados.rados.0.smithi137.stdout:2122: finishing write tid 4 to smithi13779221-8 2024-09-21T08:32:07.847 INFO:tasks.rados.rados.0.smithi137.stdout:2122: oid 8 updating version 0 to 465 2024-09-21T08:32:07.847 INFO:tasks.rados.rados.0.smithi137.stdout:2122: oid 8 updating version 465 to 466 2024-09-21T08:32:07.847 INFO:tasks.rados.rados.0.smithi137.stdout:2122: oid 8 version 466 is already newer than 464 2024-09-21T08:32:07.847 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 466 (ObjNum 745 snap 235 seq_num 745) dirty exists 2024-09-21T08:32:07.847 INFO:tasks.rados.rados.0.smithi137.stdout:2122: left oid 8 (ObjNum 745 snap 235 seq_num 745) 2024-09-21T08:32:07.848 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:07.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:07.849 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:07.853 INFO:tasks.rados.rados.0.smithi137.stdout:2123: finishing copy_from racing read to smithi13779221-48 2024-09-21T08:32:07.860 INFO:tasks.rados.rados.0.smithi137.stdout:2123: finishing copy_from to smithi13779221-48 2024-09-21T08:32:07.860 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 734 (ObjNum 566 snap 168 seq_num 566) dirty exists 2024-09-21T08:32:07.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:07.971 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:07.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:07.972 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:07 smithi165 ceph-mon[32893]: pgmap v704: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 607 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-21T08:32:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:07 smithi165 ceph-mon[32893]: osdmap e533: 8 total, 7 up, 5 in 2024-09-21T08:32:08.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:07 smithi165 ceph-mon[32893]: from='osd.0 v1:172.21.15.137:6801/865601427' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-09-21T08:32:08.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:07 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3998500860' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:32:08.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:32:08.073 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:08.097 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:08.098 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:08.098 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:08.199 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:08.221 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:08.222 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:08.222 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:08.322 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:08.345 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:08.346 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:08.346 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:08.354 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:08.446 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:08.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:08.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:08.469 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:08.480 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.0 dump_ops_in_flight 2024-09-21T08:32:08.569 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:08.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:08.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:08.592 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:08.671 INFO:tasks.rados.rados.0.smithi137.stdout:2122: done (2 left) 2024-09-21T08:32:08.671 INFO:tasks.rados.rados.0.smithi137.stdout:2123: done (1 left) 2024-09-21T08:32:08.671 INFO:tasks.rados.rados.0.smithi137.stdout:2124: done (0 left) 2024-09-21T08:32:08.671 INFO:tasks.rados.rados.0.smithi137.stdout:2125: delete oid 15 current snap is 236 2024-09-21T08:32:08.675 INFO:tasks.rados.rados.0.smithi137.stdout:2125: done (0 left) 2024-09-21T08:32:08.675 INFO:tasks.rados.rados.0.smithi137.stdout:2126: snap_remove snap 235 2024-09-21T08:32:08.692 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:08.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:08.715 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:08.716 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:08.816 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:08.839 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:08.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:08.840 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:08.843 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:08.857 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:32:08.858 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:32:08.878 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:32:08.878 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:32:08.879 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:08 smithi137 ceph-mon[29634]: from='osd.0 v1:172.21.15.137:6801/865601427' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T08:32:08.879 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:08 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3998500860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:32:08.879 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:08 smithi137 ceph-mon[29634]: osdmap e534: 8 total, 7 up, 5 in 2024-09-21T08:32:08.879 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:08 smithi137 ceph-mon[29634]: from='osd.0 v1:172.21.15.137:6801/865601427' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:32:08.879 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:08 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:32:08.879 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:08 smithi137 ceph-mon[41044]: from='osd.0 v1:172.21.15.137:6801/865601427' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T08:32:08.879 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:08 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3998500860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:32:08.879 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:08 smithi137 ceph-mon[41044]: osdmap e534: 8 total, 7 up, 5 in 2024-09-21T08:32:08.879 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:08 smithi137 ceph-mon[41044]: from='osd.0 v1:172.21.15.137:6801/865601427' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:32:08.879 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:08 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:32:08.895 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:32:08.895 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:32:08.911 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:32:08.912 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:32:08.927 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:32:08.928 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:32:08.941 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:08.944 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:32:08.944 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:32:08.960 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:32:08.960 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:32:08.961 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:08.962 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:08.962 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:08.977 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:32:08.977 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:32:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:08 smithi165 ceph-mon[32893]: from='osd.0 v1:172.21.15.137:6801/865601427' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-09-21T08:32:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:08 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3998500860' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:32:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:08 smithi165 ceph-mon[32893]: osdmap e534: 8 total, 7 up, 5 in 2024-09-21T08:32:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:08 smithi165 ceph-mon[32893]: from='osd.0 v1:172.21.15.137:6801/865601427' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi137", "root=default"]}]: dispatch 2024-09-21T08:32:09.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:08 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:32:09.064 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:09.085 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:09.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:09.086 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:09.187 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:09.209 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:09.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:09.209 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:09.310 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:09.331 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:09.332 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:09.332 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:09.433 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:09.455 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:09.456 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:09.456 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:09.557 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:09.579 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:09.579 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:09.580 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:09.675 INFO:tasks.rados.rados.0.smithi137.stdout:2126: done (0 left) 2024-09-21T08:32:09.675 INFO:tasks.rados.rados.0.smithi137.stdout:2127: snap_remove snap 208 2024-09-21T08:32:09.680 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:09.704 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:09.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:09.705 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:09.806 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:09.827 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:09.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:09.828 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:09.929 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:09.951 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:09.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:09.952 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:09.994 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: 7.2b scrub starts 2024-09-21T08:32:09.994 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: 7.2b scrub ok 2024-09-21T08:32:09.994 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: pgmap v707: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 607 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-21T08:32:09.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: 5.0 scrub starts 2024-09-21T08:32:09.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: 5.0 scrub ok 2024-09-21T08:32:09.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: 7.2c scrub starts 2024-09-21T08:32:09.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:32:09.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: osd.0 v1:172.21.15.137:6801/865601427 boot 2024-09-21T08:32:09.995 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: osdmap e535: 8 total, 8 up, 5 in 2024-09-21T08:32:09.996 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: 7.2c scrub ok 2024-09-21T08:32:09.996 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2470985280' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:09.996 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: 6.0 scrub starts 2024-09-21T08:32:09.996 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: 6.0 scrub ok 2024-09-21T08:32:09.996 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: 7.6 scrub starts 2024-09-21T08:32:09.996 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[41044]: 7.6 scrub ok 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: 7.2b scrub starts 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: 7.2b scrub ok 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: pgmap v707: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 607 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: 5.0 scrub starts 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: 5.0 scrub ok 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: 7.2c scrub starts 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: osd.0 v1:172.21.15.137:6801/865601427 boot 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: osdmap e535: 8 total, 8 up, 5 in 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: 7.2c scrub ok 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2470985280' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:09.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: 6.0 scrub starts 2024-09-21T08:32:09.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: 6.0 scrub ok 2024-09-21T08:32:09.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: 7.6 scrub starts 2024-09-21T08:32:09.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:09 smithi137 ceph-mon[29634]: 7.6 scrub ok 2024-09-21T08:32:09.998 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:32:09.998 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:32:09.998 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:32:09.998 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:32:09.998 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:32:10.001 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: 7.2b scrub starts 2024-09-21T08:32:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: 7.2b scrub ok 2024-09-21T08:32:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: pgmap v707: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 607 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 2.8 MiB/s rd, 1.1 MiB/s wr, 9 op/s 2024-09-21T08:32:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: 5.0 scrub starts 2024-09-21T08:32:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: 5.0 scrub ok 2024-09-21T08:32:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: 7.2c scrub starts 2024-09-21T08:32:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-09-21T08:32:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: osd.0 v1:172.21.15.137:6801/865601427 boot 2024-09-21T08:32:10.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: osdmap e535: 8 total, 8 up, 5 in 2024-09-21T08:32:10.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: 7.2c scrub ok 2024-09-21T08:32:10.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2470985280' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:10.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: 6.0 scrub starts 2024-09-21T08:32:10.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: 6.0 scrub ok 2024-09-21T08:32:10.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: 7.6 scrub starts 2024-09-21T08:32:10.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:09 smithi165 ceph-mon[32893]: 7.6 scrub ok 2024-09-21T08:32:10.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:10.074 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:10.075 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:10.075 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:10.176 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:10.197 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:10.197 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:10.198 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:10.299 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:10.304 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:32:10.320 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:10.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:10.320 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:10.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:10.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:10.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:10.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:10.548 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:10.569 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:10.570 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:10.570 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:10.671 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:10.676 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:10 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:32:10.677 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:10 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:32:10.677 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:10 smithi137 ceph-mon[29634]: osdmap e536: 8 total, 8 up, 5 in 2024-09-21T08:32:10.681 INFO:tasks.rados.rados.0.smithi137.stdout:2127: done (0 left) 2024-09-21T08:32:10.681 INFO:tasks.rados.rados.0.smithi137.stdout:2128: copy_from oid 20 from oid 46 current snap is 236 2024-09-21T08:32:10.681 INFO:tasks.rados.rados.0.smithi137.stdout:2129: write (excl) oid 35 current snap is 236 2024-09-21T08:32:10.681 INFO:tasks.rados.rados.0.smithi137.stdout:2129: seq_num 747 ranges {554829=413597,1396049=705252} 2024-09-21T08:32:10.686 INFO:tasks.rados.rados.0.smithi137.stdout:2129: writing smithi13779221-35 from 554829 to 968426 tid 1 2024-09-21T08:32:10.693 INFO:tasks.rados.rados.0.smithi137.stdout:2129: writing smithi13779221-35 from 1396049 to 2101301 tid 2 2024-09-21T08:32:10.693 INFO:tasks.rados.rados.0.smithi137.stdout:2130: copy_from oid 22 from oid 45 current snap is 236 2024-09-21T08:32:10.693 INFO:tasks.rados.rados.0.smithi137.stdout:2129: finishing write tid 1 to smithi13779221-35 2024-09-21T08:32:10.694 INFO:tasks.rados.rados.0.smithi137.stdout:2131: write oid 1 current snap is 236 2024-09-21T08:32:10.694 INFO:tasks.rados.rados.0.smithi137.stdout:2131: seq_num 748 ranges {571636=488846,1854531=565263} 2024-09-21T08:32:10.699 INFO:tasks.rados.rados.0.smithi137.stdout:2131: writing smithi13779221-1 from 571636 to 1060482 tid 1 2024-09-21T08:32:10.706 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:2131: writing smithi13779221-1 from 1854531 to 2419794 tid 2 2024-09-21T08:32:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:2132: write oid 32 current snap is 236 2024-09-21T08:32:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:2129: finishing write tid 2 to smithi13779221-35 2024-09-21T08:32:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:2129: finishing write tid 3 to smithi13779221-35 2024-09-21T08:32:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:2129: finishing write tid 4 to smithi13779221-35 2024-09-21T08:32:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:2129: oid 35 updating version 0 to 750 2024-09-21T08:32:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:2129: oid 35 updating version 750 to 751 2024-09-21T08:32:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:2129: oid 35 version 751 is already newer than 749 2024-09-21T08:32:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 751 (ObjNum 747 snap 236 seq_num 747) dirty exists 2024-09-21T08:32:10.707 INFO:tasks.rados.rados.0.smithi137.stdout:2129: left oid 35 (ObjNum 747 snap 236 seq_num 747) 2024-09-21T08:32:10.708 INFO:tasks.rados.rados.0.smithi137.stdout:2128: finishing copy_from racing read to smithi13779221-20 2024-09-21T08:32:10.708 INFO:tasks.rados.rados.0.smithi137.stdout:2131: finishing write tid 1 to smithi13779221-1 2024-09-21T08:32:10.708 INFO:tasks.rados.rados.0.smithi137.stdout:2132: seq_num 749 ranges {588443=564095,1796660=708921} 2024-09-21T08:32:10.708 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:10.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:10.713 INFO:tasks.rados.rados.0.smithi137.stdout:2132: writing smithi13779221-32 from 588443 to 1152538 tid 1 2024-09-21T08:32:10.721 INFO:tasks.rados.rados.0.smithi137.stdout:2132: writing smithi13779221-32 from 1796660 to 2505581 tid 2 2024-09-21T08:32:10.721 INFO:tasks.rados.rados.0.smithi137.stdout:2129: done (4 left) 2024-09-21T08:32:10.721 INFO:tasks.rados.rados.0.smithi137.stdout:2133: snap_remove snap 144 2024-09-21T08:32:10.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:10.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:10.832 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:10.832 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:10.870 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.1 dump_historic_ops 2024-09-21T08:32:10.932 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:32:10 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:32:10] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:32:10.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:10 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:32:10.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:10 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:32:10.932 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:10 smithi137 ceph-mon[41044]: osdmap e536: 8 total, 8 up, 5 in 2024-09-21T08:32:10.932 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:10.953 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:10.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:10.954 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:10 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:32:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:10 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:32:11.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:10 smithi165 ceph-mon[32893]: osdmap e536: 8 total, 8 up, 5 in 2024-09-21T08:32:11.055 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:11.077 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:11.078 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:11.078 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:11.119 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:32:11.119 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:32:11.119 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:32:11.119 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:32:11.179 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:11.200 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:11.200 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:11.201 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:11.301 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:11.330 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:11.331 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:11.331 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:11.432 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:11.453 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:11.454 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:11.454 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:11.555 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:11.577 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:11.578 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:11.578 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:11.613 INFO:tasks.rados.rados.0.smithi137.stdout:2133: done (4 left) 2024-09-21T08:32:11.613 INFO:tasks.rados.rados.0.smithi137.stdout:2134: write oid 30 current snap is 236 2024-09-21T08:32:11.613 INFO:tasks.rados.rados.0.smithi137.stdout:2128: finishing copy_from to smithi13779221-20 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 845 (ObjNum 499 snap 140 seq_num 499) dirty exists 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2130: finishing copy_from racing read to smithi13779221-22 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2131: finishing write tid 2 to smithi13779221-1 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2131: finishing write tid 3 to smithi13779221-1 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2130: finishing copy_from to smithi13779221-22 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 463 (ObjNum 740 snap 234 seq_num 740) dirty exists 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2131: finishing write tid 4 to smithi13779221-1 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2131: oid 1 updating version 0 to 541 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2131: oid 1 version 541 is already newer than 540 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2131: oid 1 updating version 541 to 542 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 542 (ObjNum 748 snap 236 seq_num 748) dirty exists 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2131: left oid 1 (ObjNum 748 snap 236 seq_num 748) 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2132: finishing write tid 1 to smithi13779221-32 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2132: finishing write tid 2 to smithi13779221-32 2024-09-21T08:32:11.614 INFO:tasks.rados.rados.0.smithi137.stdout:2132: finishing write tid 3 to smithi13779221-32 2024-09-21T08:32:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:2132: finishing write tid 4 to smithi13779221-32 2024-09-21T08:32:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:2132: oid 32 updating version 0 to 548 2024-09-21T08:32:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:2132: oid 32 version 548 is already newer than 546 2024-09-21T08:32:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:2132: oid 32 version 548 is already newer than 547 2024-09-21T08:32:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 548 (ObjNum 749 snap 236 seq_num 749) dirty exists 2024-09-21T08:32:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:2132: left oid 32 (ObjNum 749 snap 236 seq_num 749) 2024-09-21T08:32:11.615 INFO:tasks.rados.rados.0.smithi137.stdout:2134: seq_num 750 ranges {605250=639344,1855142=452579} 2024-09-21T08:32:11.620 INFO:tasks.rados.rados.0.smithi137.stdout:2134: writing smithi13779221-30 from 605250 to 1244594 tid 1 2024-09-21T08:32:11.627 INFO:tasks.rados.rados.0.smithi137.stdout:2134: writing smithi13779221-30 from 1855142 to 2307721 tid 2 2024-09-21T08:32:11.628 INFO:tasks.rados.rados.0.smithi137.stdout:2128: done (4 left) 2024-09-21T08:32:11.628 INFO:tasks.rados.rados.0.smithi137.stdout:2130: done (3 left) 2024-09-21T08:32:11.628 INFO:tasks.rados.rados.0.smithi137.stdout:2131: done (2 left) 2024-09-21T08:32:11.628 INFO:tasks.rados.rados.0.smithi137.stdout:2132: done (1 left) 2024-09-21T08:32:11.628 INFO:tasks.rados.rados.0.smithi137.stdout:2135: write oid 20 current snap is 236 2024-09-21T08:32:11.628 INFO:tasks.rados.rados.0.smithi137.stdout:2135: seq_num 751 ranges {622057=714593,1913624=708433} 2024-09-21T08:32:11.636 INFO:tasks.rados.rados.0.smithi137.stdout:2135: writing smithi13779221-20 from 622057 to 1336650 tid 1 2024-09-21T08:32:11.644 INFO:tasks.rados.rados.0.smithi137.stdout:2135: writing smithi13779221-20 from 1913624 to 2622057 tid 2 2024-09-21T08:32:11.644 INFO:tasks.rados.rados.0.smithi137.stdout:2136: read oid 7 snap -1 2024-09-21T08:32:11.644 INFO:tasks.rados.rados.0.smithi137.stdout:2136: expect (ObjNum 728 snap 228 seq_num 728) 2024-09-21T08:32:11.644 INFO:tasks.rados.rados.0.smithi137.stdout:2137: read oid 48 snap -1 2024-09-21T08:32:11.644 INFO:tasks.rados.rados.0.smithi137.stdout:2137: expect (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:32:11.644 INFO:tasks.rados.rados.0.smithi137.stdout:2138: write (excl) oid 24 current snap is 236 2024-09-21T08:32:11.644 INFO:tasks.rados.rados.0.smithi137.stdout:2134: finishing write tid 1 to smithi13779221-30 2024-09-21T08:32:11.644 INFO:tasks.rados.rados.0.smithi137.stdout:2138: seq_num 752 ranges {638864=789842,1972106=456248} 2024-09-21T08:32:11.652 INFO:tasks.rados.rados.0.smithi137.stdout:2138: writing smithi13779221-24 from 638864 to 1428706 tid 1 2024-09-21T08:32:11.658 INFO:tasks.rados.rados.0.smithi137.stdout:2138: writing smithi13779221-24 from 1972106 to 2428354 tid 2 2024-09-21T08:32:11.658 INFO:tasks.rados.rados.0.smithi137.stdout:2139: snap_remove snap 163 2024-09-21T08:32:11.679 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:11.702 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:11.702 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:11.703 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:11.803 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:11.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:32:11.826 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:11.827 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:11.827 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:11.850 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd reweight 2 1 2024-09-21T08:32:11.874 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:11.909 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:11.928 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:11.950 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:11.950 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:11.951 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:11.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[29634]: 7.14 scrub starts 2024-09-21T08:32:11.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[29634]: 7.14 scrub ok 2024-09-21T08:32:11.997 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[29634]: pgmap v710: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 625 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail 2024-09-21T08:32:11.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[29634]: osdmap e537: 8 total, 8 up, 5 in 2024-09-21T08:32:11.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:11.998 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[29634]: osdmap e538: 8 total, 8 up, 5 in 2024-09-21T08:32:11.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[41044]: 7.14 scrub starts 2024-09-21T08:32:11.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[41044]: 7.14 scrub ok 2024-09-21T08:32:11.998 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[41044]: pgmap v710: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 625 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail 2024-09-21T08:32:11.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[41044]: osdmap e537: 8 total, 8 up, 5 in 2024-09-21T08:32:11.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:11.999 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:11 smithi137 ceph-mon[41044]: osdmap e538: 8 total, 8 up, 5 in 2024-09-21T08:32:12.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:11 smithi165 ceph-mon[32893]: 7.14 scrub starts 2024-09-21T08:32:12.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:11 smithi165 ceph-mon[32893]: 7.14 scrub ok 2024-09-21T08:32:12.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:11 smithi165 ceph-mon[32893]: pgmap v710: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 625 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail 2024-09-21T08:32:12.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:11 smithi165 ceph-mon[32893]: osdmap e537: 8 total, 8 up, 5 in 2024-09-21T08:32:12.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:11 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:12.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:11 smithi165 ceph-mon[32893]: osdmap e538: 8 total, 8 up, 5 in 2024-09-21T08:32:12.052 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:12.074 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:12.075 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:12.075 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:12.176 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:12.199 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:12.199 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:12.200 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:12.300 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:12.323 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:12.323 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:12.323 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:12.424 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:12.448 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:12.448 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:12.449 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:12.513 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:32:12 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:32:12.550 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:12.571 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:12.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:12.573 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:12.621 INFO:tasks.rados.rados.0.smithi137.stdout:2139: done (5 left) 2024-09-21T08:32:12.622 INFO:tasks.rados.rados.0.smithi137.stdout:2140: write oid 8 current snap is 236 2024-09-21T08:32:12.622 INFO:tasks.rados.rados.0.smithi137.stdout:2134: finishing write tid 2 to smithi13779221-30 2024-09-21T08:32:12.622 INFO:tasks.rados.rados.0.smithi137.stdout:2134: finishing write tid 3 to smithi13779221-30 2024-09-21T08:32:12.622 INFO:tasks.rados.rados.0.smithi137.stdout:2134: finishing write tid 4 to smithi13779221-30 2024-09-21T08:32:12.622 INFO:tasks.rados.rados.0.smithi137.stdout:2134: oid 30 updating version 0 to 849 2024-09-21T08:32:12.622 INFO:tasks.rados.rados.0.smithi137.stdout:2134: oid 30 updating version 849 to 850 2024-09-21T08:32:12.622 INFO:tasks.rados.rados.0.smithi137.stdout:2134: oid 30 version 850 is already newer than 848 2024-09-21T08:32:12.622 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 850 (ObjNum 750 snap 236 seq_num 750) dirty exists 2024-09-21T08:32:12.622 INFO:tasks.rados.rados.0.smithi137.stdout:2134: left oid 30 (ObjNum 750 snap 236 seq_num 750) 2024-09-21T08:32:12.622 INFO:tasks.rados.rados.0.smithi137.stdout:2140: seq_num 753 ranges {655671=581444,2030588=625083} 2024-09-21T08:32:12.629 INFO:tasks.rados.rados.0.smithi137.stdout:2140: writing smithi13779221-8 from 655671 to 1237115 tid 1 2024-09-21T08:32:12.638 INFO:tasks.rados.rados.0.smithi137.stdout:2140: writing smithi13779221-8 from 2030588 to 2655671 tid 2 2024-09-21T08:32:12.638 INFO:tasks.rados.rados.0.smithi137.stdout:2136: expect (ObjNum 728 snap 228 seq_num 728) 2024-09-21T08:32:12.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:12.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:12.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:12.696 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:12.769 INFO:tasks.rados.rados.0.smithi137.stdout:2137: expect (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:32:12.796 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:12.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:12.818 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:12.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:12.826 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:32:12.826 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:422 7.d 7:b31d99ce:::smithi13779221-21:head [write 443718~615603] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 436.31957676000002, 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.034885076000000001, 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.827 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 422 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504976+0000", 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.828 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.504978+0000", 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0090000000000002e-06 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506268+0000", 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001290444 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506279+0000", 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0526999999999999e-05 2024-09-21T08:32:12.829 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506288+0000", 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5470000000000002e-06 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506343+0000", 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5287000000000001e-05 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506354+0000", 2024-09-21T08:32:12.830 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0798999999999999e-05 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.525308+0000", 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018953616 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529390+0000", 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0040819029999999996 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.831 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533318+0000", 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0039282190000000002 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533380+0000", 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1890000000000005e-05 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.533709+0000", 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00032948400000000001 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.832 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536916+0000", 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032063399999999998 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539553+0000", 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0026370909999999998 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539658+0000", 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105585 2024-09-21T08:32:12.833 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539723+0000", 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4343000000000004e-05 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539861+0000", 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137992 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.834 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:423 7.d 7:b31d99ce:::smithi13779221-21:head [write 1564138~470341] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 436.31818709200002, 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033782404000000002, 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 423 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.835 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506365+0000", 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.506366+0000", 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0449999999999999e-06 2024-09-21T08:32:12.836 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507012+0000", 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064568899999999996 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507021+0000", 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.2550000000000003e-06 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507029+0000", 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2559999999999996e-06 2024-09-21T08:32:12.837 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507079+0000", 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0957e-05 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.507088+0000", 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.3650000000000004e-06 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.838 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529457+0000", 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022369185999999999 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529526+0000", 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.8683999999999995e-05 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535157+0000", 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0056315849999999997 2024-09-21T08:32:12.839 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535244+0000", 2024-09-21T08:32:12.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6840000000000002e-05 2024-09-21T08:32:12.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.840 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.840 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:32:12.840 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.535398+0000", 2024-09-21T08:32:12.840 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015404399999999999 2024-09-21T08:32:12.840 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.841 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.841 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539569+0000", 2024-09-21T08:32:12.841 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.004170361 2024-09-21T08:32:12.841 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.841 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539613+0000", 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4101999999999999e-05 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539985+0000", 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037279300000000001 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540037+0000", 2024-09-21T08:32:12.842 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1345999999999999e-05 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540148+0000", 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110896 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:424 7.d 7:b31d99ce:::smithi13779221-21:head [write 2633066~610652] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:32:12.843 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 436.31119807800002, 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027015847999999999, 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 424 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:32:12.844 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513354+0000", 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513356+0000", 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8789999999999999e-06 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.845 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513753+0000", 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039670999999999999 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513764+0000", 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.154e-05 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513772+0000", 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9149999999999996e-06 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.846 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513833+0000", 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1072000000000001e-05 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513843+0000", 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8349999999999999e-06 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529644+0000", 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.015801084 2024-09-21T08:32:12.847 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529688+0000", 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3568e-05 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536031+0000", 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063434659999999999 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.848 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536135+0000", 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103599 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.536291+0000", 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000155668 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539575+0000", 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032840450000000002 2024-09-21T08:32:12.849 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.539925+0000", 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00034988899999999998 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540200+0000", 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00027585000000000001 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.850 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540250+0000", 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9413000000000003e-05 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540370+0000", 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000120315 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.851 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:425 7.d 7:b31d99ce:::smithi13779221-21:head [setxattr _header (56) in=7b,truncate 3243718] snapc a=[9] ondisk+write+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 436.31069543799998, 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.030130239, 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 425 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.852 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513857+0000", 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.853 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513858+0000", 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9199999999999999e-07 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513863+0000", 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7650000000000001e-06 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513869+0000", 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9880000000000001e-06 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.854 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.513874+0000", 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.643e-06 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515915+0000", 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020407279999999999 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515925+0000", 2024-09-21T08:32:12.855 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0051000000000001e-05 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529722+0000", 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013796780999999999 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529804+0000", 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.1663999999999996e-05 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540423+0000", 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010619635000000001 2024-09-21T08:32:12.856 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540671+0000", 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024766600000000001 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.540795+0000", 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012383899999999999 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.857 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541324+0000", 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052910200000000004 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.542654+0000", 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0013304370000000001 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543785+0000", 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001130603 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.858 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543841+0000", 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5939000000000001e-05 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543987+0000", 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014640599999999999 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:426 7.d 7:b31d99ce:::smithi13779221-21:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:32:12.859 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 436.30873649300003, 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028482668999999999, 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 426 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.860 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515816+0000", 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515818+0000", 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7999999999999999e-06 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.861 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515825+0000", 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0790000000000003e-06 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515840+0000", 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4765e-05 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515857+0000", 2024-09-21T08:32:12.862 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7414999999999998e-05 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515968+0000", 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111161 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515975+0000", 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7499999999999997e-06 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.863 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529841+0000", 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013866387000000001 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529951+0000", 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010953100000000001 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541110+0000", 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011159591 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.864 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541190+0000", 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9243e-05 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544032+0000", 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0028424050000000001 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544118+0000", 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5798999999999996e-05 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.865 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544299+0000", 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000180743 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:430 7.d 7:b0b32695:::smithi13779221-19:head [delete] snapc a=[9] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e103)", 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 436.30856581500001, 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028508957000000001, 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.866 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 430 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:32:12.867 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515987+0000", 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1900000000000001e-07 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515991+0000", 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6899999999999998e-06 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.515997+0000", 2024-09-21T08:32:12.868 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2999999999999998e-06 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516003+0000", 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0220000000000001e-06 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516045+0000", 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1069000000000003e-05 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.516052+0000", 2024-09-21T08:32:12.869 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9890000000000003e-06 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.529980+0000", 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013927999999999999 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for degraded object", 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.532305+0000", 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0023250580000000001 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.870 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541672+0000", 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093675080000000001 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541749+0000", 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6829000000000006e-05 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.541934+0000", 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018531 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.871 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543560+0000", 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00162525 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.543661+0000", 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101604 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544339+0000", 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00067745299999999995 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.872 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544394+0000", 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.5515e-05 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:56.544496+0000", 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010144100000000001 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.873 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:571 7.1 7:88bf4ad2:::smithi13779221-6:head [copy-from ver 13] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 422.27749916099998, 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037746406000000003, 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 571 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.874 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547053+0000", 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547056+0000", 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.146e-06 2024-09-21T08:32:12.875 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547067+0000", 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0609e-05 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547090+0000", 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2714999999999999e-05 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547100+0000", 2024-09-21T08:32:12.876 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9920000000000002e-06 2024-09-21T08:32:12.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547203+0000", 2024-09-21T08:32:12.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000102828 2024-09-21T08:32:12.877 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.877 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.877 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:32:12.877 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547210+0000", 2024-09-21T08:32:12.877 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3629999999999996e-06 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553577+0000", 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0063672829999999996 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553669+0000", 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1856999999999998e-05 2024-09-21T08:32:12.878 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,7", 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.573941+0000", 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020272168 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.583019+0000", 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0090777630000000008 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584643+0000", 2024-09-21T08:32:12.879 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0016245319999999999 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584686+0000", 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2259999999999999e-05 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584748+0000", 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2049000000000002e-05 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.584800+0000", 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.1841000000000001e-05 2024-09-21T08:32:12.880 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:572 7.1 7:88bf4ad2:::smithi13779221-6:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 422.27743013200001, 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.032210599999999999, 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.881 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 572 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547122+0000", 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.882 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547123+0000", 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5300000000000003e-07 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547138+0000", 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4718999999999999e-05 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.883 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.886 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547145+0000", 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2520000000000002e-06 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547151+0000", 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7649999999999999e-06 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547263+0000", 2024-09-21T08:32:12.887 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112001 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.547269+0000", 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8919999999999997e-06 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553824+0000", 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0065557499999999999 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:32:12.888 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.553873+0000", 2024-09-21T08:32:12.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8727999999999998e-05 2024-09-21T08:32:12.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.889 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.889 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579133+0000", 2024-09-21T08:32:12.889 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025259964999999999 2024-09-21T08:32:12.889 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.889 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579227+0000", 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3517999999999995e-05 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579333+0000", 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010615699999999999 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.890 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3155 7.9 7:9247b5b1:::smithi13779221-17:head [copy-from ver 183] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:32:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:32:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 220.252056318, 2024-09-21T08:32:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.029797267999999998, 2024-09-21T08:32:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.891 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3155 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.892 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572496+0000", 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572497+0000", 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4899999999999995e-07 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.893 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572500+0000", 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.05e-06 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572505+0000", 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2820000000000004e-06 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.572510+0000", 2024-09-21T08:32:12.894 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8969999999999999e-06 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573522+0000", 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001012481 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573635+0000", 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112253 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.895 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.583897+0000", 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010262288 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594011+0000", 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010113641 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594202+0000", 2024-09-21T08:32:12.896 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000191418 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602174+0000", 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079717079999999992 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602233+0000", 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9565999999999999e-05 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.897 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602293+0000", 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0134999999999999e-05 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3441 7.15 7:a9dd6c7d:::smithi13779221-47:head [write 783325~557161,stat] snapc 84=[84,83,82,81,7f,7e,7d,7c,78,77,72,71,69] ondisk+write+known_if_redirected+supports_pool_eio e343)", 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 196.26120648400001, 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028447395, 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.898 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3441 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.899 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563346+0000", 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563352+0000", 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9039999999999997e-06 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.900 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563878+0000", 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00052587200000000004 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563908+0000", 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.0685999999999997e-05 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.563922+0000", 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3738e-05 2024-09-21T08:32:12.901 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.564007+0000", 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.5283000000000003e-05 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.564017+0000", 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.4909999999999994e-06 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.902 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571275+0000", 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0072579860000000001 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571419+0000", 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00014457700000000001 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-09-21T08:32:12.903 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.571666+0000", 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00024687599999999999 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591189+0000", 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019522923000000001 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591633+0000", 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00044407499999999999 2024-09-21T08:32:12.904 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591717+0000", 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4010000000000004e-05 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591793+0000", 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5974000000000001e-05 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.905 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4000 7.b 7:d6c5a045:::smithi13779221-46:head [copy-from ver 529] snapc a7=[a7,a6,a5,a4,a3,a2,a1,9e,9d,9c,9b,9a,99,98,94,93,92,91,90,8c,8b,88,86,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e398)", 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 141.25622813800001, 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.037495776000000001, 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.906 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4000 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568324+0000", 2024-09-21T08:32:12.907 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568328+0000", 2024-09-21T08:32:12.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.349e-06 2024-09-21T08:32:12.908 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.908 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.908 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.908 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568346+0000", 2024-09-21T08:32:12.908 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.8839e-05 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568362+0000", 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5432999999999999e-05 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568371+0000", 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3009999999999995e-06 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.909 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568435+0000", 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4199e-05 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.568441+0000", 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0050000000000001e-06 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.910 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.575820+0000", 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0073784970000000004 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.575934+0000", 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011455699999999999 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587076+0000", 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011141287 2024-09-21T08:32:12.911 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.595892+0000", 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0088164750000000007 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605259+0000", 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0093663830000000007 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.912 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605662+0000", 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040333900000000001 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605744+0000", 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.2560999999999998e-05 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.605820+0000", 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5550999999999995e-05 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.913 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4168 7.b 7:d7506421:::smithi13779221-13:head [write 447218~719401,stat] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.581231+0000", 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 130.24332124899999, 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.085401058000000002, 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.914 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4168 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581231+0000", 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.915 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581231+0000", 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581234+0000", 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9909999999999998e-06 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581576+0000", 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000342109 2024-09-21T08:32:12.916 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581675+0000", 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.8947e-05 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.581686+0000", 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1039999999999999e-05 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.917 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582031+0000", 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000344333 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582110+0000", 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9197999999999996e-05 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582254+0000", 2024-09-21T08:32:12.918 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000143881 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.584348+0000", 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020943440000000002 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.585410+0000", 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001061782 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.919 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.666486+0000", 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081075990000000001 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.666565+0000", 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9076e-05 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.666632+0000", 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7366999999999994e-05 2024-09-21T08:32:12.920 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4169 7.b 7:d7506421:::smithi13779221-13:head [write 1889370~660531] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.588798+0000", 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 130.23575478199999, 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.081088987000000001, 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.921 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4169 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588798+0000", 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.922 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588798+0000", 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588800+0000", 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.903e-06 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.923 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588978+0000", 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017865599999999999 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.588995+0000", 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6696000000000001e-05 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589003+0000", 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8520000000000004e-06 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.924 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589056+0000", 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3665000000000002e-05 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589145+0000", 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8455000000000006e-05 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589276+0000", 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000131032 2024-09-21T08:32:12.925 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.590553+0000", 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001277572 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.591751+0000", 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001197137 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.926 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669765+0000", 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.078014230000000004 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669824+0000", 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9651999999999997e-05 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669887+0000", 2024-09-21T08:32:12.927 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2137000000000005e-05 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4170 7.b 7:d7506421:::smithi13779221-13:head [write 2963386~614253] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.595002+0000", 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 130.22955066899999, 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.075006211000000003, 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.928 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4170 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595002+0000", 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.929 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595002+0000", 2024-09-21T08:32:12.930 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.930 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.930 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.930 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.930 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595004+0000", 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.4559999999999999e-06 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595379+0000", 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00037446400000000003 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595396+0000", 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7303999999999999e-05 2024-09-21T08:32:12.931 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595403+0000", 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7689999999999999e-06 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595459+0000", 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6373999999999999e-05 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.932 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595547+0000", 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7869999999999997e-05 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595681+0000", 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000133857 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599798+0000", 2024-09-21T08:32:12.933 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041171790000000003 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599868+0000", 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9839000000000007e-05 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669931+0000", 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070063149000000005 2024-09-21T08:32:12.934 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669973+0000", 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1838999999999997e-05 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670008+0000", 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5111000000000003e-05 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.935 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4171 7.b 7:d7506421:::smithi13779221-13:head [setxattr _header (58) in=7b,truncate 3647218] snapc ab=[ab,aa,a9,a7,a5,a4,a3,9e,9c,9b,9a,99,98,94,93,92,91,90,8b,88,85,84,7e,7d,71] ondisk+write+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.595420+0000", 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 130.22913265400001, 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074711124000000004, 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.936 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4171 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595420+0000", 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595420+0000", 2024-09-21T08:32:12.937 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595421+0000", 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3099999999999996e-07 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595425+0000", 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3109999999999999e-06 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.938 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595431+0000", 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3600000000000001e-06 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595435+0000", 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2780000000000002e-06 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.939 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596387+0000", 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00095243599999999997 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596459+0000", 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2077999999999997e-05 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596640+0000", 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00018089500000000001 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.940 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599908+0000", 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032677769999999999 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.599918+0000", 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0059e-05 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.941 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670040+0000", 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.070122219 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670079+0000", 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.8436000000000003e-05 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670131+0000", 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2343999999999999e-05 2024-09-21T08:32:12.942 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4172 7.b 7:d7506421:::smithi13779221-13:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e409)", 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.595449+0000", 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 130.22910293499999, 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.074831616000000004, 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:32:12.943 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4172 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595449+0000", 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.944 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595449+0000", 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595450+0000", 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.85e-07 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595453+0000", 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7879999999999998e-06 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.945 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595460+0000", 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.131e-06 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.595466+0000", 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7910000000000001e-06 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.946 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.596937+0000", 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001470823 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.597004+0000", 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7550000000000002e-05 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670154+0000", 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.073150017999999997 2024-09-21T08:32:12.947 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670214+0000", 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9565999999999999e-05 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.670281+0000", 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.7163999999999996e-05 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.948 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4344 7.b 7:d4aadaae:::smithi13779221-7:head [copy-from ver 229] snapc af=[af,ae,ad,ac,ab,a9,a7,a5,a4,a3,9c,9b,9a,99,94,93,92,8b,84,7e] ondisk+write+known_if_redirected+supports_pool_eio e422)", 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:15.597787+0000", 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 117.226765063, 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033249305, 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.949 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4344 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597787+0000", 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597787+0000", 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.950 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597788+0000", 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3000000000000001e-07 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597792+0000", 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1749999999999998e-06 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.951 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597796+0000", 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9219999999999999e-06 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597800+0000", 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2069999999999997e-06 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.952 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598500+0000", 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00069988899999999998 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598508+0000", 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8310000000000001e-06 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610088+0000", 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011580653 2024-09-21T08:32:12.953 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610205+0000", 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000116882 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,5", 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.618623+0000", 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0084175770000000007 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.954 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.629957+0000", 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011334222999999999 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.630882+0000", 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00092455300000000005 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.630902+0000", 2024-09-21T08:32:12.955 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0522000000000001e-05 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.630982+0000", 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9491000000000002e-05 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.631037+0000", 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4950000000000001e-05 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.956 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.957 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.957 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.957 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.957 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4345 7.b 7:d4aadaae:::smithi13779221-7:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e422)", 2024-09-21T08:32:12.957 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:15.597855+0000", 2024-09-21T08:32:12.957 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 117.226696932, 2024-09-21T08:32:12.957 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.027086114000000001, 2024-09-21T08:32:12.957 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.957 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.957 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4345 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597855+0000", 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.958 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597855+0000", 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597856+0000", 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6199999999999996e-07 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.597860+0000", 2024-09-21T08:32:12.959 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6270000000000002e-06 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598442+0000", 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00058186999999999996 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.598450+0000", 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0420000000000003e-06 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.960 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.599133+0000", 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00068317699999999998 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.599141+0000", 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.9240000000000007e-06 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.961 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610453+0000", 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011312546999999999 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.610511+0000", 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7732999999999998e-05 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.624495+0000", 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.013983647 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.962 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.624674+0000", 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00017928099999999999 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:15.624942+0000", 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000267404 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.963 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:4390 7.10 7:0979c720:::smithi13779221-16:head [copy-from ver 498] snapc b2=[b2,b1,b0,af,ae,ad,ac,a9,a7,a5,a4,a3,9c,9b,99,94,93,92,84,7e] ondisk+write+known_if_redirected+supports_pool_eio e428)", 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:21.587905+0000", 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 111.236647826, 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.028087958, 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.964 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 4390 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587905+0000", 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587905+0000", 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.965 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587906+0000", 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.404e-06 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587911+0000", 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.305e-06 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.966 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587921+0000", 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3780000000000007e-06 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587926+0000", 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0889999999999999e-06 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.587979+0000", 2024-09-21T08:32:12.967 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3261999999999999e-05 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.588091+0000", 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112184 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2", 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.608584+0000", 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020492429999999999 2024-09-21T08:32:12.968 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615479+0000", 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00689502 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615773+0000", 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00029482400000000001 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.969 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615854+0000", 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0430000000000001e-05 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:21.615993+0000", 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000138632 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.970 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:5437 7.15 7:aaa5c288:::smithi13779221-20:head [copy-from ver 845] snapc ed=[ec,eb,ea,e9,e8,e5,e4,e2,e1,dd,dc,da,d9,d7,d6,d3,d1,cb,ca,c9,c2,c1,b9,b8,b3,b2,b0,af,a5,9b,92] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:32:10.680979+0000", 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 2.1435732010000002, 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.026753015000000002, 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 5437 2024-09-21T08:32:12.971 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.680979+0000", 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.680979+0000", 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.972 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.680981+0000", 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2800000000000002e-06 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.680991+0000", 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1530000000000003e-06 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.681005+0000", 2024-09-21T08:32:12.973 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4800999999999999e-05 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.681017+0000", 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1463e-05 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.681082+0000", 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4814999999999995e-05 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.974 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.681090+0000", 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6170000000000007e-06 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.685376+0000", 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0042859860000000003 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:12.975 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.685489+0000", 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000112785 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 2,7", 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.700489+0000", 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014999704000000001 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.706907+0000", 2024-09-21T08:32:12.976 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.006418666 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.707241+0000", 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000333371 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.707639+0000", 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00039861200000000001 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.977 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.707690+0000", 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0998000000000002e-05 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:32:10.707732+0000", 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.1764000000000002e-05 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.978 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:12.979 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:12.979 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:32:12.979 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:13.004 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:13.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:13.005 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:13.006 INFO:tasks.rados.rados.0.smithi137.stdout:2135: finishing write tid 1 to smithi13779221-20 2024-09-21T08:32:13.006 INFO:tasks.rados.rados.0.smithi137.stdout:2135: finishing write tid 2 to smithi13779221-20 2024-09-21T08:32:13.006 INFO:tasks.rados.rados.0.smithi137.stdout:2134: done (5 left) 2024-09-21T08:32:13.006 INFO:tasks.rados.rados.0.smithi137.stdout:2136: done (4 left) 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:2137: done (3 left) 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:2141: rollback oid 21 current snap is 236 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 21 to 218 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:2135: finishing write tid 3 to smithi13779221-20 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:2135: finishing write tid 4 to smithi13779221-20 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:2135: oid 20 updating version 0 to 846 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:2135: oid 20 updating version 846 to 847 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:2135: oid 20 updating version 847 to 848 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 20 v 848 (ObjNum 751 snap 236 seq_num 751) dirty exists 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:2135: left oid 20 (ObjNum 751 snap 236 seq_num 751) 2024-09-21T08:32:13.007 INFO:tasks.rados.rados.0.smithi137.stdout:2138: finishing write tid 1 to smithi13779221-24 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:2138: finishing write tid 2 to smithi13779221-24 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:2138: finishing write tid 3 to smithi13779221-24 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:2138: finishing write tid 4 to smithi13779221-24 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:2138: oid 24 updating version 0 to 743 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:2138: oid 24 version 743 is already newer than 741 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:2138: oid 24 version 743 is already newer than 742 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 743 (ObjNum 752 snap 236 seq_num 752) dirty exists 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:2138: left oid 24 (ObjNum 752 snap 236 seq_num 752) 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:2140: finishing write tid 1 to smithi13779221-8 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:2140: finishing write tid 2 to smithi13779221-8 2024-09-21T08:32:13.008 INFO:tasks.rados.rados.0.smithi137.stdout:2140: finishing write tid 3 to smithi13779221-8 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:2140: finishing write tid 4 to smithi13779221-8 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:2140: oid 8 updating version 0 to 468 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:2140: oid 8 updating version 468 to 469 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:2140: oid 8 version 469 is already newer than 467 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 8 v 469 (ObjNum 753 snap 236 seq_num 753) dirty exists 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:2140: left oid 8 (ObjNum 753 snap 236 seq_num 753) 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:2135: done (3 left) 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:2138: done (2 left) 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:2140: done (1 left) 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:2142: write oid 32 current snap is 236 2024-09-21T08:32:13.009 INFO:tasks.rados.rados.0.smithi137.stdout:2142: seq_num 754 ranges {672478=656693,2089070=459917} 2024-09-21T08:32:13.012 INFO:tasks.rados.rados.0.smithi137.stdout:2142: writing smithi13779221-32 from 672478 to 1329171 tid 1 2024-09-21T08:32:13.020 INFO:tasks.rados.rados.0.smithi137.stdout:2142: writing smithi13779221-32 from 2089070 to 2548987 tid 2 2024-09-21T08:32:13.020 INFO:tasks.rados.rados.0.smithi137.stdout:2143: copy_from oid 46 from oid 37 current snap is 236 2024-09-21T08:32:13.020 INFO:tasks.rados.rados.0.smithi137.stdout:2141: finishing rollback tid 0 to smithi13779221-21 2024-09-21T08:32:13.020 INFO:tasks.rados.rados.0.smithi137.stdout:2141: finishing rollback tid 1 to smithi13779221-21 2024-09-21T08:32:13.020 INFO:tasks.rados.rados.0.smithi137.stdout:2141: finishing rollback tid 2 to smithi13779221-21 2024-09-21T08:32:13.020 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 21 v 522 (ObjNum 677 snap 205 seq_num 677) dirty exists 2024-09-21T08:32:13.020 INFO:tasks.rados.rados.0.smithi137.stdout:2142: finishing write tid 1 to smithi13779221-32 2024-09-21T08:32:13.020 INFO:tasks.rados.rados.0.smithi137.stdout:2141: done (2 left) 2024-09-21T08:32:13.020 INFO:tasks.rados.rados.0.smithi137.stdout:2144: snap_create 2024-09-21T08:32:13.023 INFO:tasks.rados.rados.0.smithi137.stdout:2142: finishing write tid 2 to smithi13779221-32 2024-09-21T08:32:13.023 INFO:tasks.rados.rados.0.smithi137.stdout:2142: finishing write tid 3 to smithi13779221-32 2024-09-21T08:32:13.023 INFO:tasks.rados.rados.0.smithi137.stdout:2142: finishing write tid 4 to smithi13779221-32 2024-09-21T08:32:13.023 INFO:tasks.rados.rados.0.smithi137.stdout:2142: oid 32 updating version 0 to 549 2024-09-21T08:32:13.023 INFO:tasks.rados.rados.0.smithi137.stdout:2142: oid 32 updating version 549 to 550 2024-09-21T08:32:13.023 INFO:tasks.rados.rados.0.smithi137.stdout:2142: oid 32 updating version 550 to 551 2024-09-21T08:32:13.023 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 551 (ObjNum 754 snap 236 seq_num 754) dirty exists 2024-09-21T08:32:13.023 INFO:tasks.rados.rados.0.smithi137.stdout:2142: left oid 32 (ObjNum 754 snap 236 seq_num 754) 2024-09-21T08:32:13.026 INFO:tasks.rados.rados.0.smithi137.stdout:2143: finishing copy_from racing read to smithi13779221-46 2024-09-21T08:32:13.032 INFO:tasks.rados.rados.0.smithi137.stdout:2143: finishing copy_from to smithi13779221-46 2024-09-21T08:32:13.033 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 510 (ObjNum 618 snap 182 seq_num 618) dirty exists 2024-09-21T08:32:13.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:13.129 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:13.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:13.130 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:13.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:13.254 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:13.254 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:13.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:13.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:13.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:13.378 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:13.378 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:13.479 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:13.503 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:13.504 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:13.504 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:13.605 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:13.630 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:13.630 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:13.631 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:13.680 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:32:13.681 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:13 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:13.681 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:13 smithi137 ceph-mon[29634]: pgmap v713: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 631 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail 2024-09-21T08:32:13.681 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:13 smithi137 ceph-mon[29634]: osdmap e539: 8 total, 8 up, 5 in 2024-09-21T08:32:13.681 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:13 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2599788483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:13.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:13 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:13.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:13 smithi137 ceph-mon[41044]: pgmap v713: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 631 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail 2024-09-21T08:32:13.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:13 smithi137 ceph-mon[41044]: osdmap e539: 8 total, 8 up, 5 in 2024-09-21T08:32:13.681 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:13 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2599788483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:13.684 INFO:tasks.rados.rados.0.smithi137.stdout:2142: done (2 left) 2024-09-21T08:32:13.684 INFO:tasks.rados.rados.0.smithi137.stdout:2143: done (1 left) 2024-09-21T08:32:13.684 INFO:tasks.rados.rados.0.smithi137.stdout:2144: done (0 left) 2024-09-21T08:32:13.684 INFO:tasks.rados.rados.0.smithi137.stdout:2145: rollback oid 5 current snap is 237 2024-09-21T08:32:13.684 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 5 to 191 2024-09-21T08:32:13.685 INFO:tasks.rados.rados.0.smithi137.stdout:2146: snap_remove snap 173 2024-09-21T08:32:13.731 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:13.753 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:13.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:13.754 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:13.854 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:13.876 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:13.877 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:13.878 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:13.978 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:14.000 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:14.000 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:14.001 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:14.010 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_ops_in_flight 2024-09-21T08:32:14.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:13 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:14.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:13 smithi165 ceph-mon[32893]: pgmap v713: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 631 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail 2024-09-21T08:32:14.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:13 smithi165 ceph-mon[32893]: osdmap e539: 8 total, 8 up, 5 in 2024-09-21T08:32:14.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:13 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2599788483' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:14.102 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:14.124 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:14.124 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:14.124 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:14.225 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:14.247 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:14.248 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:14.248 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:14.349 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:14.371 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:14.372 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:14.372 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:14.473 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:14.495 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:14.496 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:14.496 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:14.597 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:14.620 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:14.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:14.621 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:14.686 INFO:tasks.rados.rados.0.smithi137.stdout:2146: done (1 left) 2024-09-21T08:32:14.686 INFO:tasks.rados.rados.0.smithi137.stdout:2147: snap_remove snap 219 2024-09-21T08:32:14.722 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:14.743 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:14.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:14.744 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:14.845 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:14.867 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:14.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:14.868 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:14.953 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[29634]: 5.b scrub starts 2024-09-21T08:32:14.953 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[29634]: 5.b scrub ok 2024-09-21T08:32:14.953 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:32:14.953 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[29634]: Cluster is now healthy 2024-09-21T08:32:14.953 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2599788483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:14.953 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[29634]: osdmap e540: 8 total, 8 up, 5 in 2024-09-21T08:32:14.954 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[29634]: 4.c scrub starts 2024-09-21T08:32:14.954 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[29634]: 4.c scrub ok 2024-09-21T08:32:14.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[41044]: 5.b scrub starts 2024-09-21T08:32:14.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[41044]: 5.b scrub ok 2024-09-21T08:32:14.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:32:14.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[41044]: Cluster is now healthy 2024-09-21T08:32:14.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2599788483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:14.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[41044]: osdmap e540: 8 total, 8 up, 5 in 2024-09-21T08:32:14.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[41044]: 4.c scrub starts 2024-09-21T08:32:14.954 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:14 smithi137 ceph-mon[41044]: 4.c scrub ok 2024-09-21T08:32:14.969 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:14.971 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:32:14.972 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:32:14.972 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:32:14.972 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:32:14.977 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:14.991 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:32:14.991 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:32:14.994 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:14.995 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:14.995 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:15.010 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:32:15.010 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:32:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:14 smithi165 ceph-mon[32893]: 5.b scrub starts 2024-09-21T08:32:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:14 smithi165 ceph-mon[32893]: 5.b scrub ok 2024-09-21T08:32:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:14 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:32:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:14 smithi165 ceph-mon[32893]: Cluster is now healthy 2024-09-21T08:32:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:14 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2599788483' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:14 smithi165 ceph-mon[32893]: osdmap e540: 8 total, 8 up, 5 in 2024-09-21T08:32:15.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:14 smithi165 ceph-mon[32893]: 4.c scrub starts 2024-09-21T08:32:15.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:14 smithi165 ceph-mon[32893]: 4.c scrub ok 2024-09-21T08:32:15.016 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:15.031 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:32:15.031 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:32:15.048 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:32:15.048 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:32:15.064 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:32:15.064 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:32:15.081 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:32:15.081 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:32:15.096 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:15.119 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:15.119 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:15.120 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:15.220 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:15.243 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:15.243 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:15.244 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:15.344 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:15.366 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:15.366 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:15.367 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:15.468 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:15.489 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:15.489 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:15.490 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:15.590 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:15.613 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:15.613 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:15.614 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:15.690 INFO:tasks.rados.rados.0.smithi137.stdout:2147: done (1 left) 2024-09-21T08:32:15.691 INFO:tasks.rados.rados.0.smithi137.stdout:2148: snap_create 2024-09-21T08:32:15.691 INFO:tasks.rados.rados.0.smithi137.stdout:2145: finishing rollback tid 1 to smithi13779221-5 2024-09-21T08:32:15.691 INFO:tasks.rados.rados.0.smithi137.stdout:2145: finishing rollback tid 2 to smithi13779221-5 2024-09-21T08:32:15.691 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 543 (ObjNum 497 snap 137 seq_num 497) dirty exists 2024-09-21T08:32:15.714 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:15.735 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:15.736 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:15.736 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:15.814 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:32:15.814 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:32:15.837 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:15.859 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:15.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:15.860 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:15.960 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:15.981 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: pgmap v716: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 631 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 5.6 MiB/s rd, 3.1 MiB/s wr, 13 op/s 2024-09-21T08:32:15.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: 6.12 scrub starts 2024-09-21T08:32:15.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: 6.12 scrub ok 2024-09-21T08:32:15.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: 6.2 scrub starts 2024-09-21T08:32:15.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: 6.2 scrub ok 2024-09-21T08:32:15.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: osdmap e541: 8 total, 8 up, 5 in 2024-09-21T08:32:15.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3666365457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:15.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: 4.1f deep-scrub starts 2024-09-21T08:32:15.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: 4.1f deep-scrub ok 2024-09-21T08:32:15.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:15.982 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:15 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:32:15.983 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:15.983 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:15.984 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: pgmap v716: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 631 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 5.6 MiB/s rd, 3.1 MiB/s wr, 13 op/s 2024-09-21T08:32:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: 6.12 scrub starts 2024-09-21T08:32:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: 6.12 scrub ok 2024-09-21T08:32:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: 6.2 scrub starts 2024-09-21T08:32:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: 6.2 scrub ok 2024-09-21T08:32:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: osdmap e541: 8 total, 8 up, 5 in 2024-09-21T08:32:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3666365457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: 4.1f deep-scrub starts 2024-09-21T08:32:16.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: 4.1f deep-scrub ok 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: pgmap v716: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 631 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 5.6 MiB/s rd, 3.1 MiB/s wr, 13 op/s 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: 6.12 scrub starts 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: 6.12 scrub ok 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: 6.2 scrub starts 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: 6.2 scrub ok 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: osdmap e541: 8 total, 8 up, 5 in 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3666365457' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: 4.1f deep-scrub starts 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: 4.1f deep-scrub ok 2024-09-21T08:32:16.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:16.020 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:15 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:32:16.084 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:16.106 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:16.106 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:16.107 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:16.208 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:16.230 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:16.230 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:16.231 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:16.331 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:16.353 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:16.354 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:16.354 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:16.455 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:16.476 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:16.477 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:16.477 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:16.494 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_blocked_ops 2024-09-21T08:32:16.578 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:16.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:16.600 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:16.600 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:16.611 INFO:tasks.rados.rados.0.smithi137.stdout:2145: done (1 left) 2024-09-21T08:32:16.611 INFO:tasks.rados.rados.0.smithi137.stdout:2148: done (0 left) 2024-09-21T08:32:16.611 INFO:tasks.rados.rados.0.smithi137.stdout:2149: write oid 22 current snap is 238 2024-09-21T08:32:16.611 INFO:tasks.rados.rados.0.smithi137.stdout:2149: seq_num 755 ranges {689285=731942,2147552=541733} 2024-09-21T08:32:16.619 INFO:tasks.rados.rados.0.smithi137.stdout:2149: writing smithi13779221-22 from 689285 to 1421227 tid 1 2024-09-21T08:32:16.628 INFO:tasks.rados.rados.0.smithi137.stdout:2149: writing smithi13779221-22 from 2147552 to 2689285 tid 2 2024-09-21T08:32:16.628 INFO:tasks.rados.rados.0.smithi137.stdout:2150: copy_from oid 28 from oid 34 current snap is 238 2024-09-21T08:32:16.628 INFO:tasks.rados.rados.0.smithi137.stdout:2151: read oid 20 snap 226 2024-09-21T08:32:16.628 INFO:tasks.rados.rados.0.smithi137.stdout:2151: expect (ObjNum 708 snap 218 seq_num 708) 2024-09-21T08:32:16.628 INFO:tasks.rados.rados.0.smithi137.stdout:2152: write oid 43 current snap is 238 2024-09-21T08:32:16.628 INFO:tasks.rados.rados.0.smithi137.stdout:2152: seq_num 756 ranges {706092=407191,1806034=463586} 2024-09-21T08:32:16.635 INFO:tasks.rados.rados.0.smithi137.stdout:2152: writing smithi13779221-43 from 706092 to 1113283 tid 1 2024-09-21T08:32:16.643 INFO:tasks.rados.rados.0.smithi137.stdout:2152: writing smithi13779221-43 from 1806034 to 2269620 tid 2 2024-09-21T08:32:16.643 INFO:tasks.rados.rados.0.smithi137.stdout:2153: snap_create 2024-09-21T08:32:16.643 INFO:tasks.rados.rados.0.smithi137.stdout:2149: finishing write tid 1 to smithi13779221-22 2024-09-21T08:32:16.654 INFO:tasks.rados.rados.0.smithi137.stdout:2150: finishing copy_from racing read to smithi13779221-28 2024-09-21T08:32:16.657 INFO:tasks.rados.rados.0.smithi137.stdout:2149: finishing write tid 2 to smithi13779221-22 2024-09-21T08:32:16.657 INFO:tasks.rados.rados.0.smithi137.stdout:2149: finishing write tid 3 to smithi13779221-22 2024-09-21T08:32:16.658 INFO:tasks.rados.rados.0.smithi137.stdout:2152: finishing write tid 1 to smithi13779221-43 2024-09-21T08:32:16.659 INFO:tasks.rados.rados.0.smithi137.stdout:2151: expect (ObjNum 708 snap 218 seq_num 708) 2024-09-21T08:32:16.702 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:16.726 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:16.727 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:16.727 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:16.731 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:32:16.767 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:16 smithi137 ceph-mon[29634]: osdmap e542: 8 total, 8 up, 5 in 2024-09-21T08:32:16.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:16 smithi137 ceph-mon[29634]: osdmap e543: 8 total, 8 up, 5 in 2024-09-21T08:32:16.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:16 smithi137 ceph-mon[41044]: osdmap e542: 8 total, 8 up, 5 in 2024-09-21T08:32:16.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:16 smithi137 ceph-mon[41044]: osdmap e543: 8 total, 8 up, 5 in 2024-09-21T08:32:16.806 INFO:tasks.rados.rados.0.smithi137.stdout:2152: finishing write tid 2 to smithi13779221-43 2024-09-21T08:32:16.806 INFO:tasks.rados.rados.0.smithi137.stdout:2149: finishing write tid 4 to smithi13779221-22 2024-09-21T08:32:16.806 INFO:tasks.rados.rados.0.smithi137.stdout:2149: oid 22 updating version 0 to 511 2024-09-21T08:32:16.806 INFO:tasks.rados.rados.0.smithi137.stdout:2149: oid 22 updating version 511 to 512 2024-09-21T08:32:16.806 INFO:tasks.rados.rados.0.smithi137.stdout:2149: oid 22 updating version 512 to 513 2024-09-21T08:32:16.806 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 513 (ObjNum 755 snap 238 seq_num 755) dirty exists 2024-09-21T08:32:16.806 INFO:tasks.rados.rados.0.smithi137.stdout:2149: left oid 22 (ObjNum 755 snap 238 seq_num 755) 2024-09-21T08:32:16.807 INFO:tasks.rados.rados.0.smithi137.stdout:2152: finishing write tid 3 to smithi13779221-43 2024-09-21T08:32:16.807 INFO:tasks.rados.rados.0.smithi137.stdout:2150: finishing copy_from to smithi13779221-28 2024-09-21T08:32:16.807 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 455 (ObjNum 620 snap 182 seq_num 620) dirty exists 2024-09-21T08:32:16.807 INFO:tasks.rados.rados.0.smithi137.stdout:2152: finishing write tid 4 to smithi13779221-43 2024-09-21T08:32:16.807 INFO:tasks.rados.rados.0.smithi137.stdout:2152: oid 43 updating version 0 to 415 2024-09-21T08:32:16.807 INFO:tasks.rados.rados.0.smithi137.stdout:2152: oid 43 updating version 415 to 417 2024-09-21T08:32:16.807 INFO:tasks.rados.rados.0.smithi137.stdout:2152: oid 43 version 417 is already newer than 416 2024-09-21T08:32:16.807 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 417 (ObjNum 756 snap 238 seq_num 756) dirty exists 2024-09-21T08:32:16.807 INFO:tasks.rados.rados.0.smithi137.stdout:2152: left oid 43 (ObjNum 756 snap 238 seq_num 756) 2024-09-21T08:32:16.828 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:16.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:16.850 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:16.850 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:16.951 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:16.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:16.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:16.974 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:16 smithi165 ceph-mon[32893]: osdmap e542: 8 total, 8 up, 5 in 2024-09-21T08:32:17.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:16 smithi165 ceph-mon[32893]: osdmap e543: 8 total, 8 up, 5 in 2024-09-21T08:32:17.075 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:17.096 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:17.096 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:17.097 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:17.192 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:17.197 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:17.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:17.220 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:17.220 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:17.321 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:17.343 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:17.343 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:17.344 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:17.444 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:17.466 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:17.467 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:17.467 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:17.568 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:17.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:17.591 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:17.591 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:17.622 INFO:tasks.rados.rados.0.smithi137.stdout:2149: done (4 left) 2024-09-21T08:32:17.622 INFO:tasks.rados.rados.0.smithi137.stdout:2150: done (3 left) 2024-09-21T08:32:17.622 INFO:tasks.rados.rados.0.smithi137.stdout:2151: done (2 left) 2024-09-21T08:32:17.622 INFO:tasks.rados.rados.0.smithi137.stdout:2152: done (1 left) 2024-09-21T08:32:17.622 INFO:tasks.rados.rados.0.smithi137.stdout:2153: done (0 left) 2024-09-21T08:32:17.622 INFO:tasks.rados.rados.0.smithi137.stdout:2154: rollback oid 16 current snap is 239 2024-09-21T08:32:17.622 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 16 to 216 2024-09-21T08:32:17.622 INFO:tasks.rados.rados.0.smithi137.stdout:2155: read oid 29 snap -1 2024-09-21T08:32:17.622 INFO:tasks.rados.rados.0.smithi137.stdout:2155: expect (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:32:17.622 INFO:tasks.rados.rados.0.smithi137.stdout:2156: write oid 26 current snap is 239 2024-09-21T08:32:17.623 INFO:tasks.rados.rados.0.smithi137.stdout:2156: seq_num 757 ranges {722899=482440,1748163=723597} 2024-09-21T08:32:17.623 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.2 to 1 (10000) 2024-09-21T08:32:17.629 INFO:tasks.rados.rados.0.smithi137.stdout:2156: writing smithi13779221-26 from 722899 to 1205339 tid 1 2024-09-21T08:32:17.637 INFO:tasks.rados.rados.0.smithi137.stdout:2156: writing smithi13779221-26 from 1748163 to 2471760 tid 2 2024-09-21T08:32:17.638 INFO:tasks.rados.rados.0.smithi137.stdout:2157: read oid 27 snap -1 2024-09-21T08:32:17.638 INFO:tasks.rados.rados.0.smithi137.stdout:2157: expect (ObjNum 714 snap 222 seq_num 714) 2024-09-21T08:32:17.638 INFO:tasks.rados.rados.0.smithi137.stdout:2158: snap_remove snap 218 2024-09-21T08:32:17.671 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:32:17.671 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:32:17.692 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:17.715 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:17.715 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:17.716 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:17.817 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:17.840 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:17.841 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:17.841 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:17.942 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:17.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: pgmap v719: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 608 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 13 op/s 2024-09-21T08:32:17.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: 5.12 scrub starts 2024-09-21T08:32:17.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: 5.12 scrub ok 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: 3.15 scrub starts 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: 3.15 scrub ok 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3609482380' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: from='client.26041 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: 4.11 scrub starts 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: 4.11 scrub ok 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: from='client.26041 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[29634]: osdmap e544: 8 total, 8 up, 5 in 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: pgmap v719: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 608 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 13 op/s 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: 5.12 scrub starts 2024-09-21T08:32:17.946 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: 5.12 scrub ok 2024-09-21T08:32:17.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: 3.15 scrub starts 2024-09-21T08:32:17.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: 3.15 scrub ok 2024-09-21T08:32:17.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3609482380' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:32:17.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: from='client.26041 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:32:17.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: 4.11 scrub starts 2024-09-21T08:32:17.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: 4.11 scrub ok 2024-09-21T08:32:17.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: from='client.26041 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T08:32:17.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:17 smithi137 ceph-mon[41044]: osdmap e544: 8 total, 8 up, 5 in 2024-09-21T08:32:17.964 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:17.964 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:17.964 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: pgmap v719: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 608 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 13 op/s 2024-09-21T08:32:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: 5.12 scrub starts 2024-09-21T08:32:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: 5.12 scrub ok 2024-09-21T08:32:18.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: 3.15 scrub starts 2024-09-21T08:32:18.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: 3.15 scrub ok 2024-09-21T08:32:18.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3609482380' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:32:18.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: from='client.26041 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]: dispatch 2024-09-21T08:32:18.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: 4.11 scrub starts 2024-09-21T08:32:18.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: 4.11 scrub ok 2024-09-21T08:32:18.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: from='client.26041 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 2, "weight": 1.0}]': finished 2024-09-21T08:32:18.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:17 smithi165 ceph-mon[32893]: osdmap e544: 8 total, 8 up, 5 in 2024-09-21T08:32:18.065 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:18.086 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:18.086 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:18.087 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:18.188 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:18.210 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:18.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:18.211 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:18.311 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:18.333 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:18.333 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:18.333 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:18.376 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd reweight 5 1 2024-09-21T08:32:18.434 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:18.457 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:18.457 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:18.458 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:18.558 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:18.580 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:18.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:18.581 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:18.583 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:18.621 INFO:tasks.rados.rados.0.smithi137.stdout:2158: done (4 left) 2024-09-21T08:32:18.622 INFO:tasks.rados.rados.0.smithi137.stdout:2159: read oid 6 snap -1 2024-09-21T08:32:18.622 INFO:tasks.rados.rados.0.smithi137.stdout:2159: expect deleted 2024-09-21T08:32:18.622 INFO:tasks.rados.rados.0.smithi137.stdout:2160: write (excl) oid 44 current snap is 239 2024-09-21T08:32:18.622 INFO:tasks.rados.rados.0.smithi137.stdout:2160: seq_num 758 ranges {739706=557689,1806645=467255} 2024-09-21T08:32:18.629 INFO:tasks.rados.rados.0.smithi137.stdout:2160: writing smithi13779221-44 from 739706 to 1297395 tid 1 2024-09-21T08:32:18.637 INFO:tasks.rados.rados.0.smithi137.stdout:2160: writing smithi13779221-44 from 1806645 to 2273900 tid 2 2024-09-21T08:32:18.638 INFO:tasks.rados.rados.0.smithi137.stdout:2161: snap_create 2024-09-21T08:32:18.638 INFO:tasks.rados.rados.0.smithi137.stdout:2155: expect (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:32:18.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:18.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:18.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:18.704 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:18.805 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:18.831 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:18.831 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:18.832 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:18.870 INFO:tasks.rados.rados.0.smithi137.stdout:2154: finishing rollback tid 0 to smithi13779221-16 2024-09-21T08:32:18.870 INFO:tasks.rados.rados.0.smithi137.stdout:2154: finishing rollback tid 1 to smithi13779221-16 2024-09-21T08:32:18.870 INFO:tasks.rados.rados.0.smithi137.stdout:2154: finishing rollback tid 2 to smithi13779221-16 2024-09-21T08:32:18.870 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 16 v 511 (ObjNum 701 snap 216 seq_num 701) dirty exists 2024-09-21T08:32:18.870 INFO:tasks.rados.rados.0.smithi137.stdout:2156: finishing write tid 1 to smithi13779221-26 2024-09-21T08:32:18.870 INFO:tasks.rados.rados.0.smithi137.stdout:2156: finishing write tid 2 to smithi13779221-26 2024-09-21T08:32:18.870 INFO:tasks.rados.rados.0.smithi137.stdout:2156: finishing write tid 3 to smithi13779221-26 2024-09-21T08:32:18.870 INFO:tasks.rados.rados.0.smithi137.stdout:2157: expect (ObjNum 714 snap 222 seq_num 714) 2024-09-21T08:32:18.930 INFO:tasks.rados.rados.0.smithi137.stdout:2156: finishing write tid 4 to smithi13779221-26 2024-09-21T08:32:18.930 INFO:tasks.rados.rados.0.smithi137.stdout:2156: oid 26 updating version 0 to 420 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:2156: oid 26 version 420 is already newer than 418 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:2156: oid 26 version 420 is already newer than 419 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 26 v 420 (ObjNum 757 snap 239 seq_num 757) dirty exists 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:2156: left oid 26 (ObjNum 757 snap 239 seq_num 757) 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:2160: finishing write tid 1 to smithi13779221-44 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:2160: finishing write tid 2 to smithi13779221-44 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:2160: finishing write tid 3 to smithi13779221-44 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:2160: finishing write tid 4 to smithi13779221-44 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:2160: oid 44 updating version 0 to 754 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:2160: oid 44 version 754 is already newer than 752 2024-09-21T08:32:18.931 INFO:tasks.rados.rados.0.smithi137.stdout:2160: oid 44 version 754 is already newer than 753 2024-09-21T08:32:18.932 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 754 (ObjNum 758 snap 239 seq_num 758) dirty exists 2024-09-21T08:32:18.932 INFO:tasks.rados.rados.0.smithi137.stdout:2160: left oid 44 (ObjNum 758 snap 239 seq_num 758) 2024-09-21T08:32:18.933 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:18.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:18.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:18.955 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:19.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:18 smithi165 ceph-mon[32893]: 4.1e scrub starts 2024-09-21T08:32:19.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:18 smithi165 ceph-mon[32893]: 4.1e scrub ok 2024-09-21T08:32:19.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:18 smithi165 ceph-mon[32893]: 5.1b scrub starts 2024-09-21T08:32:19.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:18 smithi165 ceph-mon[32893]: 5.1b scrub ok 2024-09-21T08:32:19.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:18 smithi165 ceph-mon[32893]: osdmap e545: 8 total, 8 up, 5 in 2024-09-21T08:32:19.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:18 smithi137 ceph-mon[41044]: 4.1e scrub starts 2024-09-21T08:32:19.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:18 smithi137 ceph-mon[41044]: 4.1e scrub ok 2024-09-21T08:32:19.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:18 smithi137 ceph-mon[41044]: 5.1b scrub starts 2024-09-21T08:32:19.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:18 smithi137 ceph-mon[41044]: 5.1b scrub ok 2024-09-21T08:32:19.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:18 smithi137 ceph-mon[41044]: osdmap e545: 8 total, 8 up, 5 in 2024-09-21T08:32:19.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:18 smithi137 ceph-mon[29634]: 4.1e scrub starts 2024-09-21T08:32:19.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:18 smithi137 ceph-mon[29634]: 4.1e scrub ok 2024-09-21T08:32:19.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:18 smithi137 ceph-mon[29634]: 5.1b scrub starts 2024-09-21T08:32:19.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:18 smithi137 ceph-mon[29634]: 5.1b scrub ok 2024-09-21T08:32:19.019 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:18 smithi137 ceph-mon[29634]: osdmap e545: 8 total, 8 up, 5 in 2024-09-21T08:32:19.056 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:19.068 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:19.081 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:19.081 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:19.082 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:19.182 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:19.189 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:32:19.204 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:19.204 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:19.205 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:19.306 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:19.327 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:19.327 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:19.328 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:19.428 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:19.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:19.452 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:19.452 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:19.495 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.2 dump_historic_ops 2024-09-21T08:32:19.532 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:19.553 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:19.580 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:19.580 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:19.581 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:19.619 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:32:19.629 INFO:tasks.rados.rados.0.smithi137.stdout:2154: done (6 left) 2024-09-21T08:32:19.629 INFO:tasks.rados.rados.0.smithi137.stdout:2155: done (5 left) 2024-09-21T08:32:19.629 INFO:tasks.rados.rados.0.smithi137.stdout:2156: done (4 left) 2024-09-21T08:32:19.630 INFO:tasks.rados.rados.0.smithi137.stdout:2157: done (3 left) 2024-09-21T08:32:19.630 INFO:tasks.rados.rados.0.smithi137.stdout:2159: done (2 left) 2024-09-21T08:32:19.630 INFO:tasks.rados.rados.0.smithi137.stdout:2160: done (1 left) 2024-09-21T08:32:19.630 INFO:tasks.rados.rados.0.smithi137.stdout:2161: done (0 left) 2024-09-21T08:32:19.630 INFO:tasks.rados.rados.0.smithi137.stdout:2162: snap_remove snap 199 2024-09-21T08:32:19.681 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:19.703 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:19.704 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:19.704 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:19.732 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[41044]: pgmap v722: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 608 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 13 op/s 2024-09-21T08:32:19.733 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[41044]: 4.14 scrub starts 2024-09-21T08:32:19.733 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[41044]: 4.14 scrub ok 2024-09-21T08:32:19.733 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[41044]: 6.16 scrub starts 2024-09-21T08:32:19.733 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[41044]: 6.16 scrub ok 2024-09-21T08:32:19.733 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2154509744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:32:19.733 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[41044]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:19.734 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2154509744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:32:19.734 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[41044]: osdmap e546: 8 total, 8 up, 5 in 2024-09-21T08:32:19.734 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[29634]: pgmap v722: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 608 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 13 op/s 2024-09-21T08:32:19.734 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[29634]: 4.14 scrub starts 2024-09-21T08:32:19.734 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[29634]: 4.14 scrub ok 2024-09-21T08:32:19.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[29634]: 6.16 scrub starts 2024-09-21T08:32:19.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[29634]: 6.16 scrub ok 2024-09-21T08:32:19.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2154509744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:32:19.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[29634]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:19.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2154509744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:32:19.735 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:19 smithi137 ceph-mon[29634]: osdmap e546: 8 total, 8 up, 5 in 2024-09-21T08:32:19.805 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:19.829 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:19.830 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:19.831 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:19.931 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:19.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:19.954 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:19.954 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:19 smithi165 ceph-mon[32893]: pgmap v722: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 608 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 5.0 MiB/s rd, 2.6 MiB/s wr, 13 op/s 2024-09-21T08:32:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:19 smithi165 ceph-mon[32893]: 4.14 scrub starts 2024-09-21T08:32:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:19 smithi165 ceph-mon[32893]: 4.14 scrub ok 2024-09-21T08:32:20.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:19 smithi165 ceph-mon[32893]: 6.16 scrub starts 2024-09-21T08:32:20.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:19 smithi165 ceph-mon[32893]: 6.16 scrub ok 2024-09-21T08:32:20.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:19 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2154509744' entity='client.admin' cmd=[{"prefix": "osd set", "key": "noscrub"}]: dispatch 2024-09-21T08:32:20.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:19 smithi165 ceph-mon[32893]: Health check failed: noscrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:20.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:19 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2154509744' entity='client.admin' cmd='[{"prefix": "osd set", "key": "noscrub"}]': finished 2024-09-21T08:32:20.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:19 smithi165 ceph-mon[32893]: osdmap e546: 8 total, 8 up, 5 in 2024-09-21T08:32:20.056 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:20.082 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:20.082 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:20.083 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:20.184 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:20.209 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:20.210 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:20.210 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:20.311 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:20.336 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:20.337 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:20.338 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:20.438 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:20.460 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:20.462 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:20.462 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:20.493 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:32:20.494 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:32:20.511 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:32:20.511 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:32:20.526 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:32:20.527 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:32:20.542 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:32:20.542 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:32:20.559 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:32:20.559 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:32:20.563 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:20.576 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:32:20.576 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:32:20.588 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:20.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:20.589 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:20.592 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:32:20.592 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:32:20.608 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:32:20.608 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:32:20.625 INFO:tasks.rados.rados.0.smithi137.stdout:2162: done (0 left) 2024-09-21T08:32:20.625 INFO:tasks.rados.rados.0.smithi137.stdout:2163: read oid 12 snap -1 2024-09-21T08:32:20.625 INFO:tasks.rados.rados.0.smithi137.stdout:2163: expect (ObjNum 741 snap 235 seq_num 741) 2024-09-21T08:32:20.625 INFO:tasks.rados.rados.0.smithi137.stdout:2164: snap_create 2024-09-21T08:32:20.641 INFO:tasks.rados.rados.0.smithi137.stdout:2163: expect (ObjNum 741 snap 235 seq_num 741) 2024-09-21T08:32:20.690 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:20.716 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:20.717 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:20.717 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:20 smithi137 ceph-mon[29634]: pgmap v725: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 605 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 255 B/s rd, 311 KiB/s wr, 1 op/s 2024-09-21T08:32:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:20 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/147969136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:20.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:20 smithi137 ceph-mon[29634]: osdmap e547: 8 total, 8 up, 5 in 2024-09-21T08:32:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:20 smithi137 ceph-mon[41044]: pgmap v725: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 605 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 255 B/s rd, 311 KiB/s wr, 1 op/s 2024-09-21T08:32:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:20 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/147969136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:20.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:20 smithi137 ceph-mon[41044]: osdmap e547: 8 total, 8 up, 5 in 2024-09-21T08:32:20.769 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:32:20 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:32:20] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:32:20.818 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:20.841 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:20.842 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:20.843 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:20.944 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:20.966 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:20.967 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:20.967 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:20 smithi165 ceph-mon[32893]: pgmap v725: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 605 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 255 B/s rd, 311 KiB/s wr, 1 op/s 2024-09-21T08:32:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:20 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/147969136' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:21.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:20 smithi165 ceph-mon[32893]: osdmap e547: 8 total, 8 up, 5 in 2024-09-21T08:32:21.069 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:21.090 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:21.090 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:21.091 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:21.191 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:21.216 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:21.216 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:21.216 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:21.317 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:21.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:21.343 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:21.343 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:21.391 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:21.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:21.468 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:21.469 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:21.469 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:21.570 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:21.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:21.599 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:21.599 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:21.613 INFO:tasks.rados.rados.0.smithi137.stdout:2163: done (1 left) 2024-09-21T08:32:21.613 INFO:tasks.rados.rados.0.smithi137.stdout:2164: done (0 left) 2024-09-21T08:32:21.613 INFO:tasks.rados.rados.0.smithi137.stdout:2165: write (excl) oid 48 current snap is 241 2024-09-21T08:32:21.613 INFO:tasks.rados.rados.0.smithi137.stdout:2165: seq_num 759 ranges {756513=632938,1865127=727266} 2024-09-21T08:32:21.614 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.5 to 1 (10000) 2024-09-21T08:32:21.621 INFO:tasks.rados.rados.0.smithi137.stdout:2165: writing smithi13779221-48 from 756513 to 1389451 tid 1 2024-09-21T08:32:21.630 INFO:tasks.rados.rados.0.smithi137.stdout:2165: writing smithi13779221-48 from 1865127 to 2592393 tid 2 2024-09-21T08:32:21.630 INFO:tasks.rados.rados.0.smithi137.stdout:2166: read oid 32 snap -1 2024-09-21T08:32:21.630 INFO:tasks.rados.rados.0.smithi137.stdout:2166: expect (ObjNum 754 snap 236 seq_num 754) 2024-09-21T08:32:21.630 INFO:tasks.rados.rados.0.smithi137.stdout:2167: copy_from oid 39 from oid 24 current snap is 241 2024-09-21T08:32:21.631 INFO:tasks.rados.rados.0.smithi137.stdout:2168: snap_remove snap 191 2024-09-21T08:32:21.700 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:21.726 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:21.726 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:21.727 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:21.827 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:21.849 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:21.850 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:21.851 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:21.951 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:21.955 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:21 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3666195452' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-21T08:32:21.955 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:21 smithi137 ceph-mon[29634]: from='client.46067 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-21T08:32:21.955 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:21 smithi137 ceph-mon[29634]: from='client.46067 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T08:32:21.955 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:21 smithi137 ceph-mon[29634]: osdmap e548: 8 total, 8 up, 5 in 2024-09-21T08:32:21.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:21 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3666195452' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-21T08:32:21.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:21 smithi137 ceph-mon[41044]: from='client.46067 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-21T08:32:21.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:21 smithi137 ceph-mon[41044]: from='client.46067 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T08:32:21.955 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:21 smithi137 ceph-mon[41044]: osdmap e548: 8 total, 8 up, 5 in 2024-09-21T08:32:21.978 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:21.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:21.979 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:22.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:21 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3666195452' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-21T08:32:22.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:21 smithi165 ceph-mon[32893]: from='client.46067 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]: dispatch 2024-09-21T08:32:22.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:21 smithi165 ceph-mon[32893]: from='client.46067 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 5, "weight": 1.0}]': finished 2024-09-21T08:32:22.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:21 smithi165 ceph-mon[32893]: osdmap e548: 8 total, 8 up, 5 in 2024-09-21T08:32:22.063 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:32:22.063 INFO:teuthology.orchestra.run.smithi137.stdout: "size": 20, 2024-09-21T08:32:22.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 600, 2024-09-21T08:32:22.063 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [ 2024-09-21T08:32:22.063 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.063 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:260 7.8 7:1fd29771:::smithi13779221-22:head [copy-from ver 15] snapc 5=[5] ondisk+write+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:32:22.063 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:32:22.063 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 455.49811653099999, 2024-09-21T08:32:22.063 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.025065418999999999, 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 260 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:32:22.064 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554322+0000", 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554325+0000", 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.667e-06 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554332+0000", 2024-09-21T08:32:22.065 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.1600000000000001e-06 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554346+0000", 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4011e-05 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554355+0000", 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9199999999999993e-06 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.066 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554524+0000", 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000169469 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554534+0000", 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5489999999999995e-06 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557746+0000", 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.003212164 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.067 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557822+0000", 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6108999999999999e-05 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.568552+0000", 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.01072972 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574773+0000", 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0062208760000000002 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.068 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.576999+0000", 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0022266059999999999 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579275+0000", 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.002276054 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579335+0000", 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9775000000000003e-05 2024-09-21T08:32:22.069 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.579388+0000", 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2339000000000002e-05 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:261 7.8 7:1fd29771:::smithi13779221-22:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e93)", 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 455.49797196999998, 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020259856, 2024-09-21T08:32:22.070 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 261 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.071 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554467+0000", 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554468+0000", 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0219999999999999e-06 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554475+0000", 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.81e-06 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.072 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554484+0000", 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.0550000000000005e-06 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554491+0000", 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.2080000000000003e-06 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554607+0000", 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011585400000000001 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.073 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.554612+0000", 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.0130000000000001e-06 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.557969+0000", 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0033576320000000002 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.558007+0000", 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.735e-05 2024-09-21T08:32:22.074 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574538+0000", 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.016531694 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574630+0000", 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1902000000000002e-05 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:46.574727+0000", 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6316000000000004e-05 2024-09-21T08:32:22.075 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:332 7.3 7:c851e2c3:::smithi13779221-25:head [read 0~2403368,checksum,omap-get-vals-by-keys,omap-get-keys,omap-get-vals,omap-get-header,getxattrs] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 454.366295149, 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90930888700000001, 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.076 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 332 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686144+0000", 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.077 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686147+0000", 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.5370000000000002e-06 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686160+0000", 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2542e-05 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686178+0000", 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.7927999999999998e-05 2024-09-21T08:32:22.078 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686187+0000", 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.3610000000000007e-06 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686261+0000", 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4197999999999997e-05 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686271+0000", 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.5259999999999999e-06 2024-09-21T08:32:22.079 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.590700+0000", 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90442903500000005 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.590838+0000", 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000137863 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595452+0000", 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0046148969999999997 2024-09-21T08:32:22.080 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:333 7.3 7:c851e2c3:::smithi13779221-25:head [read 0~2403368,checksum] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 454.36623162000001, 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91060290700000002, 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.081 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 333 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686207+0000", 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.082 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686208+0000", 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.4900000000000005e-07 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686212+0000", 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9940000000000002e-06 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686221+0000", 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8789999999999992e-06 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.083 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686225+0000", 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.5059999999999997e-06 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686303+0000", 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7855000000000006e-05 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686310+0000", 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0829999999999998e-06 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.084 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595485+0000", 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.90917505300000001 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.595590+0000", 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000105059 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.596810+0000", 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012197289999999999 2024-09-21T08:32:22.085 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:334 7.3 7:c851e2c3:::smithi13779221-25:head [sparse-read 0~1282710] snapc 0=[] ondisk+read+known_if_redirected+supports_pool_eio e94)", 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 454.36619461399999, 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.91097041300000003, 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.086 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 334 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686244+0000", 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.087 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686245+0000", 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.2600000000000002e-07 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686247+0000", 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7429999999999998e-06 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686255+0000", 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.8369999999999997e-06 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.088 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686259+0000", 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0509999999999998e-06 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686326+0000", 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.6845000000000001e-05 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:47.686331+0000", 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.13e-06 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.089 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.596911+0000", 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.910579845 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.597015+0000", 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000103538 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:24:48.597215+0000", 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000199798 2024-09-21T08:32:22.090 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:569 7.e 7:78d05d33:::smithi13779221-41:head [copy-from ver 7] snapc 14=[14,13,12,11,10,f,d] ondisk+write+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 431.50574588900002, 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.033085196999999997, 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.091 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 569 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546693+0000", 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.092 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546695+0000", 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3920000000000001e-06 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546704+0000", 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.9330000000000007e-06 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546720+0000", 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5651999999999999e-05 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.093 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546731+0000", 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1147e-05 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546794+0000", 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.3214999999999997e-05 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op must wait for map", 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546801+0000", 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.3350000000000001e-06 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.094 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556379+0000", 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0095778679999999998 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556505+0000", 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012529699999999999 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 0,5", 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.564765+0000", 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0082601020000000001 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.095 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576097+0000", 2024-09-21T08:32:22.096 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011331938999999999 2024-09-21T08:32:22.096 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.096 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.096 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:22.096 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.576715+0000", 2024-09-21T08:32:22.097 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00061843800000000002 2024-09-21T08:32:22.097 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.097 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579665+0000", 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0029497 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579729+0000", 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.4337000000000004e-05 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.579778+0000", 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.8841999999999998e-05 2024-09-21T08:32:22.098 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:570 7.e 7:78d05d33:::smithi13779221-41:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e117)", 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 431.50560479799998, 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.024747969000000002, 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.099 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 570 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546834+0000", 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.100 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546835+0000", 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.4e-07 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546838+0000", 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.4829999999999999e-06 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546845+0000", 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5810000000000002e-06 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.101 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546857+0000", 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2034e-05 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546900+0000", 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2945000000000003e-05 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting_for_map not empty", 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.546905+0000", 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2700000000000004e-06 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.102 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556676+0000", 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0097709930000000004 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for blocked object", 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.556717+0000", 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.104e-05 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571353+0000", 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.014636287 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.103 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571465+0000", 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000111943 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:25:10.571582+0000", 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00011655299999999999 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1571 7.18 7:1fd29771:::smithi13779221-22:head [copy-from ver 143] snapc 32=[32,31] ondisk+write+known_if_redirected+supports_pool_eio e182)", 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:16.499943+0000", 2024-09-21T08:32:22.104 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 365.55249555500001, 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.022751851999999999, 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1571 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499943+0000", 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.105 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499943+0000", 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499945+0000", 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.2230000000000001e-06 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499952+0000", 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.764e-06 2024-09-21T08:32:22.106 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499963+0000", 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1124000000000001e-05 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.499969+0000", 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.276e-06 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.500029+0000", 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.0927000000000001e-05 2024-09-21T08:32:22.107 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.500130+0000", 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00010014900000000001 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.509954+0000", 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0098240830000000008 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.518293+0000", 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0083390770000000003 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.108 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522396+0000", 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0041036809999999996 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522612+0000", 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00021587900000000001 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522652+0000", 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.9635000000000003e-05 2024-09-21T08:32:22.109 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:16.522695+0000", 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.3034000000000002e-05 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:1919 7.8 7:144c2ec5:::smithi13779221-45:head [delete] snapc 43=[40,3d,39] ondisk+write+peerstat_old+known_if_redirected+supports_pool_eio e215)", 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:26:48.572348+0000", 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 333.48009059399999, 2024-09-21T08:32:22.110 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.0652029459999999, 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 1919 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572348+0000", 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.111 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572348+0000", 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572350+0000", 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.928e-06 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572356+0000", 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.9449999999999998e-06 2024-09-21T08:32:22.112 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572366+0000", 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.0237e-05 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572375+0000", 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.7940000000000008e-06 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572776+0000", 2024-09-21T08:32:22.113 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040072699999999997 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:48.572789+0000", 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2974e-05 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.526784+0000", 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.95399481799999997 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for peered", 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:49.526798+0000", 2024-09-21T08:32:22.114 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.4428e-05 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.636113+0000", 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.109315396 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.636265+0000", 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00015202999999999999 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:32:22.115 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.636448+0000", 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000182957 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637097+0000", 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00064855999999999996 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637266+0000", 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016917399999999999 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.116 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637374+0000", 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107899 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637451+0000", 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.7486999999999999e-05 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:26:50.637551+0000", 2024-09-21T08:32:22.117 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.9592000000000005e-05 2024-09-21T08:32:22.117 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:22.119 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.119 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.119 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.119 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.119 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.119 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:2116 7.b e228/157 7:df2b42d9:::smithi13779221-36:head v 228'352, mlcod=225'348)", 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:01.532769+0000", 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 320.51966992600001, 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020877380000000001, 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.532769+0000", 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.120 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.532769+0000", 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.532772+0000", 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2729999999999998e-06 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542209+0000", 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0094370849999999996 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.121 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542235+0000", 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.6072999999999999e-05 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542247+0000", 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.2233e-05 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542328+0000", 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.0271000000000004e-05 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.122 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.542360+0000", 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.2166999999999999e-05 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553599+0000", 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011239416 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:01.553646+0000", 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.6862e-05 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.123 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2233 7.8 7:17170b1f:::smithi13779221-4:head [write 2798226~596838] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.553031+0000", 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 310.49940725099998, 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021618093000000001, 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.124 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2233 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553031+0000", 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553031+0000", 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.125 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553036+0000", 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7879999999999997e-06 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553403+0000", 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00036713799999999998 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553435+0000", 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.1219e-05 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.126 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553451+0000", 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.6841e-05 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553587+0000", 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013595600000000001 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553713+0000", 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00012593700000000001 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.127 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553874+0000", 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000160798 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.558304+0000", 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0044295280000000003 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.569637+0000", 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011333609 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.128 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574517+0000", 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0048793129999999997 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574592+0000", 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.5236999999999998e-05 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574650+0000", 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7729000000000002e-05 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.129 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2234 7.8 7:17170b1f:::smithi13779221-4:head [setxattr _header (56) in=7b,truncate 3445468] snapc 4e=[4e,4d,48,44,3d] ondisk+write+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.553487+0000", 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 310.49895175199998, 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.021391090000000001, 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2234 2024-09-21T08:32:22.130 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553487+0000", 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553487+0000", 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553489+0000", 2024-09-21T08:32:22.131 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.215e-06 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553495+0000", 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.7389999999999996e-06 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553500+0000", 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.2360000000000004e-06 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553505+0000", 2024-09-21T08:32:22.132 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3009999999999997e-06 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.554931+0000", 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00142592 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555047+0000", 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000115139 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 3,7", 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555183+0000", 2024-09-21T08:32:22.133 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00013642099999999999 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.558431+0000", 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0032475770000000002 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.569645+0000", 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011214783000000001 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574711+0000", 2024-09-21T08:32:22.134 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0050659449999999996 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574768+0000", 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6415000000000001e-05 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574878+0000", 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000110399 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.135 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2235 7.8 7:17170b1f:::smithi13779221-4:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e238)", 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 310.49892599600003, 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.02162265, 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "started", 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2235 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.136 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553513+0000", 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.3000000000000001e-07 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.137 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553516+0000", 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.5900000000000002e-06 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553522+0000", 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.8980000000000001e-06 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.553526+0000", 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7360000000000001e-06 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.138 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555581+0000", 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020548799999999998 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for rw locks", 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.555650+0000", 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.9046000000000003e-05 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.574919+0000", 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019268464999999999 2024-09-21T08:32:22.139 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.575027+0000", 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000108526 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:27:11.575135+0000", 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000107979 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.140 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:2831 7.8 7:144c2ec5:::smithi13779221-45:head [copy-from ver 295] snapc 64=[] ondisk+write+known_if_redirected+supports_pool_eio e288)", 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:01.538037+0000", 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 260.51440163000001, 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020800902, 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 2831 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.141 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538037+0000", 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538037+0000", 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538039+0000", 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.9099999999999999e-06 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.142 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538047+0000", 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.79e-06 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538058+0000", 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1141999999999999e-05 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538068+0000", 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6330000000000008e-06 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.143 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538117+0000", 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9360000000000002e-05 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.538218+0000", 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000101282 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.548519+0000", 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.010300971000000001 2024-09-21T08:32:22.144 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.557646+0000", 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0091271570000000003 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558736+0000", 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.001089996 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558773+0000", 2024-09-21T08:32:22.145 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.6785000000000001e-05 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558810+0000", 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.7058999999999999e-05 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:01.558838+0000", 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7817000000000002e-05 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.146 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3157 7.12 7:4ccee2c4:::smithi13779221-50:head [copy-from ver 217] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.573639+0000", 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 229.47879922000001, 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.02873676, 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3157 2024-09-21T08:32:22.147 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573639+0000", 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573639+0000", 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573642+0000", 2024-09-21T08:32:22.148 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.041e-06 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573649+0000", 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.6340000000000001e-06 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573662+0000", 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3263e-05 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573670+0000", 2024-09-21T08:32:22.149 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.418e-06 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573726+0000", 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.6103000000000001e-05 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573839+0000", 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000113186 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.586583+0000", 2024-09-21T08:32:22.150 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.012743775000000001 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.594268+0000", 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0076849559999999997 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602175+0000", 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0079072770000000007 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.151 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602261+0000", 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.6271000000000001e-05 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602327+0000", 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.5827000000000005e-05 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.602376+0000", 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9008999999999998e-05 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.152 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_op(client.14844.0:3159 7.12 7:4d793d31:::smithi13779221-9:head [copy-from ver 303] snapc 75=[75,74,73,72,71,6e,69] ondisk+write+known_if_redirected+supports_pool_eio e319)", 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:32.573715+0000", 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 229.478723573, 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019914487000000002, 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "client_info": { 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "client": "client.14844", 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "tid": 3159 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.153 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573715+0000", 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573715+0000", 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573716+0000", 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.06e-07 2024-09-21T08:32:22.154 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573719+0000", 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 3.371e-06 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573724+0000", 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4299999999999997e-06 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.573729+0000", 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9400000000000001e-06 2024-09-21T08:32:22.155 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.574567+0000", 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00083730199999999995 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.574658+0000", 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.1775999999999997e-05 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.580114+0000", 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0054551759999999999 2024-09-21T08:32:22.156 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "op_commit", 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.591315+0000", 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.011201825 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593389+0000", 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0020731130000000001 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593531+0000", 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000142144 2024-09-21T08:32:22.157 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593575+0000", 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.4229e-05 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:32.593630+0000", 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 5.4675000000000001e-05 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.158 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:3441 7.15 e343/339 7:a9dd6c7d:::smithi13779221-47:head v 343'399, mlcod=0'0)", 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:28:56.572197+0000", 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 205.480241581, 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019269069999999999, 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572197+0000", 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.159 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572197+0000", 2024-09-21T08:32:22.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.160 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572198+0000", 2024-09-21T08:32:22.160 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.7100000000000002e-07 2024-09-21T08:32:22.160 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.160 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.160 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572500+0000", 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00030198600000000002 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572508+0000", 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8799999999999997e-06 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572518+0000", 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 9.6269999999999995e-06 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.161 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572589+0000", 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0644999999999999e-05 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.572612+0000", 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3623000000000001e-05 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591357+0000", 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.018744255000000001 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.162 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:28:56.591466+0000", 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.000109583 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:4006 7.9 e398/339 7:9247b5b1:::smithi13779221-17:head v 398'338, mlcod=376'335)", 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:29:51.586649+0000", 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 150.46579010799999, 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.020193633999999999, 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.163 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586649+0000", 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586649+0000", 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.164 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.586650+0000", 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.545e-06 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587156+0000", 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00050577 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587177+0000", 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.1353999999999999e-05 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.165 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587192+0000", 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.5214e-05 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587422+0000", 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00022968399999999999 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.587586+0000", 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00016406799999999999 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.166 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.606793+0000", 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.019206392999999999 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:29:51.606842+0000", 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.9605999999999999e-05 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:4168 7.b e409/293 7:d7506421:::smithi13779221-13:head v 409'639, mlcod=407'636)", 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 139.47006007799999, 2024-09-21T08:32:22.167 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.083502604999999994, 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.168 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582379+0000", 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.2e-07 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582940+0000", 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00056103800000000003 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582952+0000", 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.1732999999999999e-05 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.169 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.582961+0000", 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 8.8000000000000004e-06 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.583031+0000", 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 7.0870000000000004e-05 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.583059+0000", 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.7206e-05 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.170 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665820+0000", 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.082760995000000004 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665881+0000", 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 6.1543000000000004e-05 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: "description": "osd_repop(client.14844.0:4169 7.b e409/293 7:d7506421:::smithi13779221-13:head v 409'640, mlcod=407'636)", 2024-09-21T08:32:22.171 INFO:teuthology.orchestra.run.smithi137.stdout: "initiated_at": "2024-09-21T08:30:02.589410+0000", 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "age": 139.463029202, 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.080226373000000004, 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "continuous": false, 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "type_data": { 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "events": [ 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "initiated", 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589410+0000", 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "header_read", 2024-09-21T08:32:22.172 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589410+0000", 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "throttled", 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589412+0000", 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.9840000000000001e-06 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "all_read", 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.589813+0000", 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.00040080299999999998 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "dispatched", 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665555+0000", 2024-09-21T08:32:22.173 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.075741557000000001 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "queued_for_pg", 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.665569+0000", 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 1.3961000000000001e-05 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "reached_pg", 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.668283+0000", 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0027142099999999999 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "started", 2024-09-21T08:32:22.174 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.668307+0000", 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 2.3487999999999999e-05 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "commit_sent", 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669596+0000", 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 0.0012892540000000001 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: }, 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: { 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: "event": "done", 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: "time": "2024-09-21T08:30:02.669636+0000", 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: "duration": 4.0116e-05 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.175 INFO:teuthology.orchestra.run.smithi137.stdout: } 2024-09-21T08:32:22.176 INFO:teuthology.orchestra.run.smithi137.stdout: ] 2024-09-21T08:32:22.176 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:32:22.177 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:22.177 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:22.177 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:22.278 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:22.300 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:22.300 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:22.301 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:22.401 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:22.421 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:22.422 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:22.422 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:22.513 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:32:22 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:32:22.523 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:22.544 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:22.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:22.545 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:22.620 INFO:tasks.rados.rados.0.smithi137.stdout:2168: done (3 left) 2024-09-21T08:32:22.620 INFO:tasks.rados.rados.0.smithi137.stdout:2169: rollback oid 37 current snap is 241 2024-09-21T08:32:22.620 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 37 to 230 2024-09-21T08:32:22.620 INFO:tasks.rados.rados.0.smithi137.stdout:2170: write oid 43 current snap is 241 2024-09-21T08:32:22.620 INFO:tasks.rados.rados.0.smithi137.stdout:2170: seq_num 760 ranges {773320=708187,1923609=470924} 2024-09-21T08:32:22.628 INFO:tasks.rados.rados.0.smithi137.stdout:2170: writing smithi13779221-43 from 773320 to 1481507 tid 1 2024-09-21T08:32:22.633 INFO:tasks.rados.rados.0.smithi137.stdout:2170: writing smithi13779221-43 from 1923609 to 2394533 tid 2 2024-09-21T08:32:22.633 INFO:tasks.rados.rados.0.smithi137.stdout:2171: write oid 46 current snap is 241 2024-09-21T08:32:22.633 INFO:tasks.rados.rados.0.smithi137.stdout:2171: seq_num 761 ranges {790127=499789,1698444=730935} 2024-09-21T08:32:22.640 INFO:tasks.rados.rados.0.smithi137.stdout:2171: writing smithi13779221-46 from 790127 to 1289916 tid 1 2024-09-21T08:32:22.646 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:22.647 INFO:tasks.rados.rados.0.smithi137.stdout:2171: writing smithi13779221-46 from 1698444 to 2429379 tid 2 2024-09-21T08:32:22.647 INFO:tasks.rados.rados.0.smithi137.stdout:2172: read oid 8 snap -1 2024-09-21T08:32:22.647 INFO:tasks.rados.rados.0.smithi137.stdout:2172: expect (ObjNum 753 snap 236 seq_num 753) 2024-09-21T08:32:22.647 INFO:tasks.rados.rados.0.smithi137.stdout:2173: rollback oid 49 current snap is 241 2024-09-21T08:32:22.647 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 49 to 209 2024-09-21T08:32:22.648 INFO:tasks.rados.rados.0.smithi137.stdout:2165: finishing write tid 1 to smithi13779221-48 2024-09-21T08:32:22.648 INFO:tasks.rados.rados.0.smithi137.stdout:2174: copy_from oid 40 from oid 6 current snap is 241 2024-09-21T08:32:22.648 INFO:tasks.rados.rados.0.smithi137.stdout:2165: finishing write tid 2 to smithi13779221-48 2024-09-21T08:32:22.648 INFO:tasks.rados.rados.0.smithi137.stdout:2166: expect (ObjNum 754 snap 236 seq_num 754) 2024-09-21T08:32:22.669 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:22.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:22.670 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:22.771 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:22.791 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:22.792 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:22.792 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:22.868 INFO:tasks.rados.rados.0.smithi137.stdout:2165: finishing write tid 3 to smithi13779221-48 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2165: finishing write tid 4 to smithi13779221-48 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2165: oid 48 updating version 0 to 830 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2165: oid 48 version 830 is already newer than 828 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2165: oid 48 version 830 is already newer than 829 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 830 (ObjNum 759 snap 241 seq_num 759) dirty exists 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2165: left oid 48 (ObjNum 759 snap 241 seq_num 759) 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2167: finishing copy_from racing read to smithi13779221-39 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2167: finishing copy_from to smithi13779221-39 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 39 v 743 (ObjNum 752 snap 236 seq_num 752) dirty exists 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2169: finishing rollback tid 0 to smithi13779221-37 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2169: finishing rollback tid 1 to smithi13779221-37 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2169: finishing rollback tid 2 to smithi13779221-37 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 866 (ObjNum 618 snap 182 seq_num 618) dirty exists 2024-09-21T08:32:22.869 INFO:tasks.rados.rados.0.smithi137.stdout:2170: finishing write tid 1 to smithi13779221-43 2024-09-21T08:32:22.870 INFO:tasks.rados.rados.0.smithi137.stdout:2165: done (8 left) 2024-09-21T08:32:22.870 INFO:tasks.rados.rados.0.smithi137.stdout:2166: done (7 left) 2024-09-21T08:32:22.870 INFO:tasks.rados.rados.0.smithi137.stdout:2167: done (6 left) 2024-09-21T08:32:22.870 INFO:tasks.rados.rados.0.smithi137.stdout:2169: done (5 left) 2024-09-21T08:32:22.870 INFO:tasks.rados.rados.0.smithi137.stdout:2175: snap_remove snap 192 2024-09-21T08:32:22.894 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:22.917 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:22.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:22.918 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:23.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:23.033 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd reweight 7 1 2024-09-21T08:32:23.041 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:23.042 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:23.042 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:23.143 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:23.165 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:23.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:23.166 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:23.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:23.288 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:23.289 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:23.289 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:23.390 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:23.399 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set nodeep-scrub 2024-09-21T08:32:23.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:23.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:23.417 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:23.473 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_ops_in_flight 2024-09-21T08:32:23.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:23.540 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:23.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:23.541 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:23.642 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:23.667 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:23.668 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:23.668 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:23.719 INFO:tasks.rados.rados.0.smithi137.stdout:2175: done (5 left) 2024-09-21T08:32:23.719 INFO:tasks.rados.rados.0.smithi137.stdout:2176: write (excl) oid 33 current snap is 241 2024-09-21T08:32:23.719 INFO:tasks.rados.rados.0.smithi137.stdout:2170: finishing write tid 2 to smithi13779221-43 2024-09-21T08:32:23.719 INFO:tasks.rados.rados.0.smithi137.stdout:2170: finishing write tid 3 to smithi13779221-43 2024-09-21T08:32:23.719 INFO:tasks.rados.rados.0.smithi137.stdout:2170: finishing write tid 4 to smithi13779221-43 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2170: oid 43 updating version 0 to 423 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2170: oid 43 version 423 is already newer than 422 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2170: oid 43 version 423 is already newer than 421 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 43 v 423 (ObjNum 760 snap 241 seq_num 760) dirty exists 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2170: left oid 43 (ObjNum 760 snap 241 seq_num 760) 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2171: finishing write tid 1 to smithi13779221-46 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2174: finishing copy_from to smithi13779221-40 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2174: got expected ENOENT (src dne) 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2174: finishing copy_from racing read to smithi13779221-40 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2173: finishing rollback tid 0 to smithi13779221-49 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2173: finishing rollback tid 1 to smithi13779221-49 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2173: finishing rollback tid 2 to smithi13779221-49 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 49 v 851 (ObjNum 560 snap 167 seq_num 560) dirty exists 2024-09-21T08:32:23.720 INFO:tasks.rados.rados.0.smithi137.stdout:2171: finishing write tid 2 to smithi13779221-46 2024-09-21T08:32:23.721 INFO:tasks.rados.rados.0.smithi137.stdout:2171: finishing write tid 3 to smithi13779221-46 2024-09-21T08:32:23.721 INFO:tasks.rados.rados.0.smithi137.stdout:2171: finishing write tid 4 to smithi13779221-46 2024-09-21T08:32:23.721 INFO:tasks.rados.rados.0.smithi137.stdout:2171: oid 46 updating version 0 to 916 2024-09-21T08:32:23.721 INFO:tasks.rados.rados.0.smithi137.stdout:2171: oid 46 version 916 is already newer than 915 2024-09-21T08:32:23.721 INFO:tasks.rados.rados.0.smithi137.stdout:2171: oid 46 updating version 916 to 917 2024-09-21T08:32:23.721 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 46 v 917 (ObjNum 761 snap 241 seq_num 761) dirty exists 2024-09-21T08:32:23.721 INFO:tasks.rados.rados.0.smithi137.stdout:2171: left oid 46 (ObjNum 761 snap 241 seq_num 761) 2024-09-21T08:32:23.721 INFO:tasks.rados.rados.0.smithi137.stdout:2172: expect (ObjNum 753 snap 236 seq_num 753) 2024-09-21T08:32:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:23 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:23 smithi137 ceph-mon[41044]: pgmap v728: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 609 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-21T08:32:23.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:23 smithi137 ceph-mon[41044]: osdmap e549: 8 total, 8 up, 5 in 2024-09-21T08:32:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:23 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:23 smithi137 ceph-mon[29634]: pgmap v728: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 609 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-21T08:32:23.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:23 smithi137 ceph-mon[29634]: osdmap e549: 8 total, 8 up, 5 in 2024-09-21T08:32:23.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:23.793 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:23.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:23.794 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:23.832 INFO:tasks.rados.rados.0.smithi137.stdout:2176: seq_num 762 ranges {406934=575038,1640573=474593,2642791=164143} 2024-09-21T08:32:23.837 INFO:tasks.rados.rados.0.smithi137.stdout:2176: writing smithi13779221-33 from 406934 to 981972 tid 1 2024-09-21T08:32:23.843 INFO:tasks.rados.rados.0.smithi137.stdout:2176: writing smithi13779221-33 from 1640573 to 2115166 tid 2 2024-09-21T08:32:23.848 INFO:tasks.rados.rados.0.smithi137.stdout:2176: writing smithi13779221-33 from 2642791 to 2806934 tid 3 2024-09-21T08:32:23.848 INFO:tasks.rados.rados.0.smithi137.stdout:2170: done (5 left) 2024-09-21T08:32:23.848 INFO:tasks.rados.rados.0.smithi137.stdout:2171: done (4 left) 2024-09-21T08:32:23.848 INFO:tasks.rados.rados.0.smithi137.stdout:2172: done (3 left) 2024-09-21T08:32:23.848 INFO:tasks.rados.rados.0.smithi137.stdout:2173: done (2 left) 2024-09-21T08:32:23.848 INFO:tasks.rados.rados.0.smithi137.stdout:2174: done (1 left) 2024-09-21T08:32:23.848 INFO:tasks.rados.rados.0.smithi137.stdout:2177: snap_create 2024-09-21T08:32:23.848 INFO:tasks.rados.rados.0.smithi137.stdout:2176: finishing write tid 1 to smithi13779221-33 2024-09-21T08:32:23.848 INFO:tasks.rados.rados.0.smithi137.stdout:2176: finishing write tid 2 to smithi13779221-33 2024-09-21T08:32:23.849 INFO:tasks.rados.rados.0.smithi137.stdout:2176: finishing write tid 3 to smithi13779221-33 2024-09-21T08:32:23.849 INFO:tasks.rados.rados.0.smithi137.stdout:2176: finishing write tid 4 to smithi13779221-33 2024-09-21T08:32:23.850 INFO:tasks.rados.rados.0.smithi137.stdout:2176: finishing write tid 5 to smithi13779221-33 2024-09-21T08:32:23.850 INFO:tasks.rados.rados.0.smithi137.stdout:2176: oid 33 updating version 0 to 364 2024-09-21T08:32:23.850 INFO:tasks.rados.rados.0.smithi137.stdout:2176: oid 33 version 364 is already newer than 362 2024-09-21T08:32:23.850 INFO:tasks.rados.rados.0.smithi137.stdout:2176: oid 33 version 364 is already newer than 363 2024-09-21T08:32:23.850 INFO:tasks.rados.rados.0.smithi137.stdout:2176: oid 33 version 364 is already newer than 361 2024-09-21T08:32:23.850 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 364 (ObjNum 762 snap 241 seq_num 762) dirty exists 2024-09-21T08:32:23.850 INFO:tasks.rados.rados.0.smithi137.stdout:2176: left oid 33 (ObjNum 762 snap 241 seq_num 762) 2024-09-21T08:32:23.895 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:23.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:23.918 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:23.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:23.974 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:23.988 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:23 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:23 smithi165 ceph-mon[32893]: pgmap v728: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 609 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 1.6 MiB/s wr, 15 op/s 2024-09-21T08:32:24.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:23 smithi165 ceph-mon[32893]: osdmap e549: 8 total, 8 up, 5 in 2024-09-21T08:32:24.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:24.041 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:24.042 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:24.042 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:24.142 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:24.166 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:24.166 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:24.167 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:24.267 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:24.289 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:24.290 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:24.290 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:24.391 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:24.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:24.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:24.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:24.518 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:24.543 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:24.544 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:24.544 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:24.645 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:24.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:24.671 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:24.672 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:24.721 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.7 to 1 (10000) 2024-09-21T08:32:24.723 INFO:tasks.rados.rados.0.smithi137.stdout:2176: done (1 left) 2024-09-21T08:32:24.723 INFO:tasks.rados.rados.0.smithi137.stdout:2177: done (0 left) 2024-09-21T08:32:24.724 INFO:tasks.rados.rados.0.smithi137.stdout:2178: delete oid 20 current snap is 242 2024-09-21T08:32:24.731 INFO:tasks.rados.rados.0.smithi137.stdout:2178: done (0 left) 2024-09-21T08:32:24.731 INFO:tasks.rados.rados.0.smithi137.stdout:2179: snap_create 2024-09-21T08:32:24.772 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:24.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:24.794 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:24.794 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:24.895 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:24.916 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:24.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:24.917 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:24 smithi165 ceph-mon[32893]: osdmap e550: 8 total, 8 up, 5 in 2024-09-21T08:32:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:24 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:32:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:24 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:32:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:24 smithi165 ceph-mon[32893]: pgmap v731: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 609 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-21T08:32:25.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:24 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/294292971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-21T08:32:25.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:24 smithi137 ceph-mon[29634]: osdmap e550: 8 total, 8 up, 5 in 2024-09-21T08:32:25.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:24 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:32:25.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:24 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:32:25.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:24 smithi137 ceph-mon[29634]: pgmap v731: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 609 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-21T08:32:25.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:24 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/294292971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-21T08:32:25.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:24 smithi137 ceph-mon[41044]: osdmap e550: 8 total, 8 up, 5 in 2024-09-21T08:32:25.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:24 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:32:25.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:24 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:32:25.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:24 smithi137 ceph-mon[41044]: pgmap v731: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 609 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 10 MiB/s rd, 1.3 MiB/s wr, 14 op/s 2024-09-21T08:32:25.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:24 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/294292971' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]: dispatch 2024-09-21T08:32:25.018 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:25.044 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:25.045 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:25.045 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:25.113 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:25.146 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:25.167 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:25.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:25.168 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:25.269 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:25.290 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:25.291 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:25.291 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:25.392 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:25.412 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:25.413 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:25.413 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:25.514 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:25.540 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:25.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:25.541 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:25.642 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:25.663 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:25.663 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:25.664 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:25.729 INFO:tasks.rados.rados.0.smithi137.stdout:2179: done (0 left) 2024-09-21T08:32:25.729 INFO:tasks.rados.rados.0.smithi137.stdout:2180: snap_remove snap 209 2024-09-21T08:32:25.729 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is set 2024-09-21T08:32:25.765 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:25 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/294292971' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T08:32:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:25 smithi137 ceph-mon[29634]: osdmap e551: 8 total, 8 up, 5 in 2024-09-21T08:32:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:25 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/23144580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:25.768 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:25 smithi137 ceph-mon[29634]: from='client.46079 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:25 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/294292971' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T08:32:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:25 smithi137 ceph-mon[41044]: osdmap e551: 8 total, 8 up, 5 in 2024-09-21T08:32:25.768 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:25 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/23144580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:25.769 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:25 smithi137 ceph-mon[41044]: from='client.46079 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:25.788 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:25.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:25.789 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:25.876 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd reweight 1 1 2024-09-21T08:32:25.889 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:25.929 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:32:25.940 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:25.941 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:25.941 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:25 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/294292971' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 7, "weight": 1.0}]': finished 2024-09-21T08:32:26.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:25 smithi165 ceph-mon[32893]: osdmap e551: 8 total, 8 up, 5 in 2024-09-21T08:32:26.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:25 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/23144580' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:26.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:25 smithi165 ceph-mon[32893]: from='client.46079 ' entity='client.admin' cmd=[{"prefix": "osd set", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:26.042 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:26.054 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:32:26.054 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:32:26.054 INFO:teuthology.orchestra.run.smithi137.stdout: "num_ops": 0 2024-09-21T08:32:26.054 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:32:26.067 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:26.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:26.068 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:26.169 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:26.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:26.192 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:26.192 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:26.292 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:26.318 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:26.318 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:26.319 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:26.420 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:26.444 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:26.445 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:26.445 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:26.546 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:26.567 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:26.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:26.568 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:26.669 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:26.699 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:26.699 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:26.700 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:26.744 INFO:tasks.rados.rados.0.smithi137.stdout:2180: done (0 left) 2024-09-21T08:32:26.744 INFO:tasks.rados.rados.0.smithi137.stdout:2181: rollback oid 11 current snap is 243 2024-09-21T08:32:26.744 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 11 to 233 2024-09-21T08:32:26.744 INFO:tasks.rados.rados.0.smithi137.stdout:2182: read oid 8 snap -1 2024-09-21T08:32:26.744 INFO:tasks.rados.rados.0.smithi137.stdout:2182: expect (ObjNum 753 snap 236 seq_num 753) 2024-09-21T08:32:26.744 INFO:tasks.rados.rados.0.smithi137.stdout:2183: snap_create 2024-09-21T08:32:26.747 INFO:tasks.rados.rados.0.smithi137.stdout:2181: finishing rollback tid 1 to smithi13779221-11 2024-09-21T08:32:26.747 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 11 v 536 (ObjNum 129 snap 0 seq_num 3761188048) dirty dne 2024-09-21T08:32:26.753 INFO:tasks.rados.rados.0.smithi137.stdout:2182: expect (ObjNum 753 snap 236 seq_num 753) 2024-09-21T08:32:26.800 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:26.824 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:26.825 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:26.825 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:26.831 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:26 smithi137 ceph-mon[29634]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:26.831 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:26 smithi137 ceph-mon[29634]: from='client.46079 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:26.831 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:26 smithi137 ceph-mon[29634]: osdmap e552: 8 total, 8 up, 5 in 2024-09-21T08:32:26.831 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:26 smithi137 ceph-mon[29634]: pgmap v734: 188 pgs: 1 remapped+peering, 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 154 active+clean; 599 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 5.3 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-21T08:32:26.831 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:26 smithi137 ceph-mon[41044]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:26.831 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:26 smithi137 ceph-mon[41044]: from='client.46079 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:26.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:26 smithi137 ceph-mon[41044]: osdmap e552: 8 total, 8 up, 5 in 2024-09-21T08:32:26.832 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:26 smithi137 ceph-mon[41044]: pgmap v734: 188 pgs: 1 remapped+peering, 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 154 active+clean; 599 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 5.3 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-21T08:32:26.857 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_blocked_ops 2024-09-21T08:32:26.926 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:26.949 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:26.950 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:26.950 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:26 smithi165 ceph-mon[32893]: Health check update: noscrub,nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:26 smithi165 ceph-mon[32893]: from='client.46079 ' entity='client.admin' cmd='[{"prefix": "osd set", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:26 smithi165 ceph-mon[32893]: osdmap e552: 8 total, 8 up, 5 in 2024-09-21T08:32:27.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:26 smithi165 ceph-mon[32893]: pgmap v734: 188 pgs: 1 remapped+peering, 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 154 active+clean; 599 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 5.3 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-21T08:32:27.051 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:27.071 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:27.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:27.072 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:27.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:27.194 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:27.195 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:27.195 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:27.296 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:27.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:27.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:27.320 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:27.421 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:27.443 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:27.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:27.444 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:27.447 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:27.467 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:27.545 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:27.567 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:27.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:27.568 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:27.668 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:27.694 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:27.695 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:27.695 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:27.741 INFO:tasks.rados.rados.0.smithi137.stdout:2181: done (2 left) 2024-09-21T08:32:27.741 INFO:tasks.rados.rados.0.smithi137.stdout:2182: done (1 left) 2024-09-21T08:32:27.741 INFO:tasks.rados.rados.0.smithi137.stdout:2183: done (0 left) 2024-09-21T08:32:27.741 INFO:tasks.rados.rados.0.smithi137.stdout:2184: read oid 39 snap -1 2024-09-21T08:32:27.741 INFO:tasks.rados.rados.0.smithi137.stdout:2184: expect (ObjNum 752 snap 236 seq_num 752) 2024-09-21T08:32:27.741 INFO:tasks.rados.rados.0.smithi137.stdout:2185: write (excl) oid 13 current snap is 244 2024-09-21T08:32:27.741 INFO:tasks.rados.rados.0.smithi137.stdout:2185: seq_num 764 ranges {440548=725536,1757537=478262,2813990=26558} 2024-09-21T08:32:27.748 INFO:tasks.rados.rados.0.smithi137.stdout:2185: writing smithi13779221-13 from 440548 to 1166084 tid 1 2024-09-21T08:32:27.754 INFO:tasks.rados.rados.0.smithi137.stdout:2185: writing smithi13779221-13 from 1757537 to 2235799 tid 2 2024-09-21T08:32:27.758 INFO:tasks.rados.rados.0.smithi137.stdout:2185: writing smithi13779221-13 from 2813990 to 2840548 tid 3 2024-09-21T08:32:27.758 INFO:tasks.rados.rados.0.smithi137.stdout:2186: rollback oid 28 current snap is 244 2024-09-21T08:32:27.758 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 28 to 215 2024-09-21T08:32:27.758 INFO:tasks.rados.rados.0.smithi137.stdout:2185: finishing write tid 1 to smithi13779221-13 2024-09-21T08:32:27.758 INFO:tasks.rados.rados.0.smithi137.stdout:2187: write oid 33 current snap is 244 2024-09-21T08:32:27.759 INFO:tasks.rados.rados.0.smithi137.stdout:2187: seq_num 765 ranges {457355=400785,1416019=621920,2725060=132295} 2024-09-21T08:32:27.763 INFO:tasks.rados.rados.0.smithi137.stdout:2187: writing smithi13779221-33 from 457355 to 858140 tid 1 2024-09-21T08:32:27.769 INFO:tasks.rados.rados.0.smithi137.stdout:2187: writing smithi13779221-33 from 1416019 to 2037939 tid 2 2024-09-21T08:32:27.774 INFO:tasks.rados.rados.0.smithi137.stdout:2187: writing smithi13779221-33 from 2725060 to 2857355 tid 3 2024-09-21T08:32:27.774 INFO:tasks.rados.rados.0.smithi137.stdout:2188: read oid 26 snap -1 2024-09-21T08:32:27.774 INFO:tasks.rados.rados.0.smithi137.stdout:2188: expect (ObjNum 757 snap 239 seq_num 757) 2024-09-21T08:32:27.774 INFO:tasks.rados.rados.0.smithi137.stdout:2189: copy_from oid 9 from oid 35 current snap is 244 2024-09-21T08:32:27.774 INFO:tasks.rados.rados.0.smithi137.stdout:2185: finishing write tid 2 to smithi13779221-13 2024-09-21T08:32:27.775 INFO:tasks.rados.rados.0.smithi137.stdout:2184: expect (ObjNum 752 snap 236 seq_num 752) 2024-09-21T08:32:27.796 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:27.818 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:27.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:27.819 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:2185: finishing write tid 3 to smithi13779221-13 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:2185: finishing write tid 4 to smithi13779221-13 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:2184: done (5 left) 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:2190: snap_create 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:2185: finishing write tid 5 to smithi13779221-13 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:2185: oid 13 updating version 0 to 929 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:2185: oid 13 updating version 929 to 930 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:2185: oid 13 updating version 930 to 932 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:2185: oid 13 version 932 is already newer than 931 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 932 (ObjNum 764 snap 244 seq_num 764) dirty exists 2024-09-21T08:32:27.920 INFO:tasks.rados.rados.0.smithi137.stdout:2185: left oid 13 (ObjNum 764 snap 244 seq_num 764) 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2186: finishing rollback tid 0 to smithi13779221-28 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2186: finishing rollback tid 1 to smithi13779221-28 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2186: finishing rollback tid 2 to smithi13779221-28 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 28 v 458 (ObjNum 656 snap 196 seq_num 656) dirty exists 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2187: finishing write tid 1 to smithi13779221-33 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2187: finishing write tid 2 to smithi13779221-33 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2187: finishing write tid 3 to smithi13779221-33 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2187: finishing write tid 4 to smithi13779221-33 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2187: finishing write tid 5 to smithi13779221-33 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2187: oid 33 updating version 0 to 367 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2187: oid 33 version 367 is already newer than 365 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2187: oid 33 updating version 367 to 368 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2187: oid 33 version 368 is already newer than 366 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 33 v 368 (ObjNum 765 snap 244 seq_num 765) dirty exists 2024-09-21T08:32:27.921 INFO:tasks.rados.rados.0.smithi137.stdout:2187: left oid 33 (ObjNum 765 snap 244 seq_num 765) 2024-09-21T08:32:27.922 INFO:tasks.rados.rados.0.smithi137.stdout:2188: expect (ObjNum 757 snap 239 seq_num 757) 2024-09-21T08:32:27.922 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:27.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:27 smithi137 ceph-mon[29634]: osdmap e553: 8 total, 8 up, 5 in 2024-09-21T08:32:27.946 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:27 smithi137 ceph-mon[41044]: osdmap e553: 8 total, 8 up, 5 in 2024-09-21T08:32:27.947 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:27.947 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:27.948 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:28.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:27 smithi165 ceph-mon[32893]: osdmap e553: 8 total, 8 up, 5 in 2024-09-21T08:32:28.047 INFO:tasks.rados.rados.0.smithi137.stdout:2189: finishing copy_from racing read to smithi13779221-9 2024-09-21T08:32:28.048 INFO:tasks.rados.rados.0.smithi137.stdout:2189: finishing copy_from to smithi13779221-9 2024-09-21T08:32:28.048 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 9 v 751 (ObjNum 747 snap 236 seq_num 747) dirty exists 2024-09-21T08:32:28.049 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:28.072 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:28.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:28.073 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:28.148 INFO:teuthology.orchestra.run.smithi137.stdout:{ 2024-09-21T08:32:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "ops": [], 2024-09-21T08:32:28.148 INFO:teuthology.orchestra.run.smithi137.stdout: "complaint_time": 30, 2024-09-21T08:32:28.149 INFO:teuthology.orchestra.run.smithi137.stdout: "num_blocked_ops": 0 2024-09-21T08:32:28.149 INFO:teuthology.orchestra.run.smithi137.stdout:} 2024-09-21T08:32:28.174 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:28.197 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:28.198 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:28.198 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:28.299 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:28.322 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:28.322 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:28.322 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:28.423 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:28.447 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:28.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:28.448 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:28.549 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:28.572 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:28.573 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:28.573 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:28.674 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:28.695 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:28.696 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:28.696 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:28.753 INFO:tasks.rados.rados.0.smithi137.stdout:2185: done (5 left) 2024-09-21T08:32:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:2186: done (4 left) 2024-09-21T08:32:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:2187: done (3 left) 2024-09-21T08:32:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:2188: done (2 left) 2024-09-21T08:32:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:2189: done (1 left) 2024-09-21T08:32:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:2190: done (0 left) 2024-09-21T08:32:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:2191: copy_from oid 5 from oid 11 current snap is 245 2024-09-21T08:32:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:2192: write (excl) oid 34 current snap is 245 2024-09-21T08:32:28.754 INFO:tasks.rados.rados.0.smithi137.stdout:2192: seq_num 766 ranges {474162=476034,1358148=481931,2352483=465303} 2024-09-21T08:32:28.759 INFO:tasks.rados.rados.0.smithi137.stdout:2192: writing smithi13779221-34 from 474162 to 950196 tid 1 2024-09-21T08:32:28.764 INFO:tasks.rados.rados.0.smithi137.stdout:2192: writing smithi13779221-34 from 1358148 to 1840079 tid 2 2024-09-21T08:32:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:2192: writing smithi13779221-34 from 2352483 to 2817786 tid 3 2024-09-21T08:32:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:2193: read oid 10 snap -1 2024-09-21T08:32:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:2193: expect deleted 2024-09-21T08:32:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:2194: snap_create 2024-09-21T08:32:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:2191: finishing copy_from to smithi13779221-5 2024-09-21T08:32:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:2191: got expected ENOENT (src dne) 2024-09-21T08:32:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:2191: finishing copy_from racing read to smithi13779221-5 2024-09-21T08:32:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:2192: finishing write tid 1 to smithi13779221-34 2024-09-21T08:32:28.770 INFO:tasks.rados.rados.0.smithi137.stdout:2192: finishing write tid 2 to smithi13779221-34 2024-09-21T08:32:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:2192: finishing write tid 3 to smithi13779221-34 2024-09-21T08:32:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:2192: finishing write tid 4 to smithi13779221-34 2024-09-21T08:32:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:2192: finishing write tid 5 to smithi13779221-34 2024-09-21T08:32:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:2192: oid 34 updating version 0 to 856 2024-09-21T08:32:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:2192: oid 34 version 856 is already newer than 854 2024-09-21T08:32:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:2192: oid 34 version 856 is already newer than 855 2024-09-21T08:32:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:2192: oid 34 version 856 is already newer than 853 2024-09-21T08:32:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 34 v 856 (ObjNum 766 snap 245 seq_num 766) dirty exists 2024-09-21T08:32:28.772 INFO:tasks.rados.rados.0.smithi137.stdout:2192: left oid 34 (ObjNum 766 snap 245 seq_num 766) 2024-09-21T08:32:28.797 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:28.818 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:28.819 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:28.819 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:28.871 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.3 dump_historic_ops 2024-09-21T08:32:28.920 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:28.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:28.945 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:28.945 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:29.004 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:28 smithi137 ceph-mon[29634]: osdmap e554: 8 total, 8 up, 5 in 2024-09-21T08:32:29.005 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:28 smithi137 ceph-mon[29634]: pgmap v737: 188 pgs: 1 remapped+peering, 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 154 active+clean; 599 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 5.3 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-21T08:32:29.005 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:28 smithi137 ceph-mon[41044]: osdmap e554: 8 total, 8 up, 5 in 2024-09-21T08:32:29.005 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:28 smithi137 ceph-mon[41044]: pgmap v737: 188 pgs: 1 remapped+peering, 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 154 active+clean; 599 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 5.3 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-21T08:32:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:28 smithi165 ceph-mon[32893]: osdmap e554: 8 total, 8 up, 5 in 2024-09-21T08:32:29.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:28 smithi165 ceph-mon[32893]: pgmap v737: 188 pgs: 1 remapped+peering, 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 154 active+clean; 599 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 5.3 MiB/s rd, 1.8 MiB/s wr, 13 op/s 2024-09-21T08:32:29.045 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:29.067 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:29.068 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:29.068 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:29.120 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:32:29.121 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:32:29.123 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:32:29.123 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:32:29.126 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:29.147 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:32:29.147 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:32:29.167 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:32:29.168 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:32:29.169 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:29.184 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:32:29.185 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:32:29.193 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:29.194 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:29.194 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:29.202 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:32:29.202 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:32:29.218 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:32:29.218 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:32:29.235 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:32:29.235 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:32:29.295 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:29.320 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:29.321 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:29.321 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:29.408 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset noscrub 2024-09-21T08:32:29.422 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:29.446 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:29.447 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:29.447 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:29.548 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:29.574 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:29.575 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:29.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:29.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:29.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:29.697 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:29.756 INFO:tasks.rados.rados.0.smithi137.stdout:2191: done (3 left) 2024-09-21T08:32:29.756 INFO:tasks.rados.rados.0.smithi137.stdout:2192: done (2 left) 2024-09-21T08:32:29.756 INFO:tasks.rados.rados.0.smithi137.stdout:2193: done (1 left) 2024-09-21T08:32:29.756 INFO:tasks.rados.rados.0.smithi137.stdout:2194: done (0 left) 2024-09-21T08:32:29.756 INFO:tasks.rados.rados.0.smithi137.stdout:2195: snap_remove snap 241 2024-09-21T08:32:29.799 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:29.821 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:29.822 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:29.822 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:29.923 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:29.944 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:29.944 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:29.945 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:30.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:29 smithi165 ceph-mon[32893]: osdmap e555: 8 total, 8 up, 5 in 2024-09-21T08:32:30.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:29 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2797711585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:30.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:29 smithi137 ceph-mon[29634]: osdmap e555: 8 total, 8 up, 5 in 2024-09-21T08:32:30.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:29 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2797711585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:30.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:29 smithi137 ceph-mon[41044]: osdmap e555: 8 total, 8 up, 5 in 2024-09-21T08:32:30.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:29 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2797711585' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:30.045 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:30.071 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:30.072 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:30.072 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:30.173 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:30.195 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:30.196 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:30.196 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:30.278 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:30.282 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:30.297 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:30.319 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:30.320 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:30.321 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:30.421 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:30.451 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:30.451 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:30.451 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:30.552 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:30.573 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:30.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:30.574 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:30.675 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:30.697 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:30.697 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:30.697 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:30.734 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:32:30] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:32:30.754 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.1 to 1 (10000) 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2195: done (0 left) 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2196: read oid 10 snap 237 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2196: expect deleted 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2197: copy_from oid 23 from oid 38 current snap is 246 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2198: read oid 50 snap -1 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2198: expect (ObjNum 660 snap 197 seq_num 660) 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2199: read oid 28 snap -1 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2199: expect (ObjNum 656 snap 196 seq_num 656) 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2200: copy_from oid 13 from oid 36 current snap is 246 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2201: write (excl) oid 29 current snap is 246 2024-09-21T08:32:30.760 INFO:tasks.rados.rados.0.smithi137.stdout:2201: seq_num 767 ranges {490969=551283,1816630=625589} 2024-09-21T08:32:30.766 INFO:tasks.rados.rados.0.smithi137.stdout:2201: writing smithi13779221-29 from 490969 to 1042252 tid 1 2024-09-21T08:32:30.775 INFO:tasks.rados.rados.0.smithi137.stdout:2201: writing smithi13779221-29 from 1816630 to 2442219 tid 2 2024-09-21T08:32:30.776 INFO:tasks.rados.rados.0.smithi137.stdout:2196: done (5 left) 2024-09-21T08:32:30.776 INFO:tasks.rados.rados.0.smithi137.stdout:2202: snap_remove snap 213 2024-09-21T08:32:30.798 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:30.819 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:30.820 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:30.820 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:30.921 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:30.943 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:30.943 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:30.943 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:30.991 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[29634]: osdmap e556: 8 total, 8 up, 5 in 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3085180477' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[29634]: pgmap v740: 188 pgs: 1 remapped+peering, 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 154 active+clean; 601 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 255 B/s rd, 301 KiB/s wr, 1 op/s 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[41044]: osdmap e556: 8 total, 8 up, 5 in 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3085180477' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[41044]: pgmap v740: 188 pgs: 1 remapped+peering, 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 154 active+clean; 601 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 255 B/s rd, 301 KiB/s wr, 1 op/s 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:32:30.992 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:30 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:32:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:30 smithi165 ceph-mon[32893]: osdmap e556: 8 total, 8 up, 5 in 2024-09-21T08:32:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:30 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3085180477' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]: dispatch 2024-09-21T08:32:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:30 smithi165 ceph-mon[32893]: pgmap v740: 188 pgs: 1 remapped+peering, 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 154 active+clean; 601 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 255 B/s rd, 301 KiB/s wr, 1 op/s 2024-09-21T08:32:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:30 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:31.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-09-21T08:32:31.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:30 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-09-21T08:32:31.044 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:31.065 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:31.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:31.066 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:31.167 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:31.189 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:31.189 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:31.190 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:31.290 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:31.313 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:31.313 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:31.314 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:31.413 INFO:teuthology.orchestra.run.smithi137.stdout:ERROR: (22) Invalid argument 2024-09-21T08:32:31.413 INFO:teuthology.orchestra.run.smithi137.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:32:31.414 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:31.437 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:31.438 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:31.438 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:31.539 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:31.564 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:31.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:31.565 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:31.619 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is unset 2024-09-21T08:32:31.620 INFO:tasks.rados.rados.0.smithi137.stdout:2202: done (5 left) 2024-09-21T08:32:31.620 INFO:tasks.rados.rados.0.smithi137.stdout:2203: write oid 31 current snap is 246 2024-09-21T08:32:31.620 INFO:tasks.rados.rados.0.smithi137.stdout:2203: seq_num 768 ranges {507776=742885,1991465=485600} 2024-09-21T08:32:31.627 INFO:tasks.rados.rados.0.smithi137.stdout:2203: writing smithi13779221-31 from 507776 to 1250661 tid 1 2024-09-21T08:32:31.634 INFO:tasks.rados.rados.0.smithi137.stdout:2203: writing smithi13779221-31 from 1991465 to 2477065 tid 2 2024-09-21T08:32:31.637 INFO:tasks.rados.rados.0.smithi137.stdout:2201: finishing write tid 1 to smithi13779221-29 2024-09-21T08:32:31.637 INFO:tasks.rados.rados.0.smithi137.stdout:2201: finishing write tid 2 to smithi13779221-29 2024-09-21T08:32:31.637 INFO:tasks.rados.rados.0.smithi137.stdout:2200: finishing copy_from racing read to smithi13779221-13 2024-09-21T08:32:31.637 INFO:tasks.rados.rados.0.smithi137.stdout:2201: finishing write tid 3 to smithi13779221-29 2024-09-21T08:32:31.637 INFO:tasks.rados.rados.0.smithi137.stdout:2201: finishing write tid 4 to smithi13779221-29 2024-09-21T08:32:31.637 INFO:tasks.rados.rados.0.smithi137.stdout:2201: oid 29 updating version 0 to 846 2024-09-21T08:32:31.638 INFO:tasks.rados.rados.0.smithi137.stdout:2201: oid 29 version 846 is already newer than 845 2024-09-21T08:32:31.638 INFO:tasks.rados.rados.0.smithi137.stdout:2201: oid 29 updating version 846 to 847 2024-09-21T08:32:31.638 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 847 (ObjNum 767 snap 246 seq_num 767) dirty exists 2024-09-21T08:32:31.638 INFO:tasks.rados.rados.0.smithi137.stdout:2201: left oid 29 (ObjNum 767 snap 246 seq_num 767) 2024-09-21T08:32:31.638 INFO:tasks.rados.rados.0.smithi137.stdout:2198: expect (ObjNum 660 snap 197 seq_num 660) 2024-09-21T08:32:31.666 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:31.689 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:31.689 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:31.690 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:31.790 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:31.802 INFO:tasks.rados.rados.0.smithi137.stdout:2200: finishing copy_from to smithi13779221-13 2024-09-21T08:32:31.802 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 13 v 734 (ObjNum 566 snap 168 seq_num 566) dirty exists 2024-09-21T08:32:31.802 INFO:tasks.rados.rados.0.smithi137.stdout:2199: expect (ObjNum 656 snap 196 seq_num 656) 2024-09-21T08:32:31.813 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:31.814 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:31.814 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:31.915 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:31.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/3085180477' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T08:32:31.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[41044]: osdmap e557: 8 total, 8 up, 5 in 2024-09-21T08:32:31.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:32:31.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[41044]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:32:31.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[41044]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:31.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2438683196' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:32:31.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[41044]: from='client.26089 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:32:31.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[41044]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:31.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[41044]: from='client.26089 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:32:31.925 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[41044]: osdmap e558: 8 total, 8 up, 5 in 2024-09-21T08:32:31.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/3085180477' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T08:32:31.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[29634]: osdmap e557: 8 total, 8 up, 5 in 2024-09-21T08:32:31.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:32:31.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[29634]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:32:31.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[29634]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:31.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2438683196' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:32:31.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[29634]: from='client.26089 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:32:31.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[29634]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:31.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[29634]: from='client.26089 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:32:31.926 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:31 smithi137 ceph-mon[29634]: osdmap e558: 8 total, 8 up, 5 in 2024-09-21T08:32:31.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:31.938 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:31.938 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:31 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/3085180477' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 1, "weight": 1.0}]': finished 2024-09-21T08:32:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:31 smithi165 ceph-mon[32893]: osdmap e557: 8 total, 8 up, 5 in 2024-09-21T08:32:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:31 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-09-21T08:32:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:31 smithi165 ceph-mon[32893]: from='mgr.44528 v1:172.21.15.137:0/3660164692' entity='mgr.y' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-09-21T08:32:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:31 smithi165 ceph-mon[32893]: from='mgr.44528 ' entity='mgr.y' 2024-09-21T08:32:32.013 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:31 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2438683196' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:32:32.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:31 smithi165 ceph-mon[32893]: from='client.26089 ' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "noscrub"}]: dispatch 2024-09-21T08:32:32.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:31 smithi165 ceph-mon[32893]: Health check update: nodeep-scrub flag(s) set (OSDMAP_FLAGS) 2024-09-21T08:32:32.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:31 smithi165 ceph-mon[32893]: from='client.26089 ' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "noscrub"}]': finished 2024-09-21T08:32:32.014 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:31 smithi165 ceph-mon[32893]: osdmap e558: 8 total, 8 up, 5 in 2024-09-21T08:32:32.027 INFO:tasks.rados.rados.0.smithi137.stdout:2197: finishing copy_from racing read to smithi13779221-23 2024-09-21T08:32:32.027 INFO:tasks.rados.rados.0.smithi137.stdout:2197: finishing copy_from to smithi13779221-23 2024-09-21T08:32:32.027 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 838 (ObjNum 734 snap 230 seq_num 734) dirty exists 2024-09-21T08:32:32.027 INFO:tasks.rados.rados.0.smithi137.stdout:2197: done (5 left) 2024-09-21T08:32:32.028 INFO:tasks.rados.rados.0.smithi137.stdout:2198: done (4 left) 2024-09-21T08:32:32.028 INFO:tasks.rados.rados.0.smithi137.stdout:2199: done (3 left) 2024-09-21T08:32:32.028 INFO:tasks.rados.rados.0.smithi137.stdout:2200: done (2 left) 2024-09-21T08:32:32.028 INFO:tasks.rados.rados.0.smithi137.stdout:2201: done (1 left) 2024-09-21T08:32:32.028 INFO:tasks.rados.rados.0.smithi137.stdout:2204: read oid 20 snap -1 2024-09-21T08:32:32.028 INFO:tasks.rados.rados.0.smithi137.stdout:2204: expect deleted 2024-09-21T08:32:32.028 INFO:tasks.rados.rados.0.smithi137.stdout:2205: write oid 36 current snap is 246 2024-09-21T08:32:32.028 INFO:tasks.rados.rados.0.smithi137.stdout:2205: seq_num 769 ranges {524583=418134,1649947=629258,2834752=89831} 2024-09-21T08:32:32.032 INFO:tasks.rados.rados.0.smithi137.stdout:2205: writing smithi13779221-36 from 524583 to 942717 tid 1 2024-09-21T08:32:32.039 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:32.040 INFO:tasks.rados.rados.0.smithi137.stdout:2205: writing smithi13779221-36 from 1649947 to 2279205 tid 2 2024-09-21T08:32:32.043 INFO:tasks.rados.rados.0.smithi137.stdout:2205: writing smithi13779221-36 from 2834752 to 2924583 tid 3 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2206: rollback oid 1 current snap is 246 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 1 to 231 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2203: finishing write tid 1 to smithi13779221-31 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2203: finishing write tid 2 to smithi13779221-31 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2203: finishing write tid 3 to smithi13779221-31 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2203: finishing write tid 4 to smithi13779221-31 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2203: oid 31 updating version 0 to 516 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2203: oid 31 version 516 is already newer than 515 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2203: oid 31 version 516 is already newer than 514 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 516 (ObjNum 768 snap 246 seq_num 768) dirty exists 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2203: left oid 31 (ObjNum 768 snap 246 seq_num 768) 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2203: done (3 left) 2024-09-21T08:32:32.044 INFO:tasks.rados.rados.0.smithi137.stdout:2207: delete oid 8 current snap is 246 2024-09-21T08:32:32.045 INFO:tasks.rados.rados.0.smithi137.stdout:2205: finishing write tid 1 to smithi13779221-36 2024-09-21T08:32:32.045 INFO:tasks.rados.rados.0.smithi137.stdout:2205: finishing write tid 2 to smithi13779221-36 2024-09-21T08:32:32.046 INFO:tasks.rados.rados.0.smithi137.stdout:2206: finishing rollback tid 0 to smithi13779221-1 2024-09-21T08:32:32.046 INFO:tasks.rados.rados.0.smithi137.stdout:2205: finishing write tid 3 to smithi13779221-36 2024-09-21T08:32:32.046 INFO:tasks.rados.rados.0.smithi137.stdout:2204: done (3 left) 2024-09-21T08:32:32.046 INFO:tasks.rados.rados.0.smithi137.stdout:2207: done (2 left) 2024-09-21T08:32:32.046 INFO:tasks.rados.rados.0.smithi137.stdout:2208: read oid 25 snap 232 2024-09-21T08:32:32.047 INFO:tasks.rados.rados.0.smithi137.stdout:2208: expect (ObjNum 688 snap 210 seq_num 688) 2024-09-21T08:32:32.047 INFO:tasks.rados.rados.0.smithi137.stdout:2209: read oid 43 snap 238 2024-09-21T08:32:32.047 INFO:tasks.rados.rados.0.smithi137.stdout:2209: expect (ObjNum 756 snap 238 seq_num 756) 2024-09-21T08:32:32.047 INFO:tasks.rados.rados.0.smithi137.stdout:2210: write (excl) oid 17 current snap is 246 2024-09-21T08:32:32.047 INFO:tasks.rados.rados.0.smithi137.stdout:2205: finishing write tid 4 to smithi13779221-36 2024-09-21T08:32:32.047 INFO:tasks.rados.rados.0.smithi137.stdout:2210: seq_num 771 ranges {558197=568632,1650558=632927,2773245=184952} 2024-09-21T08:32:32.053 INFO:tasks.rados.rados.0.smithi137.stdout:2210: writing smithi13779221-17 from 558197 to 1126829 tid 1 2024-09-21T08:32:32.059 INFO:tasks.rados.rados.0.smithi137.stdout:2210: writing smithi13779221-17 from 1650558 to 2283485 tid 2 2024-09-21T08:32:32.063 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:32.063 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:32.064 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:32.064 INFO:tasks.rados.rados.0.smithi137.stdout:2210: writing smithi13779221-17 from 2773245 to 2958197 tid 3 2024-09-21T08:32:32.064 INFO:tasks.rados.rados.0.smithi137.stdout:2211: write oid 32 current snap is 246 2024-09-21T08:32:32.064 INFO:tasks.rados.rados.0.smithi137.stdout:2206: finishing rollback tid 1 to smithi13779221-1 2024-09-21T08:32:32.064 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 1 v 544 (ObjNum 257 snap 0 seq_num 2160955462) dirty dne 2024-09-21T08:32:32.064 INFO:tasks.rados.rados.0.smithi137.stdout:2211: seq_num 772 ranges {575004=643881,1709040=492938,2917021=57983} 2024-09-21T08:32:32.070 INFO:tasks.rados.rados.0.smithi137.stdout:2211: writing smithi13779221-32 from 575004 to 1218885 tid 1 2024-09-21T08:32:32.076 INFO:tasks.rados.rados.0.smithi137.stdout:2211: writing smithi13779221-32 from 1709040 to 2201978 tid 2 2024-09-21T08:32:32.080 INFO:tasks.rados.rados.0.smithi137.stdout:2211: writing smithi13779221-32 from 2917021 to 2975004 tid 3 2024-09-21T08:32:32.081 INFO:tasks.rados.rados.0.smithi137.stdout:2206: done (5 left) 2024-09-21T08:32:32.081 INFO:tasks.rados.rados.0.smithi137.stdout:2212: copy_from oid 35 from oid 21 current snap is 246 2024-09-21T08:32:32.081 INFO:tasks.rados.rados.0.smithi137.stdout:2205: finishing write tid 5 to smithi13779221-36 2024-09-21T08:32:32.081 INFO:tasks.rados.rados.0.smithi137.stdout:2205: oid 36 updating version 0 to 876 2024-09-21T08:32:32.081 INFO:tasks.rados.rados.0.smithi137.stdout:2205: oid 36 updating version 876 to 878 2024-09-21T08:32:32.081 INFO:tasks.rados.rados.0.smithi137.stdout:2205: oid 36 updating version 878 to 879 2024-09-21T08:32:32.081 INFO:tasks.rados.rados.0.smithi137.stdout:2205: oid 36 version 879 is already newer than 877 2024-09-21T08:32:32.081 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 36 v 879 (ObjNum 769 snap 246 seq_num 769) dirty exists 2024-09-21T08:32:32.081 INFO:tasks.rados.rados.0.smithi137.stdout:2205: left oid 36 (ObjNum 769 snap 246 seq_num 769) 2024-09-21T08:32:32.081 INFO:tasks.rados.rados.0.smithi137.stdout:2209: expect (ObjNum 756 snap 238 seq_num 756) 2024-09-21T08:32:32.165 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:32.173 INFO:tasks.rados.rados.0.smithi137.stdout:2208: expect (ObjNum 688 snap 210 seq_num 688) 2024-09-21T08:32:32.186 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:32.187 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:32.187 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:32.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:32:32.289 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:32.310 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:32.311 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:32.311 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:32.347 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd reweight 4 1 2024-09-21T08:32:32.412 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:32.433 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:32.433 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:32.434 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:32.443 INFO:tasks.rados.rados.0.smithi137.stdout:2210: finishing write tid 1 to smithi13779221-17 2024-09-21T08:32:32.443 INFO:tasks.rados.rados.0.smithi137.stdout:2205: done (5 left) 2024-09-21T08:32:32.443 INFO:tasks.rados.rados.0.smithi137.stdout:2208: done (4 left) 2024-09-21T08:32:32.443 INFO:tasks.rados.rados.0.smithi137.stdout:2209: done (3 left) 2024-09-21T08:32:32.443 INFO:tasks.rados.rados.0.smithi137.stdout:2213: delete oid 4 current snap is 246 2024-09-21T08:32:32.443 INFO:tasks.rados.rados.0.smithi137.stdout:2210: finishing write tid 2 to smithi13779221-17 2024-09-21T08:32:32.443 INFO:tasks.rados.rados.0.smithi137.stdout:2210: finishing write tid 3 to smithi13779221-17 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2210: finishing write tid 4 to smithi13779221-17 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2210: finishing write tid 5 to smithi13779221-17 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2210: oid 17 updating version 0 to 758 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2210: oid 17 version 758 is already newer than 757 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2210: oid 17 version 758 is already newer than 755 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2210: oid 17 version 758 is already newer than 756 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 758 (ObjNum 771 snap 246 seq_num 771) dirty exists 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2210: left oid 17 (ObjNum 771 snap 246 seq_num 771) 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2211: finishing write tid 1 to smithi13779221-32 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2211: finishing write tid 2 to smithi13779221-32 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2211: finishing write tid 3 to smithi13779221-32 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2211: finishing write tid 4 to smithi13779221-32 2024-09-21T08:32:32.444 INFO:tasks.rados.rados.0.smithi137.stdout:2211: finishing write tid 5 to smithi13779221-32 2024-09-21T08:32:32.445 INFO:tasks.rados.rados.0.smithi137.stdout:2211: oid 32 updating version 0 to 555 2024-09-21T08:32:32.445 INFO:tasks.rados.rados.0.smithi137.stdout:2211: oid 32 version 555 is already newer than 552 2024-09-21T08:32:32.445 INFO:tasks.rados.rados.0.smithi137.stdout:2211: oid 32 version 555 is already newer than 553 2024-09-21T08:32:32.445 INFO:tasks.rados.rados.0.smithi137.stdout:2211: oid 32 version 555 is already newer than 554 2024-09-21T08:32:32.445 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 32 v 555 (ObjNum 772 snap 246 seq_num 772) dirty exists 2024-09-21T08:32:32.445 INFO:tasks.rados.rados.0.smithi137.stdout:2211: left oid 32 (ObjNum 772 snap 246 seq_num 772) 2024-09-21T08:32:32.445 INFO:tasks.rados.rados.0.smithi137.stdout:2212: finishing copy_from racing read to smithi13779221-35 2024-09-21T08:32:32.445 INFO:tasks.rados.rados.0.smithi137.stdout:2212: finishing copy_from to smithi13779221-35 2024-09-21T08:32:32.445 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 522 (ObjNum 677 snap 205 seq_num 677) dirty exists 2024-09-21T08:32:32.446 INFO:tasks.rados.rados.0.smithi137.stdout:2210: done (3 left) 2024-09-21T08:32:32.446 INFO:tasks.rados.rados.0.smithi137.stdout:2211: done (2 left) 2024-09-21T08:32:32.446 INFO:tasks.rados.rados.0.smithi137.stdout:2212: done (1 left) 2024-09-21T08:32:32.446 INFO:tasks.rados.rados.0.smithi137.stdout:2213: done (0 left) 2024-09-21T08:32:32.447 INFO:tasks.rados.rados.0.smithi137.stdout:2214: read oid 5 snap 215 2024-09-21T08:32:32.447 INFO:tasks.rados.rados.0.smithi137.stdout:2214: expect deleted 2024-09-21T08:32:32.447 INFO:tasks.rados.rados.0.smithi137.stdout:2215: rollback oid 31 current snap is 246 2024-09-21T08:32:32.447 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 31 to 232 2024-09-21T08:32:32.447 INFO:tasks.rados.rados.0.smithi137.stdout:2216: rollback oid 22 current snap is 246 2024-09-21T08:32:32.447 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 22 to 182 2024-09-21T08:32:32.447 INFO:tasks.rados.rados.0.smithi137.stdout:2217: rollback oid 24 current snap is 246 2024-09-21T08:32:32.447 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 24 to 200 2024-09-21T08:32:32.447 INFO:tasks.rados.rados.0.smithi137.stdout:2218: write (excl) oid 17 current snap is 246 2024-09-21T08:32:32.447 INFO:tasks.rados.rados.0.smithi137.stdout:2218: seq_num 774 ranges {608618=794379,2109651=780254} 2024-09-21T08:32:32.452 INFO:tasks.rados.rados.0.smithi137.stdout:2218: writing smithi13779221-17 from 608618 to 1402997 tid 1 2024-09-21T08:32:32.461 INFO:tasks.rados.rados.0.smithi137.stdout:2218: writing smithi13779221-17 from 2109651 to 2889905 tid 2 2024-09-21T08:32:32.461 INFO:tasks.rados.rados.0.smithi137.stdout:2219: rollback oid 37 current snap is 246 2024-09-21T08:32:32.461 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 37 to 244 2024-09-21T08:32:32.461 INFO:tasks.rados.rados.0.smithi137.stdout:2215: finishing rollback tid 0 to smithi13779221-31 2024-09-21T08:32:32.462 INFO:tasks.rados.rados.0.smithi137.stdout:2217: finishing rollback tid 0 to smithi13779221-24 2024-09-21T08:32:32.462 INFO:tasks.rados.rados.0.smithi137.stdout:2214: done (5 left) 2024-09-21T08:32:32.462 INFO:tasks.rados.rados.0.smithi137.stdout:2220: write oid 48 current snap is 246 2024-09-21T08:32:32.462 INFO:tasks.rados.rados.0.smithi137.stdout:2215: finishing rollback tid 1 to smithi13779221-31 2024-09-21T08:32:32.462 INFO:tasks.rados.rados.0.smithi137.stdout:2217: finishing rollback tid 1 to smithi13779221-24 2024-09-21T08:32:32.462 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 745 (ObjNum 33 snap 0 seq_num 2161149253) dirty dne 2024-09-21T08:32:32.462 INFO:tasks.rados.rados.0.smithi137.stdout:2220: seq_num 775 ranges {625425=469628,1768133=640265} 2024-09-21T08:32:32.467 INFO:tasks.rados.rados.0.smithi137.stdout:2220: writing smithi13779221-48 from 625425 to 1095053 tid 1 2024-09-21T08:32:32.475 INFO:tasks.rados.rados.0.smithi137.stdout:2220: writing smithi13779221-48 from 1768133 to 2408398 tid 2 2024-09-21T08:32:32.475 INFO:tasks.rados.rados.0.smithi137.stdout:2217: done (5 left) 2024-09-21T08:32:32.475 INFO:tasks.rados.rados.0.smithi137.stdout:2221: read oid 14 snap -1 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2221: expect (ObjNum 744 snap 235 seq_num 744) 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2222: copy_from oid 23 from oid 18 current snap is 246 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2215: finishing rollback tid 2 to smithi13779221-31 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 31 v 519 (ObjNum 618 snap 182 seq_num 618) dirty exists 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2216: finishing rollback tid 0 to smithi13779221-22 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2216: finishing rollback tid 1 to smithi13779221-22 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2216: finishing rollback tid 2 to smithi13779221-22 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 522 (ObjNum 582 snap 171 seq_num 582) dirty exists 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2215: done (6 left) 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2216: done (5 left) 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2223: read oid 35 snap -1 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2223: expect (ObjNum 677 snap 205 seq_num 677) 2024-09-21T08:32:32.476 INFO:tasks.rados.rados.0.smithi137.stdout:2224: snap_create 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:2218: finishing write tid 1 to smithi13779221-17 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:2219: finishing rollback tid 0 to smithi13779221-37 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:2219: finishing rollback tid 1 to smithi13779221-37 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:2219: finishing rollback tid 2 to smithi13779221-37 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 37 v 883 (ObjNum 618 snap 182 seq_num 618) dirty exists 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:2218: finishing write tid 2 to smithi13779221-17 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:2218: finishing write tid 3 to smithi13779221-17 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:2218: finishing write tid 4 to smithi13779221-17 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:2218: oid 17 updating version 0 to 760 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:2218: oid 17 updating version 760 to 761 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:2218: oid 17 version 761 is already newer than 759 2024-09-21T08:32:32.477 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 17 v 761 (ObjNum 774 snap 246 seq_num 774) dirty exists 2024-09-21T08:32:32.478 INFO:tasks.rados.rados.0.smithi137.stdout:2218: left oid 17 (ObjNum 774 snap 246 seq_num 774) 2024-09-21T08:32:32.478 INFO:tasks.rados.rados.0.smithi137.stdout:2220: finishing write tid 1 to smithi13779221-48 2024-09-21T08:32:32.479 INFO:tasks.rados.rados.0.smithi137.stdout:2221: expect (ObjNum 744 snap 235 seq_num 744) 2024-09-21T08:32:32.513 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:32:32 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:32:32.534 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:32.556 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:32.557 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:32.557 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:32.658 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:32.676 INFO:tasks.rados.rados.0.smithi137.stdout:2220: finishing write tid 2 to smithi13779221-48 2024-09-21T08:32:32.676 INFO:tasks.rados.rados.0.smithi137.stdout:2220: finishing write tid 3 to smithi13779221-48 2024-09-21T08:32:32.676 INFO:tasks.rados.rados.0.smithi137.stdout:2220: finishing write tid 4 to smithi13779221-48 2024-09-21T08:32:32.676 INFO:tasks.rados.rados.0.smithi137.stdout:2220: oid 48 updating version 0 to 832 2024-09-21T08:32:32.676 INFO:tasks.rados.rados.0.smithi137.stdout:2220: oid 48 updating version 832 to 833 2024-09-21T08:32:32.676 INFO:tasks.rados.rados.0.smithi137.stdout:2220: oid 48 version 833 is already newer than 831 2024-09-21T08:32:32.676 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 48 v 833 (ObjNum 775 snap 246 seq_num 775) dirty exists 2024-09-21T08:32:32.676 INFO:tasks.rados.rados.0.smithi137.stdout:2220: left oid 48 (ObjNum 775 snap 246 seq_num 775) 2024-09-21T08:32:32.676 INFO:tasks.rados.rados.0.smithi137.stdout:2222: finishing copy_from racing read to smithi13779221-23 2024-09-21T08:32:32.676 INFO:tasks.rados.rados.0.smithi137.stdout:2223: expect (ObjNum 677 snap 205 seq_num 677) 2024-09-21T08:32:32.679 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:32.680 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:32.680 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:32.781 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:32.785 INFO:tasks.rados.rados.0.smithi137.stdout:2222: finishing copy_from to smithi13779221-23 2024-09-21T08:32:32.785 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 23 v 237 (ObjNum 670 snap 203 seq_num 670) dirty exists 2024-09-21T08:32:32.785 INFO:tasks.rados.rados.0.smithi137.stdout:2218: done (6 left) 2024-09-21T08:32:32.785 INFO:tasks.rados.rados.0.smithi137.stdout:2219: done (5 left) 2024-09-21T08:32:32.786 INFO:tasks.rados.rados.0.smithi137.stdout:2220: done (4 left) 2024-09-21T08:32:32.786 INFO:tasks.rados.rados.0.smithi137.stdout:2221: done (3 left) 2024-09-21T08:32:32.786 INFO:tasks.rados.rados.0.smithi137.stdout:2222: done (2 left) 2024-09-21T08:32:32.786 INFO:tasks.rados.rados.0.smithi137.stdout:2223: done (1 left) 2024-09-21T08:32:32.786 INFO:tasks.rados.rados.0.smithi137.stdout:2224: done (0 left) 2024-09-21T08:32:32.786 INFO:tasks.rados.rados.0.smithi137.stdout:2225: snap_remove snap 231 2024-09-21T08:32:32.794 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_ops_in_flight 2024-09-21T08:32:32.803 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:32.803 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:32.804 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:32.904 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:32.926 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:32.926 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:32.927 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:33.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:32 smithi137 ceph-mon[29634]: 5.2 scrub starts 2024-09-21T08:32:33.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:32 smithi137 ceph-mon[29634]: 5.2 scrub ok 2024-09-21T08:32:33.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:32 smithi137 ceph-mon[29634]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:33.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:32 smithi137 ceph-mon[29634]: pgmap v743: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 604 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 8.3 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-21T08:32:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:32 smithi137 ceph-mon[41044]: 5.2 scrub starts 2024-09-21T08:32:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:32 smithi137 ceph-mon[41044]: 5.2 scrub ok 2024-09-21T08:32:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:32 smithi137 ceph-mon[41044]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:33.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:32 smithi137 ceph-mon[41044]: pgmap v743: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 604 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 8.3 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-21T08:32:33.028 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:33.047 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:32:33.049 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:33.050 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:33.050 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:33.151 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:33.167 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:32 smithi165 ceph-mon[32893]: 5.2 scrub starts 2024-09-21T08:32:33.167 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:32 smithi165 ceph-mon[32893]: 5.2 scrub ok 2024-09-21T08:32:33.167 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:32 smithi165 ceph-mon[32893]: from='client.24490 v1:172.21.15.165:0/215073926' entity='client.iscsi.iscsi.a' cmd=[{"prefix": "service status", "format": "json"}]: dispatch 2024-09-21T08:32:33.167 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:32 smithi165 ceph-mon[32893]: pgmap v743: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 604 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 8.3 MiB/s rd, 1.5 MiB/s wr, 13 op/s 2024-09-21T08:32:33.173 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:33.174 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:33.175 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:33.276 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:33.297 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:33.297 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:33.298 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:33.398 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:33.421 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:33.421 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:33.422 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:33.520 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:33.522 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:33.523 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:32:33.523 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:32:33.546 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:33.546 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:33.546 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:33.647 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:33.668 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:33.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:33.669 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:33.770 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:33.780 INFO:tasks.rados.rados.0.smithi137.stdout:2225: done (0 left) 2024-09-21T08:32:33.780 INFO:tasks.rados.rados.0.smithi137.stdout:2226: write (excl) oid 30 current snap is 247 2024-09-21T08:32:33.780 INFO:tasks.rados.rados.0.smithi137.stdout:2226: seq_num 776 ranges {642232=661230,1942968=783923} 2024-09-21T08:32:33.787 INFO:tasks.rados.rados.0.smithi137.stdout:2226: writing smithi13779221-30 from 642232 to 1303462 tid 1 2024-09-21T08:32:33.795 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:33.795 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:33.796 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:33.796 INFO:tasks.rados.rados.0.smithi137.stdout:2226: writing smithi13779221-30 from 1942968 to 2726891 tid 2 2024-09-21T08:32:33.796 INFO:tasks.rados.rados.0.smithi137.stdout:2227: rollback oid 50 current snap is 247 2024-09-21T08:32:33.796 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 50 to 200 2024-09-21T08:32:33.796 INFO:tasks.rados.rados.0.smithi137.stdout:2226: finishing write tid 1 to smithi13779221-30 2024-09-21T08:32:33.797 INFO:tasks.rados.rados.0.smithi137.stdout:2228: write oid 10 current snap is 247 2024-09-21T08:32:33.797 INFO:tasks.rados.rados.0.smithi137.stdout:2228: seq_num 777 ranges {659039=736479,2001450=643934} 2024-09-21T08:32:33.804 INFO:tasks.rados.rados.0.smithi137.stdout:2228: writing smithi13779221-10 from 659039 to 1395518 tid 1 2024-09-21T08:32:33.811 INFO:tasks.rados.rados.0.smithi137.stdout:2228: writing smithi13779221-10 from 2001450 to 2645384 tid 2 2024-09-21T08:32:33.811 INFO:tasks.rados.rados.0.smithi137.stdout:2229: copy_from oid 22 from oid 42 current snap is 247 2024-09-21T08:32:33.811 INFO:tasks.rados.rados.0.smithi137.stdout:2227: finishing rollback tid 0 to smithi13779221-50 2024-09-21T08:32:33.811 INFO:tasks.rados.rados.0.smithi137.stdout:2226: finishing write tid 2 to smithi13779221-30 2024-09-21T08:32:33.811 INFO:tasks.rados.rados.0.smithi137.stdout:2227: finishing rollback tid 1 to smithi13779221-50 2024-09-21T08:32:33.811 INFO:tasks.rados.rados.0.smithi137.stdout:2227: finishing rollback tid 2 to smithi13779221-50 2024-09-21T08:32:33.811 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 50 v 754 (ObjNum 660 snap 197 seq_num 660) dirty exists 2024-09-21T08:32:33.811 INFO:tasks.rados.rados.0.smithi137.stdout:2226: finishing write tid 3 to smithi13779221-30 2024-09-21T08:32:33.811 INFO:tasks.rados.rados.0.smithi137.stdout:2226: finishing write tid 4 to smithi13779221-30 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2226: oid 30 updating version 0 to 891 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2226: oid 30 version 891 is already newer than 890 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2226: oid 30 version 891 is already newer than 889 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 30 v 891 (ObjNum 776 snap 247 seq_num 776) dirty exists 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2226: left oid 30 (ObjNum 776 snap 247 seq_num 776) 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2228: finishing write tid 1 to smithi13779221-10 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2226: done (3 left) 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2227: done (2 left) 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2230: copy_from oid 5 from oid 39 current snap is 247 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2231: read oid 4 snap -1 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2231: expect deleted 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2232: rollback oid 44 current snap is 247 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 44 to 201 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:2233: rollback oid 35 current snap is 247 2024-09-21T08:32:33.812 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 35 to 182 2024-09-21T08:32:33.813 INFO:tasks.rados.rados.0.smithi137.stdout:2234: snap_create 2024-09-21T08:32:33.814 INFO:tasks.rados.rados.0.smithi137.stdout:2228: finishing write tid 2 to smithi13779221-10 2024-09-21T08:32:33.814 INFO:tasks.rados.rados.0.smithi137.stdout:2228: finishing write tid 3 to smithi13779221-10 2024-09-21T08:32:33.814 INFO:tasks.rados.rados.0.smithi137.stdout:2228: finishing write tid 4 to smithi13779221-10 2024-09-21T08:32:33.814 INFO:tasks.rados.rados.0.smithi137.stdout:2228: oid 10 updating version 0 to 836 2024-09-21T08:32:33.814 INFO:tasks.rados.rados.0.smithi137.stdout:2228: oid 10 version 836 is already newer than 834 2024-09-21T08:32:33.814 INFO:tasks.rados.rados.0.smithi137.stdout:2228: oid 10 version 836 is already newer than 835 2024-09-21T08:32:33.815 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 10 v 836 (ObjNum 777 snap 247 seq_num 777) dirty exists 2024-09-21T08:32:33.815 INFO:tasks.rados.rados.0.smithi137.stdout:2228: left oid 10 (ObjNum 777 snap 247 seq_num 777) 2024-09-21T08:32:33.815 INFO:tasks.rados.rados.0.smithi137.stdout:2232: finishing rollback tid 0 to smithi13779221-44 2024-09-21T08:32:33.817 INFO:tasks.rados.rados.0.smithi137.stdout:2232: finishing rollback tid 1 to smithi13779221-44 2024-09-21T08:32:33.817 INFO:tasks.rados.rados.0.smithi137.stdout:2232: finishing rollback tid 2 to smithi13779221-44 2024-09-21T08:32:33.817 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 44 v 764 (ObjNum 471 snap 128 seq_num 471) dirty exists 2024-09-21T08:32:33.819 INFO:tasks.rados.rados.0.smithi137.stdout:2229: finishing copy_from racing read to smithi13779221-22 2024-09-21T08:32:33.819 INFO:tasks.rados.rados.0.smithi137.stdout:2230: finishing copy_from racing read to smithi13779221-5 2024-09-21T08:32:33.826 INFO:tasks.rados.rados.0.smithi137.stdout:2233: finishing rollback tid 0 to smithi13779221-35 2024-09-21T08:32:33.827 INFO:tasks.rados.rados.0.smithi137.stdout:2229: finishing copy_from to smithi13779221-22 2024-09-21T08:32:33.827 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 22 v 704 (ObjNum 737 snap 230 seq_num 737) dirty exists 2024-09-21T08:32:33.827 INFO:tasks.rados.rados.0.smithi137.stdout:2230: finishing copy_from to smithi13779221-5 2024-09-21T08:32:33.827 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 5 v 743 (ObjNum 752 snap 236 seq_num 752) dirty exists 2024-09-21T08:32:33.827 INFO:tasks.rados.rados.0.smithi137.stdout:2233: finishing rollback tid 1 to smithi13779221-35 2024-09-21T08:32:33.827 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 35 v 766 (ObjNum 33 snap 0 seq_num 2160932291) dirty dne 2024-09-21T08:32:33.896 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:33.918 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:33.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:33.919 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:34.020 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:34.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:33 smithi137 ceph-mon[29634]: osdmap e559: 8 total, 8 up, 5 in 2024-09-21T08:32:34.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:33 smithi137 ceph-mon[29634]: 7.18 scrub starts 2024-09-21T08:32:34.021 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:33 smithi137 ceph-mon[29634]: 7.18 scrub ok 2024-09-21T08:32:34.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:33 smithi137 ceph-mon[41044]: osdmap e559: 8 total, 8 up, 5 in 2024-09-21T08:32:34.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:33 smithi137 ceph-mon[41044]: 7.18 scrub starts 2024-09-21T08:32:34.021 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:33 smithi137 ceph-mon[41044]: 7.18 scrub ok 2024-09-21T08:32:34.044 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:34.044 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:34.045 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:34.057 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:34.146 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:34.168 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:34.168 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:34.169 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:34.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:33 smithi165 ceph-mon[32893]: osdmap e559: 8 total, 8 up, 5 in 2024-09-21T08:32:34.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:33 smithi165 ceph-mon[32893]: 7.18 scrub starts 2024-09-21T08:32:34.223 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:33 smithi165 ceph-mon[32893]: 7.18 scrub ok 2024-09-21T08:32:34.250 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_blocked_ops 2024-09-21T08:32:34.270 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:34.293 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:34.293 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:34.294 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:34.394 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:34.417 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:34.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:34.418 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:34.490 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:32:34.491 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:32:34.498 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:32:34.505 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:32:34.505 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:32:34.519 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:34.521 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:32:34.521 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:32:34.536 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:32:34.537 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:32:34.541 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:34.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:34.542 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:34.553 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:32:34.553 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:32:34.569 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:32:34.569 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:32:34.586 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:32:34.586 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:32:34.602 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:32:34.602 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:32:34.643 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:34.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:34.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:34.665 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:34.766 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:34.787 INFO:tasks.rados.rados.0.smithi137.stdout:2228: done (6 left) 2024-09-21T08:32:34.787 INFO:tasks.rados.rados.0.smithi137.stdout:2229: done (5 left) 2024-09-21T08:32:34.787 INFO:tasks.rados.rados.0.smithi137.stdout:2230: done (4 left) 2024-09-21T08:32:34.787 INFO:tasks.rados.rados.0.smithi137.stdout:2231: done (3 left) 2024-09-21T08:32:34.787 INFO:tasks.rados.rados.0.smithi137.stdout:2232: done (2 left) 2024-09-21T08:32:34.787 INFO:tasks.rados.rados.0.smithi137.stdout:2233: done (1 left) 2024-09-21T08:32:34.787 INFO:tasks.rados.rados.0.smithi137.stdout:2234: done (0 left) 2024-09-21T08:32:34.787 INFO:tasks.rados.rados.0.smithi137.stdout:2235: snap_create 2024-09-21T08:32:34.788 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:34.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:34.789 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:34.790 INFO:teuthology.orchestra.run.smithi137.stderr:reweighted osd.4 to 1 (10000) 2024-09-21T08:32:34.889 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:34.893 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:34 smithi165 ceph-mon[32893]: osdmap e560: 8 total, 8 up, 5 in 2024-09-21T08:32:34.893 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:34 smithi165 ceph-mon[32893]: pgmap v746: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 604 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 8.3 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-21T08:32:34.893 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2296579208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-21T08:32:34.893 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:34 smithi165 ceph-mon[32893]: from='client.26095 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-21T08:32:34.893 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:34 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4081422916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:34.911 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:34.912 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:34.912 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:34.994 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:32:34.995 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:32:34.995 INFO:teuthology.orchestra.run.smithi165.stdout: "complaint_time": 30, 2024-09-21T08:32:34.995 INFO:teuthology.orchestra.run.smithi165.stdout: "num_blocked_ops": 0 2024-09-21T08:32:34.995 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:32:35.013 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:34 smithi137 ceph-mon[29634]: osdmap e560: 8 total, 8 up, 5 in 2024-09-21T08:32:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:34 smithi137 ceph-mon[29634]: pgmap v746: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 604 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 8.3 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-21T08:32:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2296579208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-21T08:32:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:34 smithi137 ceph-mon[29634]: from='client.26095 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-21T08:32:35.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:34 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4081422916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:35.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:34 smithi137 ceph-mon[41044]: osdmap e560: 8 total, 8 up, 5 in 2024-09-21T08:32:35.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:34 smithi137 ceph-mon[41044]: pgmap v746: 188 pgs: 4 active+clean+snaptrim, 29 active+clean+snaptrim_wait, 155 active+clean; 604 MiB data, 1.5 GiB used, 445 GiB / 447 GiB avail; 8.3 MiB/s rd, 1.2 MiB/s wr, 12 op/s 2024-09-21T08:32:35.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2296579208' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-21T08:32:35.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:34 smithi137 ceph-mon[41044]: from='client.26095 ' entity='client.admin' cmd=[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]: dispatch 2024-09-21T08:32:35.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:34 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4081422916' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:35.035 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:35.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:35.036 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:35.137 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:35.160 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:35.160 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:35.161 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:35.262 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:35.283 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:35.283 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:35.283 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:35.384 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:35.405 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:35.406 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:35.406 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:35.507 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:35.518 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd unset nodeep-scrub 2024-09-21T08:32:35.529 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:35.530 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:35.531 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:35.560 INFO:tasks.cephadm.ceph_manager.ceph:waiting for recovery to complete 2024-09-21T08:32:35.561 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:32:35.615 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.4 dump_historic_ops 2024-09-21T08:32:35.633 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:35.664 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:35.665 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:35.665 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:35.766 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:35.787 INFO:tasks.rados.rados.0.smithi137.stdout:2235: done (0 left) 2024-09-21T08:32:35.787 INFO:tasks.rados.rados.0.smithi137.stdout:2236: read oid 39 snap 237 2024-09-21T08:32:35.787 INFO:tasks.rados.rados.0.smithi137.stdout:2236: expect (ObjNum 725 snap 226 seq_num 725) 2024-09-21T08:32:35.787 INFO:tasks.rados.rados.0.smithi137.stdout:2237: copy_from oid 40 from oid 13 current snap is 249 2024-09-21T08:32:35.787 INFO:tasks.rados.rados.0.smithi137.stdout:2238: snap_remove snap 183 2024-09-21T08:32:35.789 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:35.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:35.790 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:35.795 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:35 smithi137 ceph-mon[41044]: from='client.26095 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T08:32:35.796 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:35 smithi137 ceph-mon[41044]: osdmap e561: 8 total, 8 up, 5 in 2024-09-21T08:32:35.883 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:32:35.891 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:35.912 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:35.913 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:35.913 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:36.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:35 smithi165 ceph-mon[32893]: from='client.26095 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T08:32:36.005 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:35 smithi165 ceph-mon[32893]: osdmap e561: 8 total, 8 up, 5 in 2024-09-21T08:32:36.014 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:36.036 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:36.036 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:36.036 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:36.108 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:36.137 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:36.138 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:36.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:36.159 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:36.159 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:36.232 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:35 smithi137 ceph-mon[29634]: from='client.26095 ' entity='client.admin' cmd='[{"prefix": "osd reweight", "id": 4, "weight": 1.0}]': finished 2024-09-21T08:32:36.232 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:35 smithi137 ceph-mon[29634]: osdmap e561: 8 total, 8 up, 5 in 2024-09-21T08:32:36.260 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:36.281 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:36.282 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:36.282 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:36.367 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:32:36.367 INFO:teuthology.orchestra.run.smithi165.stdout: "size": 20, 2024-09-21T08:32:36.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 600, 2024-09-21T08:32:36.367 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [ 2024-09-21T08:32:36.367 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.367 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.1.0:1138 7.1e e472/465 7:78d05d33:::smithi13779221-41:81 v 472'316, mlcod=468'312)", 2024-09-21T08:32:36.367 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:06.690512+0000", 2024-09-21T08:32:36.367 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 89.670184097999993, 2024-09-21T08:32:36.367 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0056824299999999996, 2024-09-21T08:32:36.368 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.368 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.368 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.368 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.368 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.368 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690512+0000", 2024-09-21T08:32:36.369 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.369 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.369 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.369 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.369 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690512+0000", 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690513+0000", 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2000000000000001e-07 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690523+0000", 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.064e-05 2024-09-21T08:32:36.370 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.371 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.371 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690530+0000", 2024-09-21T08:32:36.371 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3450000000000002e-06 2024-09-21T08:32:36.371 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.371 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690538+0000", 2024-09-21T08:32:36.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.8289999999999997e-06 2024-09-21T08:32:36.372 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.372 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.372 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.372 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690718+0000", 2024-09-21T08:32:36.372 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000179087 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.690739+0000", 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1569999999999998e-05 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.696151+0000", 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0054117999999999996 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.373 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:06.696195+0000", 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.3838999999999998e-05 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:4848 7.28 e477/476 7:15ddb887:::smithi13779221-8:head v 477'387, mlcod=0'0)", 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.708762+0000", 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 85.651934322000002, 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.011385744999999999, 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.374 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.708762+0000", 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.708762+0000", 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.708763+0000", 2024-09-21T08:32:36.375 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.4e-07 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.711674+0000", 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029111330000000002 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.711703+0000", 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8404000000000001e-05 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.376 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.711713+0000", 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0875e-05 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.712110+0000", 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00039688700000000001 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.712153+0000", 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.2273e-05 2024-09-21T08:32:36.377 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720094+0000", 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0079410149999999992 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720148+0000", 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4218000000000002e-05 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.378 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:4856 7.9 e477/473 7:92b92e8a:::smithi13779221-35:head v 477'383, mlcod=0'0)", 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.713175+0000", 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 85.647521952999995, 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066575059999999997, 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713175+0000", 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.379 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713175+0000", 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713177+0000", 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4480000000000001e-06 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713187+0000", 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0399e-05 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.380 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713207+0000", 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0071000000000001e-05 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713221+0000", 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3508e-05 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.381 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713343+0000", 2024-09-21T08:32:36.382 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012158 2024-09-21T08:32:36.382 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.382 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.382 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.382 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.713368+0000", 2024-09-21T08:32:36.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5168999999999999e-05 2024-09-21T08:32:36.383 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.383 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.383 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.383 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.719777+0000", 2024-09-21T08:32:36.383 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0064095189999999998 2024-09-21T08:32:36.384 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.384 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.384 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.384 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.719832+0000", 2024-09-21T08:32:36.384 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4812000000000003e-05 2024-09-21T08:32:36.384 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.385 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.385 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.385 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.385 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.385 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:4857 7.9 e477/473 7:92b92e8a:::smithi13779221-35:head v 477'384, mlcod=0'0)", 2024-09-21T08:32:36.385 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.714415+0000", 2024-09-21T08:32:36.385 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 85.646281371000001, 2024-09-21T08:32:36.386 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0057016469999999998, 2024-09-21T08:32:36.386 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.386 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.386 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.386 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714415+0000", 2024-09-21T08:32:36.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.387 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.387 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.387 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.387 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714415+0000", 2024-09-21T08:32:36.387 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714417+0000", 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1349999999999999e-06 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714427+0000", 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.8800000000000003e-06 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.388 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714445+0000", 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7861999999999999e-05 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714454+0000", 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0289999999999994e-06 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714519+0000", 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4759999999999997e-05 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.389 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714545+0000", 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.5819000000000001e-05 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720079+0000", 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055343359999999999 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720117+0000", 2024-09-21T08:32:36.390 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.7826e-05 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(osd.5.0:969 7.28 e477/476 7:17170b1f:::smithi13779221-4:b5 v 477'389, mlcod=0'0)", 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:10.714652+0000", 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 85.646044459999999, 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055407950000000003, 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.391 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714652+0000", 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714652+0000", 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714653+0000", 2024-09-21T08:32:36.392 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4720000000000001e-06 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714662+0000", 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7639999999999994e-06 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714676+0000", 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3373000000000001e-05 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714681+0000", 2024-09-21T08:32:36.393 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.772e-06 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714744+0000", 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.2284000000000002e-05 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.714773+0000", 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.8898999999999999e-05 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.394 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720154+0000", 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0053813740000000004 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:10.720193+0000", 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8856999999999997e-05 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:4880 7.1b e478/465 7:dd97af7e:::smithi13779221-30:head v 478'785, mlcod=475'781)", 2024-09-21T08:32:36.395 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:11.616312+0000", 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 84.744384788999994, 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0068370100000000001, 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616312+0000", 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616312+0000", 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.396 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616314+0000", 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.2900000000000001e-06 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616324+0000", 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0409e-05 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.397 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616339+0000", 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.4309e-05 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616348+0000", 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1519999999999998e-06 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616417+0000", 2024-09-21T08:32:36.398 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9021000000000002e-05 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.616446+0000", 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.9037999999999999e-05 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.623098+0000", 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0066520060000000002 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.399 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:11.623149+0000", 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0785000000000003e-05 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4894 7.25 7:a6c204bd:::smithi13779221-42:head [rollback c0] snapc ce=[ce,cc,cb,ca,c9,c8,c6,c5,c4,c2,c1,c0,bf,be,b9,b8,b3,b2,b0,af,a5,a4,9b,92,84] ondisk+write+known_if_redirected+supports_pool_eio e480)", 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:13.613515+0000", 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 82.747181467000004, 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0081910390000000007, 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:32:36.400 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4894 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613515+0000", 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.401 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613515+0000", 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613516+0000", 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.3799999999999996e-07 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.402 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613519+0000", 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.1920000000000001e-06 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613530+0000", 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1277000000000001e-05 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.613549+0000", 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.8819999999999999e-05 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.403 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.614509+0000", 2024-09-21T08:32:36.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00095967999999999997 2024-09-21T08:32:36.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.404 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.404 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.614581+0000", 2024-09-21T08:32:36.404 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.2457000000000001e-05 2024-09-21T08:32:36.404 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.404 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.614894+0000", 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00031301000000000001 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.617391+0000", 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0024968049999999999 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.618189+0000", 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079810899999999997 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.405 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.621564+0000", 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0033748979999999999 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.621615+0000", 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0534000000000001e-05 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:13.621706+0000", 2024-09-21T08:32:36.406 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1519000000000002e-05 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4957 7.25 7:a6c204bd:::smithi13779221-42:head [copy-from ver 381] snapc d2=[d2,d1,d0,cf,ce,cc,cb,ca,c9,c8,c5,c4,c2,c1,c0,be,b9,b8,b3,b2,b0,af,a5,9b,92,84] ondisk+write+known_if_redirected+supports_pool_eio e487)", 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:20.644293+0000", 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.716403521000004, 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.027762614000000001, 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:32:36.407 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4957 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644293+0000", 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.408 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644293+0000", 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644295+0000", 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3949999999999999e-06 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644304+0000", 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6109999999999994e-06 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.409 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644315+0000", 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1477e-05 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644326+0000", 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0919e-05 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644658+0000", 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000331742 2024-09-21T08:32:36.410 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644790+0000", 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013230199999999999 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 1,5", 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.660702+0000", 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.015912017000000001 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.411 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.667882+0000", 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0071797939999999998 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.667894+0000", 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1807000000000001e-05 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.671798+0000", 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.003903672 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.412 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.671922+0000", 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00012392499999999999 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.672056+0000", 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000133953 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:4958 7.25 7:a6c204bd:::smithi13779221-42:head [stat] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e487)", 2024-09-21T08:32:36.413 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:20.644340+0000", 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 75.716356407000006, 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.020106951000000001, 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 4958 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.414 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644340+0000", 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644340+0000", 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644341+0000", 2024-09-21T08:32:36.415 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9100000000000003e-07 2024-09-21T08:32:36.416 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.416 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.416 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.416 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:36.417 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644344+0000", 2024-09-21T08:32:36.417 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2590000000000001e-06 2024-09-21T08:32:36.417 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.417 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.417 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644365+0000", 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1206e-05 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.644373+0000", 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6909999999999997e-06 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.418 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.645689+0000", 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001316273 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for blocked object", 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.645739+0000", 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.9679999999999999e-05 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.664230+0000", 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.018491408000000001 2024-09-21T08:32:36.419 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.664312+0000", 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.2021999999999995e-05 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:20.664447+0000", 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000134721 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:5272 7.16 e512/507 7:6a4067f0:::smithi13779221-23:head v 512'221, mlcod=0'0)", 2024-09-21T08:32:36.420 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:45.645636+0000", 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 50.715060403000003, 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052462949999999998, 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645636+0000", 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.421 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645636+0000", 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645636+0000", 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.89e-07 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645658+0000", 2024-09-21T08:32:36.422 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1725e-05 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645669+0000", 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0783e-05 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645678+0000", 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.6959999999999994e-06 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645752+0000", 2024-09-21T08:32:36.423 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.4153000000000006e-05 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.645779+0000", 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.6744999999999999e-05 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.650826+0000", 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050476289999999997 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:45.650882+0000", 2024-09-21T08:32:36.424 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6174999999999997e-05 2024-09-21T08:32:36.425 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.425 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.425 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.425 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.425 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.425 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:5325 7.7 e521/473 7:e1536981:::smithi13779221-32:head v 521'219, mlcod=502'215)", 2024-09-21T08:32:36.425 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:54.623467+0000", 2024-09-21T08:32:36.425 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.737229358, 2024-09-21T08:32:36.425 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0065027629999999999, 2024-09-21T08:32:36.425 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.623467+0000", 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.623467+0000", 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.426 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.623467+0000", 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.8500000000000002e-07 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.624193+0000", 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00072595699999999995 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.624213+0000", 2024-09-21T08:32:36.427 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.9247999999999999e-05 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.624224+0000", 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0871e-05 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.624293+0000", 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.9109000000000005e-05 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.624323+0000", 2024-09-21T08:32:36.428 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.0141000000000002e-05 2024-09-21T08:32:36.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.629892+0000", 2024-09-21T08:32:36.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0055694350000000002 2024-09-21T08:32:36.429 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.429 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.429 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.429 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.629970+0000", 2024-09-21T08:32:36.429 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7616999999999999e-05 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:5326 7.7 e521/473 7:e1536981:::smithi13779221-32:head v 521'220, mlcod=502'215)", 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:54.624697+0000", 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.735999538999998, 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0069087330000000002, 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.430 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.624697+0000", 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.624697+0000", 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.624698+0000", 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.5269999999999999e-06 2024-09-21T08:32:36.431 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625111+0000", 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041218199999999998 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625121+0000", 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.0513000000000001e-05 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.432 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625128+0000", 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.8900000000000001e-06 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625188+0000", 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.9917000000000002e-05 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625211+0000", 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.3499999999999999e-05 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.433 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.631371+0000", 2024-09-21T08:32:36.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006159678 2024-09-21T08:32:36.434 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.434 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.434 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.434 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.631606+0000", 2024-09-21T08:32:36.434 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00023452600000000001 2024-09-21T08:32:36.434 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.434 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.435 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.435 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.435 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.435 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_repop(client.14844.0:5327 7.7 e521/473 7:e1536981:::smithi13779221-32:head v 521'221, mlcod=502'215)", 2024-09-21T08:32:36.435 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:54.625769+0000", 2024-09-21T08:32:36.435 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.734927720000002, 2024-09-21T08:32:36.435 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0059634620000000001, 2024-09-21T08:32:36.435 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.435 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625769+0000", 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625769+0000", 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.436 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625770+0000", 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3069999999999999e-06 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625779+0000", 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.7860000000000002e-06 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625788+0000", 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0990000000000005e-06 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.437 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625796+0000", 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5950000000000001e-06 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625846+0000", 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0716e-05 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.625867+0000", 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.0387999999999999e-05 2024-09-21T08:32:36.438 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.631609+0000", 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.005742792 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.631732+0000", 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000122779 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.439 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "pg_update_log_missing(7.27 epoch 521/475 rep_tid 1335 entries 521'212 (0'0) error 7:e55f2143:::smithi13779221-11:head by client.14844.0:5329 0.000000 -2 [r=-2+0b] ObjectCleanRegions clean_offsets: [(0, 18446744073709551615)], clean_omap: true, new_object: false trim_to 0'0 roll_forward_to 514'209)", 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:31:54.626816+0000", 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 41.733880769000002, 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0070228499999999998, 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "reached pg", 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.440 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.626816+0000", 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.626816+0000", 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.626817+0000", 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.187e-06 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.441 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.626825+0000", 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.5699999999999993e-06 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.626843+0000", 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.7240000000000001e-05 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.442 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.626856+0000", 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.377e-05 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.631876+0000", 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0050192509999999997 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:31:54.633839+0000", 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019628319999999999 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.443 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:5445 7.f 7:f1c8dcbe:::smithi13779221-1:head [write 571636~488846] snapc ed=[ec,eb,ea,e9,e8,e5,e4,e2,e1,dd,dc,da,d9,d7,d6,d3,d1,cb,ca,c9,c2,c1,b9,b8,b3,b2,b0,af,a5,9b,92] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:32:10.699321+0000", 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.661375838000001, 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0048503820000000003, 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 5445 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.444 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.699321+0000", 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.699321+0000", 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.699325+0000", 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.4240000000000003e-06 2024-09-21T08:32:36.445 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.699905+0000", 2024-09-21T08:32:36.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00058021099999999996 2024-09-21T08:32:36.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.446 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.446 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.446 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.699933+0000", 2024-09-21T08:32:36.446 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7515999999999999e-05 2024-09-21T08:32:36.446 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.699946+0000", 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.3501e-05 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.700050+0000", 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000103686 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.700182+0000", 2024-09-21T08:32:36.447 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00013176799999999999 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.700380+0000", 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00019866299999999999 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.702067+0000", 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001686903 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.448 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.703631+0000", 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0015641120000000001 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.704042+0000", 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00041083800000000001 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.704090+0000", 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7614000000000002e-05 2024-09-21T08:32:36.449 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.704171+0000", 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.1145999999999999e-05 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:5446 7.f 7:f1c8dcbe:::smithi13779221-1:head [write 1854531~565263] snapc ed=[ec,eb,ea,e9,e8,e5,e4,e2,e1,dd,dc,da,d9,d7,d6,d3,d1,cb,ca,c9,c2,c1,b9,b8,b3,b2,b0,af,a5,9b,92] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-09-21T08:32:36.450 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:32:10.706938+0000", 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.653758530000001, 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0076819109999999996, 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 5446 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.451 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.706938+0000", 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.706938+0000", 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.706946+0000", 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6359999999999994e-06 2024-09-21T08:32:36.452 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707498+0000", 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00055276499999999998 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707520+0000", 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.1543999999999999e-05 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707529+0000", 2024-09-21T08:32:36.453 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5719999999999992e-06 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707605+0000", 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.5838000000000001e-05 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707768+0000", 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00016248400000000001 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:32:36.454 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707938+0000", 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000170101 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714228+0000", 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.006290454 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714241+0000", 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.2717999999999999e-05 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.455 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714487+0000", 2024-09-21T08:32:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00024571 2024-09-21T08:32:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714544+0000", 2024-09-21T08:32:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.6767e-05 2024-09-21T08:32:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.456 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714620+0000", 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.6322000000000006e-05 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:5447 7.f 7:f1c8dcbe:::smithi13779221-1:head [setxattr _header (57) in=7b,truncate 2571636] snapc ed=[ec,eb,ea,e9,e8,e5,e4,e2,e1,dd,dc,da,d9,d7,d6,d3,d1,cb,ca,c9,c2,c1,b9,b8,b3,b2,b0,af,a5,9b,92] ondisk+write+known_if_redirected+supports_pool_eio e537)", 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:32:10.707555+0000", 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.653141440999999, 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0072772080000000003, 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.457 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "commit sent; apply or cleanup", 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 5447 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707555+0000", 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.458 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707555+0000", 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707556+0000", 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 1.1400000000000001e-06 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707562+0000", 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.5969999999999999e-06 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.459 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707568+0000", 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.7209999999999997e-06 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707574+0000", 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.4920000000000001e-06 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.708960+0000", 2024-09-21T08:32:36.460 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.001385652 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.709091+0000", 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000131798 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for subops from 5,7", 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.709273+0000", 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00018188899999999999 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.461 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "op_commit", 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714246+0000", 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0049731879999999999 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714356+0000", 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000109151 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "sub_op_commit_rec", 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714694+0000", 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00033877499999999998 2024-09-21T08:32:36.462 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "commit_sent", 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714755+0000", 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.0287000000000001e-05 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714832+0000", 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.7517999999999999e-05 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.463 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.14844.0:5448 7.f 7:f1c8dcbe:::smithi13779221-1:head [read 0~1] snapc 0=[] ondisk+read+rwordered+known_if_redirected+supports_pool_eio e537)", 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:32:10.707581+0000", 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 25.653115662000001, 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0074757679999999998, 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.14844", 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3100918956", 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 5448 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.464 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707581+0000", 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707581+0000", 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707581+0000", 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.0200000000000002e-07 2024-09-21T08:32:36.465 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707584+0000", 2024-09-21T08:32:36.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.4980000000000001e-06 2024-09-21T08:32:36.466 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.466 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.466 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.466 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707588+0000", 2024-09-21T08:32:36.466 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9920000000000001e-06 2024-09-21T08:32:36.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.707592+0000", 2024-09-21T08:32:36.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.1269999999999996e-06 2024-09-21T08:32:36.467 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.467 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.467 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.467 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.709578+0000", 2024-09-21T08:32:36.467 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0019864430000000001 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.709667+0000", 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 8.9124999999999998e-05 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714876+0000", 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0052089340000000001 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.468 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.714942+0000", 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.5313000000000003e-05 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:10.715057+0000", 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00011483400000000001 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:2148 5.1f 5:f95f44c2:::notify.0:head [watch reconnect cookie 94837804042368 gen 15] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e561)", 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:32:35.691696+0000", 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.66900023799999997, 2024-09-21T08:32:36.469 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.13201007000000001, 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2148 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691696+0000", 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.470 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691696+0000", 2024-09-21T08:32:36.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.471 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.471 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.471 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.471 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691697+0000", 2024-09-21T08:32:36.471 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.5999999999999991e-07 2024-09-21T08:32:36.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.472 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.472 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.472 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691700+0000", 2024-09-21T08:32:36.472 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.2160000000000002e-06 2024-09-21T08:32:36.472 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.473 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691708+0000", 2024-09-21T08:32:36.473 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.272e-06 2024-09-21T08:32:36.473 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.473 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.473 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691715+0000", 2024-09-21T08:32:36.474 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.1520000000000003e-06 2024-09-21T08:32:36.474 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.474 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.474 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.474 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691762+0000", 2024-09-21T08:32:36.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 4.7290000000000003e-05 2024-09-21T08:32:36.475 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.475 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.475 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:32:36.475 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691770+0000", 2024-09-21T08:32:36.475 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8949999999999998e-06 2024-09-21T08:32:36.476 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.476 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.476 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.476 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.816815+0000", 2024-09-21T08:32:36.476 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12504514 2024-09-21T08:32:36.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.477 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.477 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-09-21T08:32:36.477 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.816955+0000", 2024-09-21T08:32:36.477 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00014012000000000001 2024-09-21T08:32:36.477 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.819835+0000", 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0028793540000000002 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.819926+0000", 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1162999999999994e-05 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.478 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.820725+0000", 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00079964900000000004 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.820790+0000", 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.4528000000000003e-05 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.823706+0000", 2024-09-21T08:32:36.479 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0029163309999999999 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: "description": "osd_op(client.44387.0:2150 5.e 5:7759931f:::notify.3:head [watch reconnect cookie 94837804037760 gen 9] snapc 0=[] ondisk+write+known_if_redirected+supports_pool_eio e561)", 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: "initiated_at": "2024-09-21T08:32:35.691731+0000", 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: "age": 0.66896546999999995, 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.12394812400000001, 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: "continuous": false, 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: "type_data": { 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: "flag_point": "started", 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client_info": { 2024-09-21T08:32:36.480 INFO:teuthology.orchestra.run.smithi165.stdout: "client": "client.44387", 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: "client_addr": "v1:172.21.15.137:0/3690342859", 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: "tid": 2150 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: "events": [ 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "initiated", 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691731+0000", 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "header_read", 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691731+0000", 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0 2024-09-21T08:32:36.481 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "throttled", 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691732+0000", 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 7.8199999999999999e-07 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "all_read", 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691735+0000", 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 2.7819999999999998e-06 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "dispatched", 2024-09-21T08:32:36.482 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691740+0000", 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.276e-06 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "queued_for_pg", 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691745+0000", 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 5.3120000000000001e-06 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691784+0000", 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 3.9269e-05 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for peered", 2024-09-21T08:32:36.483 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.691791+0000", 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 6.3369999999999996e-06 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.807418+0000", 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.115627647 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for missing object", 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.807534+0000", 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.000115121 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.484 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.811475+0000", 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0039409950000000001 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "waiting for rw locks", 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.811566+0000", 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.1629e-05 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "reached_pg", 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.812077+0000", 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.00051051099999999995 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "started", 2024-09-21T08:32:36.485 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.812167+0000", 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 9.0377000000000006e-05 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: }, 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: { 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: "event": "done", 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: "time": "2024-09-21T08:32:35.815679+0000", 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: "duration": 0.0035120860000000002 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: } 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout: ] 2024-09-21T08:32:36.486 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:32:36.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:36.509 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:36.509 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:36.610 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:36.634 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:36.635 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:36.635 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:36.736 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:36.758 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:36.758 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:36.759 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:36.792 INFO:tasks.rados.rados.0.smithi137.stdout:2238: done (2 left) 2024-09-21T08:32:36.793 INFO:tasks.rados.rados.0.smithi137.stdout:2239: read oid 6 snap -1 2024-09-21T08:32:36.793 INFO:tasks.rados.rados.0.smithi137.stdout:2239: expect deleted 2024-09-21T08:32:36.793 INFO:tasks.rados.rados.0.smithi137.stdout:2240: write oid 29 current snap is 249 2024-09-21T08:32:36.793 INFO:tasks.rados.rados.0.smithi137.stdout:2240: seq_num 778 ranges {675846=411728,1543579=787592,2848853=226993} 2024-09-21T08:32:36.798 INFO:tasks.rados.rados.0.smithi137.stdout:2240: writing smithi13779221-29 from 675846 to 1087574 tid 1 2024-09-21T08:32:36.806 INFO:tasks.rados.rados.0.smithi137.stdout:2240: writing smithi13779221-29 from 1543579 to 2331171 tid 2 2024-09-21T08:32:36.811 INFO:tasks.rados.rados.0.smithi137.stdout:2240: writing smithi13779221-29 from 2848853 to 3075846 tid 3 2024-09-21T08:32:36.811 INFO:tasks.rados.rados.0.smithi137.stdout:2241: read oid 17 snap -1 2024-09-21T08:32:36.811 INFO:tasks.rados.rados.0.smithi137.stdout:2241: expect (ObjNum 774 snap 246 seq_num 774) 2024-09-21T08:32:36.811 INFO:tasks.rados.rados.0.smithi137.stdout:2242: read oid 11 snap -1 2024-09-21T08:32:36.811 INFO:tasks.rados.rados.0.smithi137.stdout:2242: expect deleted 2024-09-21T08:32:36.811 INFO:tasks.rados.rados.0.smithi137.stdout:2243: read oid 22 snap -1 2024-09-21T08:32:36.811 INFO:tasks.rados.rados.0.smithi137.stdout:2243: expect (ObjNum 737 snap 230 seq_num 737) 2024-09-21T08:32:36.811 INFO:tasks.rados.rados.0.smithi137.stdout:2244: rollback oid 3 current snap is 249 2024-09-21T08:32:36.811 INFO:tasks.rados.rados.0.smithi137.stdout:rollback oid 3 to 226 2024-09-21T08:32:36.812 INFO:tasks.rados.rados.0.smithi137.stdout:2237: finishing copy_from racing read to smithi13779221-40 2024-09-21T08:32:36.812 INFO:tasks.rados.rados.0.smithi137.stdout:2236: expect (ObjNum 725 snap 226 seq_num 725) 2024-09-21T08:32:36.859 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:36.882 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:36.882 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:36.883 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2237: finishing copy_from to smithi13779221-40 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 40 v 734 (ObjNum 566 snap 168 seq_num 566) dirty exists 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2240: finishing write tid 1 to smithi13779221-29 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2240: finishing write tid 2 to smithi13779221-29 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2236: done (7 left) 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2237: done (6 left) 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2239: done (5 left) 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2245: snap_create 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2244: finishing rollback tid 1 to smithi13779221-3 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 3 v 502 (ObjNum 33 snap 0 seq_num 3731216216) dirty dne 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2240: finishing write tid 3 to smithi13779221-29 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2240: finishing write tid 4 to smithi13779221-29 2024-09-21T08:32:36.937 INFO:tasks.rados.rados.0.smithi137.stdout:2240: finishing write tid 5 to smithi13779221-29 2024-09-21T08:32:36.938 INFO:tasks.rados.rados.0.smithi137.stdout:2240: oid 29 updating version 0 to 854 2024-09-21T08:32:36.938 INFO:tasks.rados.rados.0.smithi137.stdout:2240: oid 29 version 854 is already newer than 852 2024-09-21T08:32:36.938 INFO:tasks.rados.rados.0.smithi137.stdout:2240: oid 29 version 854 is already newer than 853 2024-09-21T08:32:36.938 INFO:tasks.rados.rados.0.smithi137.stdout:2240: oid 29 version 854 is already newer than 851 2024-09-21T08:32:36.938 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 29 v 854 (ObjNum 778 snap 249 seq_num 778) dirty exists 2024-09-21T08:32:36.938 INFO:tasks.rados.rados.0.smithi137.stdout:2240: left oid 29 (ObjNum 778 snap 249 seq_num 778) 2024-09-21T08:32:36.938 INFO:tasks.rados.rados.0.smithi137.stdout:2241: expect (ObjNum 774 snap 246 seq_num 774) 2024-09-21T08:32:36.984 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:37.004 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:37.005 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:37.005 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:37.106 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:37.124 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:36 smithi165 ceph-mon[32893]: osdmap e562: 8 total, 8 up, 5 in 2024-09-21T08:32:37.124 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:36 smithi165 ceph-mon[32893]: pgmap v749: 188 pgs: 3 unknown, 4 active+clean+snaptrim, 23 peering, 24 active+clean+snaptrim_wait, 134 active+clean; 604 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 1.6 MiB/s wr, 0 op/s 2024-09-21T08:32:37.125 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:36 smithi165 ceph-mon[32893]: osdmap e563: 8 total, 8 up, 5 in 2024-09-21T08:32:37.130 INFO:tasks.rados.rados.0.smithi137.stdout:2243: expect (ObjNum 737 snap 230 seq_num 737) 2024-09-21T08:32:37.130 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:37.131 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:37.131 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:37.164 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:32:37.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:36 smithi137 ceph-mon[29634]: osdmap e562: 8 total, 8 up, 5 in 2024-09-21T08:32:37.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:36 smithi137 ceph-mon[29634]: pgmap v749: 188 pgs: 3 unknown, 4 active+clean+snaptrim, 23 peering, 24 active+clean+snaptrim_wait, 134 active+clean; 604 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 1.6 MiB/s wr, 0 op/s 2024-09-21T08:32:37.165 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:36 smithi137 ceph-mon[29634]: osdmap e563: 8 total, 8 up, 5 in 2024-09-21T08:32:37.165 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:36 smithi137 ceph-mon[41044]: osdmap e562: 8 total, 8 up, 5 in 2024-09-21T08:32:37.165 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:36 smithi137 ceph-mon[41044]: pgmap v749: 188 pgs: 3 unknown, 4 active+clean+snaptrim, 23 peering, 24 active+clean+snaptrim_wait, 134 active+clean; 604 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 1.6 MiB/s wr, 0 op/s 2024-09-21T08:32:37.165 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:36 smithi137 ceph-mon[41044]: osdmap e563: 8 total, 8 up, 5 in 2024-09-21T08:32:37.172 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:32:37.231 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:37.253 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:37.253 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:37.254 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:37.355 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:37.376 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:37.377 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:37.377 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:37.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:37.500 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:37.501 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:37.501 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:37.507 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_ops_in_flight 2024-09-21T08:32:37.602 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:37.624 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:37.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:37.625 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:37.725 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:37.747 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:37.747 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:37.748 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:37.762 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:32:37.797 INFO:teuthology.orchestra.run.smithi137.stderr:nodeep-scrub is unset 2024-09-21T08:32:37.806 INFO:tasks.rados.rados.0.smithi137.stdout:2240: done (5 left) 2024-09-21T08:32:37.806 INFO:tasks.rados.rados.0.smithi137.stdout:2241: done (4 left) 2024-09-21T08:32:37.807 INFO:tasks.rados.rados.0.smithi137.stdout:2242: done (3 left) 2024-09-21T08:32:37.807 INFO:tasks.rados.rados.0.smithi137.stdout:2243: done (2 left) 2024-09-21T08:32:37.807 INFO:tasks.rados.rados.0.smithi137.stdout:2244: done (1 left) 2024-09-21T08:32:37.807 INFO:tasks.rados.rados.0.smithi137.stdout:2245: done (0 left) 2024-09-21T08:32:37.807 INFO:tasks.rados.rados.0.smithi137.stdout:2246: snap_remove snap 223 2024-09-21T08:32:37.848 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:37.869 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:37.870 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:37.870 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:37.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[29634]: purged_snaps scrub starts 2024-09-21T08:32:37.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[29634]: purged_snaps scrub ok 2024-09-21T08:32:37.945 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[29634]: 6.3 scrub starts 2024-09-21T08:32:37.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[29634]: 6.3 scrub ok 2024-09-21T08:32:37.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[29634]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T08:32:37.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[29634]: from='client.26107 v1:172.21.15.137:0/521065845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:32:37.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2470999473' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:37.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[29634]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:32:37.946 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/2470999473' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:37.946 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[41044]: purged_snaps scrub starts 2024-09-21T08:32:37.946 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[41044]: purged_snaps scrub ok 2024-09-21T08:32:37.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[41044]: 6.3 scrub starts 2024-09-21T08:32:37.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[41044]: 6.3 scrub ok 2024-09-21T08:32:37.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[41044]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T08:32:37.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[41044]: from='client.26107 v1:172.21.15.137:0/521065845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:32:37.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2470999473' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:37.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[41044]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:32:37.947 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:37 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/2470999473' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:37.970 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:37.972 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":750,"stamp":"2024-09-21T08:32:36.785553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":633831653,"num_objects":495,"num_object_clones":247,"num_object_copies":1485,"num_objects_missing_on_primary":0,"num_objects_missing":16,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":495,"num_whiteouts":12,"num_read":3393,"num_read_kb":2362823,"num_write":2003,"num_write_kb":627012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1558,"num_bytes_recovered":650731801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13652,"ondisk_log_size":13652,"up":564,"acting":564,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":564,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":1643404,"kb_used_data":1389020,"kb_used_omap":44,"kb_used_meta":251859,"kb_avail":467082356,"statfs":{"total":479975178240,"available":478292332544,"internally_reserved":0,"allocated":1422356480,"data_stored":2071475265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45341,"internal_metadata":257904355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":161003,"num_objects":3,"num_object_clones":4,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":16,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":4,"num_read":-50,"num_read_kb":-104083,"num_write":4,"num_write_kb":6373,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-56,"num_bytes_recovered":-38249075,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.018826"},"pg_stats":[{"pgid":"7.2d","version":"455'154","reported_seq":987,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.747838+0000","last_change":"2024-09-21T08:32:35.747893+0000","last_active":"2024-09-21T08:32:35.747838+0000","last_peered":"2024-09-21T08:32:35.747838+0000","last_clean":"2024-09-21T08:32:35.747838+0000","last_became_active":"2024-09-21T08:31:56.670753+0000","last_became_peered":"2024-09-21T08:31:56.670753+0000","last_unstale":"2024-09-21T08:32:35.747838+0000","last_undegraded":"2024-09-21T08:32:35.747838+0000","last_fullsized":"2024-09-21T08:32:35.747838+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":6,"last_scrub":"294'93","last_scrub_stamp":"2024-09-21T08:28:11.373358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:11.373358+0000","objects_scrubbed":3,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":102,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.328085935,"stat_sum":{"num_bytes":13974387,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":16,"num_read_kb":8938,"num_write":21,"num_write_kb":5598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3285818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"2"}]},{"pgid":"7.2c","version":"558'343","reported_seq":2753,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-21T08:32:33.799690+0000","last_change":"2024-09-21T08:32:33.800506+0000","last_active":"2024-09-21T08:32:33.799690+0000","last_peered":"2024-09-21T08:32:33.799690+0000","last_clean":"2024-09-21T08:32:33.799690+0000","last_became_active":"2024-09-21T08:31:59.631276+0000","last_became_peered":"2024-09-21T08:31:59.631276+0000","last_unstale":"2024-09-21T08:32:33.799690+0000","last_undegraded":"2024-09-21T08:32:33.799690+0000","last_fullsized":"2024-09-21T08:32:33.799690+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":6,"last_scrub":"533'325","last_scrub_stamp":"2024-09-21T08:32:08.675641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:32:08.675641+0000","objects_scrubbed":9,"log_size":343,"log_dups_size":0,"ondisk_log_size":343,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:27.689822+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0011806469999999999,"stat_sum":{"num_bytes":14586669,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"7.2b","version":"560'946","reported_seq":3458,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-21T08:32:33.803309+0000","last_change":"2024-09-21T08:32:33.803597+0000","last_active":"2024-09-21T08:32:33.803309+0000","last_peered":"2024-09-21T08:32:33.803309+0000","last_clean":"2024-09-21T08:32:33.803309+0000","last_became_active":"2024-09-21T08:31:59.630195+0000","last_became_peered":"2024-09-21T08:31:59.630195+0000","last_unstale":"2024-09-21T08:32:33.803309+0000","last_undegraded":"2024-09-21T08:32:33.803309+0000","last_fullsized":"2024-09-21T08:32:33.803309+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":6,"last_scrub":"533'879","last_scrub_stamp":"2024-09-21T08:32:07.696661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:32:07.696661+0000","objects_scrubbed":19,"log_size":946,"log_dups_size":0,"ondisk_log_size":946,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:27.519758+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0035125550000000001,"stat_sum":{"num_bytes":45927961,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":12,"num_read_kb":6558,"num_write":11,"num_write_kb":8168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"7.2a","version":"472'122","reported_seq":1659,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.418353+0000","last_change":"2024-09-21T08:32:33.769192+0000","last_active":"2024-09-21T08:32:35.418353+0000","last_peered":"2024-09-21T08:32:35.418353+0000","last_clean":"2024-09-21T08:32:35.418353+0000","last_became_active":"2024-09-21T08:31:59.629605+0000","last_became_peered":"2024-09-21T08:31:59.629605+0000","last_unstale":"2024-09-21T08:32:35.418353+0000","last_undegraded":"2024-09-21T08:32:35.418353+0000","last_fullsized":"2024-09-21T08:32:35.418353+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":6,"last_scrub":"472'122","last_scrub_stamp":"2024-09-21T08:31:28.194668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:28.194668+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0003801139999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"de","length":"1"}]},{"pgid":"7.29","version":"558'403","reported_seq":3046,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.420787+0000","last_change":"2024-09-21T08:32:33.741879+0000","last_active":"2024-09-21T08:32:35.420787+0000","last_peered":"2024-09-21T08:32:35.420787+0000","last_clean":"2024-09-21T08:32:35.420787+0000","last_became_active":"2024-09-21T08:31:57.664264+0000","last_became_peered":"2024-09-21T08:31:57.664264+0000","last_unstale":"2024-09-21T08:32:35.420787+0000","last_undegraded":"2024-09-21T08:32:35.420787+0000","last_fullsized":"2024-09-21T08:32:35.420787+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011461229999999999,"stat_sum":{"num_bytes":23676791,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":104,"num_read_kb":101846,"num_write":75,"num_write_kb":24161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":33558636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"2"},{"start":"c6","length":"3"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"7.28","version":"561'476","reported_seq":2785,"reported_epoch":561,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:32:35.747932+0000","last_change":"2024-09-21T08:32:35.747932+0000","last_active":"2024-09-21T08:32:35.747932+0000","last_peered":"2024-09-21T08:32:35.747932+0000","last_clean":"2024-09-21T08:32:35.747932+0000","last_became_active":"2024-09-21T08:31:59.631478+0000","last_became_peered":"2024-09-21T08:31:59.631478+0000","last_unstale":"2024-09-21T08:32:35.747932+0000","last_undegraded":"2024-09-21T08:32:35.747932+0000","last_fullsized":"2024-09-21T08:32:35.747932+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'178","last_scrub_stamp":"2024-09-21T08:26:28.217842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.217842+0000","objects_scrubbed":4,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0065354920000000004,"stat_sum":{"num_bytes":20788182,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":101,"num_read_kb":109949,"num_write":87,"num_write_kb":32244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":44868606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"7"},{"start":"c3","length":"2"},{"start":"c6","length":"3"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"7.27","version":"553'216","reported_seq":2593,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.423028+0000","last_change":"2024-09-21T08:32:33.769952+0000","last_active":"2024-09-21T08:32:35.423028+0000","last_peered":"2024-09-21T08:32:35.423028+0000","last_clean":"2024-09-21T08:32:35.423028+0000","last_became_active":"2024-09-21T08:31:57.659296+0000","last_became_peered":"2024-09-21T08:31:57.659296+0000","last_unstale":"2024-09-21T08:32:35.423028+0000","last_undegraded":"2024-09-21T08:32:35.423028+0000","last_fullsized":"2024-09-21T08:32:35.423028+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":67,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.99920343499999997,"stat_sum":{"num_bytes":11587668,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":54,"num_read_kb":52955,"num_write":41,"num_write_kb":19786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":38107210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"25"}]},{"pgid":"7.26","version":"453'161","reported_seq":2196,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.746743+0000","last_change":"2024-09-21T08:32:35.746798+0000","last_active":"2024-09-21T08:32:35.746743+0000","last_peered":"2024-09-21T08:32:35.746743+0000","last_clean":"2024-09-21T08:32:35.746743+0000","last_became_active":"2024-09-21T08:31:56.648746+0000","last_became_peered":"2024-09-21T08:31:56.648746+0000","last_unstale":"2024-09-21T08:32:35.746743+0000","last_undegraded":"2024-09-21T08:32:35.746743+0000","last_fullsized":"2024-09-21T08:32:35.746743+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":6,"last_scrub":"294'138","last_scrub_stamp":"2024-09-21T08:28:41.855754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:41.855754+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":93,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.32741683900000002,"stat_sum":{"num_bytes":6518339,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":31642,"num_write":14,"num_write_kb":4831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":4641445,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"b"}]},{"pgid":"7.25","version":"558'365","reported_seq":2763,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788546+0000","last_change":"2024-09-21T08:32:33.783736+0000","last_active":"2024-09-21T08:32:34.788546+0000","last_peered":"2024-09-21T08:32:34.788546+0000","last_clean":"2024-09-21T08:32:34.788546+0000","last_became_active":"2024-09-21T08:31:57.648035+0000","last_became_peered":"2024-09-21T08:31:57.648035+0000","last_unstale":"2024-09-21T08:32:34.788546+0000","last_undegraded":"2024-09-21T08:32:34.788546+0000","last_fullsized":"2024-09-21T08:32:34.788546+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"487'327","last_scrub_stamp":"2024-09-21T08:31:24.989978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:24.989978+0000","objects_scrubbed":4,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:51.582411+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00079025899999999997,"stat_sum":{"num_bytes":10958659,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":8870,"num_write":6,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7502779,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"7.24","version":"554'192","reported_seq":2519,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.418784+0000","last_change":"2024-09-21T08:32:33.737274+0000","last_active":"2024-09-21T08:32:35.418784+0000","last_peered":"2024-09-21T08:32:35.418784+0000","last_clean":"2024-09-21T08:32:35.418784+0000","last_became_active":"2024-09-21T08:31:12.614215+0000","last_became_peered":"2024-09-21T08:31:12.614215+0000","last_unstale":"2024-09-21T08:32:35.418784+0000","last_undegraded":"2024-09-21T08:32:35.418784+0000","last_fullsized":"2024-09-21T08:32:35.418784+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":6,"last_scrub":"492'170","last_scrub_stamp":"2024-09-21T08:31:30.858539+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:31:30.858539+0000","objects_scrubbed":4,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:42.469710+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0064774680000000001,"stat_sum":{"num_bytes":8944469,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":14,"num_write_kb":3263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"6"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"7.23","version":"179'19","reported_seq":1282,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:35.423190+0000","last_change":"2024-09-21T08:32:35.423447+0000","last_active":"2024-09-21T08:32:35.423161+0000","last_peered":"2024-09-21T08:32:33.804649+0000","last_clean":"2024-09-21T08:32:33.804649+0000","last_became_active":"2024-09-21T08:31:57.658433+0000","last_became_peered":"2024-09-21T08:31:57.658433+0000","last_unstale":"2024-09-21T08:32:35.423190+0000","last_undegraded":"2024-09-21T08:32:35.423190+0000","last_fullsized":"2024-09-21T08:32:35.423190+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:28:10.421606+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.421606+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99473726699999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,7],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"de","length":"1"}]},{"pgid":"7.22","version":"257'264","reported_seq":2692,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788132+0000","last_change":"2024-09-21T08:32:33.784658+0000","last_active":"2024-09-21T08:32:34.788132+0000","last_peered":"2024-09-21T08:32:34.788132+0000","last_clean":"2024-09-21T08:32:34.788132+0000","last_became_active":"2024-09-21T08:31:57.651552+0000","last_became_peered":"2024-09-21T08:31:57.651552+0000","last_unstale":"2024-09-21T08:32:34.788132+0000","last_undegraded":"2024-09-21T08:32:34.788132+0000","last_fullsized":"2024-09-21T08:32:34.788132+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"257'264","last_scrub_stamp":"2024-09-21T08:27:48.727600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:48.727600+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081390599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"7.21","version":"179'43","reported_seq":2083,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.421073+0000","last_change":"2024-09-21T08:32:33.743956+0000","last_active":"2024-09-21T08:32:35.421073+0000","last_peered":"2024-09-21T08:32:35.421073+0000","last_clean":"2024-09-21T08:32:35.421073+0000","last_became_active":"2024-09-21T08:31:57.644273+0000","last_became_peered":"2024-09-21T08:31:57.644273+0000","last_unstale":"2024-09-21T08:32:35.421073+0000","last_undegraded":"2024-09-21T08:32:35.421073+0000","last_fullsized":"2024-09-21T08:32:35.421073+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:36.934372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.934372+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.012905412999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"b"},{"start":"c6","length":"3"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"7.20","version":"548'185","reported_seq":2210,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.420681+0000","last_change":"2024-09-21T08:32:33.770811+0000","last_active":"2024-09-21T08:32:35.420681+0000","last_peered":"2024-09-21T08:32:35.420681+0000","last_clean":"2024-09-21T08:32:35.420681+0000","last_became_active":"2024-09-21T08:31:56.641129+0000","last_became_peered":"2024-09-21T08:31:56.641129+0000","last_unstale":"2024-09-21T08:32:35.420681+0000","last_undegraded":"2024-09-21T08:32:35.420681+0000","last_fullsized":"2024-09-21T08:32:35.420681+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":6,"last_scrub":"251'134","last_scrub_stamp":"2024-09-21T08:27:49.318143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.318143+0000","objects_scrubbed":1,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":92,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.001246278,"stat_sum":{"num_bytes":16994774,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":29,"num_read_kb":17524,"num_write":36,"num_write_kb":7693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":8389384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"c"}]},{"pgid":"7.1b","version":"560'891","reported_seq":3499,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.746331+0000","last_change":"2024-09-21T08:32:35.746385+0000","last_active":"2024-09-21T08:32:35.746331+0000","last_peered":"2024-09-21T08:32:35.746331+0000","last_clean":"2024-09-21T08:32:35.746331+0000","last_became_active":"2024-09-21T08:30:59.601175+0000","last_became_peered":"2024-09-21T08:30:59.601175+0000","last_unstale":"2024-09-21T08:32:35.746331+0000","last_undegraded":"2024-09-21T08:32:35.746331+0000","last_fullsized":"2024-09-21T08:32:35.746331+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'499","last_scrub_stamp":"2024-09-21T08:28:09.328624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.328624+0000","objects_scrubbed":5,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.32010392399999998,"stat_sum":{"num_bytes":46461038,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":173,"num_read_kb":184879,"num_write":136,"num_write_kb":48397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":61029229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"2"},{"start":"c6","length":"3"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"3.1f","version":"0'0","reported_seq":510,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423706+0000","last_change":"2024-09-21T08:30:56.618803+0000","last_active":"2024-09-21T08:32:35.423706+0000","last_peered":"2024-09-21T08:32:35.423706+0000","last_clean":"2024-09-21T08:32:35.423706+0000","last_became_active":"2024-09-21T08:30:56.617989+0000","last_became_peered":"2024-09-21T08:30:56.617989+0000","last_unstale":"2024-09-21T08:32:35.423706+0000","last_undegraded":"2024-09-21T08:32:35.423706+0000","last_fullsized":"2024-09-21T08:32:35.423706+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":217,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789429+0000","last_change":"2024-09-21T08:30:56.662007+0000","last_active":"2024-09-21T08:32:34.789429+0000","last_peered":"2024-09-21T08:32:34.789429+0000","last_clean":"2024-09-21T08:32:34.789429+0000","last_became_active":"2024-09-21T08:30:56.628842+0000","last_became_peered":"2024-09-21T08:30:56.628842+0000","last_unstale":"2024-09-21T08:32:34.789429+0000","last_undegraded":"2024-09-21T08:32:34.789429+0000","last_fullsized":"2024-09-21T08:32:34.789429+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:44.058465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:44.058465+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":569,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421750+0000","last_change":"2024-09-21T08:31:56.651426+0000","last_active":"2024-09-21T08:32:35.421750+0000","last_peered":"2024-09-21T08:32:35.421750+0000","last_clean":"2024-09-21T08:32:35.421750+0000","last_became_active":"2024-09-21T08:31:56.651133+0000","last_became_peered":"2024-09-21T08:31:56.651133+0000","last_unstale":"2024-09-21T08:32:35.421750+0000","last_undegraded":"2024-09-21T08:32:35.421750+0000","last_fullsized":"2024-09-21T08:32:35.421750+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.424482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.424482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788693+0000","last_change":"2024-09-21T08:30:56.631377+0000","last_active":"2024-09-21T08:32:34.788693+0000","last_peered":"2024-09-21T08:32:34.788693+0000","last_clean":"2024-09-21T08:32:34.788693+0000","last_became_active":"2024-09-21T08:30:56.630218+0000","last_became_peered":"2024-09-21T08:30:56.630218+0000","last_unstale":"2024-09-21T08:32:34.788693+0000","last_undegraded":"2024-09-21T08:32:34.788693+0000","last_fullsized":"2024-09-21T08:32:34.788693+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.464969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.464969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"248'68","reported_seq":720,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788378+0000","last_change":"2024-09-21T08:32:33.783456+0000","last_active":"2024-09-21T08:32:34.788378+0000","last_peered":"2024-09-21T08:32:34.788378+0000","last_clean":"2024-09-21T08:32:34.788378+0000","last_became_active":"2024-09-21T08:31:57.648774+0000","last_became_peered":"2024-09-21T08:31:57.648774+0000","last_unstale":"2024-09-21T08:32:34.788378+0000","last_undegraded":"2024-09-21T08:32:34.788378+0000","last_fullsized":"2024-09-21T08:32:34.788378+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'68","last_scrub_stamp":"2024-09-21T08:27:35.919031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:35.919031+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010419299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":926,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.422537+0000","last_change":"2024-09-21T08:31:56.658780+0000","last_active":"2024-09-21T08:32:35.422537+0000","last_peered":"2024-09-21T08:32:35.422537+0000","last_clean":"2024-09-21T08:32:35.422537+0000","last_became_active":"2024-09-21T08:31:56.658407+0000","last_became_peered":"2024-09-21T08:31:56.658407+0000","last_unstale":"2024-09-21T08:32:35.422537+0000","last_undegraded":"2024-09-21T08:32:35.422537+0000","last_fullsized":"2024-09-21T08:32:35.422537+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.804294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.804294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":853,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421577+0000","last_change":"2024-09-21T08:31:56.643062+0000","last_active":"2024-09-21T08:32:35.421577+0000","last_peered":"2024-09-21T08:32:35.421577+0000","last_clean":"2024-09-21T08:32:35.421577+0000","last_became_active":"2024-09-21T08:31:56.642755+0000","last_became_peered":"2024-09-21T08:31:56.642755+0000","last_unstale":"2024-09-21T08:32:35.421577+0000","last_undegraded":"2024-09-21T08:32:35.421577+0000","last_fullsized":"2024-09-21T08:32:35.421577+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.146798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.146798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":843,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424380+0000","last_change":"2024-09-21T08:29:00.355702+0000","last_active":"2024-09-21T08:32:35.424380+0000","last_peered":"2024-09-21T08:32:35.424380+0000","last_clean":"2024-09-21T08:32:35.424380+0000","last_became_active":"2024-09-21T08:28:29.620169+0000","last_became_peered":"2024-09-21T08:28:29.620169+0000","last_unstale":"2024-09-21T08:32:35.424380+0000","last_undegraded":"2024-09-21T08:32:35.424380+0000","last_fullsized":"2024-09-21T08:32:35.424380+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:00.355591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:00.355591+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.803875+0000","last_change":"2024-09-21T08:32:35.750616+0000","last_active":"2024-09-21T08:32:35.750616+0000","last_peered":"2024-09-21T08:32:33.803875+0000","last_clean":"2024-09-21T08:32:33.803875+0000","last_became_active":"2024-09-21T08:31:57.642832+0000","last_became_peered":"2024-09-21T08:31:57.642832+0000","last_unstale":"2024-09-21T08:32:33.803875+0000","last_undegraded":"2024-09-21T08:32:33.803875+0000","last_fullsized":"2024-09-21T08:32:33.803875+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.340817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.340817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"507'216","reported_seq":1906,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.424525+0000","last_change":"2024-09-21T08:32:33.742643+0000","last_active":"2024-09-21T08:32:35.424525+0000","last_peered":"2024-09-21T08:32:35.424525+0000","last_clean":"2024-09-21T08:32:35.424525+0000","last_became_active":"2024-09-21T08:31:56.648297+0000","last_became_peered":"2024-09-21T08:31:56.648297+0000","last_unstale":"2024-09-21T08:32:35.424525+0000","last_undegraded":"2024-09-21T08:32:35.424525+0000","last_fullsized":"2024-09-21T08:32:35.424525+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"266'188","last_scrub_stamp":"2024-09-21T08:27:49.719124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.719124+0000","objects_scrubbed":1,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":97,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011411124999999999,"stat_sum":{"num_bytes":16452542,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":35,"num_read_kb":28684,"num_write":27,"num_write_kb":14365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":13913530,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"7"}]},{"pgid":"3.1d","version":"0'0","reported_seq":1057,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421047+0000","last_change":"2024-09-21T08:30:56.626716+0000","last_active":"2024-09-21T08:32:35.421047+0000","last_peered":"2024-09-21T08:32:35.421047+0000","last_clean":"2024-09-21T08:32:35.421047+0000","last_became_active":"2024-09-21T08:30:56.625894+0000","last_became_peered":"2024-09-21T08:30:56.625894+0000","last_unstale":"2024-09-21T08:32:35.421047+0000","last_undegraded":"2024-09-21T08:32:35.421047+0000","last_fullsized":"2024-09-21T08:32:35.421047+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.780545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.780545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:31:40.664995+0000","last_change":"2024-09-21T08:31:39.648713+0000","last_active":"2024-09-21T08:31:39.648713+0000","last_peered":"2024-09-21T08:30:54.598546+0000","last_clean":"2024-09-21T08:30:54.598546+0000","last_became_active":"2024-09-21T08:28:37.568659+0000","last_became_peered":"2024-09-21T08:28:37.568659+0000","last_unstale":"2024-09-21T08:31:40.664995+0000","last_undegraded":"2024-09-21T08:31:40.664995+0000","last_fullsized":"2024-09-21T08:31:40.664995+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:26.781541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:26.781541+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1032,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.790625+0000","last_change":"2024-09-21T08:32:35.428319+0000","last_active":"2024-09-21T08:32:35.428319+0000","last_peered":"2024-09-21T08:32:33.790625+0000","last_clean":"2024-09-21T08:32:33.790625+0000","last_became_active":"2024-09-21T08:31:57.642517+0000","last_became_peered":"2024-09-21T08:31:57.642517+0000","last_unstale":"2024-09-21T08:32:33.790625+0000","last_undegraded":"2024-09-21T08:32:33.790625+0000","last_fullsized":"2024-09-21T08:32:33.790625+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:17.851031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:32:17.851031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":972,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418269+0000","last_change":"2024-09-21T08:31:56.637990+0000","last_active":"2024-09-21T08:32:35.418269+0000","last_peered":"2024-09-21T08:32:35.418269+0000","last_clean":"2024-09-21T08:32:35.418269+0000","last_became_active":"2024-09-21T08:31:56.637161+0000","last_became_peered":"2024-09-21T08:31:56.637161+0000","last_unstale":"2024-09-21T08:32:35.418269+0000","last_undegraded":"2024-09-21T08:32:35.418269+0000","last_fullsized":"2024-09-21T08:32:35.418269+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.291226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.291226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"560'395","reported_seq":2225,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-21T08:32:33.819262+0000","last_change":"2024-09-21T08:32:33.803148+0000","last_active":"2024-09-21T08:32:33.819262+0000","last_peered":"2024-09-21T08:32:33.819262+0000","last_clean":"2024-09-21T08:32:33.819262+0000","last_became_active":"2024-09-21T08:31:56.651745+0000","last_became_peered":"2024-09-21T08:31:56.651745+0000","last_unstale":"2024-09-21T08:32:33.819262+0000","last_undegraded":"2024-09-21T08:32:33.819262+0000","last_fullsized":"2024-09-21T08:32:33.819262+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"558'389","last_scrub_stamp":"2024-09-21T08:32:33.518019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:32:33.518019+0000","objects_scrubbed":14,"log_size":395,"log_dups_size":0,"ondisk_log_size":395,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:57.936272+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.003126059,"stat_sum":{"num_bytes":37082872,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":67,"num_read_kb":68210,"num_write":94,"num_write_kb":46946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":33374951,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":1050,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419990+0000","last_change":"2024-09-21T08:31:34.816475+0000","last_active":"2024-09-21T08:32:35.419990+0000","last_peered":"2024-09-21T08:32:35.419990+0000","last_clean":"2024-09-21T08:32:35.419990+0000","last_became_active":"2024-09-21T08:30:56.627614+0000","last_became_peered":"2024-09-21T08:30:56.627614+0000","last_unstale":"2024-09-21T08:32:35.419990+0000","last_undegraded":"2024-09-21T08:32:35.419990+0000","last_fullsized":"2024-09-21T08:32:35.419990+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:34.816398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:34.816398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:35.624747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.804767+0000","last_change":"2024-09-21T08:32:35.750734+0000","last_active":"2024-09-21T08:32:35.750734+0000","last_peered":"2024-09-21T08:32:33.804767+0000","last_clean":"2024-09-21T08:32:33.804767+0000","last_became_active":"2024-09-21T08:31:57.658620+0000","last_became_peered":"2024-09-21T08:31:57.658620+0000","last_unstale":"2024-09-21T08:32:33.804767+0000","last_undegraded":"2024-09-21T08:32:33.804767+0000","last_fullsized":"2024-09-21T08:32:33.804767+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:27:26.356987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:26.356987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":655,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-21T08:32:33.799274+0000","last_change":"2024-09-21T08:31:57.655236+0000","last_active":"2024-09-21T08:32:33.799274+0000","last_peered":"2024-09-21T08:32:33.799274+0000","last_clean":"2024-09-21T08:32:33.799274+0000","last_became_active":"2024-09-21T08:31:57.654911+0000","last_became_peered":"2024-09-21T08:31:57.654911+0000","last_unstale":"2024-09-21T08:32:33.799274+0000","last_undegraded":"2024-09-21T08:32:33.799274+0000","last_fullsized":"2024-09-21T08:32:33.799274+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.850373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:34.599504+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":1056,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424581+0000","last_change":"2024-09-21T08:30:04.034656+0000","last_active":"2024-09-21T08:32:35.424581+0000","last_peered":"2024-09-21T08:32:35.424581+0000","last_clean":"2024-09-21T08:32:35.424581+0000","last_became_active":"2024-09-21T08:28:04.705450+0000","last_became_peered":"2024-09-21T08:28:04.705450+0000","last_unstale":"2024-09-21T08:32:35.424581+0000","last_undegraded":"2024-09-21T08:32:35.424581+0000","last_fullsized":"2024-09-21T08:32:35.424581+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:30:04.034575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:04.034575+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1f","version":"247'32","reported_seq":723,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789112+0000","last_change":"2024-09-21T08:32:33.784473+0000","last_active":"2024-09-21T08:32:34.789112+0000","last_peered":"2024-09-21T08:32:34.789112+0000","last_clean":"2024-09-21T08:32:34.789112+0000","last_became_active":"2024-09-21T08:30:56.617686+0000","last_became_peered":"2024-09-21T08:30:56.617686+0000","last_unstale":"2024-09-21T08:32:34.789112+0000","last_undegraded":"2024-09-21T08:32:34.789112+0000","last_fullsized":"2024-09-21T08:32:34.789112+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:27:51.078531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.078531+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00069403599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"6.1e","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788802+0000","last_change":"2024-09-21T08:31:57.646381+0000","last_active":"2024-09-21T08:32:34.788802+0000","last_peered":"2024-09-21T08:32:34.788802+0000","last_clean":"2024-09-21T08:32:34.788802+0000","last_became_active":"2024-09-21T08:31:57.645634+0000","last_became_peered":"2024-09-21T08:31:57.645634+0000","last_unstale":"2024-09-21T08:32:34.788802+0000","last_undegraded":"2024-09-21T08:32:34.788802+0000","last_fullsized":"2024-09-21T08:32:34.788802+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:35.812638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:31:35.812638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:54.761982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788599+0000","last_change":"2024-09-21T08:31:57.652500+0000","last_active":"2024-09-21T08:32:34.788599+0000","last_peered":"2024-09-21T08:32:34.788599+0000","last_clean":"2024-09-21T08:32:34.788599+0000","last_became_active":"2024-09-21T08:31:57.652209+0000","last_became_peered":"2024-09-21T08:31:57.652209+0000","last_unstale":"2024-09-21T08:32:34.788599+0000","last_undegraded":"2024-09-21T08:32:34.788599+0000","last_fullsized":"2024-09-21T08:32:34.788599+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:32.768564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:32.768564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:58.372589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":1100,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421196+0000","last_change":"2024-09-21T08:30:56.660342+0000","last_active":"2024-09-21T08:32:35.421196+0000","last_peered":"2024-09-21T08:32:35.421196+0000","last_clean":"2024-09-21T08:32:35.421196+0000","last_became_active":"2024-09-21T08:30:56.630563+0000","last_became_peered":"2024-09-21T08:30:56.630563+0000","last_unstale":"2024-09-21T08:32:35.421196+0000","last_undegraded":"2024-09-21T08:32:35.421196+0000","last_fullsized":"2024-09-21T08:32:35.421196+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:37.352879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:37.352879+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":584,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.805858+0000","last_change":"2024-09-21T08:32:35.425504+0000","last_active":"2024-09-21T08:32:35.425504+0000","last_peered":"2024-09-21T08:32:33.805858+0000","last_clean":"2024-09-21T08:32:33.805858+0000","last_became_active":"2024-09-21T08:31:57.659333+0000","last_became_peered":"2024-09-21T08:31:57.659333+0000","last_unstale":"2024-09-21T08:32:33.805858+0000","last_undegraded":"2024-09-21T08:32:33.805858+0000","last_fullsized":"2024-09-21T08:32:33.805858+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:35.108794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:35.108794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"558'368","reported_seq":2654,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.420904+0000","last_change":"2024-09-21T08:32:33.771306+0000","last_active":"2024-09-21T08:32:35.420904+0000","last_peered":"2024-09-21T08:32:35.420904+0000","last_clean":"2024-09-21T08:32:35.420904+0000","last_became_active":"2024-09-21T08:30:59.606410+0000","last_became_peered":"2024-09-21T08:30:59.606410+0000","last_unstale":"2024-09-21T08:32:35.420904+0000","last_undegraded":"2024-09-21T08:32:35.420904+0000","last_fullsized":"2024-09-21T08:32:35.420904+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'236","last_scrub_stamp":"2024-09-21T08:28:09.464279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.464279+0000","objects_scrubbed":2,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":65,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.0005313790000001,"stat_sum":{"num_bytes":29513404,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":121,"num_read_kb":114857,"num_write":92,"num_write_kb":29655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":36880118,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"27"}]},{"pgid":"6.1f","version":"0'0","reported_seq":912,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424238+0000","last_change":"2024-09-21T08:30:37.732703+0000","last_active":"2024-09-21T08:32:35.424238+0000","last_peered":"2024-09-21T08:32:35.424238+0000","last_clean":"2024-09-21T08:32:35.424238+0000","last_became_active":"2024-09-21T08:28:04.725392+0000","last_became_peered":"2024-09-21T08:28:04.725392+0000","last_unstale":"2024-09-21T08:32:35.424238+0000","last_undegraded":"2024-09-21T08:32:35.424238+0000","last_fullsized":"2024-09-21T08:32:35.424238+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:37.732599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:37.732599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":215,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788598+0000","last_change":"2024-09-21T08:31:55.884676+0000","last_active":"2024-09-21T08:32:34.788598+0000","last_peered":"2024-09-21T08:32:34.788598+0000","last_clean":"2024-09-21T08:32:34.788598+0000","last_became_active":"2024-09-21T08:30:56.630058+0000","last_became_peered":"2024-09-21T08:30:56.630058+0000","last_unstale":"2024-09-21T08:32:34.788598+0000","last_undegraded":"2024-09-21T08:32:34.788598+0000","last_fullsized":"2024-09-21T08:32:34.788598+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:55.884606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:55.884606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:24.216176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":590,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419380+0000","last_change":"2024-09-21T08:30:56.656296+0000","last_active":"2024-09-21T08:32:35.419380+0000","last_peered":"2024-09-21T08:32:35.419380+0000","last_clean":"2024-09-21T08:32:35.419380+0000","last_became_active":"2024-09-21T08:30:56.626214+0000","last_became_peered":"2024-09-21T08:30:56.626214+0000","last_unstale":"2024-09-21T08:32:35.419380+0000","last_undegraded":"2024-09-21T08:32:35.419380+0000","last_fullsized":"2024-09-21T08:32:35.419380+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:29:15.349124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:15.349124+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788279+0000","last_change":"2024-09-21T08:31:57.640029+0000","last_active":"2024-09-21T08:32:34.788279+0000","last_peered":"2024-09-21T08:32:34.788279+0000","last_clean":"2024-09-21T08:32:34.788279+0000","last_became_active":"2024-09-21T08:31:57.639712+0000","last_became_peered":"2024-09-21T08:31:57.639712+0000","last_unstale":"2024-09-21T08:32:34.788279+0000","last_undegraded":"2024-09-21T08:32:34.788279+0000","last_fullsized":"2024-09-21T08:32:34.788279+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:09.116412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.116412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1d","version":"248'84","reported_seq":958,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.420595+0000","last_change":"2024-09-21T08:32:33.768456+0000","last_active":"2024-09-21T08:32:35.420595+0000","last_peered":"2024-09-21T08:32:35.420595+0000","last_clean":"2024-09-21T08:32:35.420595+0000","last_became_active":"2024-09-21T08:31:56.656340+0000","last_became_peered":"2024-09-21T08:31:56.656340+0000","last_unstale":"2024-09-21T08:32:35.420595+0000","last_undegraded":"2024-09-21T08:32:35.420595+0000","last_fullsized":"2024-09-21T08:32:35.420595+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'84","last_scrub_stamp":"2024-09-21T08:28:12.352559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.352559+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":93,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.99689609899999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"b"}]},{"pgid":"6.1c","version":"73'1","reported_seq":554,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421209+0000","last_change":"2024-09-21T08:31:56.652728+0000","last_active":"2024-09-21T08:32:35.421209+0000","last_peered":"2024-09-21T08:32:35.421209+0000","last_clean":"2024-09-21T08:32:35.421209+0000","last_became_active":"2024-09-21T08:31:56.652400+0000","last_became_peered":"2024-09-21T08:31:56.652400+0000","last_unstale":"2024-09-21T08:32:35.421209+0000","last_undegraded":"2024-09-21T08:32:35.421209+0000","last_fullsized":"2024-09-21T08:32:35.421209+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:27:14.826047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.826047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"198'2","reported_seq":585,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.803279+0000","last_change":"2024-09-21T08:32:35.425342+0000","last_active":"2024-09-21T08:32:35.425342+0000","last_peered":"2024-09-21T08:32:33.803279+0000","last_clean":"2024-09-21T08:32:33.803279+0000","last_became_active":"2024-09-21T08:31:57.659593+0000","last_became_peered":"2024-09-21T08:31:57.659593+0000","last_unstale":"2024-09-21T08:32:33.803279+0000","last_undegraded":"2024-09-21T08:32:33.803279+0000","last_fullsized":"2024-09-21T08:32:33.803279+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'2","last_scrub_stamp":"2024-09-21T08:31:33.064915+0000","last_deep_scrub":"198'2","last_deep_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_clean_scrub_stamp":"2024-09-21T08:31:33.064915+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":1075,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421658+0000","last_change":"2024-09-21T08:32:17.238877+0000","last_active":"2024-09-21T08:32:35.421658+0000","last_peered":"2024-09-21T08:32:35.421658+0000","last_clean":"2024-09-21T08:32:35.421658+0000","last_became_active":"2024-09-21T08:31:56.657752+0000","last_became_peered":"2024-09-21T08:31:56.657752+0000","last_unstale":"2024-09-21T08:32:35.421658+0000","last_undegraded":"2024-09-21T08:32:35.421658+0000","last_fullsized":"2024-09-21T08:32:35.421658+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:32:17.238789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:32:17.238789+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:42.604117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"524'23","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:28:04.672702+0000","last_change":"2024-09-21T08:28:03.666627+0000","last_active":"2024-09-21T08:28:03.666627+0000","last_peered":"2024-09-21T08:28:02.650231+0000","last_clean":"2024-09-21T08:28:02.650231+0000","last_became_active":"2024-09-21T08:27:15.613508+0000","last_became_peered":"2024-09-21T08:27:15.613508+0000","last_unstale":"2024-09-21T08:28:04.672702+0000","last_undegraded":"2024-09-21T08:28:04.672702+0000","last_fullsized":"2024-09-21T08:28:04.672702+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'21","last_scrub_stamp":"2024-09-21T08:31:49.927821+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:31:49.927821+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'129","reported_seq":1290,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428172+0000","last_change":"2024-09-21T08:31:07.783495+0000","last_active":"2024-09-21T08:32:35.428172+0000","last_peered":"2024-09-21T08:32:35.428172+0000","last_clean":"2024-09-21T08:32:35.428172+0000","last_became_active":"2024-09-21T08:30:59.600508+0000","last_became_peered":"2024-09-21T08:30:59.600508+0000","last_unstale":"2024-09-21T08:32:35.428172+0000","last_undegraded":"2024-09-21T08:32:35.428172+0000","last_fullsized":"2024-09-21T08:32:35.428172+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:07.783361+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:07.783361+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"561'292","reported_seq":2676,"reported_epoch":561,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:32:35.739974+0000","last_change":"2024-09-21T08:32:35.739974+0000","last_active":"2024-09-21T08:32:35.739974+0000","last_peered":"2024-09-21T08:32:35.739974+0000","last_clean":"2024-09-21T08:32:35.739974+0000","last_became_active":"2024-09-21T08:31:07.701482+0000","last_became_peered":"2024-09-21T08:31:07.701482+0000","last_unstale":"2024-09-21T08:32:35.739974+0000","last_undegraded":"2024-09-21T08:32:35.739974+0000","last_fullsized":"2024-09-21T08:32:35.739974+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'254","last_scrub_stamp":"2024-09-21T08:30:46.778996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:46.778996+0000","objects_scrubbed":4,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.111565991,"stat_sum":{"num_bytes":8505355,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":89,"num_read_kb":107464,"num_write":20,"num_write_kb":7973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3117748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"6"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"4.f","version":"73'15","reported_seq":606,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.422869+0000","last_change":"2024-09-21T08:31:57.660667+0000","last_active":"2024-09-21T08:32:35.422869+0000","last_peered":"2024-09-21T08:32:35.422869+0000","last_clean":"2024-09-21T08:32:35.422869+0000","last_became_active":"2024-09-21T08:31:57.660225+0000","last_became_peered":"2024-09-21T08:31:57.660225+0000","last_unstale":"2024-09-21T08:32:35.422869+0000","last_undegraded":"2024-09-21T08:32:35.422869+0000","last_fullsized":"2024-09-21T08:32:35.422869+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:31:36.097933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:31:36.097933+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:52.132388+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":574,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420222+0000","last_change":"2024-09-21T08:31:57.653751+0000","last_active":"2024-09-21T08:32:35.420222+0000","last_peered":"2024-09-21T08:32:35.420222+0000","last_clean":"2024-09-21T08:32:35.420222+0000","last_became_active":"2024-09-21T08:31:57.653413+0000","last_became_peered":"2024-09-21T08:31:57.653413+0000","last_unstale":"2024-09-21T08:32:35.420222+0000","last_undegraded":"2024-09-21T08:32:35.420222+0000","last_fullsized":"2024-09-21T08:32:35.420222+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:32.091709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:32.091709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"524'16","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:24:38.531908+0000","last_change":"2024-09-21T08:24:37.525208+0000","last_active":"2024-09-21T08:24:37.525208+0000","last_peered":"2024-09-21T08:23:37.752724+0000","last_clean":"2024-09-21T08:23:37.752724+0000","last_became_active":"2024-09-21T08:23:34.567963+0000","last_became_peered":"2024-09-21T08:23:34.567963+0000","last_unstale":"2024-09-21T08:24:38.531908+0000","last_undegraded":"2024-09-21T08:24:38.531908+0000","last_fullsized":"2024-09-21T08:24:38.531908+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:26:44.741275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.741275+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1032,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421860+0000","last_change":"2024-09-21T08:30:21.800509+0000","last_active":"2024-09-21T08:32:35.421860+0000","last_peered":"2024-09-21T08:32:35.421860+0000","last_clean":"2024-09-21T08:32:35.421860+0000","last_became_active":"2024-09-21T08:28:29.619967+0000","last_became_peered":"2024-09-21T08:28:29.619967+0000","last_unstale":"2024-09-21T08:32:35.421860+0000","last_undegraded":"2024-09-21T08:32:35.421860+0000","last_fullsized":"2024-09-21T08:32:35.421860+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:21.800427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:21.800427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'129","reported_seq":989,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788815+0000","last_change":"2024-09-21T08:31:05.031131+0000","last_active":"2024-09-21T08:32:34.788815+0000","last_peered":"2024-09-21T08:32:34.788815+0000","last_clean":"2024-09-21T08:32:34.788815+0000","last_became_active":"2024-09-21T08:30:59.600613+0000","last_became_peered":"2024-09-21T08:30:59.600613+0000","last_unstale":"2024-09-21T08:32:34.788815+0000","last_undegraded":"2024-09-21T08:32:34.788815+0000","last_fullsized":"2024-09-21T08:32:34.788815+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:05.031027+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.031027+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"179'19","reported_seq":2258,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420415+0000","last_change":"2024-09-21T08:32:33.781223+0000","last_active":"2024-09-21T08:32:35.420415+0000","last_peered":"2024-09-21T08:32:35.420415+0000","last_clean":"2024-09-21T08:32:35.420415+0000","last_became_active":"2024-09-21T08:31:57.641840+0000","last_became_peered":"2024-09-21T08:31:57.641840+0000","last_unstale":"2024-09-21T08:32:35.420415+0000","last_undegraded":"2024-09-21T08:32:35.420415+0000","last_fullsized":"2024-09-21T08:32:35.420415+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:31:05.695533+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.695533+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064436699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"4.0","version":"73'18","reported_seq":996,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419264+0000","last_change":"2024-09-21T08:31:56.650928+0000","last_active":"2024-09-21T08:32:35.419264+0000","last_peered":"2024-09-21T08:32:35.419264+0000","last_clean":"2024-09-21T08:32:35.419264+0000","last_became_active":"2024-09-21T08:31:56.650606+0000","last_became_peered":"2024-09-21T08:31:56.650606+0000","last_unstale":"2024-09-21T08:32:35.419264+0000","last_undegraded":"2024-09-21T08:32:35.419264+0000","last_fullsized":"2024-09-21T08:32:35.419264+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-09-21T08:26:54.003173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:54.003173+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":781,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421772+0000","last_change":"2024-09-21T08:31:57.661397+0000","last_active":"2024-09-21T08:32:35.421772+0000","last_peered":"2024-09-21T08:32:35.421772+0000","last_clean":"2024-09-21T08:32:35.421772+0000","last_became_active":"2024-09-21T08:31:57.659778+0000","last_became_peered":"2024-09-21T08:31:57.659778+0000","last_unstale":"2024-09-21T08:32:35.421772+0000","last_undegraded":"2024-09-21T08:32:35.421772+0000","last_fullsized":"2024-09-21T08:32:35.421772+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:52.711745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:52.711745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:56.745978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788532+0000","last_change":"2024-09-21T08:31:57.659608+0000","last_active":"2024-09-21T08:32:34.788532+0000","last_peered":"2024-09-21T08:32:34.788532+0000","last_clean":"2024-09-21T08:32:34.788532+0000","last_became_active":"2024-09-21T08:31:57.659044+0000","last_became_peered":"2024-09-21T08:31:57.659044+0000","last_unstale":"2024-09-21T08:32:34.788532+0000","last_undegraded":"2024-09-21T08:32:34.788532+0000","last_fullsized":"2024-09-21T08:32:34.788532+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:32.813973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:32.813973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:31:57.637033+0000","last_change":"2024-09-21T08:31:57.637033+0000","last_active":"2024-09-21T08:26:49.527539+0000","last_peered":"2024-09-21T08:26:47.628755+0000","last_clean":"2024-09-21T08:26:47.628755+0000","last_became_active":"2024-09-21T08:24:56.501237+0000","last_became_peered":"2024-09-21T08:24:56.501237+0000","last_unstale":"2024-09-21T08:31:57.637033+0000","last_undegraded":"2024-09-21T08:31:57.637033+0000","last_fullsized":"2024-09-21T08:31:57.637033+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:14.680354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:32:14.680354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"475'130","reported_seq":1309,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.791095+0000","last_change":"2024-09-21T08:32:35.419419+0000","last_active":"2024-09-21T08:32:35.419419+0000","last_peered":"2024-09-21T08:32:33.791095+0000","last_clean":"2024-09-21T08:32:33.791095+0000","last_became_active":"2024-09-21T08:31:57.662728+0000","last_became_peered":"2024-09-21T08:31:57.662728+0000","last_unstale":"2024-09-21T08:32:33.791095+0000","last_undegraded":"2024-09-21T08:32:33.791095+0000","last_fullsized":"2024-09-21T08:32:33.791095+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"475'130","last_scrub_stamp":"2024-09-21T08:31:08.776209+0000","last_deep_scrub":"475'130","last_deep_scrub_stamp":"2024-09-21T08:31:08.776209+0000","last_clean_scrub_stamp":"2024-09-21T08:31:08.776209+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"560'388","reported_seq":2484,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.422948+0000","last_change":"2024-09-21T08:32:33.768779+0000","last_active":"2024-09-21T08:32:35.422948+0000","last_peered":"2024-09-21T08:32:35.422948+0000","last_clean":"2024-09-21T08:32:35.422948+0000","last_became_active":"2024-09-21T08:31:56.669347+0000","last_became_peered":"2024-09-21T08:31:56.669347+0000","last_unstale":"2024-09-21T08:32:35.422948+0000","last_undegraded":"2024-09-21T08:32:35.422948+0000","last_fullsized":"2024-09-21T08:32:35.422948+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'273","last_scrub_stamp":"2024-09-21T08:27:50.318620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.318620+0000","objects_scrubbed":3,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":95,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.99898741999999996,"stat_sum":{"num_bytes":29428059,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":115,"num_read_kb":138148,"num_write":64,"num_write_kb":29897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20331960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"9"}]},{"pgid":"4.1","version":"239'21","reported_seq":227,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788347+0000","last_change":"2024-09-21T08:31:57.658132+0000","last_active":"2024-09-21T08:32:34.788347+0000","last_peered":"2024-09-21T08:32:34.788347+0000","last_clean":"2024-09-21T08:32:34.788347+0000","last_became_active":"2024-09-21T08:31:57.657608+0000","last_became_peered":"2024-09-21T08:31:57.657608+0000","last_unstale":"2024-09-21T08:32:34.788347+0000","last_undegraded":"2024-09-21T08:32:34.788347+0000","last_fullsized":"2024-09-21T08:32:34.788347+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21","last_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_deep_scrub":"239'21","last_deep_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.969642+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"475'2","reported_seq":1026,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421696+0000","last_change":"2024-09-21T08:31:57.784895+0000","last_active":"2024-09-21T08:32:35.421696+0000","last_peered":"2024-09-21T08:32:35.421696+0000","last_clean":"2024-09-21T08:32:35.421696+0000","last_became_active":"2024-09-21T08:31:57.659208+0000","last_became_peered":"2024-09-21T08:31:57.659208+0000","last_unstale":"2024-09-21T08:32:35.421696+0000","last_undegraded":"2024-09-21T08:32:35.421696+0000","last_fullsized":"2024-09-21T08:32:35.421696+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'2","last_scrub_stamp":"2024-09-21T08:31:09.109498+0000","last_deep_scrub":"475'2","last_deep_scrub_stamp":"2024-09-21T08:31:09.109498+0000","last_clean_scrub_stamp":"2024-09-21T08:31:09.109498+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:37.452049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"463'17","reported_seq":648,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419438+0000","last_change":"2024-09-21T08:32:08.235500+0000","last_active":"2024-09-21T08:32:35.419438+0000","last_peered":"2024-09-21T08:32:35.419438+0000","last_clean":"2024-09-21T08:32:35.419438+0000","last_became_active":"2024-09-21T08:30:56.630142+0000","last_became_peered":"2024-09-21T08:30:56.630142+0000","last_unstale":"2024-09-21T08:32:35.419438+0000","last_undegraded":"2024-09-21T08:32:35.419438+0000","last_fullsized":"2024-09-21T08:32:35.419438+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'17","last_scrub_stamp":"2024-09-21T08:32:08.235411+0000","last_deep_scrub":"263'14","last_deep_scrub_stamp":"2024-09-21T08:27:36.998538+0000","last_clean_scrub_stamp":"2024-09-21T08:32:08.235411+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:19.679841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1017,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420049+0000","last_change":"2024-09-21T08:31:56.650241+0000","last_active":"2024-09-21T08:32:35.420049+0000","last_peered":"2024-09-21T08:32:35.420049+0000","last_clean":"2024-09-21T08:32:35.420049+0000","last_became_active":"2024-09-21T08:31:56.649958+0000","last_became_peered":"2024-09-21T08:31:56.649958+0000","last_unstale":"2024-09-21T08:32:35.420049+0000","last_undegraded":"2024-09-21T08:32:35.420049+0000","last_fullsized":"2024-09-21T08:32:35.420049+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.344185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.344185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"76'129","reported_seq":1296,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418971+0000","last_change":"2024-09-21T08:31:57.661231+0000","last_active":"2024-09-21T08:32:35.418971+0000","last_peered":"2024-09-21T08:32:35.418971+0000","last_clean":"2024-09-21T08:32:35.418971+0000","last_became_active":"2024-09-21T08:31:57.659273+0000","last_became_peered":"2024-09-21T08:31:57.659273+0000","last_unstale":"2024-09-21T08:32:35.418971+0000","last_undegraded":"2024-09-21T08:32:35.418971+0000","last_fullsized":"2024-09-21T08:32:35.418971+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:05.800069+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.800069+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"179'43","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.803540+0000","last_change":"2024-09-21T08:32:35.750653+0000","last_active":"2024-09-21T08:32:35.750653+0000","last_peered":"2024-09-21T08:32:33.803540+0000","last_clean":"2024-09-21T08:32:33.803540+0000","last_became_active":"2024-09-21T08:31:57.643127+0000","last_became_peered":"2024-09-21T08:31:57.643127+0000","last_unstale":"2024-09-21T08:32:33.803540+0000","last_undegraded":"2024-09-21T08:32:33.803540+0000","last_fullsized":"2024-09-21T08:32:33.803540+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:50.158232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.158232+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.99984979799999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"1"},{"start":"de","length":"1"}]},{"pgid":"4.2","version":"73'10","reported_seq":626,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.804146+0000","last_change":"2024-09-21T08:32:35.423874+0000","last_active":"2024-09-21T08:32:35.423874+0000","last_peered":"2024-09-21T08:32:33.804146+0000","last_clean":"2024-09-21T08:32:33.804146+0000","last_became_active":"2024-09-21T08:31:40.666699+0000","last_became_peered":"2024-09-21T08:31:40.666699+0000","last_unstale":"2024-09-21T08:32:33.804146+0000","last_undegraded":"2024-09-21T08:32:33.804146+0000","last_fullsized":"2024-09-21T08:32:33.804146+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:29:58.300806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.300806+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":1065,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421600+0000","last_change":"2024-09-21T08:31:56.653594+0000","last_active":"2024-09-21T08:32:35.421600+0000","last_peered":"2024-09-21T08:32:35.421600+0000","last_clean":"2024-09-21T08:32:35.421600+0000","last_became_active":"2024-09-21T08:31:56.653060+0000","last_became_peered":"2024-09-21T08:31:56.653060+0000","last_unstale":"2024-09-21T08:32:35.421600+0000","last_undegraded":"2024-09-21T08:32:35.421600+0000","last_fullsized":"2024-09-21T08:32:35.421600+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:16.003225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.003225+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"523'17","reported_seq":851,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.425149+0000","last_change":"2024-09-21T08:31:56.650297+0000","last_active":"2024-09-21T08:32:35.425149+0000","last_peered":"2024-09-21T08:32:35.425149+0000","last_clean":"2024-09-21T08:32:35.425149+0000","last_became_active":"2024-09-21T08:31:56.649917+0000","last_became_peered":"2024-09-21T08:31:56.649917+0000","last_unstale":"2024-09-21T08:32:35.425149+0000","last_undegraded":"2024-09-21T08:32:35.425149+0000","last_fullsized":"2024-09-21T08:32:35.425149+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'13","last_scrub_stamp":"2024-09-21T08:27:27.874952+0000","last_deep_scrub":"176'11","last_deep_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_clean_scrub_stamp":"2024-09-21T08:27:27.874952+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":862,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428543+0000","last_change":"2024-09-21T08:32:08.907632+0000","last_active":"2024-09-21T08:32:35.428543+0000","last_peered":"2024-09-21T08:32:35.428543+0000","last_clean":"2024-09-21T08:32:35.428543+0000","last_became_active":"2024-09-21T08:28:37.565624+0000","last_became_peered":"2024-09-21T08:28:37.565624+0000","last_unstale":"2024-09-21T08:32:35.428543+0000","last_undegraded":"2024-09-21T08:32:35.428543+0000","last_fullsized":"2024-09-21T08:32:35.428543+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:08.907565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:32:08.907565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:13.789820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"76'129","reported_seq":995,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788460+0000","last_change":"2024-09-21T08:31:57.646620+0000","last_active":"2024-09-21T08:32:34.788460+0000","last_peered":"2024-09-21T08:32:34.788460+0000","last_clean":"2024-09-21T08:32:34.788460+0000","last_became_active":"2024-09-21T08:31:57.646069+0000","last_became_peered":"2024-09-21T08:31:57.646069+0000","last_unstale":"2024-09-21T08:32:34.788460+0000","last_undegraded":"2024-09-21T08:32:34.788460+0000","last_fullsized":"2024-09-21T08:32:34.788460+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:06.980429+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:06.980429+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"553'255","reported_seq":2701,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.429257+0000","last_change":"2024-09-21T08:32:33.742313+0000","last_active":"2024-09-21T08:32:35.429257+0000","last_peered":"2024-09-21T08:32:35.429257+0000","last_clean":"2024-09-21T08:32:35.429257+0000","last_became_active":"2024-09-21T08:28:07.549600+0000","last_became_peered":"2024-09-21T08:28:07.549600+0000","last_unstale":"2024-09-21T08:32:35.429257+0000","last_undegraded":"2024-09-21T08:32:35.429257+0000","last_fullsized":"2024-09-21T08:32:35.429257+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'166","last_scrub_stamp":"2024-09-21T08:29:32.253921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.253921+0000","objects_scrubbed":3,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010877980000000001,"stat_sum":{"num_bytes":10011867,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":95330,"num_write":34,"num_write_kb":12048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1484609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"6"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"4.3","version":"73'19","reported_seq":582,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.425358+0000","last_change":"2024-09-21T08:31:56.668001+0000","last_active":"2024-09-21T08:32:35.425358+0000","last_peered":"2024-09-21T08:32:35.425358+0000","last_clean":"2024-09-21T08:32:35.425358+0000","last_became_active":"2024-09-21T08:31:56.667700+0000","last_became_peered":"2024-09-21T08:31:56.667700+0000","last_unstale":"2024-09-21T08:32:35.425358+0000","last_undegraded":"2024-09-21T08:32:35.425358+0000","last_fullsized":"2024-09-21T08:32:35.425358+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:27:55.260127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:55.260127+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":561,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421551+0000","last_change":"2024-09-21T08:30:46.074689+0000","last_active":"2024-09-21T08:32:35.421551+0000","last_peered":"2024-09-21T08:32:35.421551+0000","last_clean":"2024-09-21T08:32:35.421551+0000","last_became_active":"2024-09-21T08:28:04.723998+0000","last_became_peered":"2024-09-21T08:28:04.723998+0000","last_unstale":"2024-09-21T08:32:35.421551+0000","last_undegraded":"2024-09-21T08:32:35.421551+0000","last_fullsized":"2024-09-21T08:32:35.421551+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:46.074613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:30:46.074613+0000","last_clean_scrub_stamp":"2024-09-21T08:30:46.074613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":520,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428830+0000","last_change":"2024-09-21T08:32:31.942095+0000","last_active":"2024-09-21T08:32:35.428830+0000","last_peered":"2024-09-21T08:32:35.428830+0000","last_clean":"2024-09-21T08:32:35.428830+0000","last_became_active":"2024-09-21T08:31:56.653084+0000","last_became_peered":"2024-09-21T08:31:56.653084+0000","last_unstale":"2024-09-21T08:32:35.428830+0000","last_undegraded":"2024-09-21T08:32:35.428830+0000","last_fullsized":"2024-09-21T08:32:35.428830+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:31.941959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:32:31.941959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:42.430646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":562,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421140+0000","last_change":"2024-09-21T08:31:56.643494+0000","last_active":"2024-09-21T08:32:35.421140+0000","last_peered":"2024-09-21T08:32:35.421140+0000","last_clean":"2024-09-21T08:32:35.421140+0000","last_became_active":"2024-09-21T08:31:56.642987+0000","last_became_peered":"2024-09-21T08:31:56.642987+0000","last_unstale":"2024-09-21T08:32:35.421140+0000","last_undegraded":"2024-09-21T08:32:35.421140+0000","last_fullsized":"2024-09-21T08:32:35.421140+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:46.263984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.263984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'129","reported_seq":1289,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421780+0000","last_change":"2024-09-21T08:31:06.754191+0000","last_active":"2024-09-21T08:32:35.421780+0000","last_peered":"2024-09-21T08:32:35.421780+0000","last_clean":"2024-09-21T08:32:35.421780+0000","last_became_active":"2024-09-21T08:30:59.600493+0000","last_became_peered":"2024-09-21T08:30:59.600493+0000","last_unstale":"2024-09-21T08:32:35.421780+0000","last_undegraded":"2024-09-21T08:32:35.421780+0000","last_fullsized":"2024-09-21T08:32:35.421780+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:06.754121+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:06.754121+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"558'239","reported_seq":2645,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.422009+0000","last_change":"2024-09-21T08:32:33.770566+0000","last_active":"2024-09-21T08:32:35.422009+0000","last_peered":"2024-09-21T08:32:35.422009+0000","last_clean":"2024-09-21T08:32:35.422009+0000","last_became_active":"2024-09-21T08:31:57.660183+0000","last_became_peered":"2024-09-21T08:31:57.660183+0000","last_unstale":"2024-09-21T08:32:35.422009+0000","last_undegraded":"2024-09-21T08:32:35.422009+0000","last_fullsized":"2024-09-21T08:32:35.422009+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.000138626,"stat_sum":{"num_bytes":19575274,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":69410,"num_write":65,"num_write_kb":29412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":48166099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"1"}]},{"pgid":"4.4","version":"523'36","reported_seq":641,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421434+0000","last_change":"2024-09-21T08:31:56.651060+0000","last_active":"2024-09-21T08:32:35.421434+0000","last_peered":"2024-09-21T08:32:35.421434+0000","last_clean":"2024-09-21T08:32:35.421434+0000","last_became_active":"2024-09-21T08:31:56.650428+0000","last_became_peered":"2024-09-21T08:31:56.650428+0000","last_unstale":"2024-09-21T08:32:35.421434+0000","last_undegraded":"2024-09-21T08:32:35.421434+0000","last_fullsized":"2024-09-21T08:32:35.421434+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'32","last_scrub_stamp":"2024-09-21T08:27:11.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:11.407384+0000","objects_scrubbed":10,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788521+0000","last_change":"2024-09-21T08:30:56.627449+0000","last_active":"2024-09-21T08:32:34.788521+0000","last_peered":"2024-09-21T08:32:34.788521+0000","last_clean":"2024-09-21T08:32:34.788521+0000","last_became_active":"2024-09-21T08:30:56.627283+0000","last_became_peered":"2024-09-21T08:30:56.627283+0000","last_unstale":"2024-09-21T08:32:34.788521+0000","last_undegraded":"2024-09-21T08:32:34.788521+0000","last_fullsized":"2024-09-21T08:32:34.788521+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:14.370256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.370256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":1089,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428449+0000","last_change":"2024-09-21T08:31:56.672465+0000","last_active":"2024-09-21T08:32:35.428449+0000","last_peered":"2024-09-21T08:32:35.428449+0000","last_clean":"2024-09-21T08:32:35.428449+0000","last_became_active":"2024-09-21T08:31:56.672094+0000","last_became_peered":"2024-09-21T08:31:56.672094+0000","last_unstale":"2024-09-21T08:32:35.428449+0000","last_undegraded":"2024-09-21T08:32:35.428449+0000","last_fullsized":"2024-09-21T08:32:35.428449+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1017,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.805671+0000","last_change":"2024-09-21T08:32:35.423198+0000","last_active":"2024-09-21T08:32:35.423198+0000","last_peered":"2024-09-21T08:32:33.805671+0000","last_clean":"2024-09-21T08:32:33.805671+0000","last_became_active":"2024-09-21T08:31:57.642204+0000","last_became_peered":"2024-09-21T08:31:57.642204+0000","last_unstale":"2024-09-21T08:32:33.805671+0000","last_undegraded":"2024-09-21T08:32:33.805671+0000","last_fullsized":"2024-09-21T08:32:33.805671+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:37.057876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:37.057876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788986+0000","last_change":"2024-09-21T08:31:57.643047+0000","last_active":"2024-09-21T08:32:34.788986+0000","last_peered":"2024-09-21T08:32:34.788986+0000","last_clean":"2024-09-21T08:32:34.788986+0000","last_became_active":"2024-09-21T08:31:57.642670+0000","last_became_peered":"2024-09-21T08:31:57.642670+0000","last_unstale":"2024-09-21T08:32:34.788986+0000","last_undegraded":"2024-09-21T08:32:34.788986+0000","last_fullsized":"2024-09-21T08:32:34.788986+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.844729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.844729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1,"reported_epoch":561,"state":"unknown","last_fresh":"2024-09-21T08:32:35.421255+0000","last_change":"2024-09-21T08:32:35.750693+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T08:32:35.421255+0000","last_undegraded":"2024-09-21T08:32:35.421255+0000","last_fullsized":"2024-09-21T08:32:35.421255+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":555,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:54.732571+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:54.732571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":561,"state":"unknown","last_fresh":"2024-09-21T08:32:35.424821+0000","last_change":"2024-09-21T08:32:35.750744+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T08:32:35.424821+0000","last_undegraded":"2024-09-21T08:32:35.424821+0000","last_fullsized":"2024-09-21T08:32:35.424821+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'161","last_scrub_stamp":"2024-09-21T08:31:09.054511+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:31:09.054511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":600,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423140+0000","last_change":"2024-09-21T08:31:56.660682+0000","last_active":"2024-09-21T08:32:35.423140+0000","last_peered":"2024-09-21T08:32:35.423140+0000","last_clean":"2024-09-21T08:32:35.423140+0000","last_became_active":"2024-09-21T08:31:56.659742+0000","last_became_peered":"2024-09-21T08:31:56.659742+0000","last_unstale":"2024-09-21T08:32:35.423140+0000","last_undegraded":"2024-09-21T08:32:35.423140+0000","last_fullsized":"2024-09-21T08:32:35.423140+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-09-21T08:28:16.386950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.386950+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":567,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420131+0000","last_change":"2024-09-21T08:29:56.368212+0000","last_active":"2024-09-21T08:32:35.420131+0000","last_peered":"2024-09-21T08:32:35.420131+0000","last_clean":"2024-09-21T08:32:35.420131+0000","last_became_active":"2024-09-21T08:28:29.629320+0000","last_became_peered":"2024-09-21T08:28:29.629320+0000","last_unstale":"2024-09-21T08:32:35.420131+0000","last_undegraded":"2024-09-21T08:32:35.420131+0000","last_fullsized":"2024-09-21T08:32:35.420131+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:56.368144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:29:56.368144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":1034,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.425791+0000","last_change":"2024-09-21T08:30:56.650473+0000","last_active":"2024-09-21T08:32:35.425791+0000","last_peered":"2024-09-21T08:32:35.425791+0000","last_clean":"2024-09-21T08:32:35.425791+0000","last_became_active":"2024-09-21T08:30:56.629748+0000","last_became_peered":"2024-09-21T08:30:56.629748+0000","last_unstale":"2024-09-21T08:32:35.425791+0000","last_undegraded":"2024-09-21T08:32:35.425791+0000","last_fullsized":"2024-09-21T08:32:35.425791+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:59.262463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:27:59.262463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1018,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423407+0000","last_change":"2024-09-21T08:31:56.652762+0000","last_active":"2024-09-21T08:32:35.423407+0000","last_peered":"2024-09-21T08:32:35.423407+0000","last_clean":"2024-09-21T08:32:35.423407+0000","last_became_active":"2024-09-21T08:31:56.652434+0000","last_became_peered":"2024-09-21T08:31:56.652434+0000","last_unstale":"2024-09-21T08:32:35.423407+0000","last_undegraded":"2024-09-21T08:32:35.423407+0000","last_fullsized":"2024-09-21T08:32:35.423407+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.374076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.374076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":916,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420568+0000","last_change":"2024-09-21T08:31:56.653969+0000","last_active":"2024-09-21T08:32:35.420568+0000","last_peered":"2024-09-21T08:32:35.420568+0000","last_clean":"2024-09-21T08:32:35.420568+0000","last_became_active":"2024-09-21T08:31:56.653507+0000","last_became_peered":"2024-09-21T08:31:56.653507+0000","last_unstale":"2024-09-21T08:32:35.420568+0000","last_undegraded":"2024-09-21T08:32:35.420568+0000","last_fullsized":"2024-09-21T08:32:35.420568+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:08.831386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:08.831386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"76'129","reported_seq":1298,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423562+0000","last_change":"2024-09-21T08:31:05.106966+0000","last_active":"2024-09-21T08:32:35.423562+0000","last_peered":"2024-09-21T08:32:35.423562+0000","last_clean":"2024-09-21T08:32:35.423562+0000","last_became_active":"2024-09-21T08:30:59.605942+0000","last_became_peered":"2024-09-21T08:30:59.605942+0000","last_unstale":"2024-09-21T08:32:35.423562+0000","last_undegraded":"2024-09-21T08:32:35.423562+0000","last_fullsized":"2024-09-21T08:32:35.423562+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:05.106891+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.106891+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"462'307","reported_seq":2310,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.420777+0000","last_change":"2024-09-21T08:32:33.768029+0000","last_active":"2024-09-21T08:32:35.420777+0000","last_peered":"2024-09-21T08:32:35.420777+0000","last_clean":"2024-09-21T08:32:35.420777+0000","last_became_active":"2024-09-21T08:31:56.651630+0000","last_became_peered":"2024-09-21T08:31:56.651630+0000","last_unstale":"2024-09-21T08:32:35.420777+0000","last_undegraded":"2024-09-21T08:32:35.420777+0000","last_fullsized":"2024-09-21T08:32:35.420777+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'245","last_scrub_stamp":"2024-09-21T08:26:30.864040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.864040+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":92,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99868668299999996,"stat_sum":{"num_bytes":4186220,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":25509,"num_write":17,"num_write_kb":5036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":926989,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"c"}]},{"pgid":"4.6","version":"73'12","reported_seq":1027,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421756+0000","last_change":"2024-09-21T08:31:56.653976+0000","last_active":"2024-09-21T08:32:35.421756+0000","last_peered":"2024-09-21T08:32:35.421756+0000","last_clean":"2024-09-21T08:32:35.421756+0000","last_became_active":"2024-09-21T08:31:56.653464+0000","last_became_peered":"2024-09-21T08:31:56.653464+0000","last_unstale":"2024-09-21T08:32:35.421756+0000","last_undegraded":"2024-09-21T08:32:35.421756+0000","last_fullsized":"2024-09-21T08:32:35.421756+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:26:07.384640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:07.384640+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:31:57.618681+0000","last_change":"2024-09-21T08:31:56.619363+0000","last_active":"2024-09-21T08:31:55.638838+0000","last_peered":"2024-09-21T08:31:54.608441+0000","last_clean":"2024-09-21T08:31:54.608441+0000","last_became_active":"2024-09-21T08:31:41.652019+0000","last_became_peered":"2024-09-21T08:31:41.652019+0000","last_unstale":"2024-09-21T08:31:57.618681+0000","last_undegraded":"2024-09-21T08:31:57.618681+0000","last_fullsized":"2024-09-21T08:31:57.618681+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.917891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.917891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1590,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418699+0000","last_change":"2024-09-21T08:31:56.649841+0000","last_active":"2024-09-21T08:32:35.418699+0000","last_peered":"2024-09-21T08:32:35.418699+0000","last_clean":"2024-09-21T08:32:35.418699+0000","last_became_active":"2024-09-21T08:31:56.649459+0000","last_became_peered":"2024-09-21T08:31:56.649459+0000","last_unstale":"2024-09-21T08:32:35.418699+0000","last_undegraded":"2024-09-21T08:32:35.418699+0000","last_fullsized":"2024-09-21T08:32:35.418699+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":521,"num_read_kb":516,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1037,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423787+0000","last_change":"2024-09-21T08:31:55.771472+0000","last_active":"2024-09-21T08:32:35.423787+0000","last_peered":"2024-09-21T08:32:35.423787+0000","last_clean":"2024-09-21T08:32:35.423787+0000","last_became_active":"2024-09-21T08:30:56.626598+0000","last_became_peered":"2024-09-21T08:30:56.626598+0000","last_unstale":"2024-09-21T08:32:35.423787+0000","last_undegraded":"2024-09-21T08:32:35.423787+0000","last_fullsized":"2024-09-21T08:32:35.423787+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:55.771397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:55.771397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:22.016240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":567,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421516+0000","last_change":"2024-09-21T08:30:56.638995+0000","last_active":"2024-09-21T08:32:35.421516+0000","last_peered":"2024-09-21T08:32:35.421516+0000","last_clean":"2024-09-21T08:32:35.421516+0000","last_became_active":"2024-09-21T08:30:56.638712+0000","last_became_peered":"2024-09-21T08:30:56.638712+0000","last_unstale":"2024-09-21T08:32:35.421516+0000","last_undegraded":"2024-09-21T08:32:35.421516+0000","last_fullsized":"2024-09-21T08:32:35.421516+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:17.424515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:17.424515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"550'193","reported_seq":2044,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-21T08:32:33.803250+0000","last_change":"2024-09-21T08:32:33.803612+0000","last_active":"2024-09-21T08:32:33.803250+0000","last_peered":"2024-09-21T08:32:33.803250+0000","last_clean":"2024-09-21T08:32:33.803250+0000","last_became_active":"2024-09-21T08:31:56.655654+0000","last_became_peered":"2024-09-21T08:31:56.655654+0000","last_unstale":"2024-09-21T08:32:33.803250+0000","last_undegraded":"2024-09-21T08:32:33.803250+0000","last_fullsized":"2024-09-21T08:32:33.803250+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'181","last_scrub_stamp":"2024-09-21T08:32:09.635480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:32:09.635480+0000","objects_scrubbed":5,"log_size":193,"log_dups_size":0,"ondisk_log_size":193,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:15.432455+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0022849950000000002,"stat_sum":{"num_bytes":7204516,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"4.5","version":"524'26","reported_seq":253,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789137+0000","last_change":"2024-09-21T08:31:57.641141+0000","last_active":"2024-09-21T08:32:34.789137+0000","last_peered":"2024-09-21T08:32:34.789137+0000","last_clean":"2024-09-21T08:32:34.789137+0000","last_became_active":"2024-09-21T08:31:57.640192+0000","last_became_peered":"2024-09-21T08:31:57.640192+0000","last_unstale":"2024-09-21T08:32:34.789137+0000","last_undegraded":"2024-09-21T08:32:34.789137+0000","last_fullsized":"2024-09-21T08:32:34.789137+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-21T08:28:13.956509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.956509+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":926,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.429244+0000","last_change":"2024-09-21T08:30:56.629168+0000","last_active":"2024-09-21T08:32:35.429244+0000","last_peered":"2024-09-21T08:32:35.429244+0000","last_clean":"2024-09-21T08:32:35.429244+0000","last_became_active":"2024-09-21T08:30:56.628829+0000","last_became_peered":"2024-09-21T08:30:56.628829+0000","last_unstale":"2024-09-21T08:32:35.429244+0000","last_undegraded":"2024-09-21T08:32:35.429244+0000","last_fullsized":"2024-09-21T08:32:35.429244+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.045353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.045353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":1289,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418371+0000","last_change":"2024-09-21T08:31:57.164805+0000","last_active":"2024-09-21T08:32:35.418371+0000","last_peered":"2024-09-21T08:32:35.418371+0000","last_clean":"2024-09-21T08:32:35.418371+0000","last_became_active":"2024-09-21T08:31:56.634604+0000","last_became_peered":"2024-09-21T08:31:56.634604+0000","last_unstale":"2024-09-21T08:32:35.418371+0000","last_undegraded":"2024-09-21T08:32:35.418371+0000","last_fullsized":"2024-09-21T08:32:35.418371+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:57.164726+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:57.164726+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:21.079821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":495,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421123+0000","last_change":"2024-09-21T08:31:56.642880+0000","last_active":"2024-09-21T08:32:35.421123+0000","last_peered":"2024-09-21T08:32:35.421123+0000","last_clean":"2024-09-21T08:32:35.421123+0000","last_became_active":"2024-09-21T08:31:56.642484+0000","last_became_peered":"2024-09-21T08:31:56.642484+0000","last_unstale":"2024-09-21T08:32:35.421123+0000","last_undegraded":"2024-09-21T08:32:35.421123+0000","last_fullsized":"2024-09-21T08:32:35.421123+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1051,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423975+0000","last_change":"2024-09-21T08:31:31.838452+0000","last_active":"2024-09-21T08:32:35.423975+0000","last_peered":"2024-09-21T08:32:35.423975+0000","last_clean":"2024-09-21T08:32:35.423975+0000","last_became_active":"2024-09-21T08:30:56.627007+0000","last_became_peered":"2024-09-21T08:30:56.627007+0000","last_unstale":"2024-09-21T08:32:35.423975+0000","last_undegraded":"2024-09-21T08:32:35.423975+0000","last_fullsized":"2024-09-21T08:32:35.423975+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:31.838383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:31:31.838383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:55.776066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"558'169","reported_seq":1112,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.425103+0000","last_change":"2024-09-21T08:32:33.769666+0000","last_active":"2024-09-21T08:32:35.425103+0000","last_peered":"2024-09-21T08:32:35.425103+0000","last_clean":"2024-09-21T08:32:35.425103+0000","last_became_active":"2024-09-21T08:31:56.658567+0000","last_became_peered":"2024-09-21T08:31:56.658567+0000","last_unstale":"2024-09-21T08:32:35.425103+0000","last_undegraded":"2024-09-21T08:32:35.425103+0000","last_fullsized":"2024-09-21T08:32:35.425103+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'93","last_scrub_stamp":"2024-09-21T08:28:11.373358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:11.373358+0000","objects_scrubbed":3,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":98,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.99780659100000002,"stat_sum":{"num_bytes":10254749,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":23716,"num_write":27,"num_write_kb":7172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":25893549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"6"}]},{"pgid":"4.e","version":"73'11","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788902+0000","last_change":"2024-09-21T08:30:56.656178+0000","last_active":"2024-09-21T08:32:34.788902+0000","last_peered":"2024-09-21T08:32:34.788902+0000","last_clean":"2024-09-21T08:32:34.788902+0000","last_became_active":"2024-09-21T08:30:56.626093+0000","last_became_peered":"2024-09-21T08:30:56.626093+0000","last_unstale":"2024-09-21T08:32:34.788902+0000","last_undegraded":"2024-09-21T08:32:34.788902+0000","last_fullsized":"2024-09-21T08:32:34.788902+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:28:14.416729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.416729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788721+0000","last_change":"2024-09-21T08:31:57.659564+0000","last_active":"2024-09-21T08:32:34.788721+0000","last_peered":"2024-09-21T08:32:34.788721+0000","last_clean":"2024-09-21T08:32:34.788721+0000","last_became_active":"2024-09-21T08:31:57.658983+0000","last_became_peered":"2024-09-21T08:31:57.658983+0000","last_unstale":"2024-09-21T08:32:34.788721+0000","last_undegraded":"2024-09-21T08:32:34.788721+0000","last_fullsized":"2024-09-21T08:32:34.788721+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:28.347474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:28.347474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1051,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424069+0000","last_change":"2024-09-21T08:30:56.619286+0000","last_active":"2024-09-21T08:32:35.424069+0000","last_peered":"2024-09-21T08:32:35.424069+0000","last_clean":"2024-09-21T08:32:35.424069+0000","last_became_active":"2024-09-21T08:30:56.618484+0000","last_became_peered":"2024-09-21T08:30:56.618484+0000","last_unstale":"2024-09-21T08:32:35.424069+0000","last_undegraded":"2024-09-21T08:32:35.424069+0000","last_fullsized":"2024-09-21T08:32:35.424069+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:55.150058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:32.961381+0000","last_clean_scrub_stamp":"2024-09-21T08:29:55.150058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":905,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421306+0000","last_change":"2024-09-21T08:31:56.650787+0000","last_active":"2024-09-21T08:32:35.421306+0000","last_peered":"2024-09-21T08:32:35.421306+0000","last_clean":"2024-09-21T08:32:35.421306+0000","last_became_active":"2024-09-21T08:31:56.650460+0000","last_became_peered":"2024-09-21T08:31:56.650460+0000","last_unstale":"2024-09-21T08:32:35.421306+0000","last_undegraded":"2024-09-21T08:32:35.421306+0000","last_fullsized":"2024-09-21T08:32:35.421306+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.977956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"76'129","reported_seq":1294,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418682+0000","last_change":"2024-09-21T08:31:09.810905+0000","last_active":"2024-09-21T08:32:35.418682+0000","last_peered":"2024-09-21T08:32:35.418682+0000","last_clean":"2024-09-21T08:32:35.418682+0000","last_became_active":"2024-09-21T08:30:59.604289+0000","last_became_peered":"2024-09-21T08:30:59.604289+0000","last_unstale":"2024-09-21T08:32:35.418682+0000","last_undegraded":"2024-09-21T08:32:35.418682+0000","last_fullsized":"2024-09-21T08:32:35.418682+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:09.810830+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:09.810830+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"256'199","reported_seq":2572,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423004+0000","last_change":"2024-09-21T08:32:33.781203+0000","last_active":"2024-09-21T08:32:35.423004+0000","last_peered":"2024-09-21T08:32:35.423004+0000","last_clean":"2024-09-21T08:32:35.423004+0000","last_became_active":"2024-09-21T08:28:32.558440+0000","last_became_peered":"2024-09-21T08:28:32.558440+0000","last_unstale":"2024-09-21T08:32:35.423004+0000","last_undegraded":"2024-09-21T08:32:35.423004+0000","last_fullsized":"2024-09-21T08:32:35.423004+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'199","last_scrub_stamp":"2024-09-21T08:31:26.832551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:26.832551+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066135800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"4.d","version":"345'25","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.806029+0000","last_change":"2024-09-21T08:32:35.750762+0000","last_active":"2024-09-21T08:32:35.750762+0000","last_peered":"2024-09-21T08:32:33.806029+0000","last_clean":"2024-09-21T08:32:33.806029+0000","last_became_active":"2024-09-21T08:31:40.667724+0000","last_became_peered":"2024-09-21T08:31:40.667724+0000","last_unstale":"2024-09-21T08:32:33.806029+0000","last_undegraded":"2024-09-21T08:32:33.806029+0000","last_fullsized":"2024-09-21T08:32:33.806029+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'25","last_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_deep_scrub":"345'25","last_deep_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.241931+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789186+0000","last_change":"2024-09-21T08:30:56.631627+0000","last_active":"2024-09-21T08:32:34.789186+0000","last_peered":"2024-09-21T08:32:34.789186+0000","last_clean":"2024-09-21T08:32:34.789186+0000","last_became_active":"2024-09-21T08:30:56.630646+0000","last_became_peered":"2024-09-21T08:30:56.630646+0000","last_unstale":"2024-09-21T08:32:34.789186+0000","last_undegraded":"2024-09-21T08:32:34.789186+0000","last_fullsized":"2024-09-21T08:32:34.789186+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:30.909902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.909902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":580,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421334+0000","last_change":"2024-09-21T08:31:57.658547+0000","last_active":"2024-09-21T08:32:35.421334+0000","last_peered":"2024-09-21T08:32:35.421334+0000","last_clean":"2024-09-21T08:32:35.421334+0000","last_became_active":"2024-09-21T08:31:57.658222+0000","last_became_peered":"2024-09-21T08:31:57.658222+0000","last_unstale":"2024-09-21T08:32:35.421334+0000","last_undegraded":"2024-09-21T08:32:35.421334+0000","last_fullsized":"2024-09-21T08:32:35.421334+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:08.138148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:08.138148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789179+0000","last_change":"2024-09-21T08:30:56.617373+0000","last_active":"2024-09-21T08:32:34.789179+0000","last_peered":"2024-09-21T08:32:34.789179+0000","last_clean":"2024-09-21T08:32:34.789179+0000","last_became_active":"2024-09-21T08:30:56.617090+0000","last_became_peered":"2024-09-21T08:30:56.617090+0000","last_unstale":"2024-09-21T08:32:34.789179+0000","last_undegraded":"2024-09-21T08:32:34.789179+0000","last_fullsized":"2024-09-21T08:32:34.789179+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:00.276774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:00.276774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"76'129","reported_seq":989,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418204+0000","last_change":"2024-09-21T08:31:57.659835+0000","last_active":"2024-09-21T08:32:35.418204+0000","last_peered":"2024-09-21T08:32:35.418204+0000","last_clean":"2024-09-21T08:32:35.418204+0000","last_became_active":"2024-09-21T08:31:57.659334+0000","last_became_peered":"2024-09-21T08:31:57.659334+0000","last_unstale":"2024-09-21T08:32:35.418204+0000","last_undegraded":"2024-09-21T08:32:35.418204+0000","last_fullsized":"2024-09-21T08:32:35.418204+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:06.742324+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:06.742324+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"558'124","reported_seq":829,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788993+0000","last_change":"2024-09-21T08:32:33.784397+0000","last_active":"2024-09-21T08:32:34.788993+0000","last_peered":"2024-09-21T08:32:34.788993+0000","last_clean":"2024-09-21T08:32:34.788993+0000","last_became_active":"2024-09-21T08:31:57.647290+0000","last_became_peered":"2024-09-21T08:31:57.647290+0000","last_unstale":"2024-09-21T08:32:34.788993+0000","last_undegraded":"2024-09-21T08:32:34.788993+0000","last_fullsized":"2024-09-21T08:32:34.788993+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:28:10.372149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.372149+0000","objects_scrubbed":1,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074721300000000004,"stat_sum":{"num_bytes":7412283,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":30,"num_read_kb":28275,"num_write":45,"num_write_kb":17501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":30320741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"4.c","version":"73'10","reported_seq":231,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788257+0000","last_change":"2024-09-21T08:32:13.710827+0000","last_active":"2024-09-21T08:32:34.788257+0000","last_peered":"2024-09-21T08:32:34.788257+0000","last_clean":"2024-09-21T08:32:34.788257+0000","last_became_active":"2024-09-21T08:31:57.652431+0000","last_became_peered":"2024-09-21T08:31:57.652431+0000","last_unstale":"2024-09-21T08:32:34.788257+0000","last_undegraded":"2024-09-21T08:32:34.788257+0000","last_fullsized":"2024-09-21T08:32:34.788257+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:32:13.710699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:32:13.710699+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:19.044630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788230+0000","last_change":"2024-09-21T08:31:57.651745+0000","last_active":"2024-09-21T08:32:34.788230+0000","last_peered":"2024-09-21T08:32:34.788230+0000","last_clean":"2024-09-21T08:32:34.788230+0000","last_became_active":"2024-09-21T08:31:57.651452+0000","last_became_peered":"2024-09-21T08:31:57.651452+0000","last_unstale":"2024-09-21T08:32:34.788230+0000","last_undegraded":"2024-09-21T08:32:34.788230+0000","last_fullsized":"2024-09-21T08:32:34.788230+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:54.216528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:54.216528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"523'11","reported_seq":568,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421894+0000","last_change":"2024-09-21T08:31:56.669546+0000","last_active":"2024-09-21T08:32:35.421894+0000","last_peered":"2024-09-21T08:32:35.421894+0000","last_clean":"2024-09-21T08:32:35.421894+0000","last_became_active":"2024-09-21T08:31:56.668783+0000","last_became_peered":"2024-09-21T08:31:56.668783+0000","last_unstale":"2024-09-21T08:32:35.421894+0000","last_undegraded":"2024-09-21T08:32:35.421894+0000","last_fullsized":"2024-09-21T08:32:35.421894+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:27:51.312273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.312273+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788696+0000","last_change":"2024-09-21T08:30:56.627132+0000","last_active":"2024-09-21T08:32:34.788696+0000","last_peered":"2024-09-21T08:32:34.788696+0000","last_clean":"2024-09-21T08:32:34.788696+0000","last_became_active":"2024-09-21T08:30:56.626799+0000","last_became_peered":"2024-09-21T08:30:56.626799+0000","last_unstale":"2024-09-21T08:32:34.788696+0000","last_undegraded":"2024-09-21T08:32:34.788696+0000","last_fullsized":"2024-09-21T08:32:34.788696+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:40.458054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:40.458054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"472'386","reported_seq":2733,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789483+0000","last_change":"2024-09-21T08:32:33.782483+0000","last_active":"2024-09-21T08:32:34.789483+0000","last_peered":"2024-09-21T08:32:34.789483+0000","last_clean":"2024-09-21T08:32:34.789483+0000","last_became_active":"2024-09-21T08:31:57.644069+0000","last_became_peered":"2024-09-21T08:31:57.644069+0000","last_unstale":"2024-09-21T08:32:34.789483+0000","last_undegraded":"2024-09-21T08:32:34.789483+0000","last_fullsized":"2024-09-21T08:32:34.789483+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'386","last_scrub_stamp":"2024-09-21T08:31:08.026519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:08.026519+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055346200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"4.b","version":"73'9","reported_seq":1098,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421430+0000","last_change":"2024-09-21T08:31:25.139313+0000","last_active":"2024-09-21T08:32:35.421430+0000","last_peered":"2024-09-21T08:32:35.421430+0000","last_clean":"2024-09-21T08:32:35.421430+0000","last_became_active":"2024-09-21T08:30:56.626474+0000","last_became_peered":"2024-09-21T08:30:56.626474+0000","last_unstale":"2024-09-21T08:32:35.421430+0000","last_undegraded":"2024-09-21T08:32:35.421430+0000","last_fullsized":"2024-09-21T08:32:35.421430+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:31:25.139208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:31:25.139208+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:52.945547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1060,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419840+0000","last_change":"2024-09-21T08:28:59.403264+0000","last_active":"2024-09-21T08:32:35.419840+0000","last_peered":"2024-09-21T08:32:35.419840+0000","last_clean":"2024-09-21T08:32:35.419840+0000","last_became_active":"2024-09-21T08:28:29.609526+0000","last_became_peered":"2024-09-21T08:28:29.609526+0000","last_unstale":"2024-09-21T08:32:35.419840+0000","last_undegraded":"2024-09-21T08:32:35.419840+0000","last_fullsized":"2024-09-21T08:32:35.419840+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:59.403194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.403194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.803395+0000","last_change":"2024-09-21T08:32:35.750678+0000","last_active":"2024-09-21T08:32:35.750678+0000","last_peered":"2024-09-21T08:32:33.803395+0000","last_clean":"2024-09-21T08:32:33.803395+0000","last_became_active":"2024-09-21T08:31:57.658412+0000","last_became_peered":"2024-09-21T08:31:57.658412+0000","last_unstale":"2024-09-21T08:32:33.803395+0000","last_undegraded":"2024-09-21T08:32:33.803395+0000","last_fullsized":"2024-09-21T08:32:33.803395+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:51.177738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:51.177738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":518,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418936+0000","last_change":"2024-09-21T08:31:56.649066+0000","last_active":"2024-09-21T08:32:35.418936+0000","last_peered":"2024-09-21T08:32:35.418936+0000","last_clean":"2024-09-21T08:32:35.418936+0000","last_became_active":"2024-09-21T08:31:56.648780+0000","last_became_peered":"2024-09-21T08:31:56.648780+0000","last_unstale":"2024-09-21T08:32:35.418936+0000","last_undegraded":"2024-09-21T08:32:35.418936+0000","last_fullsized":"2024-09-21T08:32:35.418936+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:36.308161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:36.308161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"560'442","reported_seq":2940,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:35.426216+0000","last_change":"2024-09-21T08:32:35.426828+0000","last_active":"2024-09-21T08:32:35.426192+0000","last_peered":"2024-09-21T08:32:33.826749+0000","last_clean":"2024-09-21T08:32:33.826749+0000","last_became_active":"2024-09-21T08:31:57.658394+0000","last_became_peered":"2024-09-21T08:31:57.658394+0000","last_unstale":"2024-09-21T08:32:35.426216+0000","last_undegraded":"2024-09-21T08:32:35.426216+0000","last_fullsized":"2024-09-21T08:32:35.426216+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99822385599999996,"stat_sum":{"num_bytes":24497846,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":131,"num_read_kb":146962,"num_write":96,"num_write_kb":30729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":51049312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,7],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"24"}]},{"pgid":"4.a","version":"73'19","reported_seq":582,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.422746+0000","last_change":"2024-09-21T08:31:56.659733+0000","last_active":"2024-09-21T08:32:35.422746+0000","last_peered":"2024-09-21T08:32:35.422746+0000","last_clean":"2024-09-21T08:32:35.422746+0000","last_became_active":"2024-09-21T08:31:56.658689+0000","last_became_peered":"2024-09-21T08:31:56.658689+0000","last_unstale":"2024-09-21T08:32:35.422746+0000","last_undegraded":"2024-09-21T08:32:35.422746+0000","last_fullsized":"2024-09-21T08:32:35.422746+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":931,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420834+0000","last_change":"2024-09-21T08:31:57.665653+0000","last_active":"2024-09-21T08:32:35.420834+0000","last_peered":"2024-09-21T08:32:35.420834+0000","last_clean":"2024-09-21T08:32:35.420834+0000","last_became_active":"2024-09-21T08:31:57.665365+0000","last_became_peered":"2024-09-21T08:31:57.665365+0000","last_unstale":"2024-09-21T08:32:35.420834+0000","last_undegraded":"2024-09-21T08:32:35.420834+0000","last_fullsized":"2024-09-21T08:32:35.420834+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.901202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.901202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1024,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419991+0000","last_change":"2024-09-21T08:32:13.055288+0000","last_active":"2024-09-21T08:32:35.419991+0000","last_peered":"2024-09-21T08:32:35.419991+0000","last_clean":"2024-09-21T08:32:35.419991+0000","last_became_active":"2024-09-21T08:31:56.652406+0000","last_became_peered":"2024-09-21T08:31:56.652406+0000","last_unstale":"2024-09-21T08:32:35.419991+0000","last_undegraded":"2024-09-21T08:32:35.419991+0000","last_fullsized":"2024-09-21T08:32:35.419991+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:13.055198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:32:13.055198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:36.426414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":511,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418078+0000","last_change":"2024-09-21T08:31:56.638200+0000","last_active":"2024-09-21T08:32:35.418078+0000","last_peered":"2024-09-21T08:32:35.418078+0000","last_clean":"2024-09-21T08:32:35.418078+0000","last_became_active":"2024-09-21T08:31:56.637710+0000","last_became_peered":"2024-09-21T08:31:56.637710+0000","last_unstale":"2024-09-21T08:32:35.418078+0000","last_undegraded":"2024-09-21T08:32:35.418078+0000","last_fullsized":"2024-09-21T08:32:35.418078+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"558'138","reported_seq":1657,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.428699+0000","last_change":"2024-09-21T08:32:33.741917+0000","last_active":"2024-09-21T08:32:35.428699+0000","last_peered":"2024-09-21T08:32:35.428699+0000","last_clean":"2024-09-21T08:32:35.428699+0000","last_became_active":"2024-09-21T08:31:07.701495+0000","last_became_peered":"2024-09-21T08:31:07.701495+0000","last_unstale":"2024-09-21T08:32:35.428699+0000","last_undegraded":"2024-09-21T08:32:35.428699+0000","last_fullsized":"2024-09-21T08:32:35.428699+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'124","last_scrub_stamp":"2024-09-21T08:31:32.865091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:32.865091+0000","objects_scrubbed":3,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:36.640976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0096207959999999992,"stat_sum":{"num_bytes":9724792,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12895,"num_write":5,"num_write_kb":3606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"6"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"4.9","version":"73'12","reported_seq":231,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788622+0000","last_change":"2024-09-21T08:31:57.659339+0000","last_active":"2024-09-21T08:32:34.788622+0000","last_peered":"2024-09-21T08:32:34.788622+0000","last_clean":"2024-09-21T08:32:34.788622+0000","last_became_active":"2024-09-21T08:31:57.658547+0000","last_became_peered":"2024-09-21T08:31:57.658547+0000","last_unstale":"2024-09-21T08:32:34.788622+0000","last_undegraded":"2024-09-21T08:32:34.788622+0000","last_fullsized":"2024-09-21T08:32:34.788622+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:31:34.776208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:31:34.776208+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:59.616638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.805045+0000","last_change":"2024-09-21T08:32:35.750805+0000","last_active":"2024-09-21T08:32:35.750805+0000","last_peered":"2024-09-21T08:32:33.805045+0000","last_clean":"2024-09-21T08:32:33.805045+0000","last_became_active":"2024-09-21T08:31:57.642317+0000","last_became_peered":"2024-09-21T08:31:57.642317+0000","last_unstale":"2024-09-21T08:32:33.805045+0000","last_undegraded":"2024-09-21T08:32:33.805045+0000","last_fullsized":"2024-09-21T08:32:33.805045+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:53.150608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:53.150608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1023,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418756+0000","last_change":"2024-09-21T08:29:59.267735+0000","last_active":"2024-09-21T08:32:35.418756+0000","last_peered":"2024-09-21T08:32:35.418756+0000","last_clean":"2024-09-21T08:32:35.418756+0000","last_became_active":"2024-09-21T08:28:37.566192+0000","last_became_peered":"2024-09-21T08:28:37.566192+0000","last_unstale":"2024-09-21T08:32:35.418756+0000","last_undegraded":"2024-09-21T08:32:35.418756+0000","last_fullsized":"2024-09-21T08:32:35.418756+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:59.267668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:29:59.267668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":936,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.804262+0000","last_change":"2024-09-21T08:32:35.422188+0000","last_active":"2024-09-21T08:32:35.422188+0000","last_peered":"2024-09-21T08:32:33.804262+0000","last_clean":"2024-09-21T08:32:33.804262+0000","last_became_active":"2024-09-21T08:31:57.641745+0000","last_became_peered":"2024-09-21T08:31:57.641745+0000","last_unstale":"2024-09-21T08:32:33.804262+0000","last_undegraded":"2024-09-21T08:32:33.804262+0000","last_fullsized":"2024-09-21T08:32:33.804262+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:29.202113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:31:29.202113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"559'849","reported_seq":3321,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.762554+0000","last_change":"2024-09-21T08:32:35.762621+0000","last_active":"2024-09-21T08:32:35.762554+0000","last_peered":"2024-09-21T08:32:35.762554+0000","last_clean":"2024-09-21T08:32:35.762554+0000","last_became_active":"2024-09-21T08:31:07.705607+0000","last_became_peered":"2024-09-21T08:31:07.705607+0000","last_unstale":"2024-09-21T08:32:35.762554+0000","last_undegraded":"2024-09-21T08:32:35.762554+0000","last_fullsized":"2024-09-21T08:32:35.762554+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'577","last_scrub_stamp":"2024-09-21T08:29:32.265609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.265609+0000","objects_scrubbed":10,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.33863036400000002,"stat_sum":{"num_bytes":26579394,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":104,"num_read_kb":104740,"num_write":69,"num_write_kb":28178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":2192635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"2"},{"start":"d5","length":"1"},{"start":"d8","length":"1"}]},{"pgid":"4.8","version":"73'15","reported_seq":1178,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428934+0000","last_change":"2024-09-21T08:31:57.662703+0000","last_active":"2024-09-21T08:32:35.428934+0000","last_peered":"2024-09-21T08:32:35.428934+0000","last_clean":"2024-09-21T08:32:35.428934+0000","last_became_active":"2024-09-21T08:31:57.662334+0000","last_became_peered":"2024-09-21T08:31:57.662334+0000","last_unstale":"2024-09-21T08:32:35.428934+0000","last_undegraded":"2024-09-21T08:32:35.428934+0000","last_fullsized":"2024-09-21T08:32:35.428934+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:31:53.711147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:31:53.711147+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:12.459852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"476'4","reported_seq":234,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788825+0000","last_change":"2024-09-21T08:31:57.673160+0000","last_active":"2024-09-21T08:32:34.788825+0000","last_peered":"2024-09-21T08:32:34.788825+0000","last_clean":"2024-09-21T08:32:34.788825+0000","last_became_active":"2024-09-21T08:31:57.652259+0000","last_became_peered":"2024-09-21T08:31:57.652259+0000","last_unstale":"2024-09-21T08:32:34.788825+0000","last_undegraded":"2024-09-21T08:32:34.788825+0000","last_fullsized":"2024-09-21T08:32:34.788825+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'4","last_scrub_stamp":"2024-09-21T08:31:10.077516+0000","last_deep_scrub":"476'4","last_deep_scrub_stamp":"2024-09-21T08:31:10.077516+0000","last_clean_scrub_stamp":"2024-09-21T08:31:10.077516+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"524'20","reported_seq":247,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788120+0000","last_change":"2024-09-21T08:31:57.659365+0000","last_active":"2024-09-21T08:32:34.788120+0000","last_peered":"2024-09-21T08:32:34.788120+0000","last_clean":"2024-09-21T08:32:34.788120+0000","last_became_active":"2024-09-21T08:31:57.658571+0000","last_became_peered":"2024-09-21T08:31:57.658571+0000","last_unstale":"2024-09-21T08:32:34.788120+0000","last_undegraded":"2024-09-21T08:32:34.788120+0000","last_fullsized":"2024-09-21T08:32:34.788120+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'18","last_scrub_stamp":"2024-09-21T08:31:51.875726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:51.875726+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:04.968651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1018,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428962+0000","last_change":"2024-09-21T08:30:50.793864+0000","last_active":"2024-09-21T08:32:35.428962+0000","last_peered":"2024-09-21T08:32:35.428962+0000","last_clean":"2024-09-21T08:32:35.428962+0000","last_became_active":"2024-09-21T08:28:29.619471+0000","last_became_peered":"2024-09-21T08:28:29.619471+0000","last_unstale":"2024-09-21T08:32:35.428962+0000","last_undegraded":"2024-09-21T08:32:35.428962+0000","last_fullsized":"2024-09-21T08:32:35.428962+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:50.793789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:50.793789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"180'50","reported_seq":2302,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423525+0000","last_change":"2024-09-21T08:32:33.781684+0000","last_active":"2024-09-21T08:32:35.423525+0000","last_peered":"2024-09-21T08:32:35.423525+0000","last_clean":"2024-09-21T08:32:35.423525+0000","last_became_active":"2024-09-21T08:31:56.644864+0000","last_became_peered":"2024-09-21T08:31:56.644864+0000","last_unstale":"2024-09-21T08:32:35.423525+0000","last_undegraded":"2024-09-21T08:32:35.423525+0000","last_fullsized":"2024-09-21T08:32:35.423525+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:32:09.976506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:32:09.976506+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:18.336292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096369100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":808,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428842+0000","last_change":"2024-09-21T08:29:39.177460+0000","last_active":"2024-09-21T08:32:35.428842+0000","last_peered":"2024-09-21T08:32:35.428842+0000","last_clean":"2024-09-21T08:32:35.428842+0000","last_became_active":"2024-09-21T08:28:29.610987+0000","last_became_peered":"2024-09-21T08:28:29.610987+0000","last_unstale":"2024-09-21T08:32:35.428842+0000","last_undegraded":"2024-09-21T08:32:35.428842+0000","last_fullsized":"2024-09-21T08:32:35.428842+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.177382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":966,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.422622+0000","last_change":"2024-09-21T08:31:56.660387+0000","last_active":"2024-09-21T08:32:35.422622+0000","last_peered":"2024-09-21T08:32:35.422622+0000","last_clean":"2024-09-21T08:32:35.422622+0000","last_became_active":"2024-09-21T08:31:56.659159+0000","last_became_peered":"2024-09-21T08:31:56.659159+0000","last_unstale":"2024-09-21T08:32:35.422622+0000","last_undegraded":"2024-09-21T08:32:35.422622+0000","last_fullsized":"2024-09-21T08:32:35.422622+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-09-21T08:27:30.806522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.806522+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1046,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421707+0000","last_change":"2024-09-21T08:31:56.652363+0000","last_active":"2024-09-21T08:32:35.421707+0000","last_peered":"2024-09-21T08:32:35.421707+0000","last_clean":"2024-09-21T08:32:35.421707+0000","last_became_active":"2024-09-21T08:31:56.650750+0000","last_became_peered":"2024-09-21T08:31:56.650750+0000","last_unstale":"2024-09-21T08:32:35.421707+0000","last_undegraded":"2024-09-21T08:32:35.421707+0000","last_fullsized":"2024-09-21T08:32:35.421707+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.795325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":605,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-21T08:32:33.800202+0000","last_change":"2024-09-21T08:31:57.660740+0000","last_active":"2024-09-21T08:32:33.800202+0000","last_peered":"2024-09-21T08:32:33.800202+0000","last_clean":"2024-09-21T08:32:33.800202+0000","last_became_active":"2024-09-21T08:31:57.660052+0000","last_became_peered":"2024-09-21T08:31:57.660052+0000","last_unstale":"2024-09-21T08:32:33.800202+0000","last_undegraded":"2024-09-21T08:32:33.800202+0000","last_fullsized":"2024-09-21T08:32:33.800202+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:10.851510+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.851510+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:34.599361+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"555'495","reported_seq":2705,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.425478+0000","last_change":"2024-09-21T08:32:33.767630+0000","last_active":"2024-09-21T08:32:35.425478+0000","last_peered":"2024-09-21T08:32:35.425478+0000","last_clean":"2024-09-21T08:32:35.425478+0000","last_became_active":"2024-09-21T08:31:56.669810+0000","last_became_peered":"2024-09-21T08:31:56.669810+0000","last_unstale":"2024-09-21T08:32:35.425478+0000","last_undegraded":"2024-09-21T08:32:35.425478+0000","last_fullsized":"2024-09-21T08:32:35.425478+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'262","last_scrub_stamp":"2024-09-21T08:26:29.875830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.875830+0000","objects_scrubbed":3,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":96,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.995530794,"stat_sum":{"num_bytes":47593958,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":243,"num_read_kb":285869,"num_write":119,"num_write_kb":50697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":45071078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"8"}]},{"pgid":"4.16","version":"73'9","reported_seq":528,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419933+0000","last_change":"2024-09-21T08:31:56.739241+0000","last_active":"2024-09-21T08:32:35.419933+0000","last_peered":"2024-09-21T08:32:35.419933+0000","last_clean":"2024-09-21T08:32:35.419933+0000","last_became_active":"2024-09-21T08:31:56.649999+0000","last_became_peered":"2024-09-21T08:31:56.649999+0000","last_unstale":"2024-09-21T08:32:35.419933+0000","last_undegraded":"2024-09-21T08:32:35.419933+0000","last_fullsized":"2024-09-21T08:32:35.419933+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:10.183035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.183035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788253+0000","last_change":"2024-09-21T08:31:57.641282+0000","last_active":"2024-09-21T08:32:34.788253+0000","last_peered":"2024-09-21T08:32:34.788253+0000","last_clean":"2024-09-21T08:32:34.788253+0000","last_became_active":"2024-09-21T08:31:57.640958+0000","last_became_peered":"2024-09-21T08:31:57.640958+0000","last_unstale":"2024-09-21T08:32:34.788253+0000","last_undegraded":"2024-09-21T08:32:34.788253+0000","last_fullsized":"2024-09-21T08:32:34.788253+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:53.898917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:53.898917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:00.608747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":567,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423230+0000","last_change":"2024-09-21T08:31:56.660694+0000","last_active":"2024-09-21T08:32:35.423230+0000","last_peered":"2024-09-21T08:32:35.423230+0000","last_clean":"2024-09-21T08:32:35.423230+0000","last_became_active":"2024-09-21T08:31:56.659891+0000","last_became_peered":"2024-09-21T08:31:56.659891+0000","last_unstale":"2024-09-21T08:32:35.423230+0000","last_undegraded":"2024-09-21T08:32:35.423230+0000","last_fullsized":"2024-09-21T08:32:35.423230+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:05.814753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.814753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1037,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424646+0000","last_change":"2024-09-21T08:31:57.642589+0000","last_active":"2024-09-21T08:32:35.424646+0000","last_peered":"2024-09-21T08:32:35.424646+0000","last_clean":"2024-09-21T08:32:35.424646+0000","last_became_active":"2024-09-21T08:31:57.642005+0000","last_became_peered":"2024-09-21T08:31:57.642005+0000","last_unstale":"2024-09-21T08:32:35.424646+0000","last_undegraded":"2024-09-21T08:32:35.424646+0000","last_fullsized":"2024-09-21T08:32:35.424646+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:07.730202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:31:07.730202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":561,"state":"unknown","last_fresh":"2024-09-21T08:32:35.423269+0000","last_change":"2024-09-21T08:32:35.750772+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T08:32:35.423269+0000","last_undegraded":"2024-09-21T08:32:35.423269+0000","last_fullsized":"2024-09-21T08:32:35.423269+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'102","last_scrub_stamp":"2024-09-21T08:26:33.255256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.255256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":1110,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420738+0000","last_change":"2024-09-21T08:31:56.671648+0000","last_active":"2024-09-21T08:32:35.420738+0000","last_peered":"2024-09-21T08:32:35.420738+0000","last_clean":"2024-09-21T08:32:35.420738+0000","last_became_active":"2024-09-21T08:31:56.671219+0000","last_became_peered":"2024-09-21T08:31:56.671219+0000","last_unstale":"2024-09-21T08:32:35.420738+0000","last_undegraded":"2024-09-21T08:32:35.420738+0000","last_fullsized":"2024-09-21T08:32:35.420738+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:28:14.535334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.535334+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":994,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420974+0000","last_change":"2024-09-21T08:31:56.659624+0000","last_active":"2024-09-21T08:32:35.420974+0000","last_peered":"2024-09-21T08:32:35.420974+0000","last_clean":"2024-09-21T08:32:35.420974+0000","last_became_active":"2024-09-21T08:31:56.658219+0000","last_became_peered":"2024-09-21T08:31:56.658219+0000","last_unstale":"2024-09-21T08:32:35.420974+0000","last_undegraded":"2024-09-21T08:32:35.420974+0000","last_fullsized":"2024-09-21T08:32:35.420974+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.361096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.361096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"523'17","reported_seq":1041,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421290+0000","last_change":"2024-09-21T08:31:56.650821+0000","last_active":"2024-09-21T08:32:35.421290+0000","last_peered":"2024-09-21T08:32:35.421290+0000","last_clean":"2024-09-21T08:32:35.421290+0000","last_became_active":"2024-09-21T08:31:56.649221+0000","last_became_peered":"2024-09-21T08:31:56.649221+0000","last_unstale":"2024-09-21T08:32:35.421290+0000","last_undegraded":"2024-09-21T08:32:35.421290+0000","last_fullsized":"2024-09-21T08:32:35.421290+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788918+0000","last_change":"2024-09-21T08:30:56.611027+0000","last_active":"2024-09-21T08:32:34.788918+0000","last_peered":"2024-09-21T08:32:34.788918+0000","last_clean":"2024-09-21T08:32:34.788918+0000","last_became_active":"2024-09-21T08:30:56.610628+0000","last_became_peered":"2024-09-21T08:30:56.610628+0000","last_unstale":"2024-09-21T08:32:34.788918+0000","last_undegraded":"2024-09-21T08:32:34.788918+0000","last_fullsized":"2024-09-21T08:32:34.788918+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.364388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.364388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"179'63","reported_seq":1477,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.747423+0000","last_change":"2024-09-21T08:32:35.747480+0000","last_active":"2024-09-21T08:32:35.747423+0000","last_peered":"2024-09-21T08:32:35.747423+0000","last_clean":"2024-09-21T08:32:35.747423+0000","last_became_active":"2024-09-21T08:31:56.649875+0000","last_became_peered":"2024-09-21T08:31:56.649875+0000","last_unstale":"2024-09-21T08:32:35.747423+0000","last_undegraded":"2024-09-21T08:32:35.747423+0000","last_fullsized":"2024-09-21T08:32:35.747423+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'63","last_scrub_stamp":"2024-09-21T08:28:13.511512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.511512+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":97,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.32372216599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"7"}]},{"pgid":"4.14","version":"104'15","reported_seq":581,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421416+0000","last_change":"2024-09-21T08:32:18.213538+0000","last_active":"2024-09-21T08:32:35.421416+0000","last_peered":"2024-09-21T08:32:35.421416+0000","last_clean":"2024-09-21T08:32:35.421416+0000","last_became_active":"2024-09-21T08:31:56.649656+0000","last_became_peered":"2024-09-21T08:31:56.649656+0000","last_unstale":"2024-09-21T08:32:35.421416+0000","last_undegraded":"2024-09-21T08:32:35.421416+0000","last_fullsized":"2024-09-21T08:32:35.421416+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:32:18.213452+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:32:18.213452+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:26.795743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":644,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-21T08:32:33.800668+0000","last_change":"2024-09-21T08:31:57.658189+0000","last_active":"2024-09-21T08:32:33.800668+0000","last_peered":"2024-09-21T08:32:33.800668+0000","last_clean":"2024-09-21T08:32:33.800668+0000","last_became_active":"2024-09-21T08:31:57.657724+0000","last_became_peered":"2024-09-21T08:31:57.657724+0000","last_unstale":"2024-09-21T08:32:33.800668+0000","last_undegraded":"2024-09-21T08:32:33.800668+0000","last_fullsized":"2024-09-21T08:32:33.800668+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:28:16.875544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.875544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:34.599423+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"524'19","reported_seq":1194,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.791647+0000","last_change":"2024-09-21T08:32:35.421672+0000","last_active":"2024-09-21T08:32:35.421672+0000","last_peered":"2024-09-21T08:32:33.791647+0000","last_clean":"2024-09-21T08:32:33.791647+0000","last_became_active":"2024-09-21T08:31:57.663936+0000","last_became_peered":"2024-09-21T08:31:57.663936+0000","last_unstale":"2024-09-21T08:32:33.791647+0000","last_undegraded":"2024-09-21T08:32:33.791647+0000","last_fullsized":"2024-09-21T08:32:33.791647+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788891+0000","last_change":"2024-09-21T08:32:18.676090+0000","last_active":"2024-09-21T08:32:34.788891+0000","last_peered":"2024-09-21T08:32:34.788891+0000","last_clean":"2024-09-21T08:32:34.788891+0000","last_became_active":"2024-09-21T08:31:57.641682+0000","last_became_peered":"2024-09-21T08:31:57.641682+0000","last_unstale":"2024-09-21T08:32:34.788891+0000","last_undegraded":"2024-09-21T08:32:34.788891+0000","last_fullsized":"2024-09-21T08:32:34.788891+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:18.676018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:32:18.676018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:45.305272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"559'180","reported_seq":1350,"reported_epoch":561,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:32:35.420452+0000","last_change":"2024-09-21T08:32:33.771528+0000","last_active":"2024-09-21T08:32:35.420452+0000","last_peered":"2024-09-21T08:32:35.420452+0000","last_clean":"2024-09-21T08:32:35.420452+0000","last_became_active":"2024-09-21T08:31:56.641759+0000","last_became_peered":"2024-09-21T08:31:56.641759+0000","last_unstale":"2024-09-21T08:32:35.420452+0000","last_undegraded":"2024-09-21T08:32:35.420452+0000","last_fullsized":"2024-09-21T08:32:35.420452+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'94","last_scrub_stamp":"2024-09-21T08:26:27.208234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.208234+0000","objects_scrubbed":1,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":95,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.99657815400000005,"stat_sum":{"num_bytes":13517584,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":13077,"num_write":47,"num_write_kb":18807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":22296475,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"9"}]},{"pgid":"4.13","version":"73'11","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788320+0000","last_change":"2024-09-21T08:30:56.661267+0000","last_active":"2024-09-21T08:32:34.788320+0000","last_peered":"2024-09-21T08:32:34.788320+0000","last_clean":"2024-09-21T08:32:34.788320+0000","last_became_active":"2024-09-21T08:30:56.626643+0000","last_became_peered":"2024-09-21T08:30:56.626643+0000","last_unstale":"2024-09-21T08:32:34.788320+0000","last_undegraded":"2024-09-21T08:32:34.788320+0000","last_fullsized":"2024-09-21T08:32:34.788320+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:30:25.153729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:25.153729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788347+0000","last_change":"2024-09-21T08:31:57.640495+0000","last_active":"2024-09-21T08:32:34.788347+0000","last_peered":"2024-09-21T08:32:34.788347+0000","last_clean":"2024-09-21T08:32:34.788347+0000","last_became_active":"2024-09-21T08:31:57.640151+0000","last_became_peered":"2024-09-21T08:31:57.640151+0000","last_unstale":"2024-09-21T08:32:34.788347+0000","last_undegraded":"2024-09-21T08:32:34.788347+0000","last_fullsized":"2024-09-21T08:32:34.788347+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:06.889127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:06.889127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":569,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424874+0000","last_change":"2024-09-21T08:32:16.284027+0000","last_active":"2024-09-21T08:32:35.424874+0000","last_peered":"2024-09-21T08:32:35.424874+0000","last_clean":"2024-09-21T08:32:35.424874+0000","last_became_active":"2024-09-21T08:31:56.659280+0000","last_became_peered":"2024-09-21T08:31:56.659280+0000","last_unstale":"2024-09-21T08:32:35.424874+0000","last_undegraded":"2024-09-21T08:32:35.424874+0000","last_fullsized":"2024-09-21T08:32:35.424874+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:16.283928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:32:16.283928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:38.645739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":764,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424687+0000","last_change":"2024-09-21T08:30:56.618534+0000","last_active":"2024-09-21T08:32:35.424687+0000","last_peered":"2024-09-21T08:32:35.424687+0000","last_clean":"2024-09-21T08:32:35.424687+0000","last_became_active":"2024-09-21T08:30:56.618134+0000","last_became_peered":"2024-09-21T08:30:56.618134+0000","last_unstale":"2024-09-21T08:32:35.424687+0000","last_undegraded":"2024-09-21T08:32:35.424687+0000","last_fullsized":"2024-09-21T08:32:35.424687+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:23.816669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:23.816669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"561'221","reported_seq":2142,"reported_epoch":561,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:32:35.762653+0000","last_change":"2024-09-21T08:32:35.762653+0000","last_active":"2024-09-21T08:32:35.762653+0000","last_peered":"2024-09-21T08:32:35.762653+0000","last_clean":"2024-09-21T08:32:35.762653+0000","last_became_active":"2024-09-21T08:31:56.657911+0000","last_became_peered":"2024-09-21T08:31:56.657911+0000","last_unstale":"2024-09-21T08:32:35.762653+0000","last_undegraded":"2024-09-21T08:32:35.762653+0000","last_fullsized":"2024-09-21T08:32:35.762653+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'44","last_scrub_stamp":"2024-09-21T08:26:27.882963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.882963+0000","objects_scrubbed":1,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":94,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.99548273600000003,"stat_sum":{"num_bytes":28511900,"num_objects":18,"num_object_clones":16,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":112,"num_read_kb":121994,"num_write":103,"num_write_kb":41259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":42498204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"a"}]},{"pgid":"4.12","version":"73'9","reported_seq":586,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423072+0000","last_change":"2024-09-21T08:30:42.016780+0000","last_active":"2024-09-21T08:32:35.423072+0000","last_peered":"2024-09-21T08:32:35.423072+0000","last_clean":"2024-09-21T08:32:35.423072+0000","last_became_active":"2024-09-21T08:28:29.636404+0000","last_became_peered":"2024-09-21T08:28:29.636404+0000","last_unstale":"2024-09-21T08:32:35.423072+0000","last_undegraded":"2024-09-21T08:32:35.423072+0000","last_fullsized":"2024-09-21T08:32:35.423072+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:42.016697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:42.016697+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789246+0000","last_change":"2024-09-21T08:32:16.654184+0000","last_active":"2024-09-21T08:32:34.789246+0000","last_peered":"2024-09-21T08:32:34.789246+0000","last_clean":"2024-09-21T08:32:34.789246+0000","last_became_active":"2024-09-21T08:31:57.639837+0000","last_became_peered":"2024-09-21T08:31:57.639837+0000","last_unstale":"2024-09-21T08:32:34.789246+0000","last_undegraded":"2024-09-21T08:32:34.789246+0000","last_fullsized":"2024-09-21T08:32:34.789246+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:16.654080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:32:16.654080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:27.615791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":881,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421170+0000","last_change":"2024-09-21T08:31:56.199080+0000","last_active":"2024-09-21T08:32:35.421170+0000","last_peered":"2024-09-21T08:32:35.421170+0000","last_clean":"2024-09-21T08:32:35.421170+0000","last_became_active":"2024-09-21T08:30:56.626197+0000","last_became_peered":"2024-09-21T08:30:56.626197+0000","last_unstale":"2024-09-21T08:32:35.421170+0000","last_undegraded":"2024-09-21T08:32:35.421170+0000","last_fullsized":"2024-09-21T08:32:35.421170+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:56.199004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:56.199004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:12.206778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":978,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.429043+0000","last_change":"2024-09-21T08:30:56.627439+0000","last_active":"2024-09-21T08:32:35.429043+0000","last_peered":"2024-09-21T08:32:35.429043+0000","last_clean":"2024-09-21T08:32:35.429043+0000","last_became_active":"2024-09-21T08:30:56.627130+0000","last_became_peered":"2024-09-21T08:30:56.627130+0000","last_unstale":"2024-09-21T08:32:35.429043+0000","last_undegraded":"2024-09-21T08:32:35.429043+0000","last_fullsized":"2024-09-21T08:32:35.429043+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:02.065082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:02.065082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.12","version":"560'458","reported_seq":2634,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.739394+0000","last_change":"2024-09-21T08:32:35.739434+0000","last_active":"2024-09-21T08:32:35.739394+0000","last_peered":"2024-09-21T08:32:35.739394+0000","last_clean":"2024-09-21T08:32:35.739394+0000","last_became_active":"2024-09-21T08:31:56.653984+0000","last_became_peered":"2024-09-21T08:31:56.653984+0000","last_unstale":"2024-09-21T08:32:35.739394+0000","last_undegraded":"2024-09-21T08:32:35.739394+0000","last_fullsized":"2024-09-21T08:32:35.739394+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'200","last_scrub_stamp":"2024-09-21T08:26:27.417028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.417028+0000","objects_scrubbed":3,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":99,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.31123925800000002,"stat_sum":{"num_bytes":49437079,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":208,"num_read_kb":239742,"num_write":127,"num_write_kb":61159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43192796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"5"}]},{"pgid":"4.11","version":"73'11","reported_seq":609,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424780+0000","last_change":"2024-09-21T08:32:16.895092+0000","last_active":"2024-09-21T08:32:35.424780+0000","last_peered":"2024-09-21T08:32:35.424780+0000","last_clean":"2024-09-21T08:32:35.424780+0000","last_became_active":"2024-09-21T08:31:56.649307+0000","last_became_peered":"2024-09-21T08:31:56.649307+0000","last_unstale":"2024-09-21T08:32:35.424780+0000","last_undegraded":"2024-09-21T08:32:35.424780+0000","last_fullsized":"2024-09-21T08:32:35.424780+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:32:16.894954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:32:16.894954+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:33.225663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1042,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421691+0000","last_change":"2024-09-21T08:31:56.649967+0000","last_active":"2024-09-21T08:32:35.421691+0000","last_peered":"2024-09-21T08:32:35.421691+0000","last_clean":"2024-09-21T08:32:35.421691+0000","last_became_active":"2024-09-21T08:31:56.649348+0000","last_became_peered":"2024-09-21T08:31:56.649348+0000","last_unstale":"2024-09-21T08:32:35.421691+0000","last_undegraded":"2024-09-21T08:32:35.421691+0000","last_fullsized":"2024-09-21T08:32:35.421691+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":633,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-21T08:32:33.798897+0000","last_change":"2024-09-21T08:31:57.659645+0000","last_active":"2024-09-21T08:32:33.798897+0000","last_peered":"2024-09-21T08:32:33.798897+0000","last_clean":"2024-09-21T08:32:33.798897+0000","last_became_active":"2024-09-21T08:31:57.659355+0000","last_became_peered":"2024-09-21T08:31:57.659355+0000","last_unstale":"2024-09-21T08:32:33.798897+0000","last_undegraded":"2024-09-21T08:32:33.798897+0000","last_fullsized":"2024-09-21T08:32:33.798897+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:34.599469+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:31:40.661693+0000","last_change":"2024-09-21T08:26:49.529910+0000","last_active":"2024-09-21T08:26:49.529910+0000","last_peered":"2024-09-21T08:26:47.516492+0000","last_clean":"2024-09-21T08:26:47.516492+0000","last_became_active":"2024-09-21T08:24:56.508323+0000","last_became_peered":"2024-09-21T08:24:56.508323+0000","last_unstale":"2024-09-21T08:31:40.661693+0000","last_undegraded":"2024-09-21T08:31:40.661693+0000","last_fullsized":"2024-09-21T08:31:40.661693+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:58.167184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.167184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.13","version":"532'77","reported_seq":2053,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.421313+0000","last_change":"2024-09-21T08:32:33.743365+0000","last_active":"2024-09-21T08:32:35.421313+0000","last_peered":"2024-09-21T08:32:35.421313+0000","last_clean":"2024-09-21T08:32:35.421313+0000","last_became_active":"2024-09-21T08:31:56.642897+0000","last_became_peered":"2024-09-21T08:31:56.642897+0000","last_unstale":"2024-09-21T08:32:35.421313+0000","last_undegraded":"2024-09-21T08:32:35.421313+0000","last_fullsized":"2024-09-21T08:32:35.421313+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'22","last_scrub_stamp":"2024-09-21T08:26:29.353795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.353795+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":98,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012744657,"stat_sum":{"num_bytes":12261866,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":73,"num_read_kb":105134,"num_write":36,"num_write_kb":17862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":15513137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"6"}]},{"pgid":"4.10","version":"73'4","reported_seq":578,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419495+0000","last_change":"2024-09-21T08:30:56.659256+0000","last_active":"2024-09-21T08:32:35.419495+0000","last_peered":"2024-09-21T08:32:35.419495+0000","last_clean":"2024-09-21T08:32:35.419495+0000","last_became_active":"2024-09-21T08:30:56.638177+0000","last_became_peered":"2024-09-21T08:30:56.638177+0000","last_unstale":"2024-09-21T08:32:35.419495+0000","last_undegraded":"2024-09-21T08:32:35.419495+0000","last_fullsized":"2024-09-21T08:32:35.419495+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-09-21T08:29:39.416397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.416397+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1004,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423614+0000","last_change":"2024-09-21T08:30:24.826266+0000","last_active":"2024-09-21T08:32:35.423614+0000","last_peered":"2024-09-21T08:32:35.423614+0000","last_clean":"2024-09-21T08:32:35.423614+0000","last_became_active":"2024-09-21T08:28:29.611592+0000","last_became_peered":"2024-09-21T08:28:29.611592+0000","last_unstale":"2024-09-21T08:32:35.423614+0000","last_undegraded":"2024-09-21T08:32:35.423614+0000","last_fullsized":"2024-09-21T08:32:35.423614+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:24.826198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:30:24.826198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.803190+0000","last_change":"2024-09-21T08:32:35.750646+0000","last_active":"2024-09-21T08:32:35.750646+0000","last_peered":"2024-09-21T08:32:33.803190+0000","last_clean":"2024-09-21T08:32:33.803190+0000","last_became_active":"2024-09-21T08:31:57.644199+0000","last_became_peered":"2024-09-21T08:31:57.644199+0000","last_unstale":"2024-09-21T08:32:33.803190+0000","last_undegraded":"2024-09-21T08:32:33.803190+0000","last_fullsized":"2024-09-21T08:32:33.803190+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:27.881562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:27.881562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789099+0000","last_change":"2024-09-21T08:32:14.667710+0000","last_active":"2024-09-21T08:32:34.789099+0000","last_peered":"2024-09-21T08:32:34.789099+0000","last_clean":"2024-09-21T08:32:34.789099+0000","last_became_active":"2024-09-21T08:31:57.648003+0000","last_became_peered":"2024-09-21T08:31:57.648003+0000","last_unstale":"2024-09-21T08:32:34.789099+0000","last_undegraded":"2024-09-21T08:32:34.789099+0000","last_fullsized":"2024-09-21T08:32:34.789099+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:14.667636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:32:14.667636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:43.990732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"555'235","reported_seq":2303,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.739896+0000","last_change":"2024-09-21T08:32:35.739896+0000","last_active":"2024-09-21T08:32:35.739896+0000","last_peered":"2024-09-21T08:32:35.739896+0000","last_clean":"2024-09-21T08:32:35.739896+0000","last_became_active":"2024-09-21T08:31:56.647728+0000","last_became_peered":"2024-09-21T08:31:56.647728+0000","last_unstale":"2024-09-21T08:32:35.739896+0000","last_undegraded":"2024-09-21T08:32:35.739896+0000","last_fullsized":"2024-09-21T08:32:35.739896+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'167","last_scrub_stamp":"2024-09-21T08:27:49.143452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.143452+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":98,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.32011975999999998,"stat_sum":{"num_bytes":21064649,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":85,"num_read_kb":108670,"num_write":39,"num_write_kb":19543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":13106064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"6"}]},{"pgid":"6.1d","version":"0'0","reported_seq":566,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420394+0000","last_change":"2024-09-21T08:30:56.627685+0000","last_active":"2024-09-21T08:32:35.420394+0000","last_peered":"2024-09-21T08:32:35.420394+0000","last_clean":"2024-09-21T08:32:35.420394+0000","last_became_active":"2024-09-21T08:30:56.627251+0000","last_became_peered":"2024-09-21T08:30:56.627251+0000","last_unstale":"2024-09-21T08:32:35.420394+0000","last_undegraded":"2024-09-21T08:32:35.420394+0000","last_fullsized":"2024-09-21T08:32:35.420394+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:28.069450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:28.069450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":882,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418186+0000","last_change":"2024-09-21T08:31:56.638174+0000","last_active":"2024-09-21T08:32:35.418186+0000","last_peered":"2024-09-21T08:32:35.418186+0000","last_clean":"2024-09-21T08:32:35.418186+0000","last_became_active":"2024-09-21T08:31:56.637565+0000","last_became_peered":"2024-09-21T08:31:56.637565+0000","last_unstale":"2024-09-21T08:32:35.418186+0000","last_undegraded":"2024-09-21T08:32:35.418186+0000","last_fullsized":"2024-09-21T08:32:35.418186+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"541'16","reported_seq":1090,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423883+0000","last_change":"2024-09-21T08:32:14.942629+0000","last_active":"2024-09-21T08:32:35.423883+0000","last_peered":"2024-09-21T08:32:35.423883+0000","last_clean":"2024-09-21T08:32:35.423883+0000","last_became_active":"2024-09-21T08:31:56.649277+0000","last_became_peered":"2024-09-21T08:31:56.649277+0000","last_unstale":"2024-09-21T08:32:35.423883+0000","last_undegraded":"2024-09-21T08:32:35.423883+0000","last_fullsized":"2024-09-21T08:32:35.423883+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"541'16","last_scrub_stamp":"2024-09-21T08:32:14.942538+0000","last_deep_scrub":"541'16","last_deep_scrub_stamp":"2024-09-21T08:32:14.942538+0000","last_clean_scrub_stamp":"2024-09-21T08:32:14.942538+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:22.569147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":122,"reported_epoch":560,"state":"active+clean","last_fresh":"2024-09-21T08:32:33.801201+0000","last_change":"2024-09-21T08:31:56.648643+0000","last_active":"2024-09-21T08:32:33.801201+0000","last_peered":"2024-09-21T08:32:33.801201+0000","last_clean":"2024-09-21T08:32:33.801201+0000","last_became_active":"2024-09-21T08:31:56.648235+0000","last_became_peered":"2024-09-21T08:31:56.648235+0000","last_unstale":"2024-09-21T08:32:33.801201+0000","last_undegraded":"2024-09-21T08:32:33.801201+0000","last_fullsized":"2024-09-21T08:32:33.801201+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.872291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.872291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:34.599486+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":46,"stat_sum":{"num_bytes":633235146,"num_objects":296,"num_object_clones":247,"num_object_copies":888,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":296,"num_whiteouts":12,"num_read":2172,"num_read_kb":2361850,"num_write":1567,"num_write_kb":626967,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":880,"num_bytes_recovered":650713102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1572872192,"data_stored":2246094431,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11765,"ondisk_log_size":11765,"up":138,"acting":138,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":14,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":676,"num_read_kb":433,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":12498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":503808,"data_stored":13378,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3558,"internal_metadata":0},"log_size":436,"ondisk_log_size":436,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":4466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":7445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":521,"num_read_kb":516,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1291,"ondisk_log_size":1291,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":522,"seq":2241972928521,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":438384,"kb_used_data":400580,"kb_used_omap":38,"kb_used_meta":36953,"kb_avail":93306768,"statfs":{"total":95995035648,"available":95546130432,"internally_reserved":0,"allocated":410193920,"data_stored":561309367,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38961,"internal_metadata":37840847},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659259,"num_pgs":123,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327256,"kb_used_data":255808,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93417896,"statfs":{"total":95995035648,"available":95659925504,"internally_reserved":0,"allocated":261947392,"data_stored":374007491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986305,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153244,"kb_used_data":126220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93591908,"statfs":{"total":95995035648,"available":95838113792,"internally_reserved":0,"allocated":129249280,"data_stored":167992535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182533,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411467,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405540,"kb_used_data":333300,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93339612,"statfs":{"total":95995035648,"available":95579762688,"internally_reserved":0,"allocated":341299200,"data_stored":536527315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574991,"num_pgs":127,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318980,"kb_used_data":273112,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93426172,"statfs":{"total":95995035648,"available":95668400128,"internally_reserved":0,"allocated":279666688,"data_stored":431638557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":2297807503366,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3558,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":3218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":94208,"data_stored":2492,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74776576,"data_stored":86348592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275013632,"data_stored":425357645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336785408,"data_stored":533760986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70848512,"data_stored":84468350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":124731392,"data_stored":162059112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":257060864,"data_stored":367752962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":405561344,"data_stored":558476369,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:32:37.980 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph pg dump --format=json 2024-09-21T08:32:38.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:38.003 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:38.003 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:38.035 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false 2024-09-21T08:32:38.104 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:38.126 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:38.126 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:38.127 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:38.135 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:37 smithi165 ceph-mon[32893]: purged_snaps scrub starts 2024-09-21T08:32:38.135 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:37 smithi165 ceph-mon[32893]: purged_snaps scrub ok 2024-09-21T08:32:38.135 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:37 smithi165 ceph-mon[32893]: 6.3 scrub starts 2024-09-21T08:32:38.135 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:37 smithi165 ceph-mon[32893]: 6.3 scrub ok 2024-09-21T08:32:38.135 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:37 smithi165 ceph-mon[32893]: Health check failed: Reduced data availability: 4 pgs inactive, 5 pgs peering (PG_AVAILABILITY) 2024-09-21T08:32:38.135 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:37 smithi165 ceph-mon[32893]: from='client.26107 v1:172.21.15.137:0/521065845' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:32:38.136 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2470999473' entity='client.admin' cmd=[{"prefix": "osd unset", "key": "nodeep-scrub"}]: dispatch 2024-09-21T08:32:38.136 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:37 smithi165 ceph-mon[32893]: Health check cleared: OSDMAP_FLAGS (was: nodeep-scrub flag(s) set) 2024-09-21T08:32:38.136 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:37 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/2470999473' entity='client.admin' cmd='[{"prefix": "osd unset", "key": "nodeep-scrub"}]': finished 2024-09-21T08:32:38.227 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:38.235 INFO:teuthology.orchestra.run.smithi165.stdout:{ 2024-09-21T08:32:38.235 INFO:teuthology.orchestra.run.smithi165.stdout: "ops": [], 2024-09-21T08:32:38.235 INFO:teuthology.orchestra.run.smithi165.stdout: "num_ops": 0 2024-09-21T08:32:38.235 INFO:teuthology.orchestra.run.smithi165.stdout:} 2024-09-21T08:32:38.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:38.251 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:38.251 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:38.352 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:38.374 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:38.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:38.374 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:38.475 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7 2024-09-21T08:32:38.497 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:38.497 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:38.498 INFO:tasks.cephadm.osd.7:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.7' 2024-09-21T08:32:38.599 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:38.617 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:38.620 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:38.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:38.621 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:38.722 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:38.743 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:38.744 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:38.744 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:38.801 INFO:tasks.rados.rados.0.smithi137.stdout:2246: done (0 left) 2024-09-21T08:32:38.801 INFO:tasks.rados.rados.0.smithi137.stdout:2247: snap_remove snap 200 2024-09-21T08:32:38.845 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:38.851 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_blocked_ops 2024-09-21T08:32:38.866 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:38.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:38.867 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:38.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:38 smithi137 ceph-mon[29634]: osdmap e564: 8 total, 8 up, 5 in 2024-09-21T08:32:38.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:38 smithi137 ceph-mon[29634]: pgmap v752: 188 pgs: 3 unknown, 4 active+clean+snaptrim, 23 peering, 24 active+clean+snaptrim_wait, 134 active+clean; 604 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 1.6 MiB/s wr, 0 op/s 2024-09-21T08:32:38.927 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:38 smithi137 ceph-mon[29634]: osdmap e565: 8 total, 8 up, 5 in 2024-09-21T08:32:38.928 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:38 smithi137 ceph-mon[41044]: osdmap e564: 8 total, 8 up, 5 in 2024-09-21T08:32:38.928 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:38 smithi137 ceph-mon[41044]: pgmap v752: 188 pgs: 3 unknown, 4 active+clean+snaptrim, 23 peering, 24 active+clean+snaptrim_wait, 134 active+clean; 604 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 1.6 MiB/s wr, 0 op/s 2024-09-21T08:32:38.928 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:38 smithi137 ceph-mon[41044]: osdmap e565: 8 total, 8 up, 5 in 2024-09-21T08:32:38.968 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:38.990 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:38.990 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:38.991 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:39.091 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:39.113 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:32:39.114 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:39.114 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:39.115 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:39.175 INFO:teuthology.orchestra.run.smithi137.stdout:osd.0: {} 2024-09-21T08:32:39.175 INFO:teuthology.orchestra.run.smithi137.stderr:osd.0: osd_enable_op_tracker = '' 2024-09-21T08:32:39.189 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:39.192 INFO:teuthology.orchestra.run.smithi137.stdout:osd.1: {} 2024-09-21T08:32:39.192 INFO:teuthology.orchestra.run.smithi137.stderr:osd.1: osd_enable_op_tracker = '' 2024-09-21T08:32:39.210 INFO:teuthology.orchestra.run.smithi137.stdout:osd.2: {} 2024-09-21T08:32:39.210 INFO:teuthology.orchestra.run.smithi137.stderr:osd.2: osd_enable_op_tracker = '' 2024-09-21T08:32:39.215 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:39.232 INFO:teuthology.orchestra.run.smithi137.stdout:osd.3: {} 2024-09-21T08:32:39.233 INFO:teuthology.orchestra.run.smithi137.stderr:osd.3: osd_enable_op_tracker = '' 2024-09-21T08:32:39.233 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:38 smithi165 ceph-mon[32893]: osdmap e564: 8 total, 8 up, 5 in 2024-09-21T08:32:39.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:38 smithi165 ceph-mon[32893]: pgmap v752: 188 pgs: 3 unknown, 4 active+clean+snaptrim, 23 peering, 24 active+clean+snaptrim_wait, 134 active+clean; 604 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 1.6 MiB/s wr, 0 op/s 2024-09-21T08:32:39.234 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:38 smithi165 ceph-mon[32893]: osdmap e565: 8 total, 8 up, 5 in 2024-09-21T08:32:39.237 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:39.238 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:39.238 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:39.249 INFO:teuthology.orchestra.run.smithi137.stdout:osd.4: {} 2024-09-21T08:32:39.249 INFO:teuthology.orchestra.run.smithi137.stderr:osd.4: osd_enable_op_tracker = '' 2024-09-21T08:32:39.266 INFO:teuthology.orchestra.run.smithi137.stdout:osd.5: {} 2024-09-21T08:32:39.266 INFO:teuthology.orchestra.run.smithi137.stderr:osd.5: osd_enable_op_tracker = '' 2024-09-21T08:32:39.282 INFO:teuthology.orchestra.run.smithi137.stdout:osd.6: {} 2024-09-21T08:32:39.282 INFO:teuthology.orchestra.run.smithi137.stderr:osd.6: osd_enable_op_tracker = '' 2024-09-21T08:32:39.298 INFO:teuthology.orchestra.run.smithi137.stdout:osd.7: {} 2024-09-21T08:32:39.298 INFO:teuthology.orchestra.run.smithi137.stderr:osd.7: osd_enable_op_tracker = '' 2024-09-21T08:32:39.339 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:39.361 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:39.362 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:39.362 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:39.463 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:39.484 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:39.484 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:39.485 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:39.586 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:39.607 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:39.607 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:39.608 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:39.659 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:32:39.659 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:32:39.708 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:39.730 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:39.731 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:39.731 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:39.805 INFO:tasks.rados.rados.0.smithi137.stdout:2247: done (0 left) 2024-09-21T08:32:39.805 INFO:tasks.rados.rados.0.smithi137.stdout:2248: write (excl) oid 41 current snap is 250 2024-09-21T08:32:39.805 INFO:tasks.rados.rados.0.smithi137.stdout:2248: seq_num 779 ranges {692653=486977,1602061=647603,2992629=100024} 2024-09-21T08:32:39.812 INFO:tasks.rados.rados.0.smithi137.stdout:2248: writing smithi13779221-41 from 692653 to 1179630 tid 1 2024-09-21T08:32:39.818 INFO:tasks.rados.rados.0.smithi137.stdout:2248: writing smithi13779221-41 from 1602061 to 2249664 tid 2 2024-09-21T08:32:39.823 INFO:tasks.rados.rados.0.smithi137.stdout:2248: writing smithi13779221-41 from 2992629 to 3092653 tid 3 2024-09-21T08:32:39.823 INFO:tasks.rados.rados.0.smithi137.stdout:2249: read oid 31 snap -1 2024-09-21T08:32:39.823 INFO:tasks.rados.rados.0.smithi137.stdout:2249: expect (ObjNum 618 snap 182 seq_num 618) 2024-09-21T08:32:39.823 INFO:tasks.rados.rados.0.smithi137.stdout:2250: write oid 24 current snap is 250 2024-09-21T08:32:39.824 INFO:tasks.rados.rados.0.smithi137.stdout:2248: finishing write tid 1 to smithi13779221-41 2024-09-21T08:32:39.824 INFO:tasks.rados.rados.0.smithi137.stdout:2250: seq_num 780 ranges {709460=562226,2060543=791261} 2024-09-21T08:32:39.830 INFO:tasks.rados.rados.0.smithi137.stdout:2250: writing smithi13779221-24 from 709460 to 1271686 tid 1 2024-09-21T08:32:39.832 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:39.839 INFO:tasks.rados.rados.0.smithi137.stdout:2250: writing smithi13779221-24 from 2060543 to 2851804 tid 2 2024-09-21T08:32:39.839 INFO:tasks.rados.rados.0.smithi137.stdout:2251: read oid 13 snap -1 2024-09-21T08:32:39.839 INFO:tasks.rados.rados.0.smithi137.stdout:2251: expect (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:32:39.839 INFO:tasks.rados.rados.0.smithi137.stdout:2252: read oid 37 snap 201 2024-09-21T08:32:39.840 INFO:tasks.rados.rados.0.smithi137.stdout:2252: expect (ObjNum 618 snap 182 seq_num 618) 2024-09-21T08:32:39.840 INFO:tasks.rados.rados.0.smithi137.stdout:2253: snap_remove snap 243 2024-09-21T08:32:39.856 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:39.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:39.857 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:39.958 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:39.981 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:39.981 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:39.981 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:40.017 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[29634]: 5.e scrub starts 2024-09-21T08:32:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[29634]: 5.e scrub ok 2024-09-21T08:32:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[29634]: 5.15 scrub starts 2024-09-21T08:32:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[29634]: 5.15 scrub ok 2024-09-21T08:32:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[29634]: from='client.? v1:172.21.15.137:0/4095044844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[29634]: 6.15 scrub starts 2024-09-21T08:32:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[29634]: 6.15 scrub ok 2024-09-21T08:32:40.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[29634]: osdmap e566: 8 total, 8 up, 5 in 2024-09-21T08:32:40.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[41044]: 5.e scrub starts 2024-09-21T08:32:40.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[41044]: 5.e scrub ok 2024-09-21T08:32:40.018 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[41044]: 5.15 scrub starts 2024-09-21T08:32:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[41044]: 5.15 scrub ok 2024-09-21T08:32:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[41044]: from='client.? v1:172.21.15.137:0/4095044844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[41044]: 6.15 scrub starts 2024-09-21T08:32:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[41044]: 6.15 scrub ok 2024-09-21T08:32:40.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:39 smithi137 ceph-mon[41044]: osdmap e566: 8 total, 8 up, 5 in 2024-09-21T08:32:40.082 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:40.104 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:40.105 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:40.105 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:40.206 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:40.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:39 smithi165 ceph-mon[32893]: 5.e scrub starts 2024-09-21T08:32:40.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:39 smithi165 ceph-mon[32893]: 5.e scrub ok 2024-09-21T08:32:40.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:39 smithi165 ceph-mon[32893]: 5.15 scrub starts 2024-09-21T08:32:40.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:39 smithi165 ceph-mon[32893]: 5.15 scrub ok 2024-09-21T08:32:40.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:39 smithi165 ceph-mon[32893]: from='client.? v1:172.21.15.137:0/4095044844' entity='client.admin' cmd=[{"prefix": "osd ls"}]: dispatch 2024-09-21T08:32:40.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:39 smithi165 ceph-mon[32893]: 6.15 scrub starts 2024-09-21T08:32:40.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:39 smithi165 ceph-mon[32893]: 6.15 scrub ok 2024-09-21T08:32:40.221 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:39 smithi165 ceph-mon[32893]: osdmap e566: 8 total, 8 up, 5 in 2024-09-21T08:32:40.228 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:40.228 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:40.229 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:40.248 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.5 dump_historic_ops 2024-09-21T08:32:40.329 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:40.351 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:40.351 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:40.352 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:40.453 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:40.474 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:40.474 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:40.475 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:40.488 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:32:40.549 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:32:40.552 INFO:teuthology.orchestra.run.smithi137.stderr:dumped all 2024-09-21T08:32:40.574 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph osd set noscrub 2024-09-21T08:32:40.576 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:40.596 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:40.596 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:40.597 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:40.697 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:40.719 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:40.720 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:40.720 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:40.821 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:40.832 INFO:tasks.rados.rados.0.smithi137.stdout:2253: done (5 left) 2024-09-21T08:32:40.832 INFO:tasks.rados.rados.0.smithi137.stdout:2254: snap_remove snap 246 2024-09-21T08:32:40.848 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:40.848 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:40.849 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:40.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:40 smithi165 ceph-mon[32893]: 1.2 scrub starts 2024-09-21T08:32:40.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:40 smithi165 ceph-mon[32893]: 1.2 scrub ok 2024-09-21T08:32:40.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:40 smithi165 ceph-mon[32893]: 5.4 scrub starts 2024-09-21T08:32:40.866 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:40 smithi165 ceph-mon[32893]: 5.4 scrub ok 2024-09-21T08:32:40.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:40 smithi165 ceph-mon[32893]: pgmap v755: 188 pgs: 3 unknown, 4 active+clean+snaptrim, 23 peering, 24 active+clean+snaptrim_wait, 134 active+clean; 608 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 1.7 MiB/s rd, 858 KiB/s wr, 2 op/s 2024-09-21T08:32:40.867 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:40 smithi165 ceph-mon[32893]: from='client.46133 v1:172.21.15.137:0/63684736' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:32:40.949 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:40.972 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:40.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:40.973 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:40.974 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:32:40.974 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:32:41.004 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[29634]: 1.2 scrub starts 2024-09-21T08:32:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[29634]: 1.2 scrub ok 2024-09-21T08:32:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[29634]: 5.4 scrub starts 2024-09-21T08:32:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[29634]: 5.4 scrub ok 2024-09-21T08:32:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[29634]: pgmap v755: 188 pgs: 3 unknown, 4 active+clean+snaptrim, 23 peering, 24 active+clean+snaptrim_wait, 134 active+clean; 608 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 1.7 MiB/s rd, 858 KiB/s wr, 2 op/s 2024-09-21T08:32:41.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[29634]: from='client.46133 v1:172.21.15.137:0/63684736' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:32:41.018 INFO:journalctl@ceph.mgr.y.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-mgr-y[29887]: ::ffff:172.21.15.165 - - [21/Sep/2024:08:32:40] "GET /metrics HTTP/1.1" 503 1620 "" "Prometheus/2.43.0" 2024-09-21T08:32:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[41044]: 1.2 scrub starts 2024-09-21T08:32:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[41044]: 1.2 scrub ok 2024-09-21T08:32:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[41044]: 5.4 scrub starts 2024-09-21T08:32:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[41044]: 5.4 scrub ok 2024-09-21T08:32:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[41044]: pgmap v755: 188 pgs: 3 unknown, 4 active+clean+snaptrim, 23 peering, 24 active+clean+snaptrim_wait, 134 active+clean; 608 MiB data, 1.6 GiB used, 445 GiB / 447 GiB avail; 1.7 MiB/s rd, 858 KiB/s wr, 2 op/s 2024-09-21T08:32:41.019 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:40 smithi137 ceph-mon[41044]: from='client.46133 v1:172.21.15.137:0/63684736' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-09-21T08:32:41.073 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0 2024-09-21T08:32:41.094 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:41.095 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:41.095 INFO:tasks.cephadm.osd.0:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.0' 2024-09-21T08:32:41.196 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:41.218 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:41.218 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:41.219 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:41.320 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:41.341 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:41.342 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:41.342 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:41.443 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:41.464 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:41.465 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:41.465 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:41.471 INFO:teuthology.orchestra.run.smithi137.stdout:{"pg_ready":true,"pg_map":{"version":755,"stamp":"2024-09-21T08:32:40.175428+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":637344415,"num_objects":496,"num_object_clones":248,"num_object_copies":1488,"num_objects_missing_on_primary":0,"num_objects_missing":16,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":496,"num_whiteouts":12,"num_read":3401,"num_read_kb":2369820,"num_write":2004,"num_write_kb":630443,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1558,"num_bytes_recovered":650731801,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":13677,"ondisk_log_size":13677,"up":564,"acting":564,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":564,"num_osds":5,"num_per_pool_osds":5,"num_per_pool_omap_osds":5,"kb":468725760,"kb_used":1640552,"kb_used_data":1386956,"kb_used_omap":44,"kb_used_meta":251859,"kb_avail":467085208,"statfs":{"total":479975178240,"available":478295252992,"internally_reserved":0,"allocated":1420242944,"data_stored":2063147462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":45341,"internal_metadata":257904355},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3512762,"num_objects":1,"num_object_clones":1,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":8,"num_read_kb":6997,"num_write":1,"num_write_kb":3431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001045"},"pg_stats":[{"pgid":"7.2d","version":"455'154","reported_seq":987,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.747838+0000","last_change":"2024-09-21T08:32:35.747893+0000","last_active":"2024-09-21T08:32:35.747838+0000","last_peered":"2024-09-21T08:32:35.747838+0000","last_clean":"2024-09-21T08:32:35.747838+0000","last_became_active":"2024-09-21T08:31:56.670753+0000","last_became_peered":"2024-09-21T08:31:56.670753+0000","last_unstale":"2024-09-21T08:32:35.747838+0000","last_undegraded":"2024-09-21T08:32:35.747838+0000","last_fullsized":"2024-09-21T08:32:35.747838+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":6,"last_scrub":"294'93","last_scrub_stamp":"2024-09-21T08:28:11.373358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:11.373358+0000","objects_scrubbed":3,"log_size":154,"log_dups_size":0,"ondisk_log_size":154,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":102,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.328085935,"stat_sum":{"num_bytes":13974387,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":1,"num_read":16,"num_read_kb":8938,"num_write":21,"num_write_kb":5598,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3285818,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"2"}]},{"pgid":"7.2c","version":"563'347","reported_seq":2780,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-09-21T08:32:38.806700+0000","last_change":"2024-09-21T08:32:38.807087+0000","last_active":"2024-09-21T08:32:38.806700+0000","last_peered":"2024-09-21T08:32:38.806700+0000","last_clean":"2024-09-21T08:32:38.806700+0000","last_became_active":"2024-09-21T08:31:59.631276+0000","last_became_peered":"2024-09-21T08:31:59.631276+0000","last_unstale":"2024-09-21T08:32:38.806700+0000","last_undegraded":"2024-09-21T08:32:38.806700+0000","last_fullsized":"2024-09-21T08:32:38.806700+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":6,"last_scrub":"533'325","last_scrub_stamp":"2024-09-21T08:32:08.675641+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:32:08.675641+0000","objects_scrubbed":9,"log_size":347,"log_dups_size":0,"ondisk_log_size":347,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:29.634510+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0.0025610250000000002,"stat_sum":{"num_bytes":18099431,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":2,"num_read_kb":0,"num_write":1,"num_write_kb":3431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"b9","length":"11"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"7"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"7.2b","version":"565'956","reported_seq":3490,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-09-21T08:32:38.813793+0000","last_change":"2024-09-21T08:32:38.814541+0000","last_active":"2024-09-21T08:32:38.813793+0000","last_peered":"2024-09-21T08:32:38.813793+0000","last_clean":"2024-09-21T08:32:38.813793+0000","last_became_active":"2024-09-21T08:31:59.630195+0000","last_became_peered":"2024-09-21T08:31:59.630195+0000","last_unstale":"2024-09-21T08:32:38.813793+0000","last_undegraded":"2024-09-21T08:32:38.813793+0000","last_fullsized":"2024-09-21T08:32:38.813793+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":6,"last_scrub":"533'879","last_scrub_stamp":"2024-09-21T08:32:07.696661+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:32:07.696661+0000","objects_scrubbed":19,"log_size":956,"log_dups_size":0,"ondisk_log_size":956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:15.739351+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0.0059237029999999998,"stat_sum":{"num_bytes":45927961,"num_objects":20,"num_object_clones":17,"num_object_copies":60,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":20,"num_whiteouts":0,"num_read":12,"num_read_kb":6558,"num_write":11,"num_write_kb":8168,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"b9","length":"11"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"7"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"7.2a","version":"472'122","reported_seq":1659,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.418353+0000","last_change":"2024-09-21T08:32:33.769192+0000","last_active":"2024-09-21T08:32:35.418353+0000","last_peered":"2024-09-21T08:32:35.418353+0000","last_clean":"2024-09-21T08:32:35.418353+0000","last_became_active":"2024-09-21T08:31:59.629605+0000","last_became_peered":"2024-09-21T08:31:59.629605+0000","last_unstale":"2024-09-21T08:32:35.418353+0000","last_undegraded":"2024-09-21T08:32:35.418353+0000","last_fullsized":"2024-09-21T08:32:35.418353+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":6,"last_scrub":"472'122","last_scrub_stamp":"2024-09-21T08:31:28.194668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:28.194668+0000","objects_scrubbed":0,"log_size":122,"log_dups_size":0,"ondisk_log_size":122,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":11,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.0003801139999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"de","length":"1"}]},{"pgid":"7.29","version":"558'403","reported_seq":3046,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.420787+0000","last_change":"2024-09-21T08:32:33.741879+0000","last_active":"2024-09-21T08:32:35.420787+0000","last_peered":"2024-09-21T08:32:35.420787+0000","last_clean":"2024-09-21T08:32:35.420787+0000","last_became_active":"2024-09-21T08:31:57.664264+0000","last_became_peered":"2024-09-21T08:31:57.664264+0000","last_unstale":"2024-09-21T08:32:35.420787+0000","last_undegraded":"2024-09-21T08:32:35.420787+0000","last_fullsized":"2024-09-21T08:32:35.420787+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.011461229999999999,"stat_sum":{"num_bytes":23676791,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":1,"num_read":104,"num_read_kb":101846,"num_write":75,"num_write_kb":24161,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":33558636,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"2"},{"start":"c6","length":"3"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"7.28","version":"561'476","reported_seq":2785,"reported_epoch":561,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:32:35.747932+0000","last_change":"2024-09-21T08:32:35.747932+0000","last_active":"2024-09-21T08:32:35.747932+0000","last_peered":"2024-09-21T08:32:35.747932+0000","last_clean":"2024-09-21T08:32:35.747932+0000","last_became_active":"2024-09-21T08:31:59.631478+0000","last_became_peered":"2024-09-21T08:31:59.631478+0000","last_unstale":"2024-09-21T08:32:35.747932+0000","last_undegraded":"2024-09-21T08:32:35.747932+0000","last_fullsized":"2024-09-21T08:32:35.747932+0000","mapping_epoch":525,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":526,"parent":"0.0","parent_split_bits":6,"last_scrub":"194'178","last_scrub_stamp":"2024-09-21T08:26:28.217842+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:28.217842+0000","objects_scrubbed":4,"log_size":476,"log_dups_size":0,"ondisk_log_size":476,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":19,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.0065354920000000004,"stat_sum":{"num_bytes":20788182,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":2,"num_read":101,"num_read_kb":109949,"num_write":87,"num_write_kb":32244,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":58,"num_bytes_recovered":44868606,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"7"},{"start":"c3","length":"2"},{"start":"c6","length":"3"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"7.27","version":"553'216","reported_seq":2593,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.423028+0000","last_change":"2024-09-21T08:32:33.769952+0000","last_active":"2024-09-21T08:32:35.423028+0000","last_peered":"2024-09-21T08:32:35.423028+0000","last_clean":"2024-09-21T08:32:35.423028+0000","last_became_active":"2024-09-21T08:31:57.659296+0000","last_became_peered":"2024-09-21T08:31:57.659296+0000","last_unstale":"2024-09-21T08:32:35.423028+0000","last_undegraded":"2024-09-21T08:32:35.423028+0000","last_fullsized":"2024-09-21T08:32:35.423028+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":67,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.99920343499999997,"stat_sum":{"num_bytes":11587668,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":54,"num_read_kb":52955,"num_write":41,"num_write_kb":19786,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":53,"num_bytes_recovered":38107210,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"25"}]},{"pgid":"7.26","version":"453'161","reported_seq":2196,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.746743+0000","last_change":"2024-09-21T08:32:35.746798+0000","last_active":"2024-09-21T08:32:35.746743+0000","last_peered":"2024-09-21T08:32:35.746743+0000","last_clean":"2024-09-21T08:32:35.746743+0000","last_became_active":"2024-09-21T08:31:56.648746+0000","last_became_peered":"2024-09-21T08:31:56.648746+0000","last_unstale":"2024-09-21T08:32:35.746743+0000","last_undegraded":"2024-09-21T08:32:35.746743+0000","last_fullsized":"2024-09-21T08:32:35.746743+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":6,"last_scrub":"294'138","last_scrub_stamp":"2024-09-21T08:28:41.855754+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:41.855754+0000","objects_scrubbed":1,"log_size":161,"log_dups_size":0,"ondisk_log_size":161,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":93,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.32741683900000002,"stat_sum":{"num_bytes":6518339,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":26,"num_read_kb":31642,"num_write":14,"num_write_kb":4831,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13,"num_bytes_recovered":4641445,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"b"}]},{"pgid":"7.25","version":"558'365","reported_seq":2763,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788546+0000","last_change":"2024-09-21T08:32:33.783736+0000","last_active":"2024-09-21T08:32:34.788546+0000","last_peered":"2024-09-21T08:32:34.788546+0000","last_clean":"2024-09-21T08:32:34.788546+0000","last_became_active":"2024-09-21T08:31:57.648035+0000","last_became_peered":"2024-09-21T08:31:57.648035+0000","last_unstale":"2024-09-21T08:32:34.788546+0000","last_undegraded":"2024-09-21T08:32:34.788546+0000","last_fullsized":"2024-09-21T08:32:34.788546+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"487'327","last_scrub_stamp":"2024-09-21T08:31:24.989978+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:24.989978+0000","objects_scrubbed":4,"log_size":365,"log_dups_size":0,"ondisk_log_size":365,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:51.582411+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00079025899999999997,"stat_sum":{"num_bytes":10958659,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":9,"num_read_kb":8870,"num_write":6,"num_write_kb":981,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":7502779,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"7.24","version":"554'192","reported_seq":2519,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.418784+0000","last_change":"2024-09-21T08:32:33.737274+0000","last_active":"2024-09-21T08:32:35.418784+0000","last_peered":"2024-09-21T08:32:35.418784+0000","last_clean":"2024-09-21T08:32:35.418784+0000","last_became_active":"2024-09-21T08:31:12.614215+0000","last_became_peered":"2024-09-21T08:31:12.614215+0000","last_unstale":"2024-09-21T08:32:35.418784+0000","last_undegraded":"2024-09-21T08:32:35.418784+0000","last_fullsized":"2024-09-21T08:32:35.418784+0000","mapping_epoch":478,"log_start":"0'0","ondisk_log_start":"0'0","created":473,"last_epoch_clean":479,"parent":"0.0","parent_split_bits":6,"last_scrub":"492'170","last_scrub_stamp":"2024-09-21T08:31:30.858539+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:31:30.858539+0000","objects_scrubbed":4,"log_size":192,"log_dups_size":0,"ondisk_log_size":192,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:42.469710+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0064774680000000001,"stat_sum":{"num_bytes":8944469,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":14,"num_write_kb":3263,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"6"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"7.23","version":"179'19","reported_seq":1282,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:35.423190+0000","last_change":"2024-09-21T08:32:35.423447+0000","last_active":"2024-09-21T08:32:35.423161+0000","last_peered":"2024-09-21T08:32:33.804649+0000","last_clean":"2024-09-21T08:32:33.804649+0000","last_became_active":"2024-09-21T08:31:57.658433+0000","last_became_peered":"2024-09-21T08:31:57.658433+0000","last_unstale":"2024-09-21T08:32:35.423190+0000","last_undegraded":"2024-09-21T08:32:35.423190+0000","last_fullsized":"2024-09-21T08:32:35.423190+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:28:10.421606+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.421606+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99473726699999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,7],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"de","length":"1"}]},{"pgid":"7.22","version":"257'264","reported_seq":2692,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788132+0000","last_change":"2024-09-21T08:32:33.784658+0000","last_active":"2024-09-21T08:32:34.788132+0000","last_peered":"2024-09-21T08:32:34.788132+0000","last_clean":"2024-09-21T08:32:34.788132+0000","last_became_active":"2024-09-21T08:31:57.651552+0000","last_became_peered":"2024-09-21T08:31:57.651552+0000","last_unstale":"2024-09-21T08:32:34.788132+0000","last_undegraded":"2024-09-21T08:32:34.788132+0000","last_fullsized":"2024-09-21T08:32:34.788132+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"257'264","last_scrub_stamp":"2024-09-21T08:27:48.727600+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:48.727600+0000","objects_scrubbed":0,"log_size":264,"log_dups_size":0,"ondisk_log_size":264,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00081390599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"7.21","version":"179'43","reported_seq":2083,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.421073+0000","last_change":"2024-09-21T08:32:33.743956+0000","last_active":"2024-09-21T08:32:35.421073+0000","last_peered":"2024-09-21T08:32:35.421073+0000","last_clean":"2024-09-21T08:32:35.421073+0000","last_became_active":"2024-09-21T08:31:57.644273+0000","last_became_peered":"2024-09-21T08:31:57.644273+0000","last_unstale":"2024-09-21T08:32:35.421073+0000","last_undegraded":"2024-09-21T08:32:35.421073+0000","last_fullsized":"2024-09-21T08:32:35.421073+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":6,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:36.934372+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:36.934372+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.012905412999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"b"},{"start":"c6","length":"3"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"7.20","version":"548'185","reported_seq":2210,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.420681+0000","last_change":"2024-09-21T08:32:33.770811+0000","last_active":"2024-09-21T08:32:35.420681+0000","last_peered":"2024-09-21T08:32:35.420681+0000","last_clean":"2024-09-21T08:32:35.420681+0000","last_became_active":"2024-09-21T08:31:56.641129+0000","last_became_peered":"2024-09-21T08:31:56.641129+0000","last_unstale":"2024-09-21T08:32:35.420681+0000","last_undegraded":"2024-09-21T08:32:35.420681+0000","last_fullsized":"2024-09-21T08:32:35.420681+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":6,"last_scrub":"251'134","last_scrub_stamp":"2024-09-21T08:27:49.318143+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.318143+0000","objects_scrubbed":1,"log_size":185,"log_dups_size":0,"ondisk_log_size":185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":92,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.001246278,"stat_sum":{"num_bytes":16994774,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":29,"num_read_kb":17524,"num_write":36,"num_write_kb":7693,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":8389384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"c"}]},{"pgid":"7.1b","version":"560'891","reported_seq":3499,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.746331+0000","last_change":"2024-09-21T08:32:35.746385+0000","last_active":"2024-09-21T08:32:35.746331+0000","last_peered":"2024-09-21T08:32:35.746331+0000","last_clean":"2024-09-21T08:32:35.746331+0000","last_became_active":"2024-09-21T08:30:59.601175+0000","last_became_peered":"2024-09-21T08:30:59.601175+0000","last_unstale":"2024-09-21T08:32:35.746331+0000","last_undegraded":"2024-09-21T08:32:35.746331+0000","last_fullsized":"2024-09-21T08:32:35.746331+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'499","last_scrub_stamp":"2024-09-21T08:28:09.328624+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.328624+0000","objects_scrubbed":5,"log_size":891,"log_dups_size":0,"ondisk_log_size":891,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":18,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.32010392399999998,"stat_sum":{"num_bytes":46461038,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":173,"num_read_kb":184879,"num_write":136,"num_write_kb":48397,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":94,"num_bytes_recovered":61029229,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"2"},{"start":"c6","length":"3"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"3.1f","version":"0'0","reported_seq":510,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423706+0000","last_change":"2024-09-21T08:30:56.618803+0000","last_active":"2024-09-21T08:32:35.423706+0000","last_peered":"2024-09-21T08:32:35.423706+0000","last_clean":"2024-09-21T08:32:35.423706+0000","last_became_active":"2024-09-21T08:30:56.617989+0000","last_became_peered":"2024-09-21T08:30:56.617989+0000","last_unstale":"2024-09-21T08:32:35.423706+0000","last_undegraded":"2024-09-21T08:32:35.423706+0000","last_fullsized":"2024-09-21T08:32:35.423706+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:38.378418+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:25:38.378418+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.18","version":"73'9","reported_seq":217,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789429+0000","last_change":"2024-09-21T08:30:56.662007+0000","last_active":"2024-09-21T08:32:34.789429+0000","last_peered":"2024-09-21T08:32:34.789429+0000","last_clean":"2024-09-21T08:32:34.789429+0000","last_became_active":"2024-09-21T08:30:56.628842+0000","last_became_peered":"2024-09-21T08:30:56.628842+0000","last_unstale":"2024-09-21T08:32:34.789429+0000","last_undegraded":"2024-09-21T08:32:34.789429+0000","last_fullsized":"2024-09-21T08:32:34.789429+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:44.058465+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:44.058465+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.19","version":"0'0","reported_seq":569,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421750+0000","last_change":"2024-09-21T08:31:56.651426+0000","last_active":"2024-09-21T08:32:35.421750+0000","last_peered":"2024-09-21T08:32:35.421750+0000","last_clean":"2024-09-21T08:32:35.421750+0000","last_became_active":"2024-09-21T08:31:56.651133+0000","last_became_peered":"2024-09-21T08:31:56.651133+0000","last_unstale":"2024-09-21T08:32:35.421750+0000","last_undegraded":"2024-09-21T08:32:35.421750+0000","last_fullsized":"2024-09-21T08:32:35.421750+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.424482+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.424482+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.1a","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788693+0000","last_change":"2024-09-21T08:30:56.631377+0000","last_active":"2024-09-21T08:32:34.788693+0000","last_peered":"2024-09-21T08:32:34.788693+0000","last_clean":"2024-09-21T08:32:34.788693+0000","last_became_active":"2024-09-21T08:30:56.630218+0000","last_became_peered":"2024-09-21T08:30:56.630218+0000","last_unstale":"2024-09-21T08:32:34.788693+0000","last_undegraded":"2024-09-21T08:32:34.788693+0000","last_fullsized":"2024-09-21T08:32:34.788693+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.464969+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.464969+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1a","version":"248'68","reported_seq":720,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788378+0000","last_change":"2024-09-21T08:32:33.783456+0000","last_active":"2024-09-21T08:32:34.788378+0000","last_peered":"2024-09-21T08:32:34.788378+0000","last_clean":"2024-09-21T08:32:34.788378+0000","last_became_active":"2024-09-21T08:31:57.648774+0000","last_became_peered":"2024-09-21T08:31:57.648774+0000","last_unstale":"2024-09-21T08:32:34.788378+0000","last_undegraded":"2024-09-21T08:32:34.788378+0000","last_fullsized":"2024-09-21T08:32:34.788378+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'68","last_scrub_stamp":"2024-09-21T08:27:35.919031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:35.919031+0000","objects_scrubbed":0,"log_size":68,"log_dups_size":0,"ondisk_log_size":68,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010419299999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"6.1b","version":"0'0","reported_seq":926,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.422537+0000","last_change":"2024-09-21T08:31:56.658780+0000","last_active":"2024-09-21T08:32:35.422537+0000","last_peered":"2024-09-21T08:32:35.422537+0000","last_clean":"2024-09-21T08:32:35.422537+0000","last_became_active":"2024-09-21T08:31:56.658407+0000","last_became_peered":"2024-09-21T08:31:56.658407+0000","last_unstale":"2024-09-21T08:32:35.422537+0000","last_undegraded":"2024-09-21T08:32:35.422537+0000","last_fullsized":"2024-09-21T08:32:35.422537+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.804294+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.804294+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":853,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421577+0000","last_change":"2024-09-21T08:31:56.643062+0000","last_active":"2024-09-21T08:32:35.421577+0000","last_peered":"2024-09-21T08:32:35.421577+0000","last_clean":"2024-09-21T08:32:35.421577+0000","last_became_active":"2024-09-21T08:31:56.642755+0000","last_became_peered":"2024-09-21T08:31:56.642755+0000","last_unstale":"2024-09-21T08:32:35.421577+0000","last_undegraded":"2024-09-21T08:32:35.421577+0000","last_fullsized":"2024-09-21T08:32:35.421577+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.146798+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.146798+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.19","version":"73'15","reported_seq":843,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424380+0000","last_change":"2024-09-21T08:29:00.355702+0000","last_active":"2024-09-21T08:32:35.424380+0000","last_peered":"2024-09-21T08:32:35.424380+0000","last_clean":"2024-09-21T08:32:35.424380+0000","last_became_active":"2024-09-21T08:28:29.620169+0000","last_became_peered":"2024-09-21T08:28:29.620169+0000","last_unstale":"2024-09-21T08:32:35.424380+0000","last_undegraded":"2024-09-21T08:32:35.424380+0000","last_fullsized":"2024-09-21T08:32:35.424380+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:00.355591+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:00.355591+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":25,"num_read_kb":16,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.18","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.803875+0000","last_change":"2024-09-21T08:32:35.750616+0000","last_active":"2024-09-21T08:32:35.750616+0000","last_peered":"2024-09-21T08:32:33.803875+0000","last_clean":"2024-09-21T08:32:33.803875+0000","last_became_active":"2024-09-21T08:31:57.642832+0000","last_became_peered":"2024-09-21T08:31:57.642832+0000","last_unstale":"2024-09-21T08:32:33.803875+0000","last_undegraded":"2024-09-21T08:32:33.803875+0000","last_fullsized":"2024-09-21T08:32:33.803875+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.340817+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.340817+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.19","version":"507'216","reported_seq":1906,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.424525+0000","last_change":"2024-09-21T08:32:33.742643+0000","last_active":"2024-09-21T08:32:35.424525+0000","last_peered":"2024-09-21T08:32:35.424525+0000","last_clean":"2024-09-21T08:32:35.424525+0000","last_became_active":"2024-09-21T08:31:56.648297+0000","last_became_peered":"2024-09-21T08:31:56.648297+0000","last_unstale":"2024-09-21T08:32:35.424525+0000","last_undegraded":"2024-09-21T08:32:35.424525+0000","last_fullsized":"2024-09-21T08:32:35.424525+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"266'188","last_scrub_stamp":"2024-09-21T08:27:49.719124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.719124+0000","objects_scrubbed":1,"log_size":216,"log_dups_size":0,"ondisk_log_size":216,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":97,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.011411124999999999,"stat_sum":{"num_bytes":16452542,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":35,"num_read_kb":28684,"num_write":27,"num_write_kb":14365,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":13913530,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"7"}]},{"pgid":"3.1d","version":"0'0","reported_seq":1057,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421047+0000","last_change":"2024-09-21T08:30:56.626716+0000","last_active":"2024-09-21T08:32:35.421047+0000","last_peered":"2024-09-21T08:32:35.421047+0000","last_clean":"2024-09-21T08:32:35.421047+0000","last_became_active":"2024-09-21T08:30:56.625894+0000","last_became_peered":"2024-09-21T08:30:56.625894+0000","last_unstale":"2024-09-21T08:32:35.421047+0000","last_undegraded":"2024-09-21T08:32:35.421047+0000","last_fullsized":"2024-09-21T08:32:35.421047+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:47.780545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:47.780545+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1a","version":"73'9","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:31:40.664995+0000","last_change":"2024-09-21T08:31:39.648713+0000","last_active":"2024-09-21T08:31:39.648713+0000","last_peered":"2024-09-21T08:30:54.598546+0000","last_clean":"2024-09-21T08:30:54.598546+0000","last_became_active":"2024-09-21T08:28:37.568659+0000","last_became_peered":"2024-09-21T08:28:37.568659+0000","last_unstale":"2024-09-21T08:31:40.664995+0000","last_undegraded":"2024-09-21T08:31:40.664995+0000","last_fullsized":"2024-09-21T08:31:40.664995+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:26.781541+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:26.781541+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":4,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":16,"num_read_kb":10,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1b","version":"0'0","reported_seq":1032,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.790625+0000","last_change":"2024-09-21T08:32:35.428319+0000","last_active":"2024-09-21T08:32:35.428319+0000","last_peered":"2024-09-21T08:32:33.790625+0000","last_clean":"2024-09-21T08:32:33.790625+0000","last_became_active":"2024-09-21T08:31:57.642517+0000","last_became_peered":"2024-09-21T08:31:57.642517+0000","last_unstale":"2024-09-21T08:32:33.790625+0000","last_undegraded":"2024-09-21T08:32:33.790625+0000","last_fullsized":"2024-09-21T08:32:33.790625+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:17.851031+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:32:17.851031+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.18","version":"0'0","reported_seq":972,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418269+0000","last_change":"2024-09-21T08:31:56.637990+0000","last_active":"2024-09-21T08:32:35.418269+0000","last_peered":"2024-09-21T08:32:35.418269+0000","last_clean":"2024-09-21T08:32:35.418269+0000","last_became_active":"2024-09-21T08:31:56.637161+0000","last_became_peered":"2024-09-21T08:31:56.637161+0000","last_unstale":"2024-09-21T08:32:35.418269+0000","last_undegraded":"2024-09-21T08:32:35.418269+0000","last_fullsized":"2024-09-21T08:32:35.418269+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.291226+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.291226+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.18","version":"565'403","reported_seq":2257,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-09-21T08:32:38.814130+0000","last_change":"2024-09-21T08:32:38.814551+0000","last_active":"2024-09-21T08:32:38.814130+0000","last_peered":"2024-09-21T08:32:38.814130+0000","last_clean":"2024-09-21T08:32:38.814130+0000","last_became_active":"2024-09-21T08:31:56.651745+0000","last_became_peered":"2024-09-21T08:31:56.651745+0000","last_unstale":"2024-09-21T08:32:38.814130+0000","last_undegraded":"2024-09-21T08:32:38.814130+0000","last_fullsized":"2024-09-21T08:32:38.814130+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"558'389","last_scrub_stamp":"2024-09-21T08:32:33.518019+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:32:33.518019+0000","objects_scrubbed":14,"log_size":403,"log_dups_size":0,"ondisk_log_size":403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:47.940389+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0.0055929480000000004,"stat_sum":{"num_bytes":37082872,"num_objects":15,"num_object_clones":13,"num_object_copies":45,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":15,"num_whiteouts":0,"num_read":74,"num_read_kb":75207,"num_write":94,"num_write_kb":46946,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":33374951,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"b9","length":"11"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"7"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"3.1c","version":"0'0","reported_seq":1050,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419990+0000","last_change":"2024-09-21T08:31:34.816475+0000","last_active":"2024-09-21T08:32:35.419990+0000","last_peered":"2024-09-21T08:32:35.419990+0000","last_clean":"2024-09-21T08:32:35.419990+0000","last_became_active":"2024-09-21T08:30:56.627614+0000","last_became_peered":"2024-09-21T08:30:56.627614+0000","last_unstale":"2024-09-21T08:32:35.419990+0000","last_undegraded":"2024-09-21T08:32:35.419990+0000","last_fullsized":"2024-09-21T08:32:35.419990+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:34.816398+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:34.816398+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:35.624747+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1b","version":"73'5","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.804767+0000","last_change":"2024-09-21T08:32:35.750734+0000","last_active":"2024-09-21T08:32:35.750734+0000","last_peered":"2024-09-21T08:32:33.804767+0000","last_clean":"2024-09-21T08:32:33.804767+0000","last_became_active":"2024-09-21T08:31:57.658620+0000","last_became_peered":"2024-09-21T08:31:57.658620+0000","last_unstale":"2024-09-21T08:32:33.804767+0000","last_undegraded":"2024-09-21T08:32:33.804767+0000","last_fullsized":"2024-09-21T08:32:33.804767+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'5","last_scrub_stamp":"2024-09-21T08:27:26.356987+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:26.356987+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":6,"num_write":6,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.1a","version":"0'0","reported_seq":665,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-09-21T08:32:38.804495+0000","last_change":"2024-09-21T08:31:57.655236+0000","last_active":"2024-09-21T08:32:38.804495+0000","last_peered":"2024-09-21T08:32:38.804495+0000","last_clean":"2024-09-21T08:32:38.804495+0000","last_became_active":"2024-09-21T08:31:57.654911+0000","last_became_peered":"2024-09-21T08:31:57.654911+0000","last_unstale":"2024-09-21T08:32:38.804495+0000","last_undegraded":"2024-09-21T08:32:38.804495+0000","last_fullsized":"2024-09-21T08:32:38.804495+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.850373+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:09.782307+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.850373+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,1],"acting":[7,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.19","version":"73'1","reported_seq":1056,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424581+0000","last_change":"2024-09-21T08:30:04.034656+0000","last_active":"2024-09-21T08:32:35.424581+0000","last_peered":"2024-09-21T08:32:35.424581+0000","last_clean":"2024-09-21T08:32:35.424581+0000","last_became_active":"2024-09-21T08:28:04.705450+0000","last_became_peered":"2024-09-21T08:28:04.705450+0000","last_unstale":"2024-09-21T08:32:35.424581+0000","last_undegraded":"2024-09-21T08:32:35.424581+0000","last_fullsized":"2024-09-21T08:32:35.424581+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:30:04.034575+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:04.034575+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":13,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1f","version":"247'32","reported_seq":723,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789112+0000","last_change":"2024-09-21T08:32:33.784473+0000","last_active":"2024-09-21T08:32:34.789112+0000","last_peered":"2024-09-21T08:32:34.789112+0000","last_clean":"2024-09-21T08:32:34.789112+0000","last_became_active":"2024-09-21T08:30:56.617686+0000","last_became_peered":"2024-09-21T08:30:56.617686+0000","last_unstale":"2024-09-21T08:32:34.789112+0000","last_undegraded":"2024-09-21T08:32:34.789112+0000","last_fullsized":"2024-09-21T08:32:34.789112+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":5,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:27:51.078531+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.078531+0000","objects_scrubbed":0,"log_size":32,"log_dups_size":0,"ondisk_log_size":32,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00069403599999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,1],"acting":[4,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"6.1e","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788802+0000","last_change":"2024-09-21T08:31:57.646381+0000","last_active":"2024-09-21T08:32:34.788802+0000","last_peered":"2024-09-21T08:32:34.788802+0000","last_clean":"2024-09-21T08:32:34.788802+0000","last_became_active":"2024-09-21T08:31:57.645634+0000","last_became_peered":"2024-09-21T08:31:57.645634+0000","last_unstale":"2024-09-21T08:32:34.788802+0000","last_undegraded":"2024-09-21T08:32:34.788802+0000","last_fullsized":"2024-09-21T08:32:34.788802+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:35.812638+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:31:35.812638+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:54.761982+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788599+0000","last_change":"2024-09-21T08:31:57.652500+0000","last_active":"2024-09-21T08:32:34.788599+0000","last_peered":"2024-09-21T08:32:34.788599+0000","last_clean":"2024-09-21T08:32:34.788599+0000","last_became_active":"2024-09-21T08:31:57.652209+0000","last_became_peered":"2024-09-21T08:31:57.652209+0000","last_unstale":"2024-09-21T08:32:34.788599+0000","last_undegraded":"2024-09-21T08:32:34.788599+0000","last_fullsized":"2024-09-21T08:32:34.788599+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:32.768564+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:32.768564+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:58.372589+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1c","version":"73'15","reported_seq":1100,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421196+0000","last_change":"2024-09-21T08:30:56.660342+0000","last_active":"2024-09-21T08:32:35.421196+0000","last_peered":"2024-09-21T08:32:35.421196+0000","last_clean":"2024-09-21T08:32:35.421196+0000","last_became_active":"2024-09-21T08:30:56.630563+0000","last_became_peered":"2024-09-21T08:30:56.630563+0000","last_unstale":"2024-09-21T08:32:35.421196+0000","last_undegraded":"2024-09-21T08:32:35.421196+0000","last_fullsized":"2024-09-21T08:32:35.421196+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:29:37.352879+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:37.352879+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":35,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1d","version":"0'0","reported_seq":584,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.805858+0000","last_change":"2024-09-21T08:32:35.425504+0000","last_active":"2024-09-21T08:32:35.425504+0000","last_peered":"2024-09-21T08:32:33.805858+0000","last_clean":"2024-09-21T08:32:33.805858+0000","last_became_active":"2024-09-21T08:31:57.659333+0000","last_became_peered":"2024-09-21T08:31:57.659333+0000","last_unstale":"2024-09-21T08:32:33.805858+0000","last_undegraded":"2024-09-21T08:32:33.805858+0000","last_fullsized":"2024-09-21T08:32:33.805858+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:35.108794+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:35.108794+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.1e","version":"558'368","reported_seq":2654,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.420904+0000","last_change":"2024-09-21T08:32:33.771306+0000","last_active":"2024-09-21T08:32:35.420904+0000","last_peered":"2024-09-21T08:32:35.420904+0000","last_clean":"2024-09-21T08:32:35.420904+0000","last_became_active":"2024-09-21T08:30:59.606410+0000","last_became_peered":"2024-09-21T08:30:59.606410+0000","last_unstale":"2024-09-21T08:32:35.420904+0000","last_undegraded":"2024-09-21T08:32:35.420904+0000","last_fullsized":"2024-09-21T08:32:35.420904+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":5,"last_scrub":"294'236","last_scrub_stamp":"2024-09-21T08:28:09.464279+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.464279+0000","objects_scrubbed":2,"log_size":368,"log_dups_size":0,"ondisk_log_size":368,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":65,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":1.0005313790000001,"stat_sum":{"num_bytes":29513404,"num_objects":14,"num_object_clones":11,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":14,"num_whiteouts":1,"num_read":121,"num_read_kb":114857,"num_write":92,"num_write_kb":29655,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":69,"num_bytes_recovered":36880118,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"27"}]},{"pgid":"6.1f","version":"0'0","reported_seq":912,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424238+0000","last_change":"2024-09-21T08:30:37.732703+0000","last_active":"2024-09-21T08:32:35.424238+0000","last_peered":"2024-09-21T08:32:35.424238+0000","last_clean":"2024-09-21T08:32:35.424238+0000","last_became_active":"2024-09-21T08:28:04.725392+0000","last_became_peered":"2024-09-21T08:28:04.725392+0000","last_unstale":"2024-09-21T08:32:35.424238+0000","last_undegraded":"2024-09-21T08:32:35.424238+0000","last_fullsized":"2024-09-21T08:32:35.424238+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:37.732599+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:37.732599+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":215,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788598+0000","last_change":"2024-09-21T08:31:55.884676+0000","last_active":"2024-09-21T08:32:34.788598+0000","last_peered":"2024-09-21T08:32:34.788598+0000","last_clean":"2024-09-21T08:32:34.788598+0000","last_became_active":"2024-09-21T08:30:56.630058+0000","last_became_peered":"2024-09-21T08:30:56.630058+0000","last_unstale":"2024-09-21T08:32:34.788598+0000","last_undegraded":"2024-09-21T08:32:34.788598+0000","last_fullsized":"2024-09-21T08:32:34.788598+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:55.884606+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:55.884606+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:24.216176+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.1d","version":"73'12","reported_seq":590,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419380+0000","last_change":"2024-09-21T08:30:56.656296+0000","last_active":"2024-09-21T08:32:35.419380+0000","last_peered":"2024-09-21T08:32:35.419380+0000","last_clean":"2024-09-21T08:32:35.419380+0000","last_became_active":"2024-09-21T08:30:56.626214+0000","last_became_peered":"2024-09-21T08:30:56.626214+0000","last_unstale":"2024-09-21T08:32:35.419380+0000","last_undegraded":"2024-09-21T08:32:35.419380+0000","last_fullsized":"2024-09-21T08:32:35.419380+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:29:15.349124+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:15.349124+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1c","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788279+0000","last_change":"2024-09-21T08:31:57.640029+0000","last_active":"2024-09-21T08:32:34.788279+0000","last_peered":"2024-09-21T08:32:34.788279+0000","last_clean":"2024-09-21T08:32:34.788279+0000","last_became_active":"2024-09-21T08:31:57.639712+0000","last_became_peered":"2024-09-21T08:31:57.639712+0000","last_unstale":"2024-09-21T08:32:34.788279+0000","last_undegraded":"2024-09-21T08:32:34.788279+0000","last_fullsized":"2024-09-21T08:32:34.788279+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:09.116412+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:55.797084+0000","last_clean_scrub_stamp":"2024-09-21T08:28:09.116412+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1d","version":"248'84","reported_seq":958,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.420595+0000","last_change":"2024-09-21T08:32:33.768456+0000","last_active":"2024-09-21T08:32:35.420595+0000","last_peered":"2024-09-21T08:32:35.420595+0000","last_clean":"2024-09-21T08:32:35.420595+0000","last_became_active":"2024-09-21T08:31:56.656340+0000","last_became_peered":"2024-09-21T08:31:56.656340+0000","last_unstale":"2024-09-21T08:32:35.420595+0000","last_undegraded":"2024-09-21T08:32:35.420595+0000","last_fullsized":"2024-09-21T08:32:35.420595+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"248'84","last_scrub_stamp":"2024-09-21T08:28:12.352559+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.352559+0000","objects_scrubbed":0,"log_size":84,"log_dups_size":0,"ondisk_log_size":84,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":93,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.99689609899999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"b"}]},{"pgid":"6.1c","version":"73'1","reported_seq":554,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421209+0000","last_change":"2024-09-21T08:31:56.652728+0000","last_active":"2024-09-21T08:32:35.421209+0000","last_peered":"2024-09-21T08:32:35.421209+0000","last_clean":"2024-09-21T08:32:35.421209+0000","last_became_active":"2024-09-21T08:31:56.652400+0000","last_became_peered":"2024-09-21T08:31:56.652400+0000","last_unstale":"2024-09-21T08:32:35.421209+0000","last_undegraded":"2024-09-21T08:32:35.421209+0000","last_fullsized":"2024-09-21T08:32:35.421209+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'1","last_scrub_stamp":"2024-09-21T08:27:14.826047+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:14.826047+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":403,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"198'2","reported_seq":585,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.803279+0000","last_change":"2024-09-21T08:32:35.425342+0000","last_active":"2024-09-21T08:32:35.425342+0000","last_peered":"2024-09-21T08:32:33.803279+0000","last_clean":"2024-09-21T08:32:33.803279+0000","last_became_active":"2024-09-21T08:31:57.659593+0000","last_became_peered":"2024-09-21T08:31:57.659593+0000","last_unstale":"2024-09-21T08:32:33.803279+0000","last_undegraded":"2024-09-21T08:32:33.803279+0000","last_fullsized":"2024-09-21T08:32:33.803279+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"198'2","last_scrub_stamp":"2024-09-21T08:31:33.064915+0000","last_deep_scrub":"198'2","last_deep_scrub_stamp":"2024-09-21T08:26:32.819313+0000","last_clean_scrub_stamp":"2024-09-21T08:31:33.064915+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1e","version":"73'10","reported_seq":1075,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421658+0000","last_change":"2024-09-21T08:32:17.238877+0000","last_active":"2024-09-21T08:32:35.421658+0000","last_peered":"2024-09-21T08:32:35.421658+0000","last_clean":"2024-09-21T08:32:35.421658+0000","last_became_active":"2024-09-21T08:31:56.657752+0000","last_became_peered":"2024-09-21T08:31:56.657752+0000","last_unstale":"2024-09-21T08:32:35.421658+0000","last_undegraded":"2024-09-21T08:32:35.421658+0000","last_fullsized":"2024-09-21T08:32:35.421658+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:32:17.238789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:32:17.238789+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:42.604117+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.1f","version":"524'23","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:28:04.672702+0000","last_change":"2024-09-21T08:28:03.666627+0000","last_active":"2024-09-21T08:28:03.666627+0000","last_peered":"2024-09-21T08:28:02.650231+0000","last_clean":"2024-09-21T08:28:02.650231+0000","last_became_active":"2024-09-21T08:27:15.613508+0000","last_became_peered":"2024-09-21T08:27:15.613508+0000","last_unstale":"2024-09-21T08:28:04.672702+0000","last_undegraded":"2024-09-21T08:28:04.672702+0000","last_fullsized":"2024-09-21T08:28:04.672702+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"508'21","last_scrub_stamp":"2024-09-21T08:31:49.927821+0000","last_deep_scrub":"98'10","last_deep_scrub_stamp":"2024-09-21T08:24:51.594212+0000","last_clean_scrub_stamp":"2024-09-21T08:31:49.927821+0000","objects_scrubbed":1,"log_size":23,"log_dups_size":0,"ondisk_log_size":23,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.a","version":"76'129","reported_seq":1290,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428172+0000","last_change":"2024-09-21T08:31:07.783495+0000","last_active":"2024-09-21T08:32:35.428172+0000","last_peered":"2024-09-21T08:32:35.428172+0000","last_clean":"2024-09-21T08:32:35.428172+0000","last_became_active":"2024-09-21T08:30:59.600508+0000","last_became_peered":"2024-09-21T08:30:59.600508+0000","last_unstale":"2024-09-21T08:32:35.428172+0000","last_undegraded":"2024-09-21T08:32:35.428172+0000","last_fullsized":"2024-09-21T08:32:35.428172+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:07.783361+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:07.783361+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.c","version":"561'292","reported_seq":2676,"reported_epoch":561,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:32:35.739974+0000","last_change":"2024-09-21T08:32:35.739974+0000","last_active":"2024-09-21T08:32:35.739974+0000","last_peered":"2024-09-21T08:32:35.739974+0000","last_clean":"2024-09-21T08:32:35.739974+0000","last_became_active":"2024-09-21T08:31:07.701482+0000","last_became_peered":"2024-09-21T08:31:07.701482+0000","last_unstale":"2024-09-21T08:32:35.739974+0000","last_undegraded":"2024-09-21T08:32:35.739974+0000","last_fullsized":"2024-09-21T08:32:35.739974+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"446'254","last_scrub_stamp":"2024-09-21T08:30:46.778996+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:30:46.778996+0000","objects_scrubbed":4,"log_size":292,"log_dups_size":0,"ondisk_log_size":292,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.111565991,"stat_sum":{"num_bytes":8505355,"num_objects":6,"num_object_clones":5,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":1,"num_read":89,"num_read_kb":107464,"num_write":20,"num_write_kb":7973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":3117748,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"6"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"4.f","version":"73'15","reported_seq":606,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.422869+0000","last_change":"2024-09-21T08:31:57.660667+0000","last_active":"2024-09-21T08:32:35.422869+0000","last_peered":"2024-09-21T08:32:35.422869+0000","last_clean":"2024-09-21T08:32:35.422869+0000","last_became_active":"2024-09-21T08:31:57.660225+0000","last_became_peered":"2024-09-21T08:31:57.660225+0000","last_unstale":"2024-09-21T08:32:35.422869+0000","last_undegraded":"2024-09-21T08:32:35.422869+0000","last_fullsized":"2024-09-21T08:32:35.422869+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:31:36.097933+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:31:36.097933+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:52.132388+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":574,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420222+0000","last_change":"2024-09-21T08:31:57.653751+0000","last_active":"2024-09-21T08:32:35.420222+0000","last_peered":"2024-09-21T08:32:35.420222+0000","last_clean":"2024-09-21T08:32:35.420222+0000","last_became_active":"2024-09-21T08:31:57.653413+0000","last_became_peered":"2024-09-21T08:31:57.653413+0000","last_unstale":"2024-09-21T08:32:35.420222+0000","last_undegraded":"2024-09-21T08:32:35.420222+0000","last_fullsized":"2024-09-21T08:32:35.420222+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:32.091709+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:32.091709+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.e","version":"524'16","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:24:38.531908+0000","last_change":"2024-09-21T08:24:37.525208+0000","last_active":"2024-09-21T08:24:37.525208+0000","last_peered":"2024-09-21T08:23:37.752724+0000","last_clean":"2024-09-21T08:23:37.752724+0000","last_became_active":"2024-09-21T08:23:34.567963+0000","last_became_peered":"2024-09-21T08:23:34.567963+0000","last_unstale":"2024-09-21T08:24:38.531908+0000","last_undegraded":"2024-09-21T08:24:38.531908+0000","last_fullsized":"2024-09-21T08:24:38.531908+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"85'9","last_scrub_stamp":"2024-09-21T08:26:44.741275+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.741275+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":1,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,2],"acting":[4,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.d","version":"0'0","reported_seq":1032,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421860+0000","last_change":"2024-09-21T08:30:21.800509+0000","last_active":"2024-09-21T08:32:35.421860+0000","last_peered":"2024-09-21T08:32:35.421860+0000","last_clean":"2024-09-21T08:32:35.421860+0000","last_became_active":"2024-09-21T08:28:29.619967+0000","last_became_peered":"2024-09-21T08:28:29.619967+0000","last_unstale":"2024-09-21T08:32:35.421860+0000","last_undegraded":"2024-09-21T08:32:35.421860+0000","last_fullsized":"2024-09-21T08:32:35.421860+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:21.800427+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:21.800427+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.5","version":"76'129","reported_seq":989,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788815+0000","last_change":"2024-09-21T08:31:05.031131+0000","last_active":"2024-09-21T08:32:34.788815+0000","last_peered":"2024-09-21T08:32:34.788815+0000","last_clean":"2024-09-21T08:32:34.788815+0000","last_became_active":"2024-09-21T08:30:59.600613+0000","last_became_peered":"2024-09-21T08:30:59.600613+0000","last_unstale":"2024-09-21T08:32:34.788815+0000","last_undegraded":"2024-09-21T08:32:34.788815+0000","last_fullsized":"2024-09-21T08:32:34.788815+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:05.031027+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.031027+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.3","version":"179'19","reported_seq":2258,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420415+0000","last_change":"2024-09-21T08:32:33.781223+0000","last_active":"2024-09-21T08:32:35.420415+0000","last_peered":"2024-09-21T08:32:35.420415+0000","last_clean":"2024-09-21T08:32:35.420415+0000","last_became_active":"2024-09-21T08:31:57.641840+0000","last_became_peered":"2024-09-21T08:31:57.641840+0000","last_unstale":"2024-09-21T08:32:35.420415+0000","last_undegraded":"2024-09-21T08:32:35.420415+0000","last_fullsized":"2024-09-21T08:32:35.420415+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'19","last_scrub_stamp":"2024-09-21T08:31:05.695533+0000","last_deep_scrub":"179'19","last_deep_scrub_stamp":"2024-09-21T08:26:31.307776+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.695533+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00064436699999999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"4.0","version":"73'18","reported_seq":996,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419264+0000","last_change":"2024-09-21T08:31:56.650928+0000","last_active":"2024-09-21T08:32:35.419264+0000","last_peered":"2024-09-21T08:32:35.419264+0000","last_clean":"2024-09-21T08:32:35.419264+0000","last_became_active":"2024-09-21T08:31:56.650606+0000","last_became_peered":"2024-09-21T08:31:56.650606+0000","last_unstale":"2024-09-21T08:32:35.419264+0000","last_undegraded":"2024-09-21T08:32:35.419264+0000","last_fullsized":"2024-09-21T08:32:35.419264+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'18","last_scrub_stamp":"2024-09-21T08:26:54.003173+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:54.003173+0000","objects_scrubbed":0,"log_size":18,"log_dups_size":0,"ondisk_log_size":18,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":33,"num_read_kb":21,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":880,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":781,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421772+0000","last_change":"2024-09-21T08:31:57.661397+0000","last_active":"2024-09-21T08:32:35.421772+0000","last_peered":"2024-09-21T08:32:35.421772+0000","last_clean":"2024-09-21T08:32:35.421772+0000","last_became_active":"2024-09-21T08:31:57.659778+0000","last_became_peered":"2024-09-21T08:31:57.659778+0000","last_unstale":"2024-09-21T08:32:35.421772+0000","last_undegraded":"2024-09-21T08:32:35.421772+0000","last_fullsized":"2024-09-21T08:32:35.421772+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:52.711745+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:52.711745+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:56.745978+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788532+0000","last_change":"2024-09-21T08:31:57.659608+0000","last_active":"2024-09-21T08:32:34.788532+0000","last_peered":"2024-09-21T08:32:34.788532+0000","last_clean":"2024-09-21T08:32:34.788532+0000","last_became_active":"2024-09-21T08:31:57.659044+0000","last_became_peered":"2024-09-21T08:31:57.659044+0000","last_unstale":"2024-09-21T08:32:34.788532+0000","last_undegraded":"2024-09-21T08:32:34.788532+0000","last_fullsized":"2024-09-21T08:32:34.788532+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:32.813973+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:32.813973+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.2","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:31:57.637033+0000","last_change":"2024-09-21T08:31:57.637033+0000","last_active":"2024-09-21T08:26:49.527539+0000","last_peered":"2024-09-21T08:26:47.628755+0000","last_clean":"2024-09-21T08:26:47.628755+0000","last_became_active":"2024-09-21T08:24:56.501237+0000","last_became_peered":"2024-09-21T08:24:56.501237+0000","last_unstale":"2024-09-21T08:31:57.637033+0000","last_undegraded":"2024-09-21T08:31:57.637033+0000","last_fullsized":"2024-09-21T08:31:57.637033+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:14.680354+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:32:14.680354+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.4","version":"475'130","reported_seq":1309,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.791095+0000","last_change":"2024-09-21T08:32:35.419419+0000","last_active":"2024-09-21T08:32:35.419419+0000","last_peered":"2024-09-21T08:32:33.791095+0000","last_clean":"2024-09-21T08:32:33.791095+0000","last_became_active":"2024-09-21T08:31:57.662728+0000","last_became_peered":"2024-09-21T08:31:57.662728+0000","last_unstale":"2024-09-21T08:32:33.791095+0000","last_undegraded":"2024-09-21T08:32:33.791095+0000","last_fullsized":"2024-09-21T08:32:33.791095+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"475'130","last_scrub_stamp":"2024-09-21T08:31:08.776209+0000","last_deep_scrub":"475'130","last_deep_scrub_stamp":"2024-09-21T08:31:08.776209+0000","last_clean_scrub_stamp":"2024-09-21T08:31:08.776209+0000","objects_scrubbed":1,"log_size":130,"log_dups_size":0,"ondisk_log_size":130,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":327680,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.2","version":"560'388","reported_seq":2484,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.422948+0000","last_change":"2024-09-21T08:32:33.768779+0000","last_active":"2024-09-21T08:32:35.422948+0000","last_peered":"2024-09-21T08:32:35.422948+0000","last_clean":"2024-09-21T08:32:35.422948+0000","last_became_active":"2024-09-21T08:31:56.669347+0000","last_became_peered":"2024-09-21T08:31:56.669347+0000","last_unstale":"2024-09-21T08:32:35.422948+0000","last_undegraded":"2024-09-21T08:32:35.422948+0000","last_fullsized":"2024-09-21T08:32:35.422948+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"276'273","last_scrub_stamp":"2024-09-21T08:27:50.318620+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.318620+0000","objects_scrubbed":3,"log_size":388,"log_dups_size":0,"ondisk_log_size":388,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":95,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.99898741999999996,"stat_sum":{"num_bytes":29428059,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":0,"num_read":115,"num_read_kb":138148,"num_write":64,"num_write_kb":29897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":20331960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"9"}]},{"pgid":"4.1","version":"239'21","reported_seq":227,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788347+0000","last_change":"2024-09-21T08:31:57.658132+0000","last_active":"2024-09-21T08:32:34.788347+0000","last_peered":"2024-09-21T08:32:34.788347+0000","last_clean":"2024-09-21T08:32:34.788347+0000","last_became_active":"2024-09-21T08:31:57.657608+0000","last_became_peered":"2024-09-21T08:31:57.657608+0000","last_unstale":"2024-09-21T08:32:34.788347+0000","last_undegraded":"2024-09-21T08:32:34.788347+0000","last_fullsized":"2024-09-21T08:32:34.788347+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"239'21","last_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_deep_scrub":"239'21","last_deep_scrub_stamp":"2024-09-21T08:27:12.969642+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.969642+0000","objects_scrubbed":7,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":20,"num_read_kb":13,"num_write":14,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"475'2","reported_seq":1026,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421696+0000","last_change":"2024-09-21T08:31:57.784895+0000","last_active":"2024-09-21T08:32:35.421696+0000","last_peered":"2024-09-21T08:32:35.421696+0000","last_clean":"2024-09-21T08:32:35.421696+0000","last_became_active":"2024-09-21T08:31:57.659208+0000","last_became_peered":"2024-09-21T08:31:57.659208+0000","last_unstale":"2024-09-21T08:32:35.421696+0000","last_undegraded":"2024-09-21T08:32:35.421696+0000","last_fullsized":"2024-09-21T08:32:35.421696+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"475'2","last_scrub_stamp":"2024-09-21T08:31:09.109498+0000","last_deep_scrub":"475'2","last_deep_scrub_stamp":"2024-09-21T08:31:09.109498+0000","last_clean_scrub_stamp":"2024-09-21T08:31:09.109498+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:37.452049+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":46,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":184,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.0","version":"463'17","reported_seq":648,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419438+0000","last_change":"2024-09-21T08:32:08.235500+0000","last_active":"2024-09-21T08:32:35.419438+0000","last_peered":"2024-09-21T08:32:35.419438+0000","last_clean":"2024-09-21T08:32:35.419438+0000","last_became_active":"2024-09-21T08:30:56.630142+0000","last_became_peered":"2024-09-21T08:30:56.630142+0000","last_unstale":"2024-09-21T08:32:35.419438+0000","last_undegraded":"2024-09-21T08:32:35.419438+0000","last_fullsized":"2024-09-21T08:32:35.419438+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"463'17","last_scrub_stamp":"2024-09-21T08:32:08.235411+0000","last_deep_scrub":"263'14","last_deep_scrub_stamp":"2024-09-21T08:27:36.998538+0000","last_clean_scrub_stamp":"2024-09-21T08:32:08.235411+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:19.679841+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,2],"acting":[1,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.3","version":"0'0","reported_seq":1017,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420049+0000","last_change":"2024-09-21T08:31:56.650241+0000","last_active":"2024-09-21T08:32:35.420049+0000","last_peered":"2024-09-21T08:32:35.420049+0000","last_clean":"2024-09-21T08:32:35.420049+0000","last_became_active":"2024-09-21T08:31:56.649958+0000","last_became_peered":"2024-09-21T08:31:56.649958+0000","last_unstale":"2024-09-21T08:32:35.420049+0000","last_undegraded":"2024-09-21T08:32:35.420049+0000","last_fullsized":"2024-09-21T08:32:35.420049+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.344185+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.344185+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.7","version":"76'129","reported_seq":1296,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418971+0000","last_change":"2024-09-21T08:31:57.661231+0000","last_active":"2024-09-21T08:32:35.418971+0000","last_peered":"2024-09-21T08:32:35.418971+0000","last_clean":"2024-09-21T08:32:35.418971+0000","last_became_active":"2024-09-21T08:31:57.659273+0000","last_became_peered":"2024-09-21T08:31:57.659273+0000","last_unstale":"2024-09-21T08:32:35.418971+0000","last_undegraded":"2024-09-21T08:32:35.418971+0000","last_fullsized":"2024-09-21T08:32:35.418971+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:05.800069+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.800069+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.1","version":"179'43","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.803540+0000","last_change":"2024-09-21T08:32:35.750653+0000","last_active":"2024-09-21T08:32:35.750653+0000","last_peered":"2024-09-21T08:32:33.803540+0000","last_clean":"2024-09-21T08:32:33.803540+0000","last_became_active":"2024-09-21T08:31:57.643127+0000","last_became_peered":"2024-09-21T08:31:57.643127+0000","last_unstale":"2024-09-21T08:32:33.803540+0000","last_undegraded":"2024-09-21T08:32:33.803540+0000","last_fullsized":"2024-09-21T08:32:33.803540+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"179'43","last_scrub_stamp":"2024-09-21T08:27:50.158232+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:50.158232+0000","objects_scrubbed":0,"log_size":43,"log_dups_size":0,"ondisk_log_size":43,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.99984979799999996,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"1"},{"start":"de","length":"1"}]},{"pgid":"4.2","version":"73'10","reported_seq":626,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.804146+0000","last_change":"2024-09-21T08:32:35.423874+0000","last_active":"2024-09-21T08:32:35.423874+0000","last_peered":"2024-09-21T08:32:33.804146+0000","last_clean":"2024-09-21T08:32:33.804146+0000","last_became_active":"2024-09-21T08:31:40.666699+0000","last_became_peered":"2024-09-21T08:31:40.666699+0000","last_unstale":"2024-09-21T08:32:33.804146+0000","last_undegraded":"2024-09-21T08:32:33.804146+0000","last_fullsized":"2024-09-21T08:32:33.804146+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:29:58.300806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.300806+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":15,"num_read_kb":10,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,5],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"66'1","reported_seq":1065,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421600+0000","last_change":"2024-09-21T08:31:56.653594+0000","last_active":"2024-09-21T08:32:35.421600+0000","last_peered":"2024-09-21T08:32:35.421600+0000","last_clean":"2024-09-21T08:32:35.421600+0000","last_became_active":"2024-09-21T08:31:56.653060+0000","last_became_peered":"2024-09-21T08:31:56.653060+0000","last_unstale":"2024-09-21T08:32:35.421600+0000","last_undegraded":"2024-09-21T08:32:35.421600+0000","last_fullsized":"2024-09-21T08:32:35.421600+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:16.003225+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:16.003225+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":436,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.3","version":"523'17","reported_seq":851,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.425149+0000","last_change":"2024-09-21T08:31:56.650297+0000","last_active":"2024-09-21T08:32:35.425149+0000","last_peered":"2024-09-21T08:32:35.425149+0000","last_clean":"2024-09-21T08:32:35.425149+0000","last_became_active":"2024-09-21T08:31:56.649917+0000","last_became_peered":"2024-09-21T08:31:56.649917+0000","last_unstale":"2024-09-21T08:32:35.425149+0000","last_undegraded":"2024-09-21T08:32:35.425149+0000","last_fullsized":"2024-09-21T08:32:35.425149+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'13","last_scrub_stamp":"2024-09-21T08:27:27.874952+0000","last_deep_scrub":"176'11","last_deep_scrub_stamp":"2024-09-21T08:26:10.440547+0000","last_clean_scrub_stamp":"2024-09-21T08:27:27.874952+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.0","version":"0'0","reported_seq":862,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428543+0000","last_change":"2024-09-21T08:32:08.907632+0000","last_active":"2024-09-21T08:32:35.428543+0000","last_peered":"2024-09-21T08:32:35.428543+0000","last_clean":"2024-09-21T08:32:35.428543+0000","last_became_active":"2024-09-21T08:28:37.565624+0000","last_became_peered":"2024-09-21T08:28:37.565624+0000","last_unstale":"2024-09-21T08:32:35.428543+0000","last_undegraded":"2024-09-21T08:32:35.428543+0000","last_fullsized":"2024-09-21T08:32:35.428543+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:08.907565+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:32:08.907565+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:13.789820+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.6","version":"76'129","reported_seq":995,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788460+0000","last_change":"2024-09-21T08:31:57.646620+0000","last_active":"2024-09-21T08:32:34.788460+0000","last_peered":"2024-09-21T08:32:34.788460+0000","last_clean":"2024-09-21T08:32:34.788460+0000","last_became_active":"2024-09-21T08:31:57.646069+0000","last_became_peered":"2024-09-21T08:31:57.646069+0000","last_unstale":"2024-09-21T08:32:34.788460+0000","last_undegraded":"2024-09-21T08:32:34.788460+0000","last_fullsized":"2024-09-21T08:32:34.788460+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:06.980429+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:06.980429+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.0","version":"553'255","reported_seq":2701,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.429257+0000","last_change":"2024-09-21T08:32:33.742313+0000","last_active":"2024-09-21T08:32:35.429257+0000","last_peered":"2024-09-21T08:32:35.429257+0000","last_clean":"2024-09-21T08:32:35.429257+0000","last_became_active":"2024-09-21T08:28:07.549600+0000","last_became_peered":"2024-09-21T08:28:07.549600+0000","last_unstale":"2024-09-21T08:32:35.429257+0000","last_undegraded":"2024-09-21T08:32:35.429257+0000","last_fullsized":"2024-09-21T08:32:35.429257+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":294,"parent":"0.0","parent_split_bits":0,"last_scrub":"374'166","last_scrub_stamp":"2024-09-21T08:29:32.253921+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.253921+0000","objects_scrubbed":3,"log_size":255,"log_dups_size":0,"ondisk_log_size":255,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.010877980000000001,"stat_sum":{"num_bytes":10011867,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":76,"num_read_kb":95330,"num_write":34,"num_write_kb":12048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":1484609,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"6"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"4.3","version":"73'19","reported_seq":582,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.425358+0000","last_change":"2024-09-21T08:31:56.668001+0000","last_active":"2024-09-21T08:32:35.425358+0000","last_peered":"2024-09-21T08:32:35.425358+0000","last_clean":"2024-09-21T08:32:35.425358+0000","last_became_active":"2024-09-21T08:31:56.667700+0000","last_became_peered":"2024-09-21T08:31:56.667700+0000","last_unstale":"2024-09-21T08:32:35.425358+0000","last_undegraded":"2024-09-21T08:32:35.425358+0000","last_fullsized":"2024-09-21T08:32:35.425358+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:27:55.260127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:55.260127+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":38,"num_read_kb":24,"num_write":22,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":330,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":561,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421551+0000","last_change":"2024-09-21T08:30:46.074689+0000","last_active":"2024-09-21T08:32:35.421551+0000","last_peered":"2024-09-21T08:32:35.421551+0000","last_clean":"2024-09-21T08:32:35.421551+0000","last_became_active":"2024-09-21T08:28:04.723998+0000","last_became_peered":"2024-09-21T08:28:04.723998+0000","last_unstale":"2024-09-21T08:32:35.421551+0000","last_undegraded":"2024-09-21T08:32:35.421551+0000","last_fullsized":"2024-09-21T08:32:35.421551+0000","mapping_epoch":290,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":291,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:46.074613+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:30:46.074613+0000","last_clean_scrub_stamp":"2024-09-21T08:30:46.074613+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.2","version":"0'0","reported_seq":520,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428830+0000","last_change":"2024-09-21T08:32:31.942095+0000","last_active":"2024-09-21T08:32:35.428830+0000","last_peered":"2024-09-21T08:32:35.428830+0000","last_clean":"2024-09-21T08:32:35.428830+0000","last_became_active":"2024-09-21T08:31:56.653084+0000","last_became_peered":"2024-09-21T08:31:56.653084+0000","last_unstale":"2024-09-21T08:32:35.428830+0000","last_undegraded":"2024-09-21T08:32:35.428830+0000","last_fullsized":"2024-09-21T08:32:35.428830+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:31.941959+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:32:31.941959+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:42.430646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.1","version":"0'0","reported_seq":562,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421140+0000","last_change":"2024-09-21T08:31:56.643494+0000","last_active":"2024-09-21T08:32:35.421140+0000","last_peered":"2024-09-21T08:32:35.421140+0000","last_clean":"2024-09-21T08:32:35.421140+0000","last_became_active":"2024-09-21T08:31:56.642987+0000","last_became_peered":"2024-09-21T08:31:56.642987+0000","last_unstale":"2024-09-21T08:32:35.421140+0000","last_undegraded":"2024-09-21T08:32:35.421140+0000","last_fullsized":"2024-09-21T08:32:35.421140+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:46.263984+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:22.425389+0000","last_clean_scrub_stamp":"2024-09-21T08:26:46.263984+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"76'129","reported_seq":1289,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421780+0000","last_change":"2024-09-21T08:31:06.754191+0000","last_active":"2024-09-21T08:32:35.421780+0000","last_peered":"2024-09-21T08:32:35.421780+0000","last_clean":"2024-09-21T08:32:35.421780+0000","last_became_active":"2024-09-21T08:30:59.600493+0000","last_became_peered":"2024-09-21T08:30:59.600493+0000","last_unstale":"2024-09-21T08:32:35.421780+0000","last_undegraded":"2024-09-21T08:32:35.421780+0000","last_fullsized":"2024-09-21T08:32:35.421780+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:06.754121+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:06.754121+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.7","version":"558'239","reported_seq":2645,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.422009+0000","last_change":"2024-09-21T08:32:33.770566+0000","last_active":"2024-09-21T08:32:35.422009+0000","last_peered":"2024-09-21T08:32:35.422009+0000","last_clean":"2024-09-21T08:32:35.422009+0000","last_became_active":"2024-09-21T08:31:57.660183+0000","last_became_peered":"2024-09-21T08:31:57.660183+0000","last_unstale":"2024-09-21T08:32:35.422009+0000","last_undegraded":"2024-09-21T08:32:35.422009+0000","last_fullsized":"2024-09-21T08:32:35.422009+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"197'77","last_scrub_stamp":"2024-09-21T08:26:44.269615+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:44.269615+0000","objects_scrubbed":3,"log_size":239,"log_dups_size":0,"ondisk_log_size":239,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":63,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":1.000138626,"stat_sum":{"num_bytes":19575274,"num_objects":8,"num_object_clones":7,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":70,"num_read_kb":69410,"num_write":65,"num_write_kb":29412,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":48166099,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"1"}]},{"pgid":"4.4","version":"523'36","reported_seq":641,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421434+0000","last_change":"2024-09-21T08:31:56.651060+0000","last_active":"2024-09-21T08:32:35.421434+0000","last_peered":"2024-09-21T08:32:35.421434+0000","last_clean":"2024-09-21T08:32:35.421434+0000","last_became_active":"2024-09-21T08:31:56.650428+0000","last_became_peered":"2024-09-21T08:31:56.650428+0000","last_unstale":"2024-09-21T08:32:35.421434+0000","last_undegraded":"2024-09-21T08:32:35.421434+0000","last_fullsized":"2024-09-21T08:32:35.421434+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"217'32","last_scrub_stamp":"2024-09-21T08:27:11.407384+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:11.407384+0000","objects_scrubbed":10,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":358,"num_objects":10,"num_object_clones":0,"num_object_copies":30,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":10,"num_whiteouts":0,"num_read":47,"num_read_kb":32,"num_write":26,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":40,"num_bytes_recovered":1432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788521+0000","last_change":"2024-09-21T08:30:56.627449+0000","last_active":"2024-09-21T08:32:34.788521+0000","last_peered":"2024-09-21T08:32:34.788521+0000","last_clean":"2024-09-21T08:32:34.788521+0000","last_became_active":"2024-09-21T08:30:56.627283+0000","last_became_peered":"2024-09-21T08:30:56.627283+0000","last_unstale":"2024-09-21T08:32:34.788521+0000","last_undegraded":"2024-09-21T08:32:34.788521+0000","last_fullsized":"2024-09-21T08:32:34.788521+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:14.370256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.370256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.2","version":"68'2","reported_seq":1089,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428449+0000","last_change":"2024-09-21T08:31:56.672465+0000","last_active":"2024-09-21T08:32:35.428449+0000","last_peered":"2024-09-21T08:32:35.428449+0000","last_clean":"2024-09-21T08:32:35.428449+0000","last_became_active":"2024-09-21T08:31:56.672094+0000","last_became_peered":"2024-09-21T08:31:56.672094+0000","last_unstale":"2024-09-21T08:32:35.428449+0000","last_undegraded":"2024-09-21T08:32:35.428449+0000","last_fullsized":"2024-09-21T08:32:35.428449+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"68'2","last_scrub_stamp":"2024-09-21T08:26:02.760728+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:26:02.760728+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00044876600000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":114,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.5","version":"0'0","reported_seq":1017,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.805671+0000","last_change":"2024-09-21T08:32:35.423198+0000","last_active":"2024-09-21T08:32:35.423198+0000","last_peered":"2024-09-21T08:32:33.805671+0000","last_clean":"2024-09-21T08:32:33.805671+0000","last_became_active":"2024-09-21T08:31:57.642204+0000","last_became_peered":"2024-09-21T08:31:57.642204+0000","last_unstale":"2024-09-21T08:32:33.805671+0000","last_undegraded":"2024-09-21T08:32:33.805671+0000","last_fullsized":"2024-09-21T08:32:33.805671+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:37.057876+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:37.057876+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.6","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788986+0000","last_change":"2024-09-21T08:31:57.643047+0000","last_active":"2024-09-21T08:32:34.788986+0000","last_peered":"2024-09-21T08:32:34.788986+0000","last_clean":"2024-09-21T08:32:34.788986+0000","last_became_active":"2024-09-21T08:31:57.642670+0000","last_became_peered":"2024-09-21T08:31:57.642670+0000","last_unstale":"2024-09-21T08:32:34.788986+0000","last_undegraded":"2024-09-21T08:32:34.788986+0000","last_fullsized":"2024-09-21T08:32:34.788986+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:34.844729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:23.602376+0000","last_clean_scrub_stamp":"2024-09-21T08:28:34.844729+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":1,"reported_epoch":561,"state":"unknown","last_fresh":"2024-09-21T08:32:35.421255+0000","last_change":"2024-09-21T08:32:35.750693+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T08:32:35.421255+0000","last_undegraded":"2024-09-21T08:32:35.421255+0000","last_fullsized":"2024-09-21T08:32:35.421255+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":555,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:54.732571+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:54.732571+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.4","version":"0'0","reported_seq":1,"reported_epoch":561,"state":"unknown","last_fresh":"2024-09-21T08:32:35.424821+0000","last_change":"2024-09-21T08:32:35.750744+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T08:32:35.424821+0000","last_undegraded":"2024-09-21T08:32:35.424821+0000","last_fullsized":"2024-09-21T08:32:35.424821+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":510,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'161","last_scrub_stamp":"2024-09-21T08:31:09.054511+0000","last_deep_scrub":"180'50","last_deep_scrub_stamp":"2024-09-21T08:26:34.278834+0000","last_clean_scrub_stamp":"2024-09-21T08:31:09.054511+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.7","version":"73'13","reported_seq":600,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423140+0000","last_change":"2024-09-21T08:31:56.660682+0000","last_active":"2024-09-21T08:32:35.423140+0000","last_peered":"2024-09-21T08:32:35.423140+0000","last_clean":"2024-09-21T08:32:35.423140+0000","last_became_active":"2024-09-21T08:31:56.659742+0000","last_became_peered":"2024-09-21T08:31:56.659742+0000","last_unstale":"2024-09-21T08:32:35.423140+0000","last_undegraded":"2024-09-21T08:32:35.423140+0000","last_fullsized":"2024-09-21T08:32:35.423140+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'13","last_scrub_stamp":"2024-09-21T08:28:16.386950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.386950+0000","objects_scrubbed":5,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":330,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":29,"num_read_kb":18,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":567,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420131+0000","last_change":"2024-09-21T08:29:56.368212+0000","last_active":"2024-09-21T08:32:35.420131+0000","last_peered":"2024-09-21T08:32:35.420131+0000","last_clean":"2024-09-21T08:32:35.420131+0000","last_became_active":"2024-09-21T08:28:29.629320+0000","last_became_peered":"2024-09-21T08:28:29.629320+0000","last_unstale":"2024-09-21T08:32:35.420131+0000","last_undegraded":"2024-09-21T08:32:35.420131+0000","last_fullsized":"2024-09-21T08:32:35.420131+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:56.368144+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:29:56.368144+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"66'1","reported_seq":1034,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.425791+0000","last_change":"2024-09-21T08:30:56.650473+0000","last_active":"2024-09-21T08:32:35.425791+0000","last_peered":"2024-09-21T08:32:35.425791+0000","last_clean":"2024-09-21T08:32:35.425791+0000","last_became_active":"2024-09-21T08:30:56.629748+0000","last_became_peered":"2024-09-21T08:30:56.629748+0000","last_unstale":"2024-09-21T08:32:35.425791+0000","last_undegraded":"2024-09-21T08:32:35.425791+0000","last_fullsized":"2024-09-21T08:32:35.425791+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:27:59.262463+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:27:59.262463+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00073822700000000005,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,4],"acting":[1,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.6","version":"0'0","reported_seq":1018,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423407+0000","last_change":"2024-09-21T08:31:56.652762+0000","last_active":"2024-09-21T08:32:35.423407+0000","last_peered":"2024-09-21T08:32:35.423407+0000","last_clean":"2024-09-21T08:32:35.423407+0000","last_became_active":"2024-09-21T08:31:56.652434+0000","last_became_peered":"2024-09-21T08:31:56.652434+0000","last_unstale":"2024-09-21T08:32:35.423407+0000","last_undegraded":"2024-09-21T08:32:35.423407+0000","last_fullsized":"2024-09-21T08:32:35.423407+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.374076+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.374076+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.5","version":"0'0","reported_seq":916,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420568+0000","last_change":"2024-09-21T08:31:56.653969+0000","last_active":"2024-09-21T08:32:35.420568+0000","last_peered":"2024-09-21T08:32:35.420568+0000","last_clean":"2024-09-21T08:32:35.420568+0000","last_became_active":"2024-09-21T08:31:56.653507+0000","last_became_peered":"2024-09-21T08:31:56.653507+0000","last_unstale":"2024-09-21T08:32:35.420568+0000","last_undegraded":"2024-09-21T08:32:35.420568+0000","last_fullsized":"2024-09-21T08:32:35.420568+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:08.831386+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:27:08.831386+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.3","version":"76'129","reported_seq":1298,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423562+0000","last_change":"2024-09-21T08:31:05.106966+0000","last_active":"2024-09-21T08:32:35.423562+0000","last_peered":"2024-09-21T08:32:35.423562+0000","last_clean":"2024-09-21T08:32:35.423562+0000","last_became_active":"2024-09-21T08:30:59.605942+0000","last_became_peered":"2024-09-21T08:30:59.605942+0000","last_unstale":"2024-09-21T08:32:35.423562+0000","last_undegraded":"2024-09-21T08:32:35.423562+0000","last_fullsized":"2024-09-21T08:32:35.423562+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":3,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:05.106891+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:05.106891+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.5","version":"462'307","reported_seq":2310,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.420777+0000","last_change":"2024-09-21T08:32:33.768029+0000","last_active":"2024-09-21T08:32:35.420777+0000","last_peered":"2024-09-21T08:32:35.420777+0000","last_clean":"2024-09-21T08:32:35.420777+0000","last_became_active":"2024-09-21T08:31:56.651630+0000","last_became_peered":"2024-09-21T08:31:56.651630+0000","last_unstale":"2024-09-21T08:32:35.420777+0000","last_undegraded":"2024-09-21T08:32:35.420777+0000","last_fullsized":"2024-09-21T08:32:35.420777+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"180'245","last_scrub_stamp":"2024-09-21T08:26:30.864040+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:30.864040+0000","objects_scrubbed":0,"log_size":307,"log_dups_size":0,"ondisk_log_size":307,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":92,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99868668299999996,"stat_sum":{"num_bytes":4186220,"num_objects":2,"num_object_clones":2,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":24,"num_read_kb":25509,"num_write":17,"num_write_kb":5036,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":926989,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"c"}]},{"pgid":"4.6","version":"73'12","reported_seq":1027,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421756+0000","last_change":"2024-09-21T08:31:56.653976+0000","last_active":"2024-09-21T08:32:35.421756+0000","last_peered":"2024-09-21T08:32:35.421756+0000","last_clean":"2024-09-21T08:32:35.421756+0000","last_became_active":"2024-09-21T08:31:56.653464+0000","last_became_peered":"2024-09-21T08:31:56.653464+0000","last_unstale":"2024-09-21T08:32:35.421756+0000","last_undegraded":"2024-09-21T08:32:35.421756+0000","last_fullsized":"2024-09-21T08:32:35.421756+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:26:07.384640+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:26:07.384640+0000","objects_scrubbed":6,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":18,"num_read_kb":12,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:31:57.618681+0000","last_change":"2024-09-21T08:31:56.619363+0000","last_active":"2024-09-21T08:31:55.638838+0000","last_peered":"2024-09-21T08:31:54.608441+0000","last_clean":"2024-09-21T08:31:54.608441+0000","last_became_active":"2024-09-21T08:31:41.652019+0000","last_became_peered":"2024-09-21T08:31:41.652019+0000","last_unstale":"2024-09-21T08:31:57.618681+0000","last_undegraded":"2024-09-21T08:31:57.618681+0000","last_fullsized":"2024-09-21T08:31:57.618681+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:29.917891+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:29.917891+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,7],"acting":[4,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.0","version":"74'5","reported_seq":1590,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418699+0000","last_change":"2024-09-21T08:31:56.649841+0000","last_active":"2024-09-21T08:32:35.418699+0000","last_peered":"2024-09-21T08:32:35.418699+0000","last_clean":"2024-09-21T08:32:35.418699+0000","last_became_active":"2024-09-21T08:31:56.649459+0000","last_became_peered":"2024-09-21T08:31:56.649459+0000","last_unstale":"2024-09-21T08:32:35.418699+0000","last_undegraded":"2024-09-21T08:32:35.418699+0000","last_fullsized":"2024-09-21T08:32:35.418699+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":64,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"74'5","last_scrub_stamp":"2024-09-21T08:24:47.740462+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:28.509258+0000","last_clean_scrub_stamp":"2024-09-21T08:24:47.740462+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00054783499999999999,"stat_sum":{"num_bytes":389,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":521,"num_read_kb":516,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1556,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.7","version":"0'0","reported_seq":1037,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423787+0000","last_change":"2024-09-21T08:31:55.771472+0000","last_active":"2024-09-21T08:32:35.423787+0000","last_peered":"2024-09-21T08:32:35.423787+0000","last_clean":"2024-09-21T08:32:35.423787+0000","last_became_active":"2024-09-21T08:30:56.626598+0000","last_became_peered":"2024-09-21T08:30:56.626598+0000","last_unstale":"2024-09-21T08:32:35.423787+0000","last_undegraded":"2024-09-21T08:32:35.423787+0000","last_fullsized":"2024-09-21T08:32:35.423787+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:55.771397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:55.771397+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:22.016240+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.4","version":"0'0","reported_seq":567,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421516+0000","last_change":"2024-09-21T08:30:56.638995+0000","last_active":"2024-09-21T08:32:35.421516+0000","last_peered":"2024-09-21T08:32:35.421516+0000","last_clean":"2024-09-21T08:32:35.421516+0000","last_became_active":"2024-09-21T08:30:56.638712+0000","last_became_peered":"2024-09-21T08:30:56.638712+0000","last_unstale":"2024-09-21T08:32:35.421516+0000","last_undegraded":"2024-09-21T08:32:35.421516+0000","last_fullsized":"2024-09-21T08:32:35.421516+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:17.424515+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:17.424515+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.6","version":"563'196","reported_seq":2070,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-09-21T08:32:38.807355+0000","last_change":"2024-09-21T08:32:38.808374+0000","last_active":"2024-09-21T08:32:38.807355+0000","last_peered":"2024-09-21T08:32:38.807355+0000","last_clean":"2024-09-21T08:32:38.807355+0000","last_became_active":"2024-09-21T08:31:56.655654+0000","last_became_peered":"2024-09-21T08:31:56.655654+0000","last_unstale":"2024-09-21T08:32:38.807355+0000","last_undegraded":"2024-09-21T08:32:38.807355+0000","last_fullsized":"2024-09-21T08:32:38.807355+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"529'181","last_scrub_stamp":"2024-09-21T08:32:09.635480+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:32:09.635480+0000","objects_scrubbed":5,"log_size":196,"log_dups_size":0,"ondisk_log_size":196,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:29.902673+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0012405179999999999,"stat_sum":{"num_bytes":7204516,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,2],"acting":[7,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"b9","length":"11"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"7"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"4.5","version":"524'26","reported_seq":253,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789137+0000","last_change":"2024-09-21T08:31:57.641141+0000","last_active":"2024-09-21T08:32:34.789137+0000","last_peered":"2024-09-21T08:32:34.789137+0000","last_clean":"2024-09-21T08:32:34.789137+0000","last_became_active":"2024-09-21T08:31:57.640192+0000","last_became_peered":"2024-09-21T08:31:57.640192+0000","last_unstale":"2024-09-21T08:32:34.789137+0000","last_undegraded":"2024-09-21T08:32:34.789137+0000","last_fullsized":"2024-09-21T08:32:34.789137+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"242'20","last_scrub_stamp":"2024-09-21T08:28:13.956509+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.956509+0000","objects_scrubbed":8,"log_size":26,"log_dups_size":0,"ondisk_log_size":26,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":154,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":24,"num_read_kb":14,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":926,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.429244+0000","last_change":"2024-09-21T08:30:56.629168+0000","last_active":"2024-09-21T08:32:35.429244+0000","last_peered":"2024-09-21T08:32:35.429244+0000","last_clean":"2024-09-21T08:32:35.429244+0000","last_became_active":"2024-09-21T08:30:56.628829+0000","last_became_peered":"2024-09-21T08:30:56.628829+0000","last_unstale":"2024-09-21T08:32:35.429244+0000","last_undegraded":"2024-09-21T08:32:35.429244+0000","last_fullsized":"2024-09-21T08:32:35.429244+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.045353+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.045353+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.0","version":"76'129","reported_seq":1289,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418371+0000","last_change":"2024-09-21T08:31:57.164805+0000","last_active":"2024-09-21T08:32:35.418371+0000","last_peered":"2024-09-21T08:32:35.418371+0000","last_clean":"2024-09-21T08:32:35.418371+0000","last_became_active":"2024-09-21T08:31:56.634604+0000","last_became_peered":"2024-09-21T08:31:56.634604+0000","last_unstale":"2024-09-21T08:32:35.418371+0000","last_undegraded":"2024-09-21T08:32:35.418371+0000","last_fullsized":"2024-09-21T08:32:35.418371+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:57.164726+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:57.164726+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:21.079821+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.4","version":"0'0","reported_seq":495,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421123+0000","last_change":"2024-09-21T08:31:56.642880+0000","last_active":"2024-09-21T08:32:35.421123+0000","last_peered":"2024-09-21T08:32:35.421123+0000","last_clean":"2024-09-21T08:32:35.421123+0000","last_became_active":"2024-09-21T08:31:56.642484+0000","last_became_peered":"2024-09-21T08:31:56.642484+0000","last_unstale":"2024-09-21T08:32:35.421123+0000","last_undegraded":"2024-09-21T08:32:35.421123+0000","last_fullsized":"2024-09-21T08:32:35.421123+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:25:41.700570+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:41.700570+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.7","version":"0'0","reported_seq":1051,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423975+0000","last_change":"2024-09-21T08:31:31.838452+0000","last_active":"2024-09-21T08:32:35.423975+0000","last_peered":"2024-09-21T08:32:35.423975+0000","last_clean":"2024-09-21T08:32:35.423975+0000","last_became_active":"2024-09-21T08:30:56.627007+0000","last_became_peered":"2024-09-21T08:30:56.627007+0000","last_unstale":"2024-09-21T08:32:35.423975+0000","last_undegraded":"2024-09-21T08:32:35.423975+0000","last_fullsized":"2024-09-21T08:32:35.423975+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:31.838383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:31:31.838383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:55.776066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.d","version":"558'169","reported_seq":1112,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.425103+0000","last_change":"2024-09-21T08:32:33.769666+0000","last_active":"2024-09-21T08:32:35.425103+0000","last_peered":"2024-09-21T08:32:35.425103+0000","last_clean":"2024-09-21T08:32:35.425103+0000","last_became_active":"2024-09-21T08:31:56.658567+0000","last_became_peered":"2024-09-21T08:31:56.658567+0000","last_unstale":"2024-09-21T08:32:35.425103+0000","last_undegraded":"2024-09-21T08:32:35.425103+0000","last_fullsized":"2024-09-21T08:32:35.425103+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"294'93","last_scrub_stamp":"2024-09-21T08:28:11.373358+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:11.373358+0000","objects_scrubbed":3,"log_size":169,"log_dups_size":0,"ondisk_log_size":169,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":98,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.99780659100000002,"stat_sum":{"num_bytes":10254749,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":25,"num_read_kb":23716,"num_write":27,"num_write_kb":7172,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":25893549,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"6"}]},{"pgid":"4.e","version":"73'11","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788902+0000","last_change":"2024-09-21T08:30:56.656178+0000","last_active":"2024-09-21T08:32:34.788902+0000","last_peered":"2024-09-21T08:32:34.788902+0000","last_clean":"2024-09-21T08:32:34.788902+0000","last_became_active":"2024-09-21T08:30:56.626093+0000","last_became_peered":"2024-09-21T08:30:56.626093+0000","last_unstale":"2024-09-21T08:32:34.788902+0000","last_undegraded":"2024-09-21T08:32:34.788902+0000","last_fullsized":"2024-09-21T08:32:34.788902+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:28:14.416729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.416729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788721+0000","last_change":"2024-09-21T08:31:57.659564+0000","last_active":"2024-09-21T08:32:34.788721+0000","last_peered":"2024-09-21T08:32:34.788721+0000","last_clean":"2024-09-21T08:32:34.788721+0000","last_became_active":"2024-09-21T08:31:57.658983+0000","last_became_peered":"2024-09-21T08:31:57.658983+0000","last_unstale":"2024-09-21T08:32:34.788721+0000","last_undegraded":"2024-09-21T08:32:34.788721+0000","last_fullsized":"2024-09-21T08:32:34.788721+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:28.347474+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:28.347474+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.f","version":"0'0","reported_seq":1051,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424069+0000","last_change":"2024-09-21T08:30:56.619286+0000","last_active":"2024-09-21T08:32:35.424069+0000","last_peered":"2024-09-21T08:32:35.424069+0000","last_clean":"2024-09-21T08:32:35.424069+0000","last_became_active":"2024-09-21T08:30:56.618484+0000","last_became_peered":"2024-09-21T08:30:56.618484+0000","last_unstale":"2024-09-21T08:32:35.424069+0000","last_undegraded":"2024-09-21T08:32:35.424069+0000","last_fullsized":"2024-09-21T08:32:35.424069+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:55.150058+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:32.961381+0000","last_clean_scrub_stamp":"2024-09-21T08:29:55.150058+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,2],"acting":[5,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.c","version":"0'0","reported_seq":905,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421306+0000","last_change":"2024-09-21T08:31:56.650787+0000","last_active":"2024-09-21T08:32:35.421306+0000","last_peered":"2024-09-21T08:32:35.421306+0000","last_clean":"2024-09-21T08:32:35.421306+0000","last_became_active":"2024-09-21T08:31:56.650460+0000","last_became_peered":"2024-09-21T08:31:56.650460+0000","last_unstale":"2024-09-21T08:32:35.421306+0000","last_undegraded":"2024-09-21T08:32:35.421306+0000","last_fullsized":"2024-09-21T08:32:35.421306+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:27:12.977956+0000","last_clean_scrub_stamp":"2024-09-21T08:27:12.977956+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"1.8","version":"76'129","reported_seq":1294,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418682+0000","last_change":"2024-09-21T08:31:09.810905+0000","last_active":"2024-09-21T08:32:35.418682+0000","last_peered":"2024-09-21T08:32:35.418682+0000","last_clean":"2024-09-21T08:32:35.418682+0000","last_became_active":"2024-09-21T08:30:59.604289+0000","last_became_peered":"2024-09-21T08:30:59.604289+0000","last_unstale":"2024-09-21T08:32:35.418682+0000","last_undegraded":"2024-09-21T08:32:35.418682+0000","last_fullsized":"2024-09-21T08:32:35.418682+0000","mapping_epoch":465,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":466,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:09.810830+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:09.810830+0000","objects_scrubbed":1,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":262688,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.e","version":"256'199","reported_seq":2572,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423004+0000","last_change":"2024-09-21T08:32:33.781203+0000","last_active":"2024-09-21T08:32:35.423004+0000","last_peered":"2024-09-21T08:32:35.423004+0000","last_clean":"2024-09-21T08:32:35.423004+0000","last_became_active":"2024-09-21T08:28:32.558440+0000","last_became_peered":"2024-09-21T08:28:32.558440+0000","last_unstale":"2024-09-21T08:32:35.423004+0000","last_undegraded":"2024-09-21T08:32:35.423004+0000","last_fullsized":"2024-09-21T08:32:35.423004+0000","mapping_epoch":318,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":319,"parent":"0.0","parent_split_bits":0,"last_scrub":"256'199","last_scrub_stamp":"2024-09-21T08:31:26.832551+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:26.832551+0000","objects_scrubbed":0,"log_size":199,"log_dups_size":0,"ondisk_log_size":199,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00066135800000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,1],"acting":[2,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"4.d","version":"345'25","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.806029+0000","last_change":"2024-09-21T08:32:35.750762+0000","last_active":"2024-09-21T08:32:35.750762+0000","last_peered":"2024-09-21T08:32:33.806029+0000","last_clean":"2024-09-21T08:32:33.806029+0000","last_became_active":"2024-09-21T08:31:40.667724+0000","last_became_peered":"2024-09-21T08:31:40.667724+0000","last_unstale":"2024-09-21T08:32:33.806029+0000","last_undegraded":"2024-09-21T08:32:33.806029+0000","last_fullsized":"2024-09-21T08:32:33.806029+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"345'25","last_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_deep_scrub":"345'25","last_deep_scrub_stamp":"2024-09-21T08:28:59.241931+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.241931+0000","objects_scrubbed":8,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":8,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":28,"num_read_kb":18,"num_write":18,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32,"num_bytes_recovered":440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789186+0000","last_change":"2024-09-21T08:30:56.631627+0000","last_active":"2024-09-21T08:32:34.789186+0000","last_peered":"2024-09-21T08:32:34.789186+0000","last_clean":"2024-09-21T08:32:34.789186+0000","last_became_active":"2024-09-21T08:30:56.630646+0000","last_became_peered":"2024-09-21T08:30:56.630646+0000","last_unstale":"2024-09-21T08:32:34.789186+0000","last_undegraded":"2024-09-21T08:32:34.789186+0000","last_fullsized":"2024-09-21T08:32:34.789186+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:30.909902+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.909902+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.c","version":"0'0","reported_seq":580,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421334+0000","last_change":"2024-09-21T08:31:57.658547+0000","last_active":"2024-09-21T08:32:35.421334+0000","last_peered":"2024-09-21T08:32:35.421334+0000","last_clean":"2024-09-21T08:32:35.421334+0000","last_became_active":"2024-09-21T08:31:57.658222+0000","last_became_peered":"2024-09-21T08:31:57.658222+0000","last_unstale":"2024-09-21T08:32:35.421334+0000","last_undegraded":"2024-09-21T08:32:35.421334+0000","last_fullsized":"2024-09-21T08:32:35.421334+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:08.138148+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:08.138148+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.f","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789179+0000","last_change":"2024-09-21T08:30:56.617373+0000","last_active":"2024-09-21T08:32:34.789179+0000","last_peered":"2024-09-21T08:32:34.789179+0000","last_clean":"2024-09-21T08:32:34.789179+0000","last_became_active":"2024-09-21T08:30:56.617090+0000","last_became_peered":"2024-09-21T08:30:56.617090+0000","last_unstale":"2024-09-21T08:32:34.789179+0000","last_undegraded":"2024-09-21T08:32:34.789179+0000","last_fullsized":"2024-09-21T08:32:34.789179+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:00.276774+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:00.276774+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.9","version":"76'129","reported_seq":989,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418204+0000","last_change":"2024-09-21T08:31:57.659835+0000","last_active":"2024-09-21T08:32:35.418204+0000","last_peered":"2024-09-21T08:32:35.418204+0000","last_clean":"2024-09-21T08:32:35.418204+0000","last_became_active":"2024-09-21T08:31:57.659334+0000","last_became_peered":"2024-09-21T08:31:57.659334+0000","last_unstale":"2024-09-21T08:32:35.418204+0000","last_undegraded":"2024-09-21T08:32:35.418204+0000","last_fullsized":"2024-09-21T08:32:35.418204+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":455,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":4,"last_scrub":"76'129","last_scrub_stamp":"2024-09-21T08:31:06.742324+0000","last_deep_scrub":"76'129","last_deep_scrub_stamp":"2024-09-21T08:24:05.765804+0000","last_clean_scrub_stamp":"2024-09-21T08:31:06.742324+0000","objects_scrubbed":0,"log_size":129,"log_dups_size":0,"ondisk_log_size":129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,5],"acting":[2,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.f","version":"558'124","reported_seq":829,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788993+0000","last_change":"2024-09-21T08:32:33.784397+0000","last_active":"2024-09-21T08:32:34.788993+0000","last_peered":"2024-09-21T08:32:34.788993+0000","last_clean":"2024-09-21T08:32:34.788993+0000","last_became_active":"2024-09-21T08:31:57.647290+0000","last_became_peered":"2024-09-21T08:31:57.647290+0000","last_unstale":"2024-09-21T08:32:34.788993+0000","last_undegraded":"2024-09-21T08:32:34.788993+0000","last_fullsized":"2024-09-21T08:32:34.788993+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"247'32","last_scrub_stamp":"2024-09-21T08:28:10.372149+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:10.372149+0000","objects_scrubbed":1,"log_size":124,"log_dups_size":0,"ondisk_log_size":124,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00074721300000000004,"stat_sum":{"num_bytes":7412283,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":1,"num_read":30,"num_read_kb":28275,"num_write":45,"num_write_kb":17501,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":34,"num_bytes_recovered":30320741,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"4.c","version":"73'10","reported_seq":231,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788257+0000","last_change":"2024-09-21T08:32:13.710827+0000","last_active":"2024-09-21T08:32:34.788257+0000","last_peered":"2024-09-21T08:32:34.788257+0000","last_clean":"2024-09-21T08:32:34.788257+0000","last_became_active":"2024-09-21T08:31:57.652431+0000","last_became_peered":"2024-09-21T08:31:57.652431+0000","last_unstale":"2024-09-21T08:32:34.788257+0000","last_undegraded":"2024-09-21T08:32:34.788257+0000","last_fullsized":"2024-09-21T08:32:34.788257+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'10","last_scrub_stamp":"2024-09-21T08:32:13.710699+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:32:13.710699+0000","objects_scrubbed":5,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:19.044630+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":15,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788230+0000","last_change":"2024-09-21T08:31:57.651745+0000","last_active":"2024-09-21T08:32:34.788230+0000","last_peered":"2024-09-21T08:32:34.788230+0000","last_clean":"2024-09-21T08:32:34.788230+0000","last_became_active":"2024-09-21T08:31:57.651452+0000","last_became_peered":"2024-09-21T08:31:57.651452+0000","last_unstale":"2024-09-21T08:32:34.788230+0000","last_undegraded":"2024-09-21T08:32:34.788230+0000","last_fullsized":"2024-09-21T08:32:34.788230+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:54.216528+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:54.216528+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.d","version":"523'11","reported_seq":568,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421894+0000","last_change":"2024-09-21T08:31:56.669546+0000","last_active":"2024-09-21T08:32:35.421894+0000","last_peered":"2024-09-21T08:32:35.421894+0000","last_clean":"2024-09-21T08:32:35.421894+0000","last_became_active":"2024-09-21T08:31:56.668783+0000","last_became_peered":"2024-09-21T08:31:56.668783+0000","last_unstale":"2024-09-21T08:32:35.421894+0000","last_undegraded":"2024-09-21T08:32:35.421894+0000","last_fullsized":"2024-09-21T08:32:35.421894+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:27:51.312273+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:27:51.312273+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.e","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788696+0000","last_change":"2024-09-21T08:30:56.627132+0000","last_active":"2024-09-21T08:32:34.788696+0000","last_peered":"2024-09-21T08:32:34.788696+0000","last_clean":"2024-09-21T08:32:34.788696+0000","last_became_active":"2024-09-21T08:30:56.626799+0000","last_became_peered":"2024-09-21T08:30:56.626799+0000","last_unstale":"2024-09-21T08:32:34.788696+0000","last_undegraded":"2024-09-21T08:32:34.788696+0000","last_fullsized":"2024-09-21T08:32:34.788696+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:40.458054+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:40.458054+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2],"acting":[4,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.8","version":"472'386","reported_seq":2733,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789483+0000","last_change":"2024-09-21T08:32:33.782483+0000","last_active":"2024-09-21T08:32:34.789483+0000","last_peered":"2024-09-21T08:32:34.789483+0000","last_clean":"2024-09-21T08:32:34.789483+0000","last_became_active":"2024-09-21T08:31:57.644069+0000","last_became_peered":"2024-09-21T08:31:57.644069+0000","last_unstale":"2024-09-21T08:32:34.789483+0000","last_undegraded":"2024-09-21T08:32:34.789483+0000","last_fullsized":"2024-09-21T08:32:34.789483+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"472'386","last_scrub_stamp":"2024-09-21T08:31:08.026519+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:08.026519+0000","objects_scrubbed":0,"log_size":386,"log_dups_size":0,"ondisk_log_size":386,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.00055346200000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,7],"acting":[4,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"4.b","version":"73'9","reported_seq":1098,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421430+0000","last_change":"2024-09-21T08:31:25.139313+0000","last_active":"2024-09-21T08:32:35.421430+0000","last_peered":"2024-09-21T08:32:35.421430+0000","last_clean":"2024-09-21T08:32:35.421430+0000","last_became_active":"2024-09-21T08:30:56.626474+0000","last_became_peered":"2024-09-21T08:30:56.626474+0000","last_unstale":"2024-09-21T08:32:35.421430+0000","last_undegraded":"2024-09-21T08:32:35.421430+0000","last_fullsized":"2024-09-21T08:32:35.421430+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:31:25.139208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:31:25.139208+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:52.945547+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":770,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":1060,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419840+0000","last_change":"2024-09-21T08:28:59.403264+0000","last_active":"2024-09-21T08:32:35.419840+0000","last_peered":"2024-09-21T08:32:35.419840+0000","last_clean":"2024-09-21T08:32:35.419840+0000","last_became_active":"2024-09-21T08:28:29.609526+0000","last_became_peered":"2024-09-21T08:28:29.609526+0000","last_unstale":"2024-09-21T08:32:35.419840+0000","last_undegraded":"2024-09-21T08:32:35.419840+0000","last_fullsized":"2024-09-21T08:32:35.419840+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:59.403194+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:59.403194+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.a","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.803395+0000","last_change":"2024-09-21T08:32:35.750678+0000","last_active":"2024-09-21T08:32:35.750678+0000","last_peered":"2024-09-21T08:32:33.803395+0000","last_clean":"2024-09-21T08:32:33.803395+0000","last_became_active":"2024-09-21T08:31:57.658412+0000","last_became_peered":"2024-09-21T08:31:57.658412+0000","last_unstale":"2024-09-21T08:32:33.803395+0000","last_undegraded":"2024-09-21T08:32:33.803395+0000","last_fullsized":"2024-09-21T08:32:33.803395+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:51.177738+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:51.177738+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,1],"acting":[4,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.9","version":"0'0","reported_seq":518,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418936+0000","last_change":"2024-09-21T08:31:56.649066+0000","last_active":"2024-09-21T08:32:35.418936+0000","last_peered":"2024-09-21T08:32:35.418936+0000","last_clean":"2024-09-21T08:32:35.418936+0000","last_became_active":"2024-09-21T08:31:56.648780+0000","last_became_peered":"2024-09-21T08:31:56.648780+0000","last_unstale":"2024-09-21T08:32:35.418936+0000","last_undegraded":"2024-09-21T08:32:35.418936+0000","last_fullsized":"2024-09-21T08:32:35.418936+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:36.308161+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:36.308161+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7,1],"acting":[2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.9","version":"560'442","reported_seq":2940,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:35.426216+0000","last_change":"2024-09-21T08:32:35.426828+0000","last_active":"2024-09-21T08:32:35.426192+0000","last_peered":"2024-09-21T08:32:33.826749+0000","last_clean":"2024-09-21T08:32:33.826749+0000","last_became_active":"2024-09-21T08:31:57.658394+0000","last_became_peered":"2024-09-21T08:31:57.658394+0000","last_unstale":"2024-09-21T08:32:35.426216+0000","last_undegraded":"2024-09-21T08:32:35.426216+0000","last_fullsized":"2024-09-21T08:32:35.426216+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"194'169","last_scrub_stamp":"2024-09-21T08:26:31.317302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:31.317302+0000","objects_scrubbed":4,"log_size":442,"log_dups_size":0,"ondisk_log_size":442,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.99822385599999996,"stat_sum":{"num_bytes":24497846,"num_objects":13,"num_object_clones":11,"num_object_copies":39,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":13,"num_whiteouts":1,"num_read":131,"num_read_kb":146962,"num_write":96,"num_write_kb":30729,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":81,"num_bytes_recovered":51049312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,7],"acting":[1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2,4,7],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"24"}]},{"pgid":"4.a","version":"73'19","reported_seq":582,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.422746+0000","last_change":"2024-09-21T08:31:56.659733+0000","last_active":"2024-09-21T08:32:35.422746+0000","last_peered":"2024-09-21T08:32:35.422746+0000","last_clean":"2024-09-21T08:32:35.422746+0000","last_became_active":"2024-09-21T08:31:56.658689+0000","last_became_peered":"2024-09-21T08:31:56.658689+0000","last_unstale":"2024-09-21T08:32:35.422746+0000","last_undegraded":"2024-09-21T08:32:35.422746+0000","last_fullsized":"2024-09-21T08:32:35.422746+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'19","last_scrub_stamp":"2024-09-21T08:25:40.449363+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:25:40.449363+0000","objects_scrubbed":0,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":9,"num_object_clones":0,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":31,"num_read_kb":20,"num_write":20,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,5],"acting":[1,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":931,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420834+0000","last_change":"2024-09-21T08:31:57.665653+0000","last_active":"2024-09-21T08:32:35.420834+0000","last_peered":"2024-09-21T08:32:35.420834+0000","last_clean":"2024-09-21T08:32:35.420834+0000","last_became_active":"2024-09-21T08:31:57.665365+0000","last_became_peered":"2024-09-21T08:31:57.665365+0000","last_unstale":"2024-09-21T08:32:35.420834+0000","last_undegraded":"2024-09-21T08:32:35.420834+0000","last_fullsized":"2024-09-21T08:32:35.420834+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:57.901202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:57.901202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.b","version":"0'0","reported_seq":1024,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419991+0000","last_change":"2024-09-21T08:32:13.055288+0000","last_active":"2024-09-21T08:32:35.419991+0000","last_peered":"2024-09-21T08:32:35.419991+0000","last_clean":"2024-09-21T08:32:35.419991+0000","last_became_active":"2024-09-21T08:31:56.652406+0000","last_became_peered":"2024-09-21T08:31:56.652406+0000","last_unstale":"2024-09-21T08:32:35.419991+0000","last_undegraded":"2024-09-21T08:32:35.419991+0000","last_fullsized":"2024-09-21T08:32:35.419991+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:13.055198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:32:13.055198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:36.426414+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"6.8","version":"0'0","reported_seq":511,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418078+0000","last_change":"2024-09-21T08:31:56.638200+0000","last_active":"2024-09-21T08:32:35.418078+0000","last_peered":"2024-09-21T08:32:35.418078+0000","last_clean":"2024-09-21T08:32:35.418078+0000","last_became_active":"2024-09-21T08:31:56.637710+0000","last_became_peered":"2024-09-21T08:31:56.637710+0000","last_unstale":"2024-09-21T08:32:35.418078+0000","last_undegraded":"2024-09-21T08:32:35.418078+0000","last_fullsized":"2024-09-21T08:32:35.418078+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:53.701980+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:24:53.701980+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.a","version":"558'138","reported_seq":1657,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.428699+0000","last_change":"2024-09-21T08:32:33.741917+0000","last_active":"2024-09-21T08:32:35.428699+0000","last_peered":"2024-09-21T08:32:35.428699+0000","last_clean":"2024-09-21T08:32:35.428699+0000","last_became_active":"2024-09-21T08:31:07.701495+0000","last_became_peered":"2024-09-21T08:31:07.701495+0000","last_unstale":"2024-09-21T08:32:35.428699+0000","last_undegraded":"2024-09-21T08:32:35.428699+0000","last_fullsized":"2024-09-21T08:32:35.428699+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"491'124","last_scrub_stamp":"2024-09-21T08:31:32.865091+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:31:32.865091+0000","objects_scrubbed":3,"log_size":138,"log_dups_size":0,"ondisk_log_size":138,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":17,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:36.640976+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0.0096207959999999992,"stat_sum":{"num_bytes":9724792,"num_objects":4,"num_object_clones":3,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":12,"num_read_kb":12895,"num_write":5,"num_write_kb":3606,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"8"},{"start":"c3","length":"6"},{"start":"cc","length":"5"},{"start":"d5","length":"1"},{"start":"d8","length":"1"},{"start":"de","length":"1"}]},{"pgid":"4.9","version":"73'12","reported_seq":231,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788622+0000","last_change":"2024-09-21T08:31:57.659339+0000","last_active":"2024-09-21T08:32:34.788622+0000","last_peered":"2024-09-21T08:32:34.788622+0000","last_clean":"2024-09-21T08:32:34.788622+0000","last_became_active":"2024-09-21T08:31:57.658547+0000","last_became_peered":"2024-09-21T08:31:57.658547+0000","last_unstale":"2024-09-21T08:32:34.788622+0000","last_undegraded":"2024-09-21T08:32:34.788622+0000","last_fullsized":"2024-09-21T08:32:34.788622+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'12","last_scrub_stamp":"2024-09-21T08:31:34.776208+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:31:34.776208+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:32:59.616638+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":220,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":24,"num_read_kb":15,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":1320,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,7],"acting":[4,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.805045+0000","last_change":"2024-09-21T08:32:35.750805+0000","last_active":"2024-09-21T08:32:35.750805+0000","last_peered":"2024-09-21T08:32:33.805045+0000","last_clean":"2024-09-21T08:32:33.805045+0000","last_became_active":"2024-09-21T08:31:57.642317+0000","last_became_peered":"2024-09-21T08:31:57.642317+0000","last_unstale":"2024-09-21T08:32:33.805045+0000","last_undegraded":"2024-09-21T08:32:33.805045+0000","last_fullsized":"2024-09-21T08:32:33.805045+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:53.150608+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:53.150608+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.8","version":"0'0","reported_seq":1023,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418756+0000","last_change":"2024-09-21T08:29:59.267735+0000","last_active":"2024-09-21T08:32:35.418756+0000","last_peered":"2024-09-21T08:32:35.418756+0000","last_clean":"2024-09-21T08:32:35.418756+0000","last_became_active":"2024-09-21T08:28:37.566192+0000","last_became_peered":"2024-09-21T08:28:37.566192+0000","last_unstale":"2024-09-21T08:32:35.418756+0000","last_undegraded":"2024-09-21T08:32:35.418756+0000","last_fullsized":"2024-09-21T08:32:35.418756+0000","mapping_epoch":323,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":324,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:59.267668+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:29:59.267668+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.b","version":"0'0","reported_seq":936,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.804262+0000","last_change":"2024-09-21T08:32:35.422188+0000","last_active":"2024-09-21T08:32:35.422188+0000","last_peered":"2024-09-21T08:32:33.804262+0000","last_clean":"2024-09-21T08:32:33.804262+0000","last_became_active":"2024-09-21T08:31:57.641745+0000","last_became_peered":"2024-09-21T08:31:57.641745+0000","last_unstale":"2024-09-21T08:32:33.804262+0000","last_undegraded":"2024-09-21T08:32:33.804262+0000","last_fullsized":"2024-09-21T08:32:33.804262+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:29.202113+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:25:25.598341+0000","last_clean_scrub_stamp":"2024-09-21T08:31:29.202113+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,4],"acting":[1,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[4,5,7],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"7.b","version":"559'849","reported_seq":3321,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.762554+0000","last_change":"2024-09-21T08:32:35.762621+0000","last_active":"2024-09-21T08:32:35.762554+0000","last_peered":"2024-09-21T08:32:35.762554+0000","last_clean":"2024-09-21T08:32:35.762554+0000","last_became_active":"2024-09-21T08:31:07.705607+0000","last_became_peered":"2024-09-21T08:31:07.705607+0000","last_unstale":"2024-09-21T08:32:35.762554+0000","last_undegraded":"2024-09-21T08:32:35.762554+0000","last_fullsized":"2024-09-21T08:32:35.762554+0000","mapping_epoch":293,"log_start":"0'0","ondisk_log_start":"0'0","created":80,"last_epoch_clean":474,"parent":"0.0","parent_split_bits":0,"last_scrub":"376'577","last_scrub_stamp":"2024-09-21T08:29:32.265609+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:29:32.265609+0000","objects_scrubbed":10,"log_size":849,"log_dups_size":0,"ondisk_log_size":849,"stats_invalid":true,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":14,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.33863036400000002,"stat_sum":{"num_bytes":26579394,"num_objects":11,"num_object_clones":9,"num_object_copies":33,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":11,"num_whiteouts":0,"num_read":104,"num_read_kb":104740,"num_write":69,"num_write_kb":28178,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":2192635,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,2],"acting":[1,5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"2"},{"start":"d5","length":"1"},{"start":"d8","length":"1"}]},{"pgid":"4.8","version":"73'15","reported_seq":1178,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428934+0000","last_change":"2024-09-21T08:31:57.662703+0000","last_active":"2024-09-21T08:32:35.428934+0000","last_peered":"2024-09-21T08:32:35.428934+0000","last_clean":"2024-09-21T08:32:35.428934+0000","last_became_active":"2024-09-21T08:31:57.662334+0000","last_became_peered":"2024-09-21T08:31:57.662334+0000","last_unstale":"2024-09-21T08:32:35.428934+0000","last_undegraded":"2024-09-21T08:32:35.428934+0000","last_fullsized":"2024-09-21T08:32:35.428934+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'15","last_scrub_stamp":"2024-09-21T08:31:53.711147+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:31:53.711147+0000","objects_scrubbed":7,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:12.459852+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":26,"num_read_kb":17,"num_write":16,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":42,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,4],"acting":[5,7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.f","version":"476'4","reported_seq":234,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788825+0000","last_change":"2024-09-21T08:31:57.673160+0000","last_active":"2024-09-21T08:32:34.788825+0000","last_peered":"2024-09-21T08:32:34.788825+0000","last_clean":"2024-09-21T08:32:34.788825+0000","last_became_active":"2024-09-21T08:31:57.652259+0000","last_became_peered":"2024-09-21T08:31:57.652259+0000","last_unstale":"2024-09-21T08:32:34.788825+0000","last_undegraded":"2024-09-21T08:32:34.788825+0000","last_fullsized":"2024-09-21T08:32:34.788825+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"476'4","last_scrub_stamp":"2024-09-21T08:31:10.077516+0000","last_deep_scrub":"476'4","last_deep_scrub_stamp":"2024-09-21T08:31:10.077516+0000","last_clean_scrub_stamp":"2024-09-21T08:31:10.077516+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":92,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":10,"num_read_kb":10,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":368,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.9","version":"524'20","reported_seq":247,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788120+0000","last_change":"2024-09-21T08:31:57.659365+0000","last_active":"2024-09-21T08:32:34.788120+0000","last_peered":"2024-09-21T08:32:34.788120+0000","last_clean":"2024-09-21T08:32:34.788120+0000","last_became_active":"2024-09-21T08:31:57.658571+0000","last_became_peered":"2024-09-21T08:31:57.658571+0000","last_unstale":"2024-09-21T08:32:34.788120+0000","last_undegraded":"2024-09-21T08:32:34.788120+0000","last_fullsized":"2024-09-21T08:32:34.788120+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"464'18","last_scrub_stamp":"2024-09-21T08:31:51.875726+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:51.875726+0000","objects_scrubbed":1,"log_size":20,"log_dups_size":0,"ondisk_log_size":20,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:04.968651+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.a","version":"0'0","reported_seq":1018,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428962+0000","last_change":"2024-09-21T08:30:50.793864+0000","last_active":"2024-09-21T08:32:35.428962+0000","last_peered":"2024-09-21T08:32:35.428962+0000","last_clean":"2024-09-21T08:32:35.428962+0000","last_became_active":"2024-09-21T08:28:29.619471+0000","last_became_peered":"2024-09-21T08:28:29.619471+0000","last_unstale":"2024-09-21T08:32:35.428962+0000","last_undegraded":"2024-09-21T08:32:35.428962+0000","last_fullsized":"2024-09-21T08:32:35.428962+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:50.793789+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:50.793789+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.14","version":"180'50","reported_seq":2302,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423525+0000","last_change":"2024-09-21T08:32:33.781684+0000","last_active":"2024-09-21T08:32:35.423525+0000","last_peered":"2024-09-21T08:32:35.423525+0000","last_clean":"2024-09-21T08:32:35.423525+0000","last_became_active":"2024-09-21T08:31:56.644864+0000","last_became_peered":"2024-09-21T08:31:56.644864+0000","last_unstale":"2024-09-21T08:32:35.423525+0000","last_undegraded":"2024-09-21T08:32:35.423525+0000","last_fullsized":"2024-09-21T08:32:35.423525+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"180'50","last_scrub_stamp":"2024-09-21T08:32:09.976506+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:32:09.976506+0000","objects_scrubbed":0,"log_size":50,"log_dups_size":0,"ondisk_log_size":50,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:18.336292+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.00096369100000000003,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7],"acting":[2,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"73","length":"28"},{"start":"9c","length":"14"},{"start":"b1","length":"1"},{"start":"b4","length":"4"},{"start":"ba","length":"10"},{"start":"cc","length":"5"},{"start":"d2","length":"4"},{"start":"d7","length":"2"},{"start":"db","length":"6"},{"start":"e3","length":"1"},{"start":"e6","length":"2"},{"start":"e9","length":"1"},{"start":"ed","length":"1"},{"start":"f3","length":"1"}]},{"pgid":"3.10","version":"0'0","reported_seq":808,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.428842+0000","last_change":"2024-09-21T08:29:39.177460+0000","last_active":"2024-09-21T08:32:35.428842+0000","last_peered":"2024-09-21T08:32:35.428842+0000","last_clean":"2024-09-21T08:32:35.428842+0000","last_became_active":"2024-09-21T08:28:29.610987+0000","last_became_peered":"2024-09-21T08:28:29.610987+0000","last_unstale":"2024-09-21T08:32:35.428842+0000","last_undegraded":"2024-09-21T08:32:35.428842+0000","last_fullsized":"2024-09-21T08:32:35.428842+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:29:39.177382+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.177382+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,1],"acting":[5,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.17","version":"73'6","reported_seq":966,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.422622+0000","last_change":"2024-09-21T08:31:56.660387+0000","last_active":"2024-09-21T08:32:35.422622+0000","last_peered":"2024-09-21T08:32:35.422622+0000","last_clean":"2024-09-21T08:32:35.422622+0000","last_became_active":"2024-09-21T08:31:56.659159+0000","last_became_peered":"2024-09-21T08:31:56.659159+0000","last_unstale":"2024-09-21T08:32:35.422622+0000","last_undegraded":"2024-09-21T08:32:35.422622+0000","last_fullsized":"2024-09-21T08:32:35.422622+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'6","last_scrub_stamp":"2024-09-21T08:27:30.806522+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:30.806522+0000","objects_scrubbed":3,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.16","version":"0'0","reported_seq":1046,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421707+0000","last_change":"2024-09-21T08:31:56.652363+0000","last_active":"2024-09-21T08:32:35.421707+0000","last_peered":"2024-09-21T08:32:35.421707+0000","last_clean":"2024-09-21T08:32:35.421707+0000","last_became_active":"2024-09-21T08:31:56.650750+0000","last_became_peered":"2024-09-21T08:31:56.650750+0000","last_unstale":"2024-09-21T08:32:35.421707+0000","last_undegraded":"2024-09-21T08:32:35.421707+0000","last_fullsized":"2024-09-21T08:32:35.421707+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:26:53.795325+0000","last_clean_scrub_stamp":"2024-09-21T08:26:53.795325+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.15","version":"0'0","reported_seq":623,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-09-21T08:32:39.484944+0000","last_change":"2024-09-21T08:32:39.484963+0000","last_active":"2024-09-21T08:32:39.484944+0000","last_peered":"2024-09-21T08:32:39.484944+0000","last_clean":"2024-09-21T08:32:39.484944+0000","last_became_active":"2024-09-21T08:31:57.660052+0000","last_became_peered":"2024-09-21T08:31:57.660052+0000","last_unstale":"2024-09-21T08:32:39.484944+0000","last_undegraded":"2024-09-21T08:32:39.484944+0000","last_fullsized":"2024-09-21T08:32:39.484944+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:39.484893+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:32:39.484893+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:42.308877+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"7.15","version":"555'495","reported_seq":2705,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.425478+0000","last_change":"2024-09-21T08:32:33.767630+0000","last_active":"2024-09-21T08:32:35.425478+0000","last_peered":"2024-09-21T08:32:35.425478+0000","last_clean":"2024-09-21T08:32:35.425478+0000","last_became_active":"2024-09-21T08:31:56.669810+0000","last_became_peered":"2024-09-21T08:31:56.669810+0000","last_unstale":"2024-09-21T08:32:35.425478+0000","last_undegraded":"2024-09-21T08:32:35.425478+0000","last_fullsized":"2024-09-21T08:32:35.425478+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"192'262","last_scrub_stamp":"2024-09-21T08:26:29.875830+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.875830+0000","objects_scrubbed":3,"log_size":495,"log_dups_size":0,"ondisk_log_size":495,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":96,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.995530794,"stat_sum":{"num_bytes":47593958,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":1,"num_read":243,"num_read_kb":285869,"num_write":119,"num_write_kb":50697,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":54,"num_bytes_recovered":45071078,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"8"}]},{"pgid":"4.16","version":"73'9","reported_seq":528,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419933+0000","last_change":"2024-09-21T08:31:56.739241+0000","last_active":"2024-09-21T08:32:35.419933+0000","last_peered":"2024-09-21T08:32:35.419933+0000","last_clean":"2024-09-21T08:32:35.419933+0000","last_became_active":"2024-09-21T08:31:56.649999+0000","last_became_peered":"2024-09-21T08:31:56.649999+0000","last_unstale":"2024-09-21T08:32:35.419933+0000","last_undegraded":"2024-09-21T08:32:35.419933+0000","last_fullsized":"2024-09-21T08:32:35.419933+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:27:10.183035+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:27:10.183035+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788253+0000","last_change":"2024-09-21T08:31:57.641282+0000","last_active":"2024-09-21T08:32:34.788253+0000","last_peered":"2024-09-21T08:32:34.788253+0000","last_clean":"2024-09-21T08:32:34.788253+0000","last_became_active":"2024-09-21T08:31:57.640958+0000","last_became_peered":"2024-09-21T08:31:57.640958+0000","last_unstale":"2024-09-21T08:32:34.788253+0000","last_undegraded":"2024-09-21T08:32:34.788253+0000","last_fullsized":"2024-09-21T08:32:34.788253+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:53.898917+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:31:53.898917+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:00.608747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.17","version":"0'0","reported_seq":567,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423230+0000","last_change":"2024-09-21T08:31:56.660694+0000","last_active":"2024-09-21T08:32:35.423230+0000","last_peered":"2024-09-21T08:32:35.423230+0000","last_clean":"2024-09-21T08:32:35.423230+0000","last_became_active":"2024-09-21T08:31:56.659891+0000","last_became_peered":"2024-09-21T08:31:56.659891+0000","last_unstale":"2024-09-21T08:32:35.423230+0000","last_undegraded":"2024-09-21T08:32:35.423230+0000","last_fullsized":"2024-09-21T08:32:35.423230+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:05.814753+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:26:05.814753+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.14","version":"0'0","reported_seq":1037,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424646+0000","last_change":"2024-09-21T08:31:57.642589+0000","last_active":"2024-09-21T08:32:35.424646+0000","last_peered":"2024-09-21T08:32:35.424646+0000","last_clean":"2024-09-21T08:32:35.424646+0000","last_became_active":"2024-09-21T08:31:57.642005+0000","last_became_peered":"2024-09-21T08:31:57.642005+0000","last_unstale":"2024-09-21T08:32:35.424646+0000","last_undegraded":"2024-09-21T08:32:35.424646+0000","last_fullsized":"2024-09-21T08:32:35.424646+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:07.730202+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:31:07.730202+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4,7],"acting":[2,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"7.16","version":"0'0","reported_seq":1,"reported_epoch":561,"state":"unknown","last_fresh":"2024-09-21T08:32:35.423269+0000","last_change":"2024-09-21T08:32:35.750772+0000","last_active":"0.000000","last_peered":"0.000000","last_clean":"0.000000","last_became_active":"0.000000","last_became_peered":"0.000000","last_unstale":"2024-09-21T08:32:35.423269+0000","last_undegraded":"2024-09-21T08:32:35.423269+0000","last_fullsized":"2024-09-21T08:32:35.423269+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"189'102","last_scrub_stamp":"2024-09-21T08:26:33.255256+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:33.255256+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.15","version":"73'9","reported_seq":1110,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420738+0000","last_change":"2024-09-21T08:31:56.671648+0000","last_active":"2024-09-21T08:32:35.420738+0000","last_peered":"2024-09-21T08:32:35.420738+0000","last_clean":"2024-09-21T08:32:35.420738+0000","last_became_active":"2024-09-21T08:31:56.671219+0000","last_became_peered":"2024-09-21T08:31:56.671219+0000","last_unstale":"2024-09-21T08:32:35.420738+0000","last_undegraded":"2024-09-21T08:32:35.420738+0000","last_fullsized":"2024-09-21T08:32:35.420738+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:28:14.535334+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:28:14.535334+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":20,"num_bytes_recovered":550,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":994,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420974+0000","last_change":"2024-09-21T08:31:56.659624+0000","last_active":"2024-09-21T08:32:35.420974+0000","last_peered":"2024-09-21T08:32:35.420974+0000","last_clean":"2024-09-21T08:32:35.420974+0000","last_became_active":"2024-09-21T08:31:56.658219+0000","last_became_peered":"2024-09-21T08:31:56.658219+0000","last_unstale":"2024-09-21T08:32:35.420974+0000","last_undegraded":"2024-09-21T08:32:35.420974+0000","last_fullsized":"2024-09-21T08:32:35.420974+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:12.361096+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:12.361096+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"5.14","version":"523'17","reported_seq":1041,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421290+0000","last_change":"2024-09-21T08:31:56.650821+0000","last_active":"2024-09-21T08:32:35.421290+0000","last_peered":"2024-09-21T08:32:35.421290+0000","last_clean":"2024-09-21T08:32:35.421290+0000","last_became_active":"2024-09-21T08:31:56.649221+0000","last_became_peered":"2024-09-21T08:31:56.649221+0000","last_unstale":"2024-09-21T08:32:35.421290+0000","last_undegraded":"2024-09-21T08:32:35.421290+0000","last_fullsized":"2024-09-21T08:32:35.421290+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"101'10","last_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_deep_scrub":"101'10","last_deep_scrub_stamp":"2024-09-21T08:24:54.690590+0000","last_clean_scrub_stamp":"2024-09-21T08:24:54.690590+0000","objects_scrubbed":1,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.17","version":"0'0","reported_seq":207,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788918+0000","last_change":"2024-09-21T08:30:56.611027+0000","last_active":"2024-09-21T08:32:34.788918+0000","last_peered":"2024-09-21T08:32:34.788918+0000","last_clean":"2024-09-21T08:32:34.788918+0000","last_became_active":"2024-09-21T08:30:56.610628+0000","last_became_peered":"2024-09-21T08:30:56.610628+0000","last_unstale":"2024-09-21T08:32:34.788918+0000","last_undegraded":"2024-09-21T08:32:34.788918+0000","last_fullsized":"2024-09-21T08:32:34.788918+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:39.364388+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:28:39.364388+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.17","version":"179'63","reported_seq":1477,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.747423+0000","last_change":"2024-09-21T08:32:35.747480+0000","last_active":"2024-09-21T08:32:35.747423+0000","last_peered":"2024-09-21T08:32:35.747423+0000","last_clean":"2024-09-21T08:32:35.747423+0000","last_became_active":"2024-09-21T08:31:56.649875+0000","last_became_peered":"2024-09-21T08:31:56.649875+0000","last_unstale":"2024-09-21T08:32:35.747423+0000","last_undegraded":"2024-09-21T08:32:35.747423+0000","last_fullsized":"2024-09-21T08:32:35.747423+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"179'63","last_scrub_stamp":"2024-09-21T08:28:13.511512+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:28:13.511512+0000","objects_scrubbed":0,"log_size":63,"log_dups_size":0,"ondisk_log_size":63,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":97,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.32372216599999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"7"}]},{"pgid":"4.14","version":"104'15","reported_seq":581,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421416+0000","last_change":"2024-09-21T08:32:18.213538+0000","last_active":"2024-09-21T08:32:35.421416+0000","last_peered":"2024-09-21T08:32:35.421416+0000","last_clean":"2024-09-21T08:32:35.421416+0000","last_became_active":"2024-09-21T08:31:56.649656+0000","last_became_peered":"2024-09-21T08:31:56.649656+0000","last_unstale":"2024-09-21T08:32:35.421416+0000","last_undegraded":"2024-09-21T08:32:35.421416+0000","last_fullsized":"2024-09-21T08:32:35.421416+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"104'15","last_scrub_stamp":"2024-09-21T08:32:18.213452+0000","last_deep_scrub":"104'15","last_deep_scrub_stamp":"2024-09-21T08:24:57.624718+0000","last_clean_scrub_stamp":"2024-09-21T08:32:18.213452+0000","objects_scrubbed":5,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:26.795743+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":14,"num_read_kb":9,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.13","version":"66'1","reported_seq":654,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-09-21T08:32:38.807962+0000","last_change":"2024-09-21T08:31:57.658189+0000","last_active":"2024-09-21T08:32:38.807962+0000","last_peered":"2024-09-21T08:32:38.807962+0000","last_clean":"2024-09-21T08:32:38.807962+0000","last_became_active":"2024-09-21T08:31:57.657724+0000","last_became_peered":"2024-09-21T08:31:57.657724+0000","last_unstale":"2024-09-21T08:32:38.807962+0000","last_undegraded":"2024-09-21T08:32:38.807962+0000","last_fullsized":"2024-09-21T08:32:38.807962+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"66'1","last_scrub_stamp":"2024-09-21T08:28:16.875544+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:28:16.875544+0000","objects_scrubbed":1,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":993,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":7,"num_read_kb":7,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1986,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,2],"acting":[7,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"5.15","version":"524'19","reported_seq":1194,"reported_epoch":560,"state":"peering","last_fresh":"2024-09-21T08:32:33.791647+0000","last_change":"2024-09-21T08:32:35.421672+0000","last_active":"2024-09-21T08:32:35.421672+0000","last_peered":"2024-09-21T08:32:33.791647+0000","last_clean":"2024-09-21T08:32:33.791647+0000","last_became_active":"2024-09-21T08:31:57.663936+0000","last_became_peered":"2024-09-21T08:31:57.663936+0000","last_unstale":"2024-09-21T08:32:33.791647+0000","last_undegraded":"2024-09-21T08:32:33.791647+0000","last_fullsized":"2024-09-21T08:32:33.791647+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'8","last_scrub_stamp":"2024-09-21T08:25:18.620302+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:25:18.620302+0000","objects_scrubbed":1,"log_size":19,"log_dups_size":0,"ondisk_log_size":19,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"no scrub is scheduled","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1,4,7],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"6.16","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788891+0000","last_change":"2024-09-21T08:32:18.676090+0000","last_active":"2024-09-21T08:32:34.788891+0000","last_peered":"2024-09-21T08:32:34.788891+0000","last_clean":"2024-09-21T08:32:34.788891+0000","last_became_active":"2024-09-21T08:31:57.641682+0000","last_became_peered":"2024-09-21T08:31:57.641682+0000","last_unstale":"2024-09-21T08:32:34.788891+0000","last_undegraded":"2024-09-21T08:32:34.788891+0000","last_fullsized":"2024-09-21T08:32:34.788891+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:18.676018+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:32:18.676018+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:45.305272+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,5],"acting":[4,7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.10","version":"559'180","reported_seq":1350,"reported_epoch":561,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:32:35.420452+0000","last_change":"2024-09-21T08:32:33.771528+0000","last_active":"2024-09-21T08:32:35.420452+0000","last_peered":"2024-09-21T08:32:35.420452+0000","last_clean":"2024-09-21T08:32:35.420452+0000","last_became_active":"2024-09-21T08:31:56.641759+0000","last_became_peered":"2024-09-21T08:31:56.641759+0000","last_unstale":"2024-09-21T08:32:35.420452+0000","last_undegraded":"2024-09-21T08:32:35.420452+0000","last_fullsized":"2024-09-21T08:32:35.420452+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"172'94","last_scrub_stamp":"2024-09-21T08:26:27.208234+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.208234+0000","objects_scrubbed":1,"log_size":180,"log_dups_size":0,"ondisk_log_size":180,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":95,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.99657815400000005,"stat_sum":{"num_bytes":13517584,"num_objects":7,"num_object_clones":6,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":23,"num_read_kb":13077,"num_write":47,"num_write_kb":18807,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23,"num_bytes_recovered":22296475,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"9"}]},{"pgid":"4.13","version":"73'11","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788320+0000","last_change":"2024-09-21T08:30:56.661267+0000","last_active":"2024-09-21T08:32:34.788320+0000","last_peered":"2024-09-21T08:32:34.788320+0000","last_clean":"2024-09-21T08:32:34.788320+0000","last_became_active":"2024-09-21T08:30:56.626643+0000","last_became_peered":"2024-09-21T08:30:56.626643+0000","last_unstale":"2024-09-21T08:32:34.788320+0000","last_undegraded":"2024-09-21T08:32:34.788320+0000","last_fullsized":"2024-09-21T08:32:34.788320+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:30:25.153729+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:25.153729+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,5],"acting":[4,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":211,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.788347+0000","last_change":"2024-09-21T08:31:57.640495+0000","last_active":"2024-09-21T08:32:34.788347+0000","last_peered":"2024-09-21T08:32:34.788347+0000","last_clean":"2024-09-21T08:32:34.788347+0000","last_became_active":"2024-09-21T08:31:57.640151+0000","last_became_peered":"2024-09-21T08:31:57.640151+0000","last_unstale":"2024-09-21T08:32:34.788347+0000","last_undegraded":"2024-09-21T08:32:34.788347+0000","last_fullsized":"2024-09-21T08:32:34.788347+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:27:06.889127+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:27:06.889127+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.12","version":"0'0","reported_seq":569,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424874+0000","last_change":"2024-09-21T08:32:16.284027+0000","last_active":"2024-09-21T08:32:35.424874+0000","last_peered":"2024-09-21T08:32:35.424874+0000","last_clean":"2024-09-21T08:32:35.424874+0000","last_became_active":"2024-09-21T08:31:56.659280+0000","last_became_peered":"2024-09-21T08:31:56.659280+0000","last_unstale":"2024-09-21T08:32:35.424874+0000","last_undegraded":"2024-09-21T08:32:35.424874+0000","last_fullsized":"2024-09-21T08:32:35.424874+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:16.283928+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:32:16.283928+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:38.645739+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,7],"acting":[1,5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.11","version":"0'0","reported_seq":764,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424687+0000","last_change":"2024-09-21T08:30:56.618534+0000","last_active":"2024-09-21T08:32:35.424687+0000","last_peered":"2024-09-21T08:32:35.424687+0000","last_clean":"2024-09-21T08:32:35.424687+0000","last_became_active":"2024-09-21T08:30:56.618134+0000","last_became_peered":"2024-09-21T08:30:56.618134+0000","last_unstale":"2024-09-21T08:32:35.424687+0000","last_undegraded":"2024-09-21T08:32:35.424687+0000","last_fullsized":"2024-09-21T08:32:35.424687+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:23.816669+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:23.816669+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.11","version":"561'221","reported_seq":2142,"reported_epoch":561,"state":"active+clean+snaptrim","last_fresh":"2024-09-21T08:32:35.762653+0000","last_change":"2024-09-21T08:32:35.762653+0000","last_active":"2024-09-21T08:32:35.762653+0000","last_peered":"2024-09-21T08:32:35.762653+0000","last_clean":"2024-09-21T08:32:35.762653+0000","last_became_active":"2024-09-21T08:31:56.657911+0000","last_became_peered":"2024-09-21T08:31:56.657911+0000","last_unstale":"2024-09-21T08:32:35.762653+0000","last_undegraded":"2024-09-21T08:32:35.762653+0000","last_fullsized":"2024-09-21T08:32:35.762653+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"194'44","last_scrub_stamp":"2024-09-21T08:26:27.882963+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.882963+0000","objects_scrubbed":1,"log_size":221,"log_dups_size":0,"ondisk_log_size":221,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":94,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":1,"snaptrim_duration":0.99548273600000003,"stat_sum":{"num_bytes":28511900,"num_objects":18,"num_object_clones":16,"num_object_copies":54,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":18,"num_whiteouts":1,"num_read":112,"num_read_kb":121994,"num_write":103,"num_write_kb":41259,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":36,"num_bytes_recovered":42498204,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,2],"acting":[1,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"73","length":"a"}]},{"pgid":"4.12","version":"73'9","reported_seq":586,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423072+0000","last_change":"2024-09-21T08:30:42.016780+0000","last_active":"2024-09-21T08:32:35.423072+0000","last_peered":"2024-09-21T08:32:35.423072+0000","last_clean":"2024-09-21T08:32:35.423072+0000","last_became_active":"2024-09-21T08:28:29.636404+0000","last_became_peered":"2024-09-21T08:28:29.636404+0000","last_unstale":"2024-09-21T08:32:35.423072+0000","last_undegraded":"2024-09-21T08:32:35.423072+0000","last_fullsized":"2024-09-21T08:32:35.423072+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'9","last_scrub_stamp":"2024-09-21T08:30:42.016697+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:30:42.016697+0000","objects_scrubbed":4,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":17,"num_read_kb":11,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":220,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,5],"acting":[1,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789246+0000","last_change":"2024-09-21T08:32:16.654184+0000","last_active":"2024-09-21T08:32:34.789246+0000","last_peered":"2024-09-21T08:32:34.789246+0000","last_clean":"2024-09-21T08:32:34.789246+0000","last_became_active":"2024-09-21T08:31:57.639837+0000","last_became_peered":"2024-09-21T08:31:57.639837+0000","last_unstale":"2024-09-21T08:32:34.789246+0000","last_undegraded":"2024-09-21T08:32:34.789246+0000","last_fullsized":"2024-09-21T08:32:34.789246+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:16.654080+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:32:16.654080+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:27.615791+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"5.13","version":"0'0","reported_seq":881,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421170+0000","last_change":"2024-09-21T08:31:56.199080+0000","last_active":"2024-09-21T08:32:35.421170+0000","last_peered":"2024-09-21T08:32:35.421170+0000","last_clean":"2024-09-21T08:32:35.421170+0000","last_became_active":"2024-09-21T08:30:56.626197+0000","last_became_peered":"2024-09-21T08:30:56.626197+0000","last_unstale":"2024-09-21T08:32:35.421170+0000","last_undegraded":"2024-09-21T08:32:35.421170+0000","last_fullsized":"2024-09-21T08:32:35.421170+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:56.199004+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:56.199004+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:12.206778+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"6.10","version":"0'0","reported_seq":978,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.429043+0000","last_change":"2024-09-21T08:30:56.627439+0000","last_active":"2024-09-21T08:32:35.429043+0000","last_peered":"2024-09-21T08:32:35.429043+0000","last_clean":"2024-09-21T08:32:35.429043+0000","last_became_active":"2024-09-21T08:30:56.627130+0000","last_became_peered":"2024-09-21T08:30:56.627130+0000","last_unstale":"2024-09-21T08:32:35.429043+0000","last_undegraded":"2024-09-21T08:32:35.429043+0000","last_fullsized":"2024-09-21T08:32:35.429043+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:02.065082+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:02.065082+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,4],"acting":[5,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"7.12","version":"560'458","reported_seq":2634,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.739394+0000","last_change":"2024-09-21T08:32:35.739434+0000","last_active":"2024-09-21T08:32:35.739394+0000","last_peered":"2024-09-21T08:32:35.739394+0000","last_clean":"2024-09-21T08:32:35.739394+0000","last_became_active":"2024-09-21T08:31:56.653984+0000","last_became_peered":"2024-09-21T08:31:56.653984+0000","last_unstale":"2024-09-21T08:32:35.739394+0000","last_undegraded":"2024-09-21T08:32:35.739394+0000","last_fullsized":"2024-09-21T08:32:35.739394+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"189'200","last_scrub_stamp":"2024-09-21T08:26:27.417028+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:27.417028+0000","objects_scrubbed":3,"log_size":458,"log_dups_size":0,"ondisk_log_size":458,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":99,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.31123925800000002,"stat_sum":{"num_bytes":49437079,"num_objects":22,"num_object_clones":18,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":208,"num_read_kb":239742,"num_write":127,"num_write_kb":61159,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":57,"num_bytes_recovered":43192796,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"5"}]},{"pgid":"4.11","version":"73'11","reported_seq":609,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.424780+0000","last_change":"2024-09-21T08:32:16.895092+0000","last_active":"2024-09-21T08:32:35.424780+0000","last_peered":"2024-09-21T08:32:35.424780+0000","last_clean":"2024-09-21T08:32:35.424780+0000","last_became_active":"2024-09-21T08:31:56.649307+0000","last_became_peered":"2024-09-21T08:31:56.649307+0000","last_unstale":"2024-09-21T08:32:35.424780+0000","last_undegraded":"2024-09-21T08:32:35.424780+0000","last_fullsized":"2024-09-21T08:32:35.424780+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'11","last_scrub_stamp":"2024-09-21T08:32:16.894954+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:32:16.894954+0000","objects_scrubbed":5,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:33.225663+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":110,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":1042,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.421691+0000","last_change":"2024-09-21T08:31:56.649967+0000","last_active":"2024-09-21T08:32:35.421691+0000","last_peered":"2024-09-21T08:32:35.421691+0000","last_clean":"2024-09-21T08:32:35.421691+0000","last_became_active":"2024-09-21T08:31:56.649348+0000","last_became_peered":"2024-09-21T08:31:56.649348+0000","last_unstale":"2024-09-21T08:32:35.421691+0000","last_undegraded":"2024-09-21T08:32:35.421691+0000","last_fullsized":"2024-09-21T08:32:35.421691+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:26:03.808061+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:26:03.808061+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,1],"acting":[5,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.10","version":"0'0","reported_seq":643,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-09-21T08:32:38.808544+0000","last_change":"2024-09-21T08:31:57.659645+0000","last_active":"2024-09-21T08:32:38.808544+0000","last_peered":"2024-09-21T08:32:38.808544+0000","last_clean":"2024-09-21T08:32:38.808544+0000","last_became_active":"2024-09-21T08:31:57.659355+0000","last_became_peered":"2024-09-21T08:31:57.659355+0000","last_unstale":"2024-09-21T08:32:38.808544+0000","last_undegraded":"2024-09-21T08:32:38.808544+0000","last_fullsized":"2024-09-21T08:32:38.808544+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:37.920950+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:37.920950+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,5],"acting":[7,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"6.13","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:31:40.661693+0000","last_change":"2024-09-21T08:26:49.529910+0000","last_active":"2024-09-21T08:26:49.529910+0000","last_peered":"2024-09-21T08:26:47.516492+0000","last_clean":"2024-09-21T08:26:47.516492+0000","last_became_active":"2024-09-21T08:24:56.508323+0000","last_became_peered":"2024-09-21T08:24:56.508323+0000","last_unstale":"2024-09-21T08:31:40.661693+0000","last_undegraded":"2024-09-21T08:31:40.661693+0000","last_fullsized":"2024-09-21T08:31:40.661693+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":507,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:29:58.167184+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:29:58.167184+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,5],"acting":[4,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.13","version":"532'77","reported_seq":2053,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.421313+0000","last_change":"2024-09-21T08:32:33.743365+0000","last_active":"2024-09-21T08:32:35.421313+0000","last_peered":"2024-09-21T08:32:35.421313+0000","last_clean":"2024-09-21T08:32:35.421313+0000","last_became_active":"2024-09-21T08:31:56.642897+0000","last_became_peered":"2024-09-21T08:31:56.642897+0000","last_unstale":"2024-09-21T08:32:35.421313+0000","last_undegraded":"2024-09-21T08:32:35.421313+0000","last_fullsized":"2024-09-21T08:32:35.421313+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":181,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"186'22","last_scrub_stamp":"2024-09-21T08:26:29.353795+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:26:29.353795+0000","objects_scrubbed":0,"log_size":77,"log_dups_size":0,"ondisk_log_size":77,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":98,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.012744657,"stat_sum":{"num_bytes":12261866,"num_objects":5,"num_object_clones":4,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":73,"num_read_kb":105134,"num_write":36,"num_write_kb":17862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":10,"num_bytes_recovered":15513137,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,2],"acting":[5,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"6"}]},{"pgid":"4.10","version":"73'4","reported_seq":578,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.419495+0000","last_change":"2024-09-21T08:30:56.659256+0000","last_active":"2024-09-21T08:32:35.419495+0000","last_peered":"2024-09-21T08:32:35.419495+0000","last_clean":"2024-09-21T08:32:35.419495+0000","last_became_active":"2024-09-21T08:30:56.638177+0000","last_became_peered":"2024-09-21T08:30:56.638177+0000","last_unstale":"2024-09-21T08:32:35.419495+0000","last_undegraded":"2024-09-21T08:32:35.419495+0000","last_fullsized":"2024-09-21T08:32:35.419495+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"73'4","last_scrub_stamp":"2024-09-21T08:29:39.416397+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:31.530408+0000","last_clean_scrub_stamp":"2024-09-21T08:29:39.416397+0000","objects_scrubbed":2,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4,5],"acting":[1,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":1004,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423614+0000","last_change":"2024-09-21T08:30:24.826266+0000","last_active":"2024-09-21T08:32:35.423614+0000","last_peered":"2024-09-21T08:32:35.423614+0000","last_clean":"2024-09-21T08:32:35.423614+0000","last_became_active":"2024-09-21T08:28:29.611592+0000","last_became_peered":"2024-09-21T08:28:29.611592+0000","last_unstale":"2024-09-21T08:32:35.423614+0000","last_undegraded":"2024-09-21T08:32:35.423614+0000","last_fullsized":"2024-09-21T08:32:35.423614+0000","mapping_epoch":315,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":316,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:24.826198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:30:24.826198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,2],"acting":[5,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.11","version":"0'0","reported_seq":2,"reported_epoch":561,"state":"peering","last_fresh":"2024-09-21T08:32:33.803190+0000","last_change":"2024-09-21T08:32:35.750646+0000","last_active":"2024-09-21T08:32:35.750646+0000","last_peered":"2024-09-21T08:32:33.803190+0000","last_clean":"2024-09-21T08:32:33.803190+0000","last_became_active":"2024-09-21T08:31:57.644199+0000","last_became_peered":"2024-09-21T08:31:57.644199+0000","last_unstale":"2024-09-21T08:32:33.803190+0000","last_undegraded":"2024-09-21T08:32:33.803190+0000","last_fullsized":"2024-09-21T08:32:33.803190+0000","mapping_epoch":561,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:31:27.881562+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:31:27.881562+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"no scrub is scheduled","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,1],"acting":[4,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"6.12","version":"0'0","reported_seq":219,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:34.789099+0000","last_change":"2024-09-21T08:32:14.667710+0000","last_active":"2024-09-21T08:32:34.789099+0000","last_peered":"2024-09-21T08:32:34.789099+0000","last_clean":"2024-09-21T08:32:34.789099+0000","last_became_active":"2024-09-21T08:31:57.648003+0000","last_became_peered":"2024-09-21T08:31:57.648003+0000","last_unstale":"2024-09-21T08:32:34.789099+0000","last_undegraded":"2024-09-21T08:32:34.789099+0000","last_fullsized":"2024-09-21T08:32:34.789099+0000","mapping_epoch":523,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":524,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:32:14.667636+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:32:14.667636+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:43.990732+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,2],"acting":[4,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"7.1c","version":"555'235","reported_seq":2303,"reported_epoch":561,"state":"active+clean+snaptrim_wait","last_fresh":"2024-09-21T08:32:35.739896+0000","last_change":"2024-09-21T08:32:35.739896+0000","last_active":"2024-09-21T08:32:35.739896+0000","last_peered":"2024-09-21T08:32:35.739896+0000","last_clean":"2024-09-21T08:32:35.739896+0000","last_became_active":"2024-09-21T08:31:56.647728+0000","last_became_peered":"2024-09-21T08:31:56.647728+0000","last_unstale":"2024-09-21T08:32:35.739896+0000","last_undegraded":"2024-09-21T08:32:35.739896+0000","last_fullsized":"2024-09-21T08:32:35.739896+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":259,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":5,"last_scrub":"273'167","last_scrub_stamp":"2024-09-21T08:27:49.143452+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:24:33.484514+0000","last_clean_scrub_stamp":"2024-09-21T08:27:49.143452+0000","objects_scrubbed":0,"log_size":235,"log_dups_size":0,"ondisk_log_size":235,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":98,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0.32011975999999998,"stat_sum":{"num_bytes":21064649,"num_objects":9,"num_object_clones":8,"num_object_copies":27,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":9,"num_whiteouts":0,"num_read":85,"num_read_kb":108670,"num_write":39,"num_write_kb":19543,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19,"num_bytes_recovered":13106064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[{"start":"73","length":"6"}]},{"pgid":"6.1d","version":"0'0","reported_seq":566,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.420394+0000","last_change":"2024-09-21T08:30:56.627685+0000","last_active":"2024-09-21T08:32:35.420394+0000","last_peered":"2024-09-21T08:32:35.420394+0000","last_clean":"2024-09-21T08:32:35.420394+0000","last_became_active":"2024-09-21T08:30:56.627251+0000","last_became_peered":"2024-09-21T08:30:56.627251+0000","last_unstale":"2024-09-21T08:32:35.420394+0000","last_undegraded":"2024-09-21T08:32:35.420394+0000","last_fullsized":"2024-09-21T08:32:35.420394+0000","mapping_epoch":462,"log_start":"0'0","ondisk_log_start":"0'0","created":71,"last_epoch_clean":463,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:30:28.069450+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:35.559396+0000","last_clean_scrub_stamp":"2024-09-21T08:30:28.069450+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"queued for scrub","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,4],"acting":[1,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":882,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.418186+0000","last_change":"2024-09-21T08:31:56.638174+0000","last_active":"2024-09-21T08:32:35.418186+0000","last_peered":"2024-09-21T08:32:35.418186+0000","last_clean":"2024-09-21T08:32:35.418186+0000","last_became_active":"2024-09-21T08:31:56.637565+0000","last_became_peered":"2024-09-21T08:31:56.637565+0000","last_unstale":"2024-09-21T08:32:35.418186+0000","last_undegraded":"2024-09-21T08:32:35.418186+0000","last_fullsized":"2024-09-21T08:32:35.418186+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":65,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:24:51.848520+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:29.515888+0000","last_clean_scrub_stamp":"2024-09-21T08:24:51.848520+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,7],"acting":[1,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1f","version":"541'16","reported_seq":1090,"reported_epoch":561,"state":"active+clean","last_fresh":"2024-09-21T08:32:35.423883+0000","last_change":"2024-09-21T08:32:14.942629+0000","last_active":"2024-09-21T08:32:35.423883+0000","last_peered":"2024-09-21T08:32:35.423883+0000","last_clean":"2024-09-21T08:32:35.423883+0000","last_became_active":"2024-09-21T08:31:56.649277+0000","last_became_peered":"2024-09-21T08:31:56.649277+0000","last_unstale":"2024-09-21T08:32:35.423883+0000","last_undegraded":"2024-09-21T08:32:35.423883+0000","last_fullsized":"2024-09-21T08:32:35.423883+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":67,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"541'16","last_scrub_stamp":"2024-09-21T08:32:14.942538+0000","last_deep_scrub":"541'16","last_deep_scrub_stamp":"2024-09-21T08:32:14.942538+0000","last_clean_scrub_stamp":"2024-09-21T08:32:14.942538+0000","objects_scrubbed":5,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-21T08:33:22.569147+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":110,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":19,"num_read_kb":12,"num_write":12,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30,"num_bytes_recovered":660,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"5.1e","version":"0'0","reported_seq":132,"reported_epoch":565,"state":"active+clean","last_fresh":"2024-09-21T08:32:38.807192+0000","last_change":"2024-09-21T08:31:56.648643+0000","last_active":"2024-09-21T08:32:38.807192+0000","last_peered":"2024-09-21T08:32:38.807192+0000","last_clean":"2024-09-21T08:32:38.807192+0000","last_became_active":"2024-09-21T08:31:56.648235+0000","last_became_peered":"2024-09-21T08:31:56.648235+0000","last_unstale":"2024-09-21T08:32:38.807192+0000","last_undegraded":"2024-09-21T08:32:38.807192+0000","last_fullsized":"2024-09-21T08:32:38.807192+0000","mapping_epoch":522,"log_start":"0'0","ondisk_log_start":"0'0","created":69,"last_epoch_clean":523,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-21T08:28:38.872291+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-21T08:23:33.544062+0000","last_clean_scrub_stamp":"2024-09-21T08:28:38.872291+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2,1],"acting":[7,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]}],"pool_stats":[{"poolid":7,"num_pg":46,"stat_sum":{"num_bytes":636747908,"num_objects":297,"num_object_clones":248,"num_object_copies":891,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":297,"num_whiteouts":12,"num_read":2180,"num_read_kb":2368847,"num_write":1568,"num_write_kb":630398,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":880,"num_bytes_recovered":650713102,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1570701312,"data_stored":2237728041,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":11790,"ondisk_log_size":11790,"up":138,"acting":138,"num_store_stats":8},{"poolid":6,"num_pg":32,"stat_sum":{"num_bytes":416,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":65,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1248,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":96,"acting":96,"num_store_stats":8},{"poolid":5,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":2,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":140,"ondisk_log_size":140,"up":96,"acting":96,"num_store_stats":8},{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":3702,"num_objects":178,"num_object_clones":0,"num_object_copies":534,"num_objects_missing_on_primary":0,"num_objects_missing":14,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":178,"num_whiteouts":0,"num_read":676,"num_read_kb":433,"num_write":417,"num_write_kb":34,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":603,"num_bytes_recovered":12498,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":499712,"data_stored":13268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3558,"internal_metadata":0},"log_size":436,"ondisk_log_size":436,"up":96,"acting":96,"num_store_stats":8},{"poolid":3,"num_pg":32,"stat_sum":{"num_bytes":1613,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":24,"num_read_kb":24,"num_write":10,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":4466,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":7445,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":10,"ondisk_log_size":10,"up":96,"acting":96,"num_store_stats":8},{"poolid":2,"num_pg":3,"stat_sum":{"num_bytes":408,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":521,"num_read_kb":516,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16,"num_bytes_recovered":1670,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":8,"ondisk_log_size":8,"up":9,"acting":9,"num_store_stats":8},{"poolid":1,"num_pg":11,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2969600,"data_stored":2951840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1291,"ondisk_log_size":1291,"up":33,"acting":33,"num_store_stats":7}],"osd_stats":[{"osd":7,"up_from":522,"seq":2241972928522,"num_pgs":128,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":435532,"kb_used_data":398516,"kb_used_omap":38,"kb_used_meta":36953,"kb_avail":93309620,"statfs":{"total":95995035648,"available":95549050880,"internally_reserved":0,"allocated":408080384,"data_stored":552981564,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":38961,"internal_metadata":37840847},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":0,"seq":210453397624,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":41,"seq":176093659259,"num_pgs":123,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":327256,"kb_used_data":255808,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93417896,"statfs":{"total":95995035648,"available":95659925504,"internally_reserved":0,"allocated":261947392,"data_stored":374007491,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":31,"seq":133143986305,"num_pgs":80,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":153244,"kb_used_data":126220,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93591908,"statfs":{"total":95995035648,"available":95838113792,"internally_reserved":0,"allocated":129249280,"data_stored":167992535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":0,"seq":107374182533,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411467,"num_pgs":106,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":405540,"kb_used_data":333300,"kb_used_omap":1,"kb_used_meta":71422,"kb_avail":93339612,"statfs":{"total":95995035648,"available":95579762688,"internally_reserved":0,"allocated":341299200,"data_stored":536527315,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":73136581},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574991,"num_pgs":127,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":318980,"kb_used_data":273112,"kb_used_omap":1,"kb_used_meta":45054,"kb_avail":93426172,"statfs":{"total":95995035648,"available":95668400128,"internally_reserved":0,"allocated":279666688,"data_stored":431638557,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":46135749},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":0,"seq":2297807503367,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":262688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":327680,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1475,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1613,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3558,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":3218,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":53248,"data_stored":1348,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":1364,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":2684,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":2162,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":2382,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":5,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":13,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":416,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":6,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":403,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":74776576,"data_stored":86348592,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":275013632,"data_stored":425357645,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":336785408,"data_stored":533760986,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":70848512,"data_stored":84468350,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":124731392,"data_stored":162059112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":257060864,"data_stored":367752962,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28094464,"data_stored":27870415,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":7,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":403390464,"data_stored":550109979,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-21T08:32:41.478 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph status --format=json 2024-09-21T08:32:41.565 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:41.586 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:41.587 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:41.587 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:41.596 DEBUG:teuthology.orchestra.run.smithi165:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph daemon osd.6 dump_ops_in_flight 2024-09-21T08:32:41.622 INFO:tasks.rados.rados.0.smithi137.stdout:2254: done (5 left) 2024-09-21T08:32:41.622 INFO:tasks.rados.rados.0.smithi137.stdout:2255: read oid 47 snap 242 2024-09-21T08:32:41.622 INFO:tasks.rados.rados.0.smithi137.stdout:2255: expect (ObjNum 718 snap 222 seq_num 718) 2024-09-21T08:32:41.622 INFO:tasks.rados.rados.0.smithi137.stdout:2256: snap_create 2024-09-21T08:32:41.622 INFO:tasks.rados.rados.0.smithi137.stdout:2248: finishing write tid 2 to smithi13779221-41 2024-09-21T08:32:41.623 INFO:tasks.rados.rados.0.smithi137.stdout:2248: finishing write tid 3 to smithi13779221-41 2024-09-21T08:32:41.623 INFO:tasks.rados.rados.0.smithi137.stdout:2248: finishing write tid 4 to smithi13779221-41 2024-09-21T08:32:41.623 INFO:tasks.rados.rados.0.smithi137.stdout:2248: finishing write tid 5 to smithi13779221-41 2024-09-21T08:32:41.623 INFO:tasks.rados.rados.0.smithi137.stdout:2248: oid 41 updating version 0 to 426 2024-09-21T08:32:41.623 INFO:tasks.rados.rados.0.smithi137.stdout:2248: oid 41 updating version 426 to 427 2024-09-21T08:32:41.623 INFO:tasks.rados.rados.0.smithi137.stdout:2248: oid 41 version 427 is already newer than 425 2024-09-21T08:32:41.623 INFO:tasks.rados.rados.0.smithi137.stdout:2248: oid 41 version 427 is already newer than 424 2024-09-21T08:32:41.623 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 41 v 427 (ObjNum 779 snap 250 seq_num 779) dirty exists 2024-09-21T08:32:41.623 INFO:tasks.rados.rados.0.smithi137.stdout:2248: left oid 41 (ObjNum 779 snap 250 seq_num 779) 2024-09-21T08:32:41.623 INFO:tasks.rados.rados.0.smithi137.stdout:2249: expect (ObjNum 618 snap 182 seq_num 618) 2024-09-21T08:32:41.688 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6 2024-09-21T08:32:41.709 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:41.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:41.709 INFO:tasks.cephadm.osd.6:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.6' 2024-09-21T08:32:41.713 INFO:tasks.rados.rados.0.smithi137.stdout:2250: finishing write tid 1 to smithi13779221-24 2024-09-21T08:32:41.713 INFO:tasks.rados.rados.0.smithi137.stdout:2250: finishing write tid 2 to smithi13779221-24 2024-09-21T08:32:41.713 INFO:tasks.rados.rados.0.smithi137.stdout:2252: expect (ObjNum 618 snap 182 seq_num 618) 2024-09-21T08:32:41.795 INFO:tasks.rados.rados.0.smithi137.stdout:2250: finishing write tid 3 to smithi13779221-24 2024-09-21T08:32:41.795 INFO:tasks.rados.rados.0.smithi137.stdout:2250: finishing write tid 4 to smithi13779221-24 2024-09-21T08:32:41.795 INFO:tasks.rados.rados.0.smithi137.stdout:2250: oid 24 updating version 0 to 747 2024-09-21T08:32:41.795 INFO:tasks.rados.rados.0.smithi137.stdout:2250: oid 24 updating version 747 to 748 2024-09-21T08:32:41.796 INFO:tasks.rados.rados.0.smithi137.stdout:2250: oid 24 version 748 is already newer than 746 2024-09-21T08:32:41.796 INFO:tasks.rados.rados.0.smithi137.stdout:update_object_version oid 24 v 748 (ObjNum 780 snap 250 seq_num 780) dirty exists 2024-09-21T08:32:41.796 INFO:tasks.rados.rados.0.smithi137.stdout:2250: left oid 24 (ObjNum 780 snap 250 seq_num 780) 2024-09-21T08:32:41.796 INFO:tasks.rados.rados.0.smithi137.stdout:2251: expect (ObjNum 566 snap 168 seq_num 566) 2024-09-21T08:32:41.810 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:41.836 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:41.836 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:41.836 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:41.855 INFO:teuthology.orchestra.run.smithi165.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.b/config 2024-09-21T08:32:41.937 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1 2024-09-21T08:32:41.960 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:41.960 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:41.960 INFO:tasks.cephadm.osd.1:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.1' 2024-09-21T08:32:41.964 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:42.018 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[29634]: 7.4 scrub starts 2024-09-21T08:32:42.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[29634]: 7.4 scrub ok 2024-09-21T08:32:42.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[29634]: osdmap e567: 8 total, 8 up, 5 in 2024-09-21T08:32:42.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[29634]: 5.14 scrub starts 2024-09-21T08:32:42.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[29634]: 5.14 scrub ok 2024-09-21T08:32:42.146 INFO:journalctl@ceph.mon.a.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[29634]: osdmap e568: 8 total, 8 up, 5 in 2024-09-21T08:32:42.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[41044]: 7.4 scrub starts 2024-09-21T08:32:42.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[41044]: 7.4 scrub ok 2024-09-21T08:32:42.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[41044]: osdmap e567: 8 total, 8 up, 5 in 2024-09-21T08:32:42.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[41044]: 5.14 scrub starts 2024-09-21T08:32:42.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[41044]: 5.14 scrub ok 2024-09-21T08:32:42.147 INFO:journalctl@ceph.mon.c.smithi137.stdout:Sep 21 08:32:41 smithi137 ceph-mon[41044]: osdmap e568: 8 total, 8 up, 5 in 2024-09-21T08:32:42.147 INFO:tasks.rados.rados.0.smithi137.stdout:2255: expect (ObjNum 718 snap 222 seq_num 718) 2024-09-21T08:32:42.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:42.148 DEBUG:teuthology.orchestra.run.smithi137:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:9e2ac70bb7aea67d34f0d50002344abbe7803736 shell --fsid ec8566d6-77f1-11ef-baf6-efdc52797490 -- ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=true 2024-09-21T08:32:42.152 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:41 smithi165 ceph-mon[32893]: 7.4 scrub starts 2024-09-21T08:32:42.152 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:41 smithi165 ceph-mon[32893]: 7.4 scrub ok 2024-09-21T08:32:42.152 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:41 smithi165 ceph-mon[32893]: osdmap e567: 8 total, 8 up, 5 in 2024-09-21T08:32:42.152 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:41 smithi165 ceph-mon[32893]: 5.14 scrub starts 2024-09-21T08:32:42.153 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:41 smithi165 ceph-mon[32893]: 5.14 scrub ok 2024-09-21T08:32:42.153 INFO:journalctl@ceph.mon.b.smithi165.stdout:Sep 21 08:32:41 smithi165 ceph-mon[32893]: osdmap e568: 8 total, 8 up, 5 in 2024-09-21T08:32:42.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi165.stdout:Sep 21 08:32:42 smithi165 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-iscsi-iscsi-a[61734]: debug there is no tcmu-runner data available 2024-09-21T08:32:42.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:42.170 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:42.170 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:42.271 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5 2024-09-21T08:32:42.293 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:42.294 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:42.294 INFO:tasks.cephadm.osd.5:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.5' 2024-09-21T08:32:42.389 INFO:teuthology.orchestra.run.smithi165.stdout:ERROR: (22) Invalid argument 2024-09-21T08:32:42.389 INFO:teuthology.orchestra.run.smithi165.stdout:op_tracker tracking is not enabled now, so no ops are tracked currently, even those get stuck. Please enable "osd_enable_op_tracker", and the tracker will start to track new ops received afterwards. 2024-09-21T08:32:42.395 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:42.418 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:42.419 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:42.419 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3' 2024-09-21T08:32:42.490 INFO:teuthology.orchestra.run.smithi137.stderr:Inferring config /var/lib/ceph/ec8566d6-77f1-11ef-baf6-efdc52797490/mon.a/config 2024-09-21T08:32:42.520 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2 2024-09-21T08:32:42.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:42.541 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:42.541 INFO:tasks.cephadm.osd.2:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.2' 2024-09-21T08:32:42.617 INFO:teuthology.orchestra.run.smithi137.stderr:noscrub is set 2024-09-21T08:32:42.621 INFO:tasks.rados.rados.0.smithi137.stdout:2248: done (6 left) 2024-09-21T08:32:42.621 INFO:tasks.rados.rados.0.smithi137.stdout:2249: done (5 left) 2024-09-21T08:32:42.621 INFO:tasks.rados.rados.0.smithi137.stdout:2250: done (4 left) 2024-09-21T08:32:42.621 INFO:tasks.rados.rados.0.smithi137.stdout:2251: done (3 left) 2024-09-21T08:32:42.621 INFO:tasks.rados.rados.0.smithi137.stdout:2252: done (2 left) 2024-09-21T08:32:42.621 INFO:tasks.rados.rados.0.smithi137.stdout:2255: done (1 left) 2024-09-21T08:32:42.621 INFO:tasks.rados.rados.0.smithi137.stdout:2256: done (0 left) 2024-09-21T08:32:42.622 INFO:tasks.rados.rados.0.smithi137.stdout:2257: read oid 40 snap 215 2024-09-21T08:32:42.622 INFO:tasks.rados.rados.0.smithi137.stdout:2257: expect (ObjNum 664 snap 199 seq_num 664) 2024-09-21T08:32:42.622 INFO:tasks.rados.rados.0.smithi137.stdout:2258: read oid 6 snap -1 2024-09-21T08:32:42.622 INFO:tasks.rados.rados.0.smithi137.stdout:2258: expect deleted 2024-09-21T08:32:42.622 INFO:tasks.rados.rados.0.smithi137.stdout:2259: delete oid 46 current snap is 251 2024-09-21T08:32:42.636 INFO:tasks.rados.rados.0.smithi137.stdout:2257: expect (ObjNum 664 snap 199 seq_num 664) 2024-09-21T08:32:42.643 DEBUG:teuthology.orchestra.run.smithi165:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4 2024-09-21T08:32:42.663 INFO:teuthology.orchestra.run.smithi165.stderr:sudo: docker: command not found 2024-09-21T08:32:42.664 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:42.664 INFO:tasks.cephadm.osd.4:Ignoring exception while sending signal: Command failed on smithi165 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.4' 2024-09-21T08:32:42.686 INFO:teuthology.orchestra.run.smithi137.stdout: 2024-09-21T08:32:42.686 INFO:teuthology.orchestra.run.smithi137.stdout:{"fsid":"ec8566d6-77f1-11ef-baf6-efdc52797490","health":{"status":"HEALTH_WARN","checks":{"OSDMAP_FLAGS":{"severity":"HEALTH_WARN","summary":{"message":"noscrub flag(s) set","count":7},"muted":false},"PG_AVAILABILITY":{"severity":"HEALTH_WARN","summary":{"message":"Reduced data availability: 5 pgs inactive, 5 pgs peering","count":10},"muted":false}},"mutes":[]},"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_age":771,"monmap":{"epoch":3,"min_mon_release_name":"squid","num_mons":3},"osdmap":{"epoch":569,"num_osds":8,"num_up_osds":8,"osd_up_since":1726907528,"num_in_osds":5,"osd_in_since":1726907455,"num_remapped_pgs":0},"pgmap":{"pgs_by_state":[{"state_name":"active+clean","count":134},{"state_name":"active+clean+snaptrim_wait","count":24},{"state_name":"peering","count":23},{"state_name":"active+clean+snaptrim","count":4},{"state_name":"unknown","count":3}],"num_pgs":188,"num_pools":7,"num_objects":496,"data_bytes":637344415,"bytes_used":1679925248,"bytes_avail":478295252992,"bytes_total":479975178240,"unknown_pgs_ratio":0.01595744676887989,"inactive_pgs_ratio":0.12234042584896088,"read_bytes_sec":1790764,"write_bytes_sec":878106,"read_op_per_sec":1,"write_op_per_sec":0},"fsmap":{"epoch":1,"btime":"2024-09-21T08:18:21:794533+0000","by_rank":[],"up:standby":0},"mgrmap":{"available":true,"num_standbys":1,"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"services":{"dashboard":"https://172.21.15.137:8443/","prometheus":"http://172.21.15.137:9283/"}},"servicemap":{"epoch":13,"modified":"2024-09-21T08:25:01.614612+0000","services":{"rgw":{"daemons":{"summary":"","44387":{"start_epoch":12,"start_stamp":"2024-09-21T08:23:37.685972+0000","gid":44387,"addr":"172.21.15.137:0/3690342859","metadata":{"arch":"x86_64","ceph_release":"squid","ceph_version":"ceph version 19.1.1-367-g9e2ac70b (9e2ac70bb7aea67d34f0d50002344abbe7803736) squid (stable)","ceph_version_short":"19.1.1-367-g9e2ac70b","container_hostname":"smithi137","container_image":"quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:07c2c7a2cb9ca9b41c609b2ddfb8539abdc0283fbb4aebf87c70eeaf670bffe5","cpu":"Intel(R) Xeon(R) CPU E5-1620 v4 @ 3.50GHz","distro":"centos","distro_description":"CentOS Stream 9","distro_version":"9","frontend_config#0":"beast port=80","frontend_type#0":"beast","hostname":"smithi137","id":"foo.a","kernel_description":"#1 SMP PREEMPT_DYNAMIC Thu Sep 12 15:49:37 UTC 2024","kernel_version":"5.14.0-508.el9.x86_64","mem_swap_kb":"0","mem_total_kb":"32487204","num_handles":"1","os":"Linux","pid":"7","realm_id":"","realm_name":"","zone_id":"b5400f1a-5eb3-485f-84fb-fd9ddf38f21b","zone_name":"default","zonegroup_id":"7bc8b587-83f6-4450-a0d5-d40477a44b6b","zonegroup_name":"default"},"task_status":{}}}}}},"progress_events":{}} 2024-09-21T08:32:42.765 DEBUG:teuthology.orchestra.run.smithi137:> sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3 2024-09-21T08:32:42.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-21T08:32:42.787 INFO:teuthology.orchestra.run.smithi137.stderr:sudo: docker: command not found 2024-09-21T08:32:42.788 INFO:tasks.cephadm.osd.3:Ignoring exception while sending signal: Command failed on smithi137 with status 1: 'sudo docker kill -s 1 ceph-ec8566d6-77f1-11ef-baf6-efdc52797490-osd.3'